Allocations and compute scheduling: Difference between revisions

<b>Note:</b> While the scheduler will compute the priority based on the usage calculated with the above bundles, users requesting multiple GPUs per node also have to take into account the physical ratios.

[[File:Slurm portal land edit.png|thumb|alt=usage portal landing view|usage portal landing view]]

[https://portal.alliancecan.ca/slurm portal.alliancecan.ca/slurm] provides an interface for exploring time-series data about jobs on Alliance national clusters. The page contains a figure that can display several usage metrics. When you first log in to the site, the figure will display CPU-equivalent days on the Cedar cluster for your account. If you have no usage on Cedar, the figure will contain the text “No Data or usage too small to have a meaningful plot”. The data appearing in the figure can be modified by control panels along the left margin of the page. There are three panels:

<br clear=all>

[[File:Slurm portal account usage edit.png|thumb|alt=usage display of a specified account|Usage display of a specified account]]

If you have access to more than one [[Running_jobs#Accounts_and_projects|Slurm account]], the "Select user’s account" pull-down menu of the "SLURM account" panel lets you select which project account will be displayed in the figure window. If the “Select user’s account” is left empty the figure will display all of your usage across accounts on the specified cluster during the selected time period. The "Select user’s account" pull-down menu is populated by a list of all the accounts that have job records on the selected cluster during the selected time interval. When you select a single project account the figure is updated and the summary panel titled "Allocation Information" is populated with details of the project account. The height of each bar in the histogram figure corresponds to the metric for that day (e.g. CPU-equivalent days) across all users in the account on the system. The top eight users are displayed in unique colors stacked on top of the summed metric for all other users in gray. You can navigate the figure using [https://plotly.com/graphing-libraries/ Plotly] tools (zoom, pan, etc) whose icons appear at the top-right when you hover your mouse over the figure window. You can also use the legend on the right-hand side to manipulate the figure. Single-clicking an item will toggle the item's presence in the figure, and double-clicking the item will toggle off or on all the other items in the figure.  

<br clear=all>

[[File:Slurm portal account usage queued edit.png|thumb|alt=Allocation target and queued resources displayed on usage figure|Allocation target and queued resources displayed on usage figure]]

When a single account has been selected for display, the "Allocation target" is shown as a horizontal red line. It can be turned off or on with the “Display allocation target by default” item in the “Parameters” panel, or by clicking on the words "Allocation target" in the legend to the right of the figure.  

You can toggle the display of the "Queued jobs" metric, which presents a sum of all resources in pending jobs at each time point, by clicking on the words "Queued jobs" in the legend to the right of the figure.  

<br clear=all>

[[File:Slurm portal select sys date.png|thumb|alt=Select a specific cluster and time interval|Select a specific cluster and time interval]]

The figure shows your usage for a single cluster over a specified time interval. The "System" pull-down menu contains entries for each of the currently-active national clusters that use Slurm as a scheduler. You can use the "Start date (incl.)" and "End date (incl.)" fields in the "Select system and dates" panel to change the time interval displayed in the figure. It will include all jobs on the specified cluster that were in a running (R) or pending (PD) state during the time interval, including both the start and end date. Selecting an end date in the future will display the <i>projection</i> of currently running and pending jobs for their requested duration into the future.

<br clear=all>

[[File:Slurm portal account use duration edit.png|thumb|alt=Displaying usage over and extended period into the future|Displaying usage over and extended period into the future]]

If you select an end time after the present time, the figure will have a transparent red area overlayed on the future time labelled "Projection". In this projection period, each job is assumed to run to the time limit requested for it. For queued resources, the projection supposes that each pending job starts at beginning of the projected time (that is, right now) and runs until its requested time limit. This is not intended to be a forecast of actual future events!

<br clear=all>

[[File:Slurm portal parameter panel.png|thumb|alt=Parameters of the usage series histogram|Parameters of the usage series histogram]]

Use the "Metric" pull-down control in the "Parameters" panel to select from the following metrics: CPU, CPU-equivalent, RGU, RGU-equivalent, Memory, Billing, gpu, and all specific GPU models available on the selected cluster.  

The "Summation" pull-down allows you to switch between the daily "Total" and "Running total". If you select "Total", each bar of the histogram represents the total usage in that one day.  If you select "Running total", each bar represents the sum of that day's usage and all previous days back to the beginning of the time interval. If the "Allocation Target" is displayed, it is similarly adjusted to show the running total of the target usage. See the next section for more.

If you set "Include Running jobs" to "No", the figure shows only data from records of completed jobs. If you set it to "Yes" it includes data from running jobs too.

<br clear=all>

[[File:Slurm portal account use cumulative edit.png|thumb|alt=Display of the  running total of account usage|Display of the  running total of account usage]]

When displaying the running total of the usage for a single account along with the "Allocation target" the usage histogram displays how an account deviates from its target share over the period displayed. The values in this view are the cumulative sum across days from "total" summation view for both the usage and allocation target. When an account is submitting jobs that request more than the account’s target share it is expected that the usage cumulative sum will oscillate above and below the target share cumulative sum if the scheduler is managing fair share properly. Because the scheduler uses a decay period for the impact of past usage, a good interval to use to inspect the scheduler’s performance in maintaining the account's fair share is to display the past 30 days.