Allocations and compute scheduling: Difference between revisions

With the 2025 [[infrastructure renewal]] it will become possible to schedule a fraction of a GPU using [[multi-instance GPU]] technology.  Different jobs, potentially belonging to different users, can run on the same GPU at the same time.  Following [https://docs.nvidia.com/datacenter/tesla/mig-user-guide/#terminology NVidia's terminology], a fraction of a GPU allocated to a single job is called a "GPU instance", also sometimes called a "MIG instance".   

Note: a GPU instance of profile <b>1g</b> is worth 1/7 of a A100 or H100 GPU. The case of <b>3g</b> takes into consideration the extra amount of memory per <b>g</b>.

* [[File:Two_core_equivalents.png|frame|Figure 2 - Two core equivalents.]] Research groups using more cores than memory (above the 1 core/4GB memory ratio), will be charged by cores.  For example, a research group requesting two cores and 2GB per core for a total of 4 GB of memory. The request requires 2 core equivalents worth of cores but only one bundle for memory.  This job request will be counted as 2 core equivalents when priority is calculated. See Figure 2. <br clear=all>

=Viewing usage of compute resources using the portal=

[[File:Slurm portal land edit.png|thumb|alt=usage portal landing view|usage portal landing view]]

[https://portal.alliancecan.ca/slurm portal.alliancecan.ca/slurm] provides an interface for exploring time-series data about jobs on Alliance national clusters. The page contains a figure that can display several usage metrics. When you first log in to the site, the figure will display CPU-equivalent days on the Cedar cluster for your account. If you have no usage on Cedar, the figure will contain the text “No Data or usage too small to have a meaningful plot”. The data appearing in the figure can be modified by control panels along the left margin of the page. There are three panels:

==Displaying a specified account==

If you have access to more than one [[Running_jobs#Accounts_and_projects|Slurm account]], the "Select user’s account" pull-down menu of the "SLURM account" panel lets you select which project account will be displayed in the figure window. If the “Select user’s account” is left empty the figure will display all of your usage across accounts on the specified cluster during the selected time period. The "Select user’s account" pull-down menu is populated by a list of all the accounts that have job records on the selected cluster during the selected time interval. When you select a single project account the figure is updated and the summary panel titled "Allocation Information" is populated with details of the project account. The height of each bar in the histogram figure corresponds to the metric for that day (e.g. CPU-equivalent days) across all users in the account on the system. The top eight users are displayed in unique colors stacked on top of the summed metric for all other users in gray. You can navigate the figure using [https://plotly.com/graphing-libraries/ Plotly] tools (zoom, pan, etc) whose icons appear at the top-right when you hover your mouse over the figure window. You can also use the legend on the right-hand side to manipulate the figure. Single-clicking an item will toggle the item's presence in the figure, and double-clicking the item will toggle off or on all the other items in the figure.  

==Displaying the allocation target and queued resources==

When a single account has been selected for display, the "Allocation target" is shown as a horizontal red line. It can be turned off or on with the “Display allocation target by default” item in the “Parameters” panel, or by clicking on the words "Allocation target" in the legend to the right of the figure.  

You can toggle the display of the "Queued jobs" metric, which presents a sum of all resources in pending jobs at each time point, by clicking on the words "Queued jobs" in the legend to the right of the figure.  

==Selecting a specific cluster and time interval==

The figure shows your usage for a single cluster over a specified time interval. The "System" pull-down menu contains entries for each of the currently-active national clusters that use Slurm as a scheduler. You can use the "Start date (incl.)" and "End date (incl.)" fields in the "Select system and dates" panel to change the time interval displayed in the figure. It will include all jobs on the specified cluster that were in a running (R) or pending (PD) state during the time interval, including both the start and end date. Selecting an end date in the future will display the <i>projection</i> of currently running and pending jobs for their requested duration into the future.

==Displaying usage over an extended time period into the future==

If you select an end time after the present time, the figure will have a transparent red area overlayed on the future time labelled "Projection". In this projection period, each job is assumed to run to the time limit requested for it. For queued resources, the projection supposes that each pending job starts at beginning of the projected time (that is, right now) and runs until its requested time limit. This is not intended to be a forecast of actual future events!

==Metrics, summation, and running jobs==

Use the "Metric" pull-down control in the "Parameters" panel to select from the following metrics: CPU, CPU-equivalent, RGU, RGU-equivalent, Memory, Billing, gpu, and all specific GPU models available on the selected cluster.  

The "Summation" pull-down allows you to switch between the daily "Total" and "Running total". If you select "Total", each bar of the histogram represents the total usage in that one day.  If you select "Running total", each bar represents the sum of that day's usage and all previous days back to the beginning of the time interval. If the "Allocation Target" is displayed, it is similarly adjusted to show the running total of the target usage.

If you set "Include Running jobs" to "No", the figure shows only data from records of completed jobs. If you set it to "Yes" it includes data from running jobs too.

==Display of the running total of account usage==

When displaying the running total of the usage for a single account along with the "Allocation target" the usage histogram displays how an account deviates from its target share over the period displayed. The values in this view are the cumulative sum across days from "total" summation view for both the usage and allocation target. When an account is submitting jobs that request more than the account’s target share it is expected that the usage cumulative total will oscilate above and below the target share cumulative sum if the scheduler is managing fair share properly. Because the scheduler uses a decay period for the impact of past usage, a good interval to use to inspect the scheduler’s performance in maintaining the accounts fair share is to display the past 30 days.

=Viewing group usage summaries of compute resources at ccdb.ca= <!--T:35-->

For resource allocation projects that have GPU usage the table is broken down into usage on various GPU models and measured in RGUs.