Updated 2021-05-03

Understanding the PACE Monthly Statement

In January 2021, Georgia Tech formalized the adoption of a new method of funding research cyberinfrastructure that is more sustainable and flexible, and the new model has the full support of Georgia Tech’s executive leadership. The new model is based on actual consumption of resources – similar to that of commercial cloud offerings. In this new model, PIs are charged only for what they actually use rather than for some fixed capacity which may remain idle. PIs will receive a monthly statement for their usage, and below we are providing an explanation of the various account information that are provided on the monthly statement to PIs.

Compute Statements

Example compute statement for account GT-gburdell3-CODA20

1. Header

This is the header for the monthly report containing various account information. On the left is the month that this report pertains to and on the right is the main account name for this report.

2. Account Balance vs. Time

This graph shows the account balance as a function of time. The section of the line that is highlighted orange shows the account balance for the month of this specific report. The rest of the line is linearly extrapolated from this report's data to show an estimation of the timeframe at which your account will no longer have any credits.

Note

If the account draws from funds shared by multiple PIs or those provided by a school/college, this graph will show the balance decline from jobs run by all research groups sharing the funds, not just your group.

3. Charges By Queue

The bar graph shows the amount of credits consumed in the month of the report. Each bar on the graph represents one day. The orange section of each bar represents the amount credits used by on the Inferno queue and the blue section represents the amount credits used by on the Embers queue.

The pie chart shows the proportion of credits consumed that came from the Inferno queue vs. the Embers queue for the month of the report.

Please note that the jobs submitted to the Embers queue do not actually consume credits, so in both of the charts, the credits depicted as being used by the Embers queue are the amount of credits that would have been used if the jobs were ran on the Inferno queue. Think of this as the amount of credits that were saved by using the free Embers queue.

Note

If the account has no utilization for the month, this section will be omitted.

4. Charges By User

This pie chart shows the distribution of credits used by each user on the account. If more than 5 users ran jobs for the account, then this chart will show the top 5 users and group all the rest of the users into one section.

Note

If the account has no utilization for the month, this section will be omitted.

5. CPU Efficiency

This is a box and whiskers plot that depicts the CPU efficiencies of jobs ran by the users on the account. The green line in the center of user's graph is the median CPU efficiency for that user. The box contains the middle 50% of a user's CPU efficiencies and the two ends of each plot represent the extremes, the lowest and highest CPU efficiencies of each user.

The formula used to calculate efficiency is the following:

CPU Efficiency = CPUTime / (processors*job duration)

CPUTime is the amount of time the processor was actually doing work. If efficiency is low, it could be the result of a few unintended causes as stated below.

  • Interactive Jobs - interactive jobs (pace-vnc-job, pace-jupyter-notebook, etc.) inherently do not run as efficiently as a batch job, but also if the job is not cancelled after finished, it will continue running.
  • Oversubscription - oversubscription of processors can cause scheduling conflict within the CPU, as multiple threads/processes must wait for available resources to run. It is generally best practice to have a 1:1 ratio for threads/processes:CPU cores.
  • Shared Memory - shared memory (multi-threaded) applications that are attempted on jobs using multiple servers cannot effectively utilize CPU cycles as the application is bound to only the first node in the hostlist.
  • GPU Jobs - the primary processor for these jobs is the GPU rather than the CPU, so there may be little tracked CPU utilization
  • Built-in MPI - if you use an application with its own MPI compiled without the Torque TM API, resource utilization from all nodes will be reported incorrectly and the CPUTime will not be tracked correctly

Note

If the account has no utilization for the month, this section will be omitted.

6. Usage Statistics

This section of the report gives an overall summary of different statistics for the whole account. The descriptions for each statistic are below:

  • Number of Jobs Run - the total amount of jobs ran in the month of the report
  • Total Walltime - walltime is the amount of time it takes for a job to complete
  • Total CPU-Time - the total amount of time processors were actually being used for jobs
  • Avg Queue Time - how long the job waited before running (Queue Time = start time - submit time)
  • Avg Walltime - the average amount of walltime per job
  • Avg Expansion Factor - a metric to normalize queue time based on job duration. Walltime in this formula is the actual job duration, not the requested walltime (Expansion Factor = 1 + (queue time / walltime)`
  • Avg Job Size - the average number of a processors that were used to complete a job

Note

If the account has no utilization for the month, this section will be omitted.

7. Charges by Node Class

This pie chart shows the distributions of credits consumed by each of the node classes that is used on the account. If jobs were run on more than 5 node classes, this chart will show the top 5 node classes and will aggregate the rest.

Note

If the account has no utilization for the month, this section will be omitted.

8. Actual Walltime

This is a box and whiskers plot that depicts the walltimes of jobs ran by the users on the account. The green line in the center of user's graph is the median walltime for that user. The box contains the middle 50% of a user's walltimes and the two ends of each plot represent the extremes, the lowest and highest walltime of each user.

Note

If the account has no utilization for the month, this section will be omitted.

9. Balance

This section shows the starting and ending balance of the account for the month of the report. It also shows the number of credits consumed.

Note

If the account draws from funds shared by multiple PIs or those provided by a school/college, the start and end balance will reflect usage by all research groups on the parent account, but the credits consumed will reflect only usage on your research group's child account. Note that this means the credits consumed will not be the difference in the two balances.

Storage Statements

Example storage statement for project p-gburdell3-0

1. Header

The header indicates the billing cycle month and the project storage name.

2. Storage Account Warnings

This section may be populated with one of four warning messages if the state of the project storage may have an impact on research efforts:

  • Project storage utilization exceeds the current quota Warning for project storage that is over quota
  • Remaining storage account balance is less than quota, so quota will be reduced to 1TB (0TB for project storage that does not receive the free-tier credit) until storage credits are sufficient to support the requested quota. Warning for project storage that is over quota
  • Remaining storage account balance will only cover current quota for one more month, after which the project quota will be reduced to 1TB (0TB for project storage that does not receive the free-tier credit) until storage credits are sufficient to support the requested quota. Warning for project storage that is over quota
  • Remaining storage account balance will only cover current quota for two more months, after which the project quota will be reduced to 1TB (0TB for project storage that does not receive the free-tier credit) until storage credits are sufficient to support the requested quota. Warning for project storage that is over quota

Important

If one of these warnings is printed on your monthly statement, please contact PACE support to purchase more storage or request a quota reduction.

Note

This section will be empty if the above conditions do not apply.

3. Storage Account Balance

If this project storage account has credits in excess of the quota, this chart will show the balance progression, quota, and projected date when the storage credits will be reduced to a value less than the quota.

Note

This section will be empty if the balance of storage credits is less than or equal to the project storage quota.

4. Project Storage Utilization

This percentage bar depicts the project storage consumption, with the used and available portions presented both in units of terabtyes (TB) and percentage of the total quota.

5. Utilization by User

This pie chart displays a breakdown of the project storage utilization by user; if there are more than five users, the top five are displayed

6. Project Storage Summary

This table details the project storage:

  • Project Quota (TB): the project storage quota on the last day of the previous month
  • Project Utilization: percentage of the project storage quota consumed by data
  • Balance on 1st of Previous Month (TB•month): Storage credit balance at start of billing cycle
  • Balance on 1st of Current Month (TB•month): Storage credit balance at end of billing cycle, determined as Start-Consumed+Purchased
  • Free Tier Contribution (TB•month): 1 if project storage account is eligible for free-tier credit, 0 otherwise
  • Storage Credits Consumed (TB•month): Project Quota - Free Tier Contribution
  • Storage Credits Purchased (TB•month): Quantity of storage credits purchased during billing cycle
  • Amount Billed ($): Dollar amount charged to worktag for credits purchased during billing cycle