New: temperature, fans, power supplies and SSD health per server. View hardware health

Try it free

A GPU that is not computing still costs money. PeakNode shows for each card since when it has been idle and which process is holding it anyway. Enter your rate per GPU hour and the amount in euros appears alongside, for 7 and for 30 days. On top of that comes the energy consumption of the GPUs in kWh.

Example: 6 GPU servers with 29 GPUs, height by utilization. Can be rotated with the mouse.

What is Idle time and energy?

For each GPU, PeakNode checks whether utilization is below 5% and shows since when the card has been idle and which processes are holding it. The stored reports add up to idle GPU hours for 7 and 30 days, and with a stored cost per GPU hour also to an amount in euros. The portal adds up the reported power draw of the GPUs over time to kWh, and with an electricity price to costs.

GPU nodes in the portal: header “1 GPU in critical state” with utilization, idle GPUs, GPU memory, power and the hottest GPU of the fleet, below the table of the six nodes with utilization, GPU memory, hottest GPU, power, GPU health and status.
Screenshot from the portal with sample data (German interface).

Utilization and idle time in the portal

  • One threshold for everything: below 5% utilization a GPU counts as idle.
  • Periods without a report count neither as idle nor as busy.
  • The energy is that of the GPUs, without CPU, power supply losses and cooling.

What PeakNode shows about idle time and energy

The figures come from the agent’s stored reports. Anything that was not measured appears as “-”, not as zero.

Idle time per GPU, with “since when”

When a card drops below 5% utilization, the portal records the start. After a gap in reporting the count starts again: a server that was switched off over the weekend is therefore not shown as “idle for 3 days”. If the driver does not report utilization, for example with MIG, the state remains unknown.

Who is holding a GPU

For every card PeakNode shows the processes with PID, name, memory used and user. This shows you who is holding a GPU on which nothing is computing. The process list is available with the Linux agent. It does not send command lines, and in containers it only sees processes of its own namespace.

Alert on long idle time

The alert metric “GPU idle time (minutes)” fires when a GPU has been idle for longer than the configured time. The card with the longest idle time is what counts; up to 10,080 minutes, that is seven days, can be set. During a maintenance window the alert pauses like all threshold alerts.

Idle cost in euros

For each server monitor you can optionally store the cost per GPU hour, for example rent or depreciation plus electricity. PeakNode multiplies it by the idle GPU hours of the last 7 and 30 days. With a data retention of 14 days, the longer period is shortened to 14 days.

Energy in kWh and euros

From the reported GPU power, the portal calculates the energy for today, this month, the selected period and since installation. With an electricity price per kWh, the costs appear alongside. This is the board power of the GPUs, not the consumption of the whole server.

All servers on the GPU nodes page

One page for the whole fleet: how many GPUs are idle, the energy since midnight and the utilization of the last 24 hours in 15-minute steps. Below that every card with “Idle since” and its processes, plus energy and idle cost per server.

Idle cost per server

Example

Server monitorCost per GPU hourIdle, 7 daysIdle, 30 days
hgx-02 (8 GPUs)€2.50412.5 GPU hours ≈ €1,031.251,520.0 GPU hours ≈ €3,800.00
ws-labor-1 (1 GPU)-96.0 GPU hours388.5 GPU hours

Labels as in the portal. Without a stored cost, the table shows GPU hours only.

How the figures are produced

All values can be recalculated from the stored reports. Nothing is estimated for idle time; for energy, missing reports are bridged only up to the offline limit.

FigureHow it is producedWhere it appears
Idle sinceTime of the first report below 5% utilization, carried forward as long as the GPU stays unusedCard per GPU in the server monitor, GPU nodes page
Idle GPUsGPUs with less than 5% utilization in the latest reportGPU nodes page
GPU idle time (minutes)longest idle time of any GPU in the serverAlert rule on the server monitor
Idle time in GPU hoursSum of idle GPUs times duration per report, for 7 and 30 days“Idle cost” card in the server monitor and on GPU nodes, API
Idle costIdle GPU hours times the stored cost per GPU hour“Idle cost” card, SLA and monthly report
GPU energyMean of two consecutive power readings times the interval between reportsServer monitor, GPU nodes, SLA and monthly report
Energy costkWh times the stored electricity priceServer monitor, GPU nodes
Stall suspectedUtilization of 90% or more while power stays below 30% of the limit over several reportsBadge on the GPU

The opposite case to idle time carries the badge “Stall suspected”: high utilization with very low power draw, often a waiting process rather than real compute load. As long as the reports of a period have not been summarized yet, it shows “being calculated”.

Set up in three steps

PeakNode measures idle time and energy from the first report. For amounts in euros, all that is missing is your rate per GPU hour and your electricity price.

  1. Server monitor with the Linux agent

    The Linux agent reads NVIDIA GPUs once per second and reports the mean for each report. This matters for bursty loads: a snapshot once a minute shows 0 or 100% there.

  2. Store the cost per GPU hour and the electricity price

    In the server monitor under “Advanced settings”. Both are optional. Without them the portal shows GPU hours and kWh, with them the amounts.

  3. Set an idle alert

    Add an alert, choose the metric “GPU idle time (minutes)” and set the duration after which you want to know. The alert goes to the monitor’s channels.

Frequently asked
questions about Idle time and energy

Continue to:

How do I see which GPU is idle?

PeakNode marks every GPU with less than 5% utilization as idle and shows since when. The GPU nodes page counts the idle cards across all servers.

How does PeakNode calculate the idle cost of a GPU?

From the stored reports: every GPU below 5% utilization in a report counts as idle for the time back to the previous report, at most for one reporting interval plus 10%. The sum is the idle GPU hours. Multiplied by your cost per GPU hour, this gives the amount for 7 and 30 days.

Does PeakNode measure the power consumption of the whole server?

No. It measures the board power of the GPUs as reported by nvidia-smi, added up over time to kWh. CPU, power supply losses and cooling are not included.

Does a GPU with a loaded model count as idle?

Yes, if its utilization is below 5%. PeakNode evaluates utilization, not memory in use. An inference server without requests therefore shows as idle, even if the model is loaded.

What happens to periods in which the server did not report?

They do not count towards idle time: a report covers at most one reporting interval plus 10%, larger intervals are gaps. For energy, the portal bridges an interval up to the offline limit with the mean of the two power readings; longer pauses do not count there either. Without a single report in the period, it shows “-” instead of 0.

Are idle time and energy also in the monthly report?

Yes. The SLA and monthly report lists the GPU energy in kWh and the idle time in GPU hours for each server, and with a stored cost also the amount. The API provides both as well.

How much does a GPU hour cost?

That depends on your own numbers: for rented GPUs the provider’s hourly rate, for your own servers for example depreciation plus electricity, divided by the hours. You enter this rate per server monitor, and PeakNode uses it to calculate the cost of idle GPU hours. For the electricity share, the portal provides the measured GPU energy in kWh.

Is 100% GPU utilization bad?

No, high utilization is what you want for training and inference. It becomes suspicious when the card draws hardly any power at the same time: at 90% utilization or more with less than 30% of the power limit over three consecutive reports, PeakNode flags the GPU as a suspected hung process. If a card throttles because of heat, hardware health shows it.

See what idle GPUs cost

Create an account, create a server monitor, install the agent, enter your rate per GPU hour. The Proof of Concept plan is free.