New: temperature, fans, power supplies and SSD health per server. View hardware health

Try it free
Hardware health in the portal: header “2 critical hardware events in 24 h”, the state of the fleet as a grid with one row per server and one column per component group (GPUs, CPU, RAM, drives, cooling, power supplies, network), the events of the last 14 days as bars and the list of events with time, server, severity and message.

No utilization chart tells you whether a GPU is computing without errors. Memory errors or a failed card show up in the server’s system log and in the error counters of the NVIDIA driver. PeakNode reads both, assigns every event to the affected card and sends critical events to your channels immediately.

Screenshot from the portal with sample data (German interface).

What is Hardware health?

Hardware health in PeakNode evaluates, per NVIDIA GPU, what decides between failure and service: Xid messages (the NVIDIA driver’s error codes in the kernel log), ECC memory errors, row remapping (replacement of faulty memory rows), the PCIe link, throttle reasons, missing cards and driver errors. For this, the Linux agent reads journalctl or dmesg and nvidia-smi. The portal assigns the events to the card, rates them as critical, warning or info and reports critical events and warnings through the server monitor’s channels.

GPUs of the node hgx-02 in the portal: table of the eight NVIDIA H100 with utilization, memory usage, temperature, power draw and clock per card, below the utilization history per GPU.
Screenshot from the portal with sample data (German interface).

Hardware health in the portal

  • Eleven Xid codes count as critical, including 79 (GPU fallen off the bus) and 48 (double-bit ECC).
  • Each event is recorded once; the same Xid message is not repeated.
  • Requirement: Linux agent and NVIDIA GPUs. Anything the driver does not provide appears as “-” in the portal.

Which GPU faults PeakNode detects

All values come from nvidia-smi and the kernel log. PeakNode does not estimate anything: an unknown value stays unknown.

Xid messages with a rating

Xid is the error code the NVIDIA driver writes to the kernel log. PeakNode reads it from journalctl or dmesg and rates it: eleven codes such as 79 (GPU fallen off the bus) or 48 (double-bit ECC) are critical, four count as info, all others as a warning.

Memory errors and remapped rows

New uncorrectable ECC errors are critical. For corrected errors, PeakNode only warns when they rise sharply compared with the card’s own history. If the GPU can no longer replace a faulty memory row (row remapping failed), it needs to go in for service.

PCIe link below maximum

The PCIe link is the connection between card and mainboard. PeakNode compares its generation and width with what the card supports: Gen4 x8 instead of Gen5 x16 gets the badge “PCIe below max”. Because links routinely downshift at idle, the finding only counts at a utilization of 50% or more.

Throttle reason, not just temperature

The portal shows why a card is slowing down: power cap, HW slowdown, HW thermal, HW power brake, SW thermal or application clock setting. If a thermal reason persists over three consecutive reports, an event is created with the temperature.

Missing GPU and driver errors

If fewer cards report than expected, that is critical. The expected number is automatically the highest number seen so far. If nvidia-smi stops responding, the agent reports the driver error, and the portal shows the last known values with a note that they are not current.

All servers on the “Hardware health” page

One page for the whole fleet: GPUs with findings, critical events and warnings of the last 24 hours and the number of servers with a readable kernel log. Below that, the events of the last 14 days, the state of all GPUs and the 150 latest events, filterable by severity.

What is reported when

Critical events go out immediately. Warnings that describe a plannable task are collected by NodeCare and sent as a summary in the morning.

WhatSourceWhen reported
Xid 44, 48, 63, 64, 74, 79, 92, 94, 95, 119, 120Kernel log (journalctl -k, otherwise dmesg)immediately, critical
Xid without a critical ratingKernel logimmediately, warning
Xid 13, 31, 43, 45Kernel logInfo in the event list, no notification
New uncorrectable ECC errorsnvidia-smi, counter since the driver was loadedimmediately, critical
Row remapping failednvidia-smi --query-remapped-rowsimmediately, critical
Row remapping pendingnvidia-smi --query-remapped-rowsWarning, in the NodeCare summary at 08:00
More remapped rows than at the last readingnvidia-smi --query-remapped-rowsimmediately, warning
Corrected ECC errors risen sharplynvidia-smi and the card’s own historyWarning, in the NodeCare summary at 08:00
Thermal throttling over three consecutive reportsnvidia-smi, throttle reasonsWarning, in the NodeCare summary at 08:00
PCIe link below maximum under loadnvidia-smi, current and maximum linkBadge on the GPU, after three consecutive reports a NodeCare “Watch” item
Fewer GPUs than expectedNumber of reported cards against the expected numberimmediately, critical
GPU driver not respondingnvidia-smi fails or times outimmediately, critical, once a day while it persists

Notifications are only sent if “Alert on hardware events” is switched on for the server monitor and a channel is set. Besides the GPUs, the page also shows CPU temperature, memory, drives, fans and power supplies per server as soon as the Linux agent version 11 or later provides the values (hwmon, EDAC, SMART, BMC). DCGM profiling metrics are not covered.

Frequently searched Xid codes and their rating

The name appears like this in the portal and in the notification. For the meaning of a code, NVIDIA’s Xid catalog is authoritative.

XidName in the portalRating in PeakNode
13Graphics engine exceptionInfo
31GPU memory page faultInfo
43GPU stopped processingInfo
45Preemptive cleanup (process terminated)Info
48Double bit ECC error (DBE)Critical
79GPU has fallen off the busCritical
94Contained ECC errorCritical
95Uncontained ECC errorCritical
109Context switch timeout errorWarning
119GSP RPC timeoutCritical

Info items only appear in the event list; warnings and critical events are reported. A code without a name in the portal appears with its number and counts as a warning.

Set up in three steps

Hardware health is not a separate module. It is active as soon as the Linux agent reports from a server with NVIDIA GPUs.

  1. Install the agent as root

    The agent needs to read the kernel log, so its cron job runs as root. If the log is not readable, as is often the case in containers, the portal shows “Kernel log not readable on this server” instead of zero events.

  2. Check the expected number of GPUs

    PeakNode sets the value to the highest number seen. If you deliberately remove a card, adjust the number in the server monitor. With 0 the check is switched off.

  3. Switch on alerts for hardware events

    Choose the channels in the server monitor and switch on the alert. From then on, critical events go out immediately.

Frequently asked
questions about Hardware health

Continue to:

What is hardware health in PeakNode?

The part of PeakNode that records and rates faults on NVIDIA GPUs: Xid messages, ECC errors, row remapping, PCIe link, throttling, missing cards and driver errors. It reports critical events immediately through the server monitor’s channels.

What does Xid 79 mean, and which Xid errors are critical?

In the portal Xid 79 reads “GPU has fallen off the bus”: the card has dropped off the bus. PeakNode reports this immediately as critical, with the affected GPU. Codes 44, 48, 63, 64, 74, 92, 94, 95, 119 and 120 are critical as well. Codes 13, 31, 43 and 45 are info without a notification, all others are warnings.

How does PeakNode handle ECC errors on the GPU?

PeakNode reports new uncorrectable errors immediately as critical. For corrected errors it only warns when they reach at least 100 in 24 hours and ten times the daily mean of the 14 days before, at the earliest after about 8 days of history. The counters are those of the driver and start again at zero after it is loaded.

What does row remapping pending or failed mean?

From the Ampere generation onwards, an NVIDIA GPU replaces faulty memory rows. Pending means: the replacement row only becomes active after a GPU reset; NodeCare lists this as a task for the next maintenance window. Failed means: the card can no longer replace a faulty row and needs to go in for service; PeakNode reports this as critical.

Why does PeakNode show only a dash for some GPUs?

Because the driver does not provide the value for that card. ECC, HBM temperature and row remapping exist on data center GPUs; consumer cards report “N/A” there. PeakNode then shows “-” and creates neither an event nor a zero from it.

Does hardware health work on Windows?

No. Xid, ECC, row remapping, PCIe and throttle reasons are only provided by the Linux agent. The Windows agent reports only basic values from NVIDIA GPUs such as utilization, memory, temperature and power; the macOS agent reports no GPU values.

How does hardware health differ from NodeCare?

Hardware health records events as soon as they occur and reports critical ones immediately. NodeCare turns these events and the history into a maintenance plan with stages, dates and recommendations. Both use the same agent readings.

What do Xid 31, 43 and 94 mean?

In the portal Xid 31 reads “GPU memory page fault”, an error during a memory access, Xid 43 “GPU stopped processing” and Xid 94 “Contained ECC error”, a memory error that was contained. PeakNode lists 31 and 43 as info without a notification and 94 as critical. The rating of further codes is in the table above.

Is there a tool to check GPU health?

For a one-off check, nvidia-smi -q shows ECC counters, remapped rows, throttle reasons and temperatures of every GPU. For continuous checks, the PeakNode Linux agent reads the same fields plus the Xid messages from the kernel log at every report, and the portal rates each event and reports critical ones. Basic values such as utilization and memory are covered by GPU server monitoring.

Is a single ECC error a reason to replace the GPU?

For PeakNode a single corrected ECC error is not a finding; it only warns when they rise sharply. New uncorrectable errors and a failed row remapping are reported as critical. Whether a card is replaced is decided by the reseller or vendor; NodeCare creates the record for the warranty claim as a PDF.

See what is in the kernel log of your GPU servers

Create an account, create a server monitor, install the agent as root. From the first report, PeakNode evaluates Xid, ECC and PCIe per card.