NVIDIA Xid Error Codes: GPU Troubleshooting Basics | GPU Debugging Series #1

When a training job crashes or a GPU disappears from nvidia-smi, where should you start? This guide introduces NVIDIA Xid messages: what they mean, where to find them, and how to use them to narrow down a GPU problem. We will work through two common troubleshooting scenarios: an unreachable GPU (Xid 79, GPU has fallen off the bus) and ECC memory errors (Xid 48/63/64/92/94/95). You will also learn what to collect before contacting support. A quick reference to common data center Xids and NVIDIA’s recommended actions appears at the end.

1. Xid: The GPU’s “Black Box” Alert

An Xid is an error or event code reported by the NVIDIA GPU driver. When a GPU encounters a problem, the application might report a CUDA error, nvidia-smi might no longer list the expected GPUs, or the host might become unresponsive. On Linux, the NVIDIA kernel driver can write an Xid message to the kernel log, using the prefix NVRM.

This message identifies the affected GPU and the type of event. Depending on the error and driver version, it may also identify the process involved. Think of it as a GPU’s black box: a record from the driver, close to the hardware, that can add context the application’s error message lacks. It is valuable evidence, although it does not always reveal the root cause. A typical message looks like this:

NVRM: Xid (PCI:0000:5a:00): 79, pid=12345, name=python, GPU has fallen off the bus.

There are four useful fields to read:

NVIDIA describes Xids as driver reports that can indicate a hardware problem, an NVIDIA software problem, or an application problem.¹ That is what makes troubleshooting difficult: a single code can have several very different causes. Keep three principles in mind before reaching for a lookup table:

1. An Xid identifies a symptom, not a diagnosis. Xid 79 can involve power delivery, a PCIe link, platform firmware, or the GPU itself. You need surrounding evidence to tell them apart.
2. An Xid does not automatically mean the hardware has failed. Xid 13/31/43 often involve application faults. Xid 45 commonly records cleanup after an application exits or another error occurs. Xid 63/92/154 are informational, but the associated memory state or recovery action may still require attention.^{2 3}
3. Read related Xids together and in timestamp order. Whether a memory error is contained, whether remapping succeeds, and what recovery action the driver requests determine whether to restart an application, reset a GPU, or reboot a host.

2. First, Find the Xid Messages

On Linux, the kernel log is the first place to look. dmesg reads the kernel’s in-memory ring buffer, and grep filters the output for Xid messages. Depending on the distribution and logging configuration, those messages may also be retained by journald or written to syslog files.

# Read the kernel ring buffer; -T shows human-readable timestamps.
sudo dmesg -T | grep -i xid

# Read kernel messages from the current and previous boots.
# The previous boot is available only if its journal was retained.
sudo journalctl -k -b | grep -i xid
sudo journalctl -k -b -1 | grep -i xid

# Traditional syslog files, if configured on this system.
sudo grep -i xid /var/log/messages     # RHEL / Rocky Linux
sudo grep -i xid /var/log/syslog       # Ubuntu / Debian

# Include 30 lines before and 5 lines after each match.
sudo dmesg -T | grep -i -B30 -A5 "xid"

Two practical habits make these logs much more useful:

• Treat the Xid line as a starting point. Earlier messages may show PCIe Advanced Error Reporting (AER) events, temperature warnings, or a second GPU failing at nearly the same time. These clues can be more useful than the Xid number alone.
• Save the logs before rebooting. The kernel ring buffer is lost at reboot, and older entries can be overwritten even during the same boot. Preserve the Xid and its surrounding messages before recovery. In production, retain kernel logs centrally or configure persistent journald storage so that evidence survives a reboot. Community troubleshooting notes also emphasize collecting this context.⁴

Once you have the messages, the next step is to connect them to what the system is doing. The following two examples show how.

3. Troubleshooting Example: Xid 79, “GPU Has Fallen Off the Bus”

This message means the driver can no longer reach the GPU over PCIe. The device may disappear from PCI enumeration, or it may still have an entry while failing to respond normally. From the application’s perspective, it resembles a USB drive being disconnected while in use: operations fail because the device is no longer accessible.

The symptom is straightforward; the cause can be anywhere from the power supply and PCIe path to firmware or the GPU itself.

3.1 What It Looks Like

# Kernel log: Xid 79 may be followed by a crash-dump collection request.
NVRM: Xid (PCI:0000:ca:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
NVRM: GPU 0000:ca:00.0: GPU has fallen off the bus.
NVRM: A GPU crash dump has been created. If possible, please run
NVRM: nvidia-bug-report.sh as root to collect this data before
NVRM: the NVIDIA kernel module is unloaded.

# nvidia-smi may fail to access the GPU.
Unable to determine the device handle for GPU 0000:ca:00.0: Unknown Error

# lspci may show failed configuration-space reads.
ca:00.0 3D controller: NVIDIA Corporation Device 2330 (rev ff)
    !!! Unknown header type 7f

Compare the physical GPU inventory with what you expect:

nvidia-smi --query-gpu=index,uuid,pci.bus_id --format=csv

Do not use the line count from nvidia-smi --list-gpus as a reliable physical GPU count: its output can include Multi-Instance GPU (MIG) instances or error messages. The query above may also fail if a GPU is unreachable; that failure is itself useful evidence.

Applications may report CUDA error: unspecified launch failure, NCCL’s unhandled cuda error, or a training job that hangs until it times out. These symptoms alone do not establish that Xid 79 occurred; confirm it in the logs.

3.2 Possible Causes

NVIDIA defines Xid 79 as a failure to access the GPU over its PCIe connection. A failed PCIe link is a common cause, but GPU hardware and driver problems are also possible.³ The categories below are troubleshooting hypotheses, not diagnoses:

GSP failures are discussed in Alibaba Cloud’s Xid 119/120 troubleshooting guide; they are a possible lead, not proof that every Xid 79 is a firmware issue.⁵

3.3 A Practical Troubleshooting Sequence

1. Establish the scope. How many GPUs are affected? Is this one host or a group of similar hosts? Did the driver, firmware, or BIOS change recently? If several GPUs fail together, investigate shared power and the PCIe/platform path early.
2. Read the surrounding kernel messages. Look for AER events, thermal warnings, or Xid 119/120 before Xid 79. Start with the preceding 30 lines and widen the time window as needed.
3. Check PCIe visibility. Run sudo lspci -s 0000:ca:00.0 -vv, replacing the address with the affected GPU’s full PCI address. A missing device or rev ff with an invalid header suggests that PCI configuration access is failing; it does not, by itself, prove a physical disconnection. If the GPU is unreachable, an ordinary nvidia-smi -r reset is unlikely to recover it. A host reboot, and sometimes a full power cycle, may be necessary.
4. Collect evidence before recovery. Run sudo nvidia-bug-report.sh before unloading the driver or rebooting so that available crash data is preserved.⁶ On bare metal, follow the host recovery procedure. In the cloud, follow the provider’s workflow: AWS documents reboot and stop/start troubleshooting, while Google Cloud’s current Xid 79 guidance uses host maintenance and repair. A VM reboot does not reboot its physical host, and stop/start is not a universal guarantee of different hardware.^{7 8} Before stopping an EC2 instance, preserve any required instance-store data, because stopping the instance erases it.⁹

5. Verify recovery before returning the GPU to service. Check the inventory again, then inspect the PCIe link:

   nvidia-smi --query-gpu=index,pci.bus_id,pcie.link.gen.current,pcie.link.width.current --format=csv
   

   Compare the results with the expected platform configuration and lspci -vv link capabilities. Current link speed and width can fall at idle because of power management; assess suspected degradation under an appropriate workload.¹⁰

6. If the problem returns, isolate the cause. With the machine powered down and following the system vendor’s service procedure, inspect or reseat the GPU, power connectors, risers, and cables. Test known-good components and supported BIOS/VBIOS/driver versions. Investigate PCIe Active State Power Management (ASPM) only when the symptoms and vendor guidance justify it; disabling it is a targeted diagnostic step, not a universal fix.¹¹ If the failure persists, contact the vendor about NVIDIA Field Diagnostics and an RMA assessment.²

4. Troubleshooting Example: ECC Memory Errors

4.1 ECC, SBE, and DBE

Memory can experience bit flips caused by electrical noise, radiation, or hardware faults. ECC (error-correcting code) adds information that lets hardware detect and, for some errors, correct corrupted data. Many NVIDIA data center GPUs provide ECC protection; the protected memory areas and reporting details depend on the GPU.

You will often see single-bit errors (SBEs) described as correctable and double-bit errors (DBEs) as uncorrectable. These are useful terms for reading Xid messages, though modern ECC schemes are more complex than this simple distinction. A corrected error generally does not interrupt the application. An uncorrectable error can terminate the affected application to avoid using corrupted data. Whether recovery also requires an immediate GPU reset or host reboot depends on containment and the driver’s recovery action.¹²

Start with two questions: Was the error corrected? If not, was it contained?

4.2 How the Messages Fit Together

ECC-related Xids often describe different parts of the same incident. The following map shows their roles; it is not a guaranteed log sequence, and not every incident produces every message:

High rate of corrected single-bit errors
   └─ Xid 92: high single-bit ECC error rate

Uncorrectable memory error
   ├─ Xid 48: uncorrectable ECC error detected
   ├─ Containment status, on supported GPUs starting with A100:
   │    Xid 94: contained; restart the affected application
   │    Xid 95: uncontained; recover the GPU or host as directed
   │    Xid 45 may also appear as application cleanup
   └─ Memory repair bookkeeping:
        Xid 63: page-retirement or row-remapping event recorded
        Xid 64: page-retirement or row-remapping recording failure

Xid 154, when reported: the required recovery action has changed
   None / Drain P2P / Drain and Reset / GPU Reset Required /
   Node Reboot Required

Row remapping replaces a faulty GPU memory row with a spare row in the same memory bank. On supported GPUs, starting with Ampere, pending remaps are applied when the GPU is reset. Earlier GPUs use page retirement, which removes affected memory pages from use. Both approaches avoid reusing faulty memory, but they operate differently. Newer architectures also have additional repair mechanisms.¹²

Xid 63 records remapping or retirement activity; check the actual pending state before scheduling recovery. Xid 64 indicates a failure in that process, which requires recovery and investigation. It does not, on its own, explain the failure or establish that the GPU must be replaced.

4.3 Check the State with nvidia-smi

Use these commands to inspect ECC counters and row-remapping status:

nvidia-smi -q -d ECC
nvidia-smi -q -d ROW_REMAPPER

The following excerpts are annotated examples, not complete command output. Field names and availability vary by GPU and driver; N/A means a field is unavailable, not that its value is zero.

ECC Errors
    Volatile                         <- Counts since the last driver load/reset of counters
        SRAM Correctable    : 0
        SRAM Uncorrectable  : 0
        DRAM Correctable    : 0
        DRAM Uncorrectable  : 0       <- If increasing, correlate with kernel logs
    Aggregate                        <- Persistent historical counts

Remapped Rows
    Correctable Error           : 0  <- Remaps associated with correctable errors
    Uncorrectable Error         : 0  <- Remaps associated with uncorrectable errors
    Pending                     : No <- Yes: a remap is waiting to be applied
    Remapping Failure Occurred  : No <- Yes: investigate and seek an RMA assessment

A nonzero aggregate count does not by itself mean the GPU is currently faulty. Correlate historical totals with recent increases, volatile counters, Xids, and remapping status. Likewise, a correctable-error remap count alone is not a reason to replace a GPU, but it should not be discarded without checking the broader state. Pay particular attention to new uncorrectable errors, Pending, and Remapping Failure Occurred.¹⁰ Crusoe’s operational guide provides additional examples of these fields.¹³

4.4 What to Do Next

Use the table below as a starting point. Read the entire incident before choosing an action: an Xid 63 does not override an accompanying uncontained error. When available, follow GPU Recovery Action, together with the guidance for the GPU model and platform.

These sources describe different hardware and recovery paths. In particular, NVIDIA’s older A100 guidance calls for an immediate node reboot for Xid 95 when MIG (Multi-Instance GPU) is disabled. With MIG enabled, it allows work on other GPU instances to drain before resetting the affected physical GPU. Do not assume that every uncontained error permits waiting for all work to finish.^{2 3}

About GPU reset: sudo nvidia-smi -r -i <index> resets the selected GPU without rebooting the host when the platform supports it. Replace <index> with the intended GPU’s current index. Root privileges are required, and no application may be using the devices being reset, including CUDA jobs, display servers, and monitoring processes such as another nvidia-smi instance. NVLink/NVSwitch topology, GPU generation, Fabric Manager, and virtualization can restrict individual resets. In a VM, reset availability depends on the hypervisor and provider. Verify GPU health afterward; a failed recovery may require a host power cycle or provider intervention.^{2 10 8}

4.5 When to Consider an RMA

An RMA (Return Merchandise Authorization) is the vendor’s process for returning faulty hardware for service or replacement. NVIDIA’s row-remapping policy defines conditions that can set the remapping-failure flag and trigger an RMA assessment. The documented conditions include:¹⁴

• A bank needs another uncorrectable-error remap after eight such rows have already been remapped.
• A previously remapped row needs to be remapped again.
• The GPU reaches 512 total remappings for uncorrectable errors.

Apply these thresholds within the policy’s supported hardware and repair model. Blackwell also introduces HBM channel repair, which can change the recovery path before row-remapping limits are exhausted. A single remap count is therefore insufficient to make a replacement decision across every GPU generation.

NVIDIA Field Diagnostics is the authoritative hardware diagnostic tool typically used before an RMA. Ask the system vendor or cloud support team when and how to run it, and retain its diagnostic log along with nvidia-bug-report.log.gz.² NVIDIA’s published Tesla RMA process also excludes failures that occur only with ECC disabled. For ECC-capable production GPUs, keep ECC enabled according to the platform’s operating guidance and confirm the applicable RMA requirements with the vendor.¹⁵

5. Putting the Troubleshooting Process Together

The goal is to identify which layer needs attention—application, driver, or hardware—and choose the appropriate recovery action. The two examples above lead to three habits:

1. Find the Xid and preserve its context. Start with sudo dmesg -T | grep -i xid, then read nearby messages and related codes. A single Xid is rarely enough to establish a root cause.
2. For Xid 79, investigate the PCIe and platform path early. Check PCI visibility, collect a bug report, recover the host or follow the cloud provider’s repair workflow, and verify the GPU before restoring service. Recurring failures need component isolation and vendor diagnostics.
3. For ECC errors, let containment and recovery state guide the response. Corrected errors and successful remapping events may permit continued work; a contained error generally requires restarting the affected application. Uncontained errors and remapping failures require prompt attention at the GPU or host level.

When you need help from a system vendor or cloud provider, describe what happened, when it happened, how it was detected, how many hosts or GPUs were affected, and how often it occurs. Include recent changes to the OS, driver, firmware, and application.² Attach:

• The OS, kernel version, GPU model, and driver version.
• Xid messages with surrounding context, preferably complete dmesg or journalctl output for the incident.
• The output of nvidia-smi -q, if the GPU is accessible.
• Troubleshooting steps already taken and their results.
• The nvidia-bug-report.log.gz file generated by sudo nvidia-bug-report.sh.

Appendix: Common Data Center Xids at a Glance

NVIDIA’s Xid Catalog lists immediate and investigative actions for Ampere and newer GPUs, with applicability marked by model. Examples include IGNORE, RESTART_APP, RESET_GPU, RESTART_BM, and CONTACT_SUPPORT.³ This table summarizes the catalog alongside GPU Debug Guidelines and provider guidance; it is not a replacement for the model-specific workflow. For Volta and older GPUs, consult the archived documentation.¹⁶

The action labels distinguish the scope of recovery:

• RESTART_APP: Restart the affected application and investigate why it failed. This action alone does not require a GPU reset.
• RESET_GPU: Reset the affected GPU with sudo nvidia-smi -r -i <index>, subject to the access and topology restrictions described above.
• RESTART_BM: Reboot the bare-metal host. Cloud users must follow the provider’s equivalent recovery procedure; rebooting a guest VM is not a physical host reboot.
• CONTACT_SUPPORT / RUN_FIELDDIAG: Contact the system vendor or cloud provider, or run NVIDIA Field Diagnostics as instructed. Diagnostics determine whether an RMA is appropriate.

Xid 154 deserves special attention. On supported driver/GPU combinations, it can follow another Xid and report a recovery-state change, for example:

Xid 154: GPU recovery action changed from 0x0 (None) to 0x2 (Node Reboot Required)

The GPU Recovery Action field in nvidia-smi -q provides the same kind of guidance. Possible states include no action, a GPU reset, a node reboot, draining peer-to-peer activity (Drain P2P), and draining work before a reset (Drain and Reset). Here, drain means stopping new affected work and letting eligible existing work finish before the next recovery step. Use the specific state reported by the driver rather than treating every drain request as the same operation. The catalog documents Xid 154 linkage for several codes in the R565 driver family; availability depends on the driver and GPU.^{3 10}

Appendix: Terminology

References