Sample diagnostic report: two GB10 nodes at less than half speed

1 · Summary

Nodes A and B were running matrix-multiply work at less than half the speed of an identical healthy node. Under full load the GPU on node B held about 851 MHz and 20.7 W, against about 2,242 MHz and 89.7 W on the healthy reference. No throttle reason was reported and no software setting differed. A normal reboot did not clear it. A full power drain did: both nodes now measure within the healthy range of our other units. No hardware replacement was needed.

2 · Findings, with measured evidence

F1. Nodes A and B deliver less than half the GPU throughput of a healthy identical node. Test: a 45-second bf16 matrix multiply (8192 × 8192 × 8192), same container and kernel on every node, run with the GPU otherwise idle and compared on TFLOP/s.

Node A showed the same pattern. Its own before-fix figure (about 42 TFLOP/s) was not captured in the same paired run; see section 7.

F2. No software signal points to the cause. Every clock throttle reason (software power cap, hardware slowdown, thermal) read "Not Active" on the slow nodes. The CPU frequency governor and the ACPI platform profile were identical to healthy nodes, and the power-mode tool found on some other NVIDIA platforms is not present. The only evidence was the throughput gap itself.

F3. The common remedies do not clear it. A software reboot brought the node back in about 100 seconds with the clock unchanged. Locking the clock higher with nvidia-smi -lgc was accepted ("All done") and changed nothing.

F4. Idle readings are misleading. GPU clocks float when idle, so idle clock readings could not tell slow and healthy nodes apart reliably. Only a paired test under full load did.

3 · Root cause class

A latched low-power state somewhere in the unit's power management; we did not identify the mechanism (see section 7). It survived a warm reboot, and no operating-system or driver setting we checked changed it. The silicon performs at spec after the drain, so it is not a permanent hardware fault, and no software setting we checked was involved. In our two cases it cleared only after a full power drain.

4 · Recommended fix

A full power drain, in this order:

• Shut the node down completely.
• Unplug everything: the cord at the wall outlet, the cord into the power brick, and the brick's cable at the back of the unit.
• Wait at least 5 minutes.
• Reconnect the wall outlet first, then the power brick, then the unit last. Power on.
• Re-run the 45-second check before putting the node back into service.

Estimated work: minutes per node, plus the 5-minute wait. No follow-on engagement needed. The step-by-step version, with a free check script, is public: Is your GB10 stuck in low-power mode?

5 · Verification

The reference column is one healthy node measured in the same session. Healthy units of this model ranged from 88.8 to 95.3 TFLOP/s across units and days, so "after" figures slightly above the reference are within that spread, not a gain beyond normal.

Later re-checks with the published check script: node A measured 91.7 TFLOP/s (healthy) on 26 Oct 2026, and both nodes passed again on 28 Oct 2026.

6 · Recoverable headroom

On each affected node, GPU matrix-multiply throughput went from 42.5 to 93.5–95.3 TFLOP/s, about 2.2 times. Before the fix, our plans for 8-node layouts had to work around only six healthy nodes. After it, all eight were usable, so 8-way tensor-parallel layouts were back on the table.

7 · What we could not confirm

• Which component latches, and what triggers it. We fixed the state; we did not identify its mechanism. We have reported the case in NVIDIA's developer forum, where NVIDIA staff have asked affected owners for details and diagnostics.
• When it started. The slowdown produced no log entry, so we cannot date its onset or say how long the nodes ran slow.
• Node A's exact before-fix numbers in the paired test. Its approximate figure comes from earlier runs, not the same session as node B's.
• Whether it will recur. Both nodes have stayed healthy at every re-check so far. Other owners report units that needed longer drains, and at least one owner reported a drain that didn't work.

8 · Method and reproducibility

Paired runs on the same container and kernel, compared on delivered throughput (TFLOP/s), never on idle clock readings. Anyone with the same hardware can re-run the test with our free check script and compare against the reference table there.

A client report also includes the session record and its fingerprint when the work touched client systems, and a scoped fixed bid or hourly estimate when the fix is real work. Neither applied here: this work ran in our own lab, and the fix took minutes.