Before We Try to Break It

Part 1 covered why I rebuilt my local AI system around WRX90, a Threadripper PRO 9955WX, 128 GB of eight-channel ECC RDIMM, two RTX 3090 Founders Edition cards, Ubuntu, air cooling, and hardware-level remote management. When almost $11,000 in tracked hardware acquisition value sits inside one case, “it boots” is not a result. It is the beginning of the test.

Part 2 covers the commissioning of the machine: verifying that the hardware is stable, establishing its real operating limits, documenting the mistakes made during testing, and deciding which workloads I can trust it to run unattended.

The build did not come together in one clean weekend. Life happened. I went back and forth to my research lab, making progress whenever I could. It was frustrating to pause my AI research while finishing hardware assembly, firmware configuration, memory qualification, networking, cooling, and recovery infrastructure.

But this is part of building a serious local AI system.

You do not assemble an expensive machine and immediately jump into training, fine-tuning, or running the newest Qwen model. First, you have to prove that the hardware underneath the research is trustworthy. Otherwise, when something eventually fails, you will not know whether the problem came from the model, the inference engine, the operating system, a GPU, system memory, cooling, or your own test harness – turtles all the way down, so to speak. You want good baseline data from a new build, and keeping it very contained and simple allows you to limit surface area for later issues when you need to diagnose the problem.

Commissioning Is Not Benchmarking

A benchmark asks how fast a system is under a defined workload. Commissioning asks whether the complete system is ready to perform its intended job safely, correctly, and repeatedly.

I did not need one giant number that made the machine look impressive. I needed a chain of evidence covering CPU correctness, GPU correctness, VRAM integrity, system-memory stability and integrity, inter-GPU communication, storage health, networking, operating-system recovery, out-of-band recovery, power-loss behavior, and the deliberately ugly combined-load corner. After spending more than $4,000 on the RAM kit, memory qualification was especially important. Every result also needed a boundary.

If a VRAM test covered 21.750 GiB, I would not call it all 24 GB. If one GPU recorded a few seconds of software thermal slowdown during an hour, I would not hide it. If a test command launched incorrectly, I would invalidate the run instead of promoting it simply because the machine did not crash. The objective was not a flawless story. The goal was a trustworthy machine. Very rarely does any build come together without issues — and if one did, I would be EXTREMELY worried. Problems always arise, and it’s best to accept the inevitable, which, believe it or not, calms you down.

The full methodology, normalized data, reusable GPU test utilities, operating limits, failure ledger, and privacy policy are published in the Part 2 commissioning repository.

The Results at a Glance

Gate	Result	Evidence headline
CPU ramp — 32 workers, 10 minutes	Pass	32/32 workers; 82.2°C peak; ~231 W package
CPU endurance — 32 workers, 60 minutes	Pass	32/32 workers; 81.9°C peak; ~233.3 W package
Individual GPU deterministic compute	Pass	~79.6 workload TFLOPS per card; zero mismatches
Dual-GPU endurance — 60 minutes	Pass with observation	518,470 verified iterations; zero mismatches; GPU 0 had less thermal margin
Verified-pattern VRAM	Pass	21.750 GiB and 63.508 TiB checked traffic per card; zero mismatches
Two-GPU NCCL	Pass	Collectives through 256 MiB; zero wrong values
Online and offline ECC memory at 4800	Pass with observation	96 GiB online plus 2/2 offline passes; zero errors or ECC events; one transient CPU thermal sample
Online and offline ECC memory at 6000	Pass	96 GiB online plus 2/2 offline passes; zero errors or ECC events
Network and recovery	Pass	Warm, cold, AC-loss, and required-service recovery exercised
Deliberate maximum synthetic CPU plus dual-GPU saturation	Thermal boundary	Computation remained correct; CPU regulated at 95°C; not representative of normal AI inference

What the 95°C result means: Normal unattended AI inference remains approved. The boundary appeared only after layering 28 synthetic CPU stress workers on top of two independent GPU stress workloads running both RTX 3090s near their stock 350 W limits. Only unattended operation of that deliberately constructed combined corner is restricted.

The final row is the easiest result to misunderstand. The CPU reaching 95°C does not mean the server cannot run unattended, nor does it mean normal AI inference pushes the machine into that condition.

That temperature appeared during a deliberately pathological synthetic test that layered 28 CPU stress workers on top of two independent GPU matrix workloads running both RTX 3090s near their stock 350 W limits. The purpose was to manufacture the worst combined thermal condition I could reasonably create—not to reproduce a normal inference workload.

The distinction matters, and I will return to it later in this report.

CPU and GPU Endurance

The Threadripper PRO 9955WX has 16 physical cores and 32 logical threads. A ten-minute ramp across all 32 logical CPUs finished with every worker correct, an 82.2°C peak, and approximately 231 W of package power. The one-hour CPU-only endurance run also finished 32 for 32. Temperature peaked at 81.9°C at approximately 233.3 W.

That established a clear result: the Noctua air cooler and chassis airflow could sustain the selected CPU-only workload without approaching the processor’s 95°C operating boundary.

Each RTX 3090 was then tested independently with the same deterministic FP16 matrix workload. GPU 0 completed 21,710 verified iterations at approximately 79.55 workload TFLOPS. GPU 1 completed 21,780 at approximately 79.80. Neither produced a mismatch.

Those TFLOPS figures belong to this particular test. They are not universal gaming scores, training-performance numbers, or a claim that every AI kernel will behave the same way.

Both GPUs then ran together for 15 minutes, followed by a one-hour dual-GPU endurance test. The long run produced 518,470 combined verified iterations with zero mismatches. GPU 1 remained straightforward: 66°C, approximately 335 W, and no slowdown-counter increase. GPU 0 reached 70°C at approximately 348 W and accumulated about 3.97 seconds of software thermal slowdown during the hour—roughly 0.11% of the run. There was no hardware thermal slowdown, Xid event, computation mismatch, or material throughput collapse.

The correct conclusion is not that GPU 0 failed. The correct conclusion is that the upper card passed with less thermal margin than the lower one. This is expected. These are non-blower style cards – which means air is blown upward inside the case. GPU 1 (the bottom GPU/second GPU) is below GPU 0 (the top GPU/first GPU). GPU 1 just dumps its air upwards towards the GPU above it, causing the discrepancy.

Testing the Memory on the GPUs

A GPU can multiply matrices correctly and still have a memory problem outside the region touched by that workload. I therefore ran verified-pattern VRAM testing on each card.

Each GPU allocated 21.750 GiB, completed 598 full pattern cycles, and moved 63.508 TiB of checked traffic with zero mismatches.

That covered approximately 92.3% of CUDA-visible memory. It is extensive coverage, but it is not every physical byte of the advertised 24 GB, and I will not round a strong result into a false one.

In practical inference, consuming every final megabyte is not desirable anyway. Inference engines need working space for compute buffers, temporary allocations, KV cache growth, and other runtime activity. If model placement leaves no headroom, a small allocation spike can end the run with the dreaded out-of-memory error.

The exact amount of required headroom depends on the model, backend, context size, batch size, cache configuration, and workload. The commissioning result proves the tested region. It does not create a universal VRAM-reserve rule.

The machine also does not contain one magical 48 GB GPU. There is no NVLink bridge installed, and direct GeForce PCIe peer access was unavailable in the qualified path. NCCL used host shared memory for communication between the cards.

All-reduce, all-gather, and reduce-scatter completed correctly through 256 MiB messages with zero wrong values. The path works. It is not equivalent to direct P2P or NVLink, and the 48 GB figure remains combined physical capacity that software must explicitly divide.

The First Test I Invalidated

The initial “96 GiB” system-memory test was wrong. I launched the tool with parameters I believed would allocate 96 GiB across multiple workers. The process ran. The terminal looked busy. The machine did not crash. Actual resident-memory use was only around 14–16 GiB.

That run is invalid.

I stopped it, documented the mistake, corrected my understanding of the tool’s allocation semantics, and relaunched the workload while independently verifying actual memory residency.

The corrected DDR5-4800 run exercised 96 GiB for one hour, completed approximately 4.097 billion operations across eight workers, and left the ECC error state clean.

Linux recorded one transient 91.5°C CPU sample while the online-memory methods changed. The next ten-second sample was 86.2°C, the slower BMC maximum was 88°C, and no threshold event appeared.

That sample did not invalidate memory correctness, but it gave me another reason to investigate CPU thermal margin under more aggressive combined loading.

Offline memory testing followed: two complete passes, 24 of 24 tests, zero memory errors, and zero ECC events in 2 hours, 29 minutes, and 48 seconds.

The lesson is larger than one incorrect command. A configuration file or busy terminal window is not evidence that the intended workload actually occurred. Admission, allocation, residency, and live behavior must be observed.

The RAM Side Quest

Only after the conservative DDR5-4800 baseline passed did I move to 6000. The first fully qualified 6000 state used the kit’s Profile #1 with its frequency reduced from 6400 to 6000. It ran at 32-39-39-80 with the BIOS-displayed 1.40 V profile.

That configuration passed a 30-minute 96 GiB online workload with eight of eight workers, approximately 2.15 billion operations, and clean ECC reporting. It then passed another two offline passes covering all 24 tests with zero errors or ECC events.

Later, I switched to the kit’s native 6000 Profile #2: 32-38-38-80 with a lower BIOS-displayed 1.35 V profile. The first attempt booted normally—but the running system revealed that memory was still operating at 4800 because frequency had been left on Auto. I returned to the BIOS, explicitly selected 6000, and verified the active state after repeated boots.

Profile #2 passed a deliberately narrower transition gate: all eight DIMMs detected, full capacity available, repeated successful boots, clean EDAC state, and clear platform fault indicators. It did not receive another complete online and offline stress cycle. The underlying 6000 frequency had already passed those tests under Profile #1, but that does not allow me to claim Profile #2 completed testing that I chose not to repeat.

Why I Stopped at DDR5-6000

At the qualified 6000 state, five pinned STREAM-style measurements reached 113.0–114.1 GB/s COPY and 119.3–119.9 GB/s TRIAD.

I did not run a controlled local DDR5-6000-versus-6400 comparison. I stopped at 6000 because the selected CPU-side workload was already topping out near 120 GB/s, far below the simple theoretical bandwidth of eight DDR5-6000 channels. Increasing memory frequency would widen the DRAM side of the path, but it would not automatically widen the part of the path limiting this 16-core, two-CCD processor.

The finding matters for large mixture-of-experts models. When model experts spill from the GPUs into system memory, decode performance can become heavily dependent on how quickly the CPU can consume that memory. More DIMM bandwidth on paper does not guarantee more usable CPU-side bandwidth.

That result made a future 24-core or 32-core Threadripper PRO more interesting—but it did not manufacture a guaranteed performance multiplier. A future CPU must still earn that claim through measurements on this machine.

The Threadripper PRO CPU Upgrade Ladder

The installed 9955WX result in the following chart was measured on this machine. The bandwidth, bottleneck, decode, and cost-efficiency figures for the remaining processors are estimates based on the measured baseline, expected CCD scaling, the installed eight-channel memory configuration, and the specific behavior of RAM-offloaded MoE inference.

They are projections for future testing—not completed benchmarks.

The Test That Found the Wall

CPU-only endurance passed. Dual-GPU endurance passed. Then I deliberately combined them. The final test ran 28 CPU stress workers while both RTX 3090s executed independent deterministic FP16 matrix workloads at their unchanged stock 350 W power limits. The computation remained correct. The GPUs did not overheat. The power supply did not trip. No cable melted. Nothing caught fire.

The CPU reached 95°C, AMD’s published maximum operating temperature for the Threadripper PRO 9955WX. GPU 0 peaked at 64°C and 349.80 W. GPU 1 peaked at 63°C and 344.75 W. Both retained PCIe Gen4 x16, neither recorded thermal slowdown, all 28 CPU workers passed, and both GPU workers completed with zero mismatches.

ECC reporting, hardware-error reporting, monitored power rails, NVMe health, and required services remained clean. The machine stayed correct. The CPU cooling margin did not.

What 95°C Actually Means

This result does not mean the server cannot operate unattended. It does not mean the CPU failed, and it does not mean a normal AI workload will automatically produce the same condition. The test intentionally combined three independent synthetic stress workloads:

• 28 CPU workers designed to sustain heavy all-core CPU activity.
• One deterministic matrix workload holding GPU 0 near its 350 W stock limit.
• One deterministic matrix workload holding GPU 1 near its 350 W stock limit.

That is a deliberately ugly thermal scenario.

GPU-resident inference can drive both GPUs hard, but the CPU generally handles orchestration, tokenization, networking, and other supporting work rather than simultaneously running a separate maximum all-core stress test. Hybrid CPU/GPU inference can use the CPU, system memory, PCIe links, and both GPUs, but it still does not automatically reproduce the artificial condition created by layering a dedicated CPU stress workload over two independent GPU stress workloads.

A custom training, preprocessing, rendering, compilation, or mixed-compute workflow could potentially approach this corner. That is why finding it matters. But it is not representative of the AI inference workloads for which this server was built and later tested. The 95°C result therefore defines one restricted corner of the operating envelope. It does not invalidate the rest of the envelope.

Normal unattended GPU inference remains approved. Sustained CPU-only work remains approved. Sustained dual-GPU work remains approved. The measured hybrid CPU/GPU inference configurations remain approved. Only unattended operation of the deliberately constructed maximum-CPU-plus-maximum-dual-GPU synthetic workload is restricted.

The Safety Wrapper Also Failed

One mistake made the combined test more uncomfortable than it needed to be. The safety wrapper detected 95°C after roughly eight minutes, but the signal originally used did not terminate the background workload. The load continued until its normal 20-minute completion.

The processor’s hardware thermal regulation did its job, and computation remained correct, but a safety wrapper that detects a stop condition and fails to stop the workload is still broken. I corrected the termination behavior and preserved the operating restriction.

That is another important distinction: the hardware did not produce incorrect work. The test-control mechanism failed to perform one of its intended actions.

The Final Operating Policy

• Sustained CPU-only work: approved.
• Sustained dual-GPU work: approved.
• GPU-dominant inference: approved.
• Measured hybrid CPU/GPU inference: approved.
• Separately scheduled heavy workloads: approved.
• Normal unattended local-AI operation: approved.
• Unattended maximum CPU plus maximum dual-GPU synthetic saturation: not approved in the current cooling configuration.

This is key to understand. The machine is essentially, chef’s kiss, perfect. The chances of repeating the max load GPU and CPU are in my mind – less than 1%. And even if I do have some insane workload to push the machine to this limit – we can slow down the process to drop the CPU temp a few degrees to put it below its 95°C threshold. In the end, it was a mission success.

Recovery Had to Be Real

The motherboard’s BMC operates independently of Ubuntu and independently of both RTX 3090s. The compute GPUs remain display-free while the management controller exposes POST, BIOS, sensor data, and console recovery through an isolated path. SSH is the normal route. Out-of-band management is the recovery route when SSH or Ubuntu is unavailable.

I exercised warm reboot, normal shutdown and cold start, complete AC removal, out-of-band reset, service recovery, and the BIOS Last State behavior in both directions. When AC disappeared while the machine had been running, it returned after power restoration. When it had been intentionally shut down, it remained off. That turned “the motherboard supports remote management” into “this machine can actually be recovered remotely.”

What Passed, What Failed, and What Changed

The hardware did not fail commissioning. The test process failed in useful ways.

• The first memory launch did not allocate what I thought it allocated.
• The first Profile #2 boot did not activate the frequency I thought it activated.
• The combined-load wrapper did not stop the workload when it thought it had stopped it.
• One proposed power-supply telemetry path turned out to be unsupported.

Each failure changed the procedure:

• Verify residency instead of trusting command-line intent.
• Verify the active configuration from the running machine instead of trusting the selected firmware profile.
• Test the emergency-stop mechanism as seriously as the workload itself.
• Reject unsupported integrations instead of forcing them into the system.
• Preserve observations that complicate the story.

A commissioning report that hides invalid runs is just marketing with extra columns. The complete failure ledger is public: read the failures and corrections.

A Passed Machine With a Known Limit

The server did not leave commissioning as a flawless object. It left as something better: a measured system with a defined operating envelope. The CPU passed sustained CPU-only work. Both GPUs passed individual and simultaneous deterministic compute. More than 21 GiB of VRAM per card survived verified-pattern testing. Eight channels of ECC memory passed online and offline qualification at the fully tested states. Storage remained healthy. NCCL worked correctly through the transport available on this hardware.

Recovery behavior was rehearsed instead of assumed. Normal unattended AI operation remains approved. When I pushed the CPU and both GPUs into the most aggressive synthetic combined corner I could reasonably construct, the CPU cooler showed me its boundary. That does not mean the server cannot operate on its own. It means I now know one specific workload combination that I will not leave unattended without changing the cooling configuration or limiting the workload.

Commissioning is not a ritual performed after a build so the report looks professional. It is the process that turns expensive hardware into infrastructure you understand. No matter what you build – it’s a machine, it has atoms, it has limits.

In Part 3, we move from synthetic validation to real open models—and find out what this machine can actually do.

Follow along at blog.zacharycangemi.com or explore the commissioning report in the Local LLM Infrastructure V2 commissioning repository.