Blog

I recently connected to my TrueNAS server to find a critical alert! My main photo backup pool was degraded, and a drive was completely missing from the configuration.

The message was terrifying, but it contained a glimmer of hope: "Sufficient replicas exist." My RAIDZ1 setup was doing its job, keeping my data accessible with one drive down. But I was now flying without a safety net. It was time to troubleshoot, and I feared the worst for my month-old 8TB WD Red Plus drive.

Phase 1: The "Zombie" Drive Boot Loop

My first instinct was to reboot the server, hoping it was just a transient glitch. That was a mistake. The server wouldn't shut down cleanly. The web interface died, but the machine stayed powered on.

Connecting a monitor revealed the issue. The operating system was hanging, waiting indefinitely for the missing drive to synchronize its cache before powering off. It was stuck in a loop, throwing I/O errors and timing out, unable to get a response from the problem disk.

[1177756.052818] sd 1:0:0:0: [sdb] tag#11 timing out command, waited 180s
[1177756.052828] I/O error, dev sdb, sector 15628052*** op 0x0:(READ) flags 0x800 phys_seg 13 prio class 0
...
[1177776.096789] sd 1:0:0:0: [sdb] Synchronizing SCSI cache
[1178136.098667] sd 1:0:0:0: [sdb] tag#22 timing out command, waited 360s

With the keyboard unresponsive, I had to perform a hard reset by holding the power button for 10 seconds. I decided to try booting it back up with the drive still connected to see if I could get SMART data, fully prepared to force-kill it again if it hung. Fortunately, the "zombie" drive responded just enough to let the system boot.

Phase 2: Investigation & Mistaken Identity

Once back in the web GUI, the pool was still degraded, but the system was live. I needed to verify which physical drive was the culprit. Since Linux drive letters (like /dev/sdb) can change between reboots based on the order drives respond, I couldn't rely on the old error logs. I needed to map the current drive letters to their serial numbers.

I jumped into the shell and used lsblk to find my missing drive:

[username]@truenas[~]$ lsblk -o NAME,MODEL,SERIAL,SIZE
NAME        MODEL                    SERIAL            SIZE
sda         WDC WD80EFPX-68C4ZN0     WD-RD2Y****       7.3T
sdb         WDC WD80EFPX-68C4ZN0     WD-RD2S****       7.3T
sdc         WDC WD80EFPX-68C4ZN0     WD-RD39****       7.3T
nvme0n1     WD Red SN700 500GB       252071*****       465.8G

Aha! The drive with serial number WD-RD39**** (the one TrueNAS complained about) was currently assigned the letter sdc. The GUI also showed sdc with multiple errors. I had my target.

Phase 3: The Smoking Gun (SMART Data)

It was time to interrogate the drive with smartctl. Given the symptoms, I was expecting to see reallocated sectors or other signs of physical death. I ran the command on the target drive letter:

sudo smartctl -a /dev/sdc

The output was surprising. The overall health test passed, and the critical physical health indicators were perfect:

• ID 5 (Reallocated_Sector_Ct): 0
• ID 197 (Current_Pending_Sector): 0

The drive platters were physically fine. But scrolling down to the vendor-specific attributes revealed the real culprit:

ID# ATTRIBUTE_NAME         FLAG     VALUE WORST THRESH TYPE      UPDATED WHEN_FAILED RAW_VALUE
...
199 UDMA_CRC_Error_Count   0x0032   200   127   000    Old_age   Always       -          185

UDMA CRC Error Count: 185. This error means data is being corrupted in transit between the hard drive's logic board and the motherboard's SATA controller. The drive was perfectly healthy, but the SATA cable it was using to talk to the server was full of static. TrueNAS had kicked the drive out because it was untrustworthy. This is almost always caused by a faulty or poorly seated SATA cable.

Phase 4: The Fix & Verification

The fix was simple, and thankfully cheaper than a new 8TB drive. I shut down the server, opened the case, and replaced the SATA data cable connected to the drive with serial WD-RD39**** with a known-good spare. After booting back up, I needed to verify the fix.

I ran the SMART check again on the drive's new location to monitor the error count.

sudo smartctl -a /dev/sda

The result confirmed the fix. The UDMA_CRC_Error_Count was still at a raw value of 185. Since this is a lifetime counter that never resets, the fact that it didn't increase to 186 meant that zero new communication errors had occurred during the boot process with the new cable. The connection was stable.

Bringing the Pool Back to Life

Even though the hardware was fixed, TrueNAS still showed the pool as DEGRADED because it remembered the old error state. I had to manually tell ZFS to clear the error history for that pool.

sudo zpool clear photo-backup

Finally, to ensure complete data integrity and prove the drive wouldn't drop offline under load, I ran a full pool scrub from the GUI. This forces ZFS to read every block of data, verify its checksum, and repair any inconsistencies from parity data.

The scrub finished in about an hour with 0 checksum errors. The pool returned to a healthy ONLINE state, and my data was safe.

Conclusion

What looked like a disastrous hard drive failure turned out to be a simple $5 faulty SATA cable. This experience was a great reminder to never assume a drive is dead without digging into the SMART data first. A high UDMA_CRC_Error_Count is a dead giveaway for connection issues, not drive failure.

The scariest part of this ordeal was that I only discovered it by chance when I logged into the dashboard. A silent failure is the worst kind. In my next post, I'll document how I set up email and push notifications in TrueNAS Scale so the server can scream for help the moment something goes wrong, long before a drive gets kicked out of the pool.