Something crazy happened tonight
1. Turned on the ceiling light in the room, where the server is located. The server is plugged into a surge protector.
2. Computer starts making this beeping noise. I don't have a motherboard speaker plugged into it, so yeah... Hard drives probably.
3. Rebooted the server and the sound went away. The server was currently doing zpool scrub, but it should be fine with a hard reboot.
4. Check the webgui on my PC and it seems the server isn't booting into the OS (headless server).
5. Move the server into my room and hooked it up to a monitor.
6. Bootloader says "ZFS: i/o error - all block copies unavailable." It can't find root file system to mount.
7. Turns out the molex to 4x SATA power adapter got unplugged (possibly during transport to my room). This mean only one HDD had power, when I tried to boot the system in my room.
8. Re-plugged the adapter back in and the server booted into OS as normal.
9. Checked SMART Info and
10. This is what I get.
ada0: New Seagate 1.5 TB (ST1500DL003-9VT16L CC3C)
ada1: My oldest Seagate 1.5 TB (ST31500341AS CC1H)
ada2, ada3, ada4: New Seagate 3 TB (ST3000DM001-9YN166 CC4B)
1. ada2's power cycles and lifetime is insanely high after this event. I bought these 3 TBs in November 2012.
2. ada2, ada3, and ada4 have really high load cycles.
1. The SMART Info reported for ada0 (newest 1.5 TB) is actually the SMART Info for my ada1 (older 1.5 TB). For some reason they are swapped.
2. Cable Errors on ada0 & ada4, did like 4-5 cable swaps already. Going to buy 3 more new SATA cables to replace the last 3 salvaged cables from older motherboards.
3. I don't even think the power cycle numbers and lifetime are even accurate. I had the older 1.5 TB for a few years and it seems the 3TB alright caught up to it in lifetime and power cycles. Doesn't make any sense to me.
ZERO DATA CORRUPTION/LOST!
pool: storage state: ONLINE scan: scrub in progress since Mon Mar 4 00:02:06 2013 2.76T scanned out of 5.90T at 392M/s, 2h20m to go 0 repaired, 46.77% done config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gpt/raidz1-1 ONLINE 0 0 0 gpt/raidz1-2 ONLINE 0 0 0 gpt/raidz1-3 ONLINE 0 0 0 errors: No known data errors
What I'm planning on doing:
1. Buy new SATA cables
2. Use Acronis Drive Manager to update the Cable Error threshold to show drives at 100% healthy again.
3. Zpool Scrub my Mirror zpool.