WhenKittensATK Posted January 26, 2015 Posted January 26, 2015 (edited) I came home last night and noticed my server (Ubuntu Server 14.04.1 LTS) was beeping. So, I just did a reboot and let it be. This morning when I went to check it out, all the hard drives on the M1015 card weren't showing up. I tried all the normal physical troubleshooting like replugging card/cables, different PCIE slots, and even 2 different computers. What's interesting is on boot-up it no longer shows the "Press key to boot into M1015 bios" like it normally does. Checked the syslogs in Ubuntu and all it showed was zfs doing auto snapshots without any errors until I did the reboot. I also tried updating motherboard BIOs, as expected that did nothing. This is the syslog of it booting up. It does show the Toshiba and Seagate HDDs on the M1015, so it has to be working in some extent. No clue where to go from here. I could try flashing it to the default mode and then back to IT mode, but that would have to wait for another day. http://pastebin.com/5mtpAj9g Edited January 26, 2015 by WhenKittensATK Share this post Link to post Share on other sites More sharing options...
Waco Posted January 26, 2015 Posted January 26, 2015 How many drives do you have? Whatever device is at 4:0:5 seems to be causing an issue...and it may very well be the RAID card. Once it's up, what does lsscsi output? Or does it not get to the point where you can bang on a command line? I'm guessing none of the drives show up in /dev based on that syslog... If you have enough SATA ports, I'd start by moving your drives to them and getting your server back up. You can always move them back to the LSI card once you get things sorted out. Share this post Link to post Share on other sites More sharing options...
WhenKittensATK Posted January 26, 2015 Posted January 26, 2015 (edited) Yeah looks like one of the Seagate drives went bad. I can remove my two 1.5TBs and move the five working 3TBs over to the motherboard. peter@UbuntuZFS:~$ lsscsi [0:0:0:0] disk ATA ST1500DL003-9VT1 CC3C /dev/sda [1:0:0:0] disk ATA ST31500341AS CC1H /dev/sdb [3:0:0:0] disk ATA WDC WD6400AAKS-0 01.0 /dev/sdc [4:0:0:0] disk ATA TOSHIBA DT01ACA3 ABB0 /dev/sdd [4:0:1:0] disk ATA TOSHIBA DT01ACA3 ABB0 /dev/sde [4:0:2:0] disk ATA TOSHIBA DT01ACA3 ABB0 /dev/sdf [4:0:3:0] disk ATA ST3000DM001-9YN1 CC4B /dev/sdg [4:0:4:0] disk ATA ST3000DM001-9YN1 CC4B /dev/sdh Edited January 26, 2015 by WhenKittensATK Share this post Link to post Share on other sites More sharing options...
WhenKittensATK Posted February 3, 2015 Posted February 3, 2015 Finally got around to trying to fix the server. Took out the second Seagate 3TB and the rest of the hard drives pulled up. Seems strange one failed hdd would cause that. I'll have to replace it sometime. Never again Seagate. Share this post Link to post Share on other sites More sharing options...
Waco Posted February 3, 2015 Posted February 3, 2015 Badle cable perhaps? Share this post Link to post Share on other sites More sharing options...
WhenKittensATK Posted February 3, 2015 Posted February 3, 2015 Yeah I'll probably buy two pairs of cable with the HDD replacement. Found the same ones for a few dollars cheaper than Amazon on eBay. HGST Deskstar NAS H3IKNAS30003272SN 3TB vs TOSHIBA PH3300U-1I72 3TB as a replacement? Share this post Link to post Share on other sites More sharing options...
Waco Posted February 4, 2015 Posted February 4, 2015 I mean, perhaps the drive is fine. Did you try another SATA cable before calling it dead? EDIT: Ah, it's a SAS->SATA breakout cable. I'd try it before calling the drive dead, unless you can pop the drive into another machine to confirm it's deadness. If you're using ZFS, I'd get the Toshiba since it's cheaper and you don't really need the TLER stuff on the Hitachi. I'd personally still go with the Seagate Barracuda though, or the cheaper bare Toshiba. EDIT: If you're comfortable with refurbs, the WD Green and Barracuda XT are ~$90 each. Share this post Link to post Share on other sites More sharing options...
WhenKittensATK Posted February 4, 2015 Posted February 4, 2015 (edited) I plugged the hdd into the motherboard's sata and its not showing up in "fdisk -l" and "parted print all." I'll try some other SATA cables once I can find where I put them all. EDIT: Also plugged the HDD into my SATA to USB HDD kit and no luck there. I think I'll go with the retail Toshiba, since those have 3 year warranty vs 2 year with OEM drives. Edited February 4, 2015 by WhenKittensATK Share this post Link to post Share on other sites More sharing options...
WhenKittensATK Posted February 15, 2015 Posted February 15, 2015 Oh man. When I hit "enter" on that zfs replace command, the HGST jumped to life and now sounds like a jet engine haha. Scared the crap out of me. All silent then a roaring sound. pool: storage state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sat Feb 14 22:22:36 2015 110G scanned out of 13.8T at 352M/s, 11h21m to go 30.4G resilvered, 0.77% done config: NAME STATE READ WRITE CKSUM storage DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 ata-ST3000DM001-9YN166_W1F16H3L ONLINE 0 0 0 replacing-1 OFFLINE 0 0 0 603394605016626068 OFFLINE 0 0 0 was /dev/sdg2 ata-HGST_HDN724030ALE640_PK1234P8JHWM3X ONLINE 0 0 0 (resilvering) ata-ST3000DM001-9YN166_W1F15KYX ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 ata-TOSHIBA_DT01ACA300_236DEX8GS ONLINE 0 0 0 ata-TOSHIBA_DT01ACA300_236BEUGGS ONLINE 0 0 0 ata-TOSHIBA_DT01ACA300_236D8NGGS ONLINE 0 0 0 errors: No known data errors Share this post Link to post Share on other sites More sharing options...
Waco Posted February 15, 2015 Posted February 15, 2015 Hopefully it completes successfully! I'm crossing my fingers this weekend on a 2 PB filesystem that lost 6 drives in a single node overnight... Share this post Link to post Share on other sites More sharing options...
Recommended Posts
Please sign in to comment
You will be able to leave a comment after signing in
Sign In Now