TS-HTGL/R5 f/w 1.35 "Can't load kernel" after replacing/rebuilding all 4 drives

[raven457]

TS-HTGL/R5, firmware 1.35, raid 5, 4 x 500gb

Starting with device in good working condition, I pulled drive 3 (500gb spinning disk), replaced it with a 500gb ssd, and then rebuilt the array with no issues or errors. I then did the same for drive 2, and then drive 4, and then drive 1 was swapped last.

After the last successful rebuild with drive 1's new ssd installed, I verified array health via the web interface and confirmed there were no errors or reported issues with the array. I then rebooted the device.

The machine went into panic mode and reported that it could not load the kernel. I powered down, replaced drive 1 with the old 500gb spinning disk, and the device booted up as expected - degraded performance mode, and in the web interface I'm being asked if I want to import disk 1 and repair the array.

I'm at a bit of a loss here - if the device and array were online and working OK with drives 2, 3, and 4, and then successfully restored itself with a new drive 1, how can the kernel be missing from the array? 

How can I replace drive 1 again and not have this issue reoccur?

[1000001101000]

I have an idea of what may have happened but it'll be tricky to confirm if I'm right.

Did the device ever boot successfully with just SSDs installed? Or even with any?

I've used a few different brands on SSDs in my TS-HTGL without any problems but did see something similar with my LS-QL. In my LS-QL the bootloader crashes if one of my SSDs is inserted when it starts, I don't know exactly why. My guess is that you're seeing something similar with your TS-HTGL. In this case I think the firmware was able to see the drives without an issue but the bootloader can't for some reason. On the LS-QL it crashes and shuts off, yours sounds like it is running successfully but just fails to read them or some reason.

That's the only scenario that makes sense to me but it may not be the only explanation. The only way I can think to confirm would be to set up a serial console on the device and observe the bootloader output.

[raven457]

"Did the device ever boot successfully with just SSDs installed? Or even with any?"

It booted OK with drive 3 as SSD. I hotswapped 2, 4, and 1, so I'm guessing when it rebooted with with just drive 3 as SSD it was able to read the kernel without issue from any of the other drives.

I hope it's not a compatibility issue like you ran into, how incredibly frustrating that would be.

[1000001101000]

hmm, so it sounds like the bootloader is failing to read the SSDs but not crashing or anything like that. Prsumably as long as you have at least one regular drive it will boot normally.

What model SSDs are these? I assume I don't have one to test with but it would be good to know either way.

[raven457]

They are SK Hynix Gold S31, 500GB.

I had to get out the door to work this morning, but I'm going to mess with it more tonight.

[raven457]

Last night I went ahead and pulled off another backup, just in case.

Today - powered down and pulled spinning disk hdd 1, and then rebooted without anything connected to bay 1. I was greeted with error "TFTP Boot Recovery E06" at this point. I then downloaded TFTP Boot Recovery, extracted initrd.buffalo from my firmware pack, etc. 

I then reconnected SSD disk 1, and launched TFTP Boot.exe. The transfer took place, and after a few minutes, the TeraStation warm booted into an array degraded state and flashed me the expected DHCP reserved IP address. I was then able to access the device via web interface, was able to re-add SSD disk 1, and start the rebuild process. I'll know more after about 350 minutes...

[raven457]

array repair complete with all SSDs, warm reboot from the webpage console, and...

"TFTP Mode E06 Lost boot image"

Reloaded the boot image with the TFTP server once again, going against the full array of SSDs, power down, reboot, same thing.

[raven457]

Well, I think it boils down to an incompatibility between the system itself and the SSDs, as you mentioned earlier, 1000001101000.

With the data safely backed up, there wasn't much to be lost - so!

• Booted up to TFTP Boot Recovery E06 error.
• Provided handholding via TFTP Boot to get the system fully booted
• Used the firmware update tool in debug mode with force update, do not check version, rebuild partition table, update boot, kernel, initrd, and rootfs, and for grins, delete user-config all selected
• update eventually fails with a "Failed to formatting (0xFFFFFFFFE) Updating is aborted" message
• Terastore shows as online with correct dhcp reserved IP, the update tool can see it, but reports "----" for firmware version, and the device is not accessible via webconsole at all

Power down, swap SSD drive 1 for for HDD drive 1, reboot....

annnnnnd, it boots up with no errors. Of course there's no array info, and the web console still doesn't work.

Between the age of the machine and this, it's looking like it's time to retire my old Terastation. Bummer, that.

[1000001101000]

I'm super curious about the particulars about what's happening in this case. The only way I can think to get more information would be to capture the console output from the boot process. If you're interested I can help walk you through that procedure.

At high level that involves:
connecting the serial port on the back to a PC (usb adapters are available for a few dollars online).
booting the device with one of the working HDDs connected to boot from
running a command that converts the rear serial port to a serial console (until the power is disconnected)
rebooting and capturing the output on the PC
then possibly replacing the hdd with an ssd again, then rebooting and capturing that output as well.

Let me know if that's something you'd be interested in trying.

I'm debating buying one of the same drive to test myself with this and other NAS models, though a quick search makes them sound a bit more expensive than I'd typically spend just to satisfy my curiosity.