Terrastation weirdness - double drive failure, but... not really?

[SkyPics]

Hello all -

Looking for some assistance in this strange issue. I have a Terrastation (TS3400D1604) running 4x4tb drives in RAID5. Yesterday, I learned it had been discovered that the Terrastation was reporting two failed drives (#3 and #4). As it turns out, #3 was a drive which had been recently replaced already. I found this a little hard to believe, and decided to double check with some other software to see what it had to say regarding the health of this drive #3. Using Crystal Disk Info, it seemed to indicate the drive had no issues and was healthy. By comparison, it definitely threw up caution flags on the other failed disk, #4, which didn't surprise me since it was an original, old drive. I put disk #3 back into the Terrastation tower and now the NAS Navigator and the control panel (or whatever you call it, the options & settings area accessed through it's IP address) is telling me the "new drive is ready to be formatted."

This being RAID5, I am aware we can't recover the RAID in the event of losing more than 1 drive, but as far as I can tell, disk #3 is both healthy and unmolested. We have everything backed up, so its not the end of the world if we had to rebuild that storage from scratch but it would be a gigantic pain and, of course, I would much rather be able to add a fresh drive #4 and just rebuild it.

My question: is there a way to get the Terrastation to recognize drive #3 as it was, with no issues (since there doesn't actually seem to be any) so that I can have it help rebuild the RAID array when I get a fresh new disk for slot #4? Or is there no way to have this TS realize that disk3 is actually ok?

[1000001101000]

generally speaking, that drive might be physically okay but could have been out of sync with the array for some reason (like if it never actually got sync's when replacing the previous failed drive) . In that scenario you could have been run in degraded mode for a while without realizing it until a second drive failed.

I haven't tried RAID 5 under the stock firmware on this device, but I would expect if that happened the device would be sounding all sort of alarms in that scenario.

you could try:
1. either connect the drives to a system running Linux or get a root shell running on the device
2. use mdadm to examine the state of each data partition and see if they all still show as part of the array and if they are in sync (or close to it)
3. use mdadm to force start/reassemble the array with the 3 drives.
4. if it comes up run an fsck and see if the filesystem shows as corrupted

see:
https://raid.wiki.kernel.org/index.php/RAID_Recovery

[SkyPics]

Ooohh lordy theres a lot I need to learn apparently. I don't think we had any issues with disk 3 (the previously-replaced one) up to now, nothing but blue lights and no warnings on our notifications/NAS panel. That said, I suppose its possible it wasn't synced - I say that because we have linkstations and this terrastation, and when this happens with the linkstations you can't just hot-swap a drive in and have it rebuild the array like the terrastation is able to, the links you have to go in and tell it to from the browser. (Don't quote me on that, I'm no expert, just going off my limited understanding) So I might have one something I didn't realize (or didn't do something I thought I did) when we just hotswapped that last defective drive in the TS.

I'm going to try investigating your suggestions and see if anything works. I'm sure I'll be back with more posts when I inevitably get stuck hah

[SkyPics]

Well, I don't think I have a way to hook this thing up to anything Linux based, and as far as setting up a shell... is there perhaps a guide for that? I really am starting from scratch in an endeavor like that.

I guess if having any hope of getting this to work requires I learn to do this, it is what it is, but is there really no other way to get this stinkin' TS system to recognize the #3 drive is as it was?

I guess I'm not even sure what it had a problem with in the first place. The system/raid array has been inaccessible since both of the drives "went" so I don't think anything has been de-synced unless the system is somehow doing something to the remaining two drives, which seems unlikely but possible I suppose possible... gah, wish there was just something to click to "check the integrity of disk X" or check for sync-ability, through the system I can access via Windows.

[1000001101000]

I'm not aware of any particularly clean/easy to troubleshoot it at this point. it may be easier to just wipe all the drives and start over if you have backups of the data already.

I don't remember if I've ever tried accessing the command line in the stock firmware for this particular model.

This script or some variation of it would probably work:
https://github.com/rogers0/OpenLinkstation/tree/master/0_get-ssh

if you can't get an interactive shell working you could try sending the desire commands one at a time (the script demonstrates how to do that)