Array2 RAID mode 'Not Configured' and disks in other (wrong) array

[RJD185]

Looking for guidance on any data recovery options.

Running a Linkstation LS-QVL with four drives, two 1Tb and two 2Tb. Up until a few days ago these were configured as simple mirror RAID (RAID 1) Array1 and Array2. The email notifications reported these accurately (see below). However, as of today, RAID Array 2 is shown as 'Not configured' and in the administration UI, Raid Array 1 is shown as containing all four disks.

Array1 really does look like it has all four disks in it, with no obvious errors showing. Sometime between 31st July and 3rd August, Array2 vanished and Array1 absorbed the two disks originally in Array2. During that time, the scheduled RAID scanning would have run (1st of the month), and I will have probably have shutdown then sometime later restarted the Linkstation (button push shutdown and restart maybe a day or so later).

I'm seeking some guidance on the most appropriate way to unpick this mess to see whether I have any recoverable content of the original Array2 file systems remaining on the drives.

• Should I remove the two disks from Array1 using the admin (would that destroy any content)?
• Are there any logs or other information files that might indicate what has happened?
• If I can extract disks from Array1, what options exist for seeking to extract content from the disk?
• Anything else someone would suggest

For reference the last email showing both RAID Arrays from 31st July:

[HDD Usage Status]
RAID Array 1 Usage Rate : 224778296 kbytes / 961280276 kbytes (Usage Rate 23%)
RAID Array 2 Usage Rate : 1115671844 kbytes / 1937365616 kbytes (Usage Rate 58%)

[DISK error status]
DISK1   0
DISK2   0
DISK3   0
DISK4   0

The next email I have which is probably after the restart shows same disk error status, but the line for RAID Array 2 is simply missing.

[RJD185]

I've pulled logs direct from the Linkstation (I know I'm not supposed to) and the sequence of events appears to be as below but looking increasingly like the original Array2 configuration will have been destroyed by a restart a few days ago.

• July 19, replaced a broken drive and RAID Array2 rebuilt normally (in notification emails)
• Up to July 31, normal running with normal access on Array2 (in notification emails)
• July 31, shutdown Linkstation for a couple of days, (e.g. from lighttpd.webui.error.log)
• August 2, restart Linkstation, which puts disk 3 and disk 4 in Array1, and dumps Array2 (from linkstation.log)

Linkstation log implies that on restart, it restored the array, but placed all four disks in array1, which obviously rendered array2 unusable (array2=off).

Jul 31 14:22:33 TelfordTF1_NAS linkstation: Stopped rarpd tftpd rarpcfgd fwupdated
Jul 31 14:22:33 TelfordTF1_NAS ups.sh: Successfully stopped!
Aug  2 11:46:15 TelfordTF1_NAS linkstation: Started inetd
Aug  2 11:46:15 TelfordTF1_NAS linkstation: Started errormon
Aug  2 11:46:16 TelfordTF1_NAS linkstation: Started kernelmon
Aug  2 11:46:18 TelfordTF1_NAS kernelmon: cmd=lanact 0 full eth0
Aug  2 11:46:18 TelfordTF1_NAS kernelmon: cmd=lanact 1000 full eth0
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh: *** diskinfo guess ***
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  >check normal state
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:   * The status is normal? *
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:    diskinfo guess is not exist normal state.
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:    skip normal status checking.
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh: *** compaire ***
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  array1=raid1 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  array1_dev=md21 ... [skip]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk1=array1 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk1_dev= ... [skip]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk2=array1 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk2_dev= ... [skip]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk3=array1 ... [NG]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh: *** restore ***
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  array1=raid1 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  array1_dev=md21 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk1=array1 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk1_dev= ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk2=array1 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk2_dev= ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk3=array1 ... [RESTORE]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  >exist entry. change diskinfo status.
Aug  2 11:46:28 TelfordTF1_NAS start_data_array.sh:   verify ... [OK]
Aug  2 11:46:28 TelfordTF1_NAS start_data_array.sh:  disk3_dev= ... [OK]
Aug  2 11:46:28 TelfordTF1_NAS start_data_array.sh:  disk4=array1 ... [RESTORE]
Aug  2 11:46:28 TelfordTF1_NAS start_data_array.sh:  >exist entry. change diskinfo status.
Aug  2 11:46:33 TelfordTF1_NAS start_data_array.sh:   verify ... [OK]
Aug  2 11:46:33 TelfordTF1_NAS start_data_array.sh:  disk4_dev= ... [OK]
Aug  2 11:46:33 TelfordTF1_NAS start_data_array.sh:  array2=off ... [RESTORE]
Aug  2 11:46:33 TelfordTF1_NAS start_data_array.sh:  >exist entry. change diskinfo status.
Aug  2 11:46:37 TelfordTF1_NAS start_data_array.sh:   verify ... [OK]
Aug  2 11:46:37 TelfordTF1_NAS start_data_array.sh:  array2_dev= ... [RESTORE]
Aug  2 11:46:37 TelfordTF1_NAS start_data_array.sh:  >exist entry. change diskinfo status.
Aug  2 11:46:41 TelfordTF1_NAS start_data_array.sh:   verify ... [OK]

[RJD185]

After some digging, this is what appeared to happen and it is irreversible (with total data loss).

• On reboot, start_data_array.sh runs to start the arrays, and in the course of this, it checks whether it thinks the RAID array configuration is 'normal'.
• In this case, whatever it found, the script decided that the configuration was not 'normal' and went on to 'guess' what the configuration should be.
• The script then decided that my disk3, which should have been in array2, was somehow not configured properly and reconfigured array1 to contain its current disks plus both disk3 and disk4 as well, thus rendering Array2 'Not configured' as it had no disks, and wiping the file system on disk3 and disk4.

I could see this sequence from the reports in linkstation.log, comparing with some diagnostic strings in the start_data_array.sh script, and validating against several other sources (some other log files on the Linkstation file system, and the monitoring emails I received when Array2 simply disappeared).

So to reiterate, the Linkstation was restarted, the initialization script for the RAID arrays decided something wasn't quite right with the configuration, had a guess what it should be, then explicitly destroyed the array2 configuration by adding the disks from array2 in to array1, thereby destroying any existing data. Job done - family records gone from both disks. I've got some more details about this analysis (which I admit involved me extracting files from the Linkstation file system that aren't normally visible, but I was desperate to find out if the lost data was recoverable; for clarity, I didn't have a non-standard configuration or anything like that), but probably aren't appropriate for this forum.

The moral of this story is that even with RAID 1, take a periodic full backup on a disconnected copy. I was willing to accept a significant house fire but failed to account for this possibility and pay the price (which is about five years of family photos).