Buffalo Forums

Products => Storage => Topic started by: RJD185 on August 08, 2016, 10:17:58 AM

Title: Array2 RAID mode 'Not Configured' and disks in other (wrong) array
Post by: RJD185 on August 08, 2016, 10:17:58 AM
Looking for guidance on any data recovery options.

Running a Linkstation LS-QVL with four drives, two 1Tb and two 2Tb. Up until a few days ago these were configured as simple mirror RAID (RAID 1) Array1 and Array2. The email notifications reported these accurately (see below). However, as of today, RAID Array 2 is shown as 'Not configured' and in the administration UI, Raid Array 1 is shown as containing all four disks.

Array1 really does look like it has all four disks in it, with no obvious errors showing. Sometime between 31st July and 3rd August, Array2 vanished and Array1 absorbed the two disks originally in Array2. During that time, the scheduled RAID scanning would have run (1st of the month), and I will have probably have shutdown then sometime later restarted the Linkstation (button push shutdown and restart maybe a day or so later).

I'm seeking some guidance on the most appropriate way to unpick this mess to see whether I have any recoverable content of the original Array2 file systems remaining on the drives.


For reference the last email showing both RAID Arrays from 31st July:

[HDD Usage Status]
RAID Array 1 Usage Rate : 224778296 kbytes / 961280276 kbytes (Usage Rate 23%)
RAID Array 2 Usage Rate : 1115671844 kbytes / 1937365616 kbytes (Usage Rate 58%)

[DISK error status]
DISK1   0
DISK2   0
DISK3   0
DISK4   0

The next email I have which is probably after the restart shows same disk error status, but the line for RAID Array 2 is simply missing.
Title: Re: Array2 RAID mode 'Not Configured' and disks in other (wrong) array
Post by: RJD185 on August 08, 2016, 12:29:34 PM
I've pulled logs direct from the Linkstation (I know I'm not supposed to) and the sequence of events appears to be as below but looking increasingly like the original Array2 configuration will have been destroyed by a restart a few days ago.


Linkstation log implies that on restart, it restored the array, but placed all four disks in array1, which obviously rendered array2 unusable (array2=off).

Jul 31 14:22:33 TelfordTF1_NAS linkstation: Stopped rarpd tftpd rarpcfgd fwupdated
Jul 31 14:22:33 TelfordTF1_NAS ups.sh: Successfully stopped!
Aug  2 11:46:15 TelfordTF1_NAS linkstation: Started inetd
Aug  2 11:46:15 TelfordTF1_NAS linkstation: Started errormon
Aug  2 11:46:16 TelfordTF1_NAS linkstation: Started kernelmon
Aug  2 11:46:18 TelfordTF1_NAS kernelmon: cmd=lanact 0 full eth0
Aug  2 11:46:18 TelfordTF1_NAS kernelmon: cmd=lanact 1000 full eth0
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh: *** diskinfo guess ***
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  >check normal state
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:   * The status is normal? *
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:    diskinfo guess is not exist normal state.
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:    skip normal status checking.
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh: *** compaire ***
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  array1=raid1 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  array1_dev=md21 ... [skip]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk1=array1 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk1_dev= ... [skip]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk2=array1 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk2_dev= ... [skip]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk3=array1 ... [NG]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh: *** restore ***
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  array1=raid1 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  array1_dev=md21 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk1=array1 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk1_dev= ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk2=array1 ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk2_dev= ... [OK]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  disk3=array1 ... [RESTORE]
Aug  2 11:46:24 TelfordTF1_NAS start_data_array.sh:  >exist entry. change diskinfo status.
Aug  2 11:46:28 TelfordTF1_NAS start_data_array.sh:   verify ... [OK]
Aug  2 11:46:28 TelfordTF1_NAS start_data_array.sh:  disk3_dev= ... [OK]
Aug  2 11:46:28 TelfordTF1_NAS start_data_array.sh:  disk4=array1 ... [RESTORE]
Aug  2 11:46:28 TelfordTF1_NAS start_data_array.sh:  >exist entry. change diskinfo status.
Aug  2 11:46:33 TelfordTF1_NAS start_data_array.sh:   verify ... [OK]
Aug  2 11:46:33 TelfordTF1_NAS start_data_array.sh:  disk4_dev= ... [OK]
Aug  2 11:46:33 TelfordTF1_NAS start_data_array.sh:  array2=off ... [RESTORE]
Aug  2 11:46:33 TelfordTF1_NAS start_data_array.sh:  >exist entry. change diskinfo status.
Aug  2 11:46:37 TelfordTF1_NAS start_data_array.sh:   verify ... [OK]
Aug  2 11:46:37 TelfordTF1_NAS start_data_array.sh:  array2_dev= ... [RESTORE]
Aug  2 11:46:37 TelfordTF1_NAS start_data_array.sh:  >exist entry. change diskinfo status.
Aug  2 11:46:41 TelfordTF1_NAS start_data_array.sh:   verify ... [OK]
Title: Re: Array2 RAID mode 'Not Configured' and disks in other (wrong) array
Post by: RJD185 on August 19, 2016, 08:07:14 AM
After some digging, this is what appeared to happen and it is irreversible (with total data loss).

I could see this sequence from the reports in linkstation.log, comparing with some diagnostic strings in the start_data_array.sh script, and validating against several other sources (some other log files on the Linkstation file system, and the monitoring emails I received when Array2 simply disappeared).

So to reiterate, the Linkstation was restarted, the initialization script for the RAID arrays decided something wasn't quite right with the configuration, had a guess what it should be, then explicitly destroyed the array2 configuration by adding the disks from array2 in to array1, thereby destroying any existing data. Job done - family records gone from both disks. I've got some more details about this analysis (which I admit involved me extracting files from the Linkstation file system that aren't normally visible, but I was desperate to find out if the lost data was recoverable; for clarity, I didn't have a non-standard configuration or anything like that), but probably aren't appropriate for this forum.

The moral of this story is that even with RAID 1, take a periodic full backup on a disconnected copy. I was willing to accept a significant house fire but failed to account for this possibility and pay the price (which is about five years of family photos).
Browser ID: smf (is_webkit)
Templates: 1: Printpage (default).
Sub templates: 4: init, print_above, main, print_below.
Language files: 1: index+Modifications.english (default).
Style sheets: 0: .
Hooks called: 59 (show)
Files included: 27 - 1055KB. (show)
Memory used: 740KB.
Tokens: post-login.
Queries used: 14.

[Show Queries]