TS-XL/R5 Multiple E30 Errors all at once

[paloaltoan]

Hi,

I have a TS-XL/R5 purchased in 2010. It has been used very lightly for home use and has been reliable for years. All of a sudden, two days ago on Christmas eve it failed with 3 drives reporting error E30. All the errors reported all at once within a couple mins of each other. What is that the syslog time stamps are not in a correct chronological sequence. I have pasted the relevant section of the syslog below.

I cannot believe that 3 disks fail all at the same time. This is highly unlikely. I dont know what to do as I cant do any diagnostics on each separate disk and I'm affraid that trying to rebuild any one disk will just makes things worse. I would really appreciate hearing from anyone who may have experienced such a situation. I am really hoping that the OS of the TS-XL has somehow become screwed up and is incorrectly reporting errors. But I don't know how to diagnose this. 

Dec 24 22:00:02 Jupiter last message repeated 2 times
Dec 24 22:00:02 Jupiter S40B_update_notifications.sh: checking and registering to cron...
Dec 24 22:00:02 Jupiter S40B_update_notifications.sh: old xml file found. deleting...
Dec 24 22:00:03 Jupiter S40B_update_notifications.sh: starting f/w cheking process ...
Dec 24 22:00:06 Jupiter S40B_update_notifications.sh: step1 result=0
Dec 24 22:00:06 Jupiter S40B_update_notifications.sh: version_is_latest
Dec 24 22:00:06 Jupiter S40B_update_notifications.sh: deleting from lcd...
Dec 24 22:30:01 Jupiter nas_status.sh: loadavg= 0.00 0.00 0.00 : CPU Utilization= 84%
Dec 24 23:00:01 Jupiter nas_status.sh: loadavg= 0.02 0.01 0.00 : CPU Utilization= 84%
Dec 24 23:30:01 Jupiter nas_status.sh: loadavg= 0.01 0.01 0.00 : CPU Utilization= 84%
Dec 25 00:00:01 Jupiter nas_status.sh: loadavg= 0.09 0.04 0.01 : CPU Utilization= 84%
Dec 25 00:30:01 Jupiter nas_status.sh: loadavg= 0.03 0.02 0.00 : CPU Utilization= 84%
Dec 25 00:43:05 Jupiter linkstation: [Success] Sync to ntp server
Dec 25 01:00:01 Jupiter nas_status.sh: loadavg= 0.07 0.03 0.01 : CPU Utilization= 84%
Dec 25 01:30:01 Jupiter nas_status.sh: loadavg= 0.00 0.00 0.00 : CPU Utilization= 84%
Dec 25 02:00:02 Jupiter nas_status.sh: loadavg= 0.00 0.01 0.00 : CPU Utilization= 84%
Dec 25 02:30:01 Jupiter nas_status.sh: loadavg= 0.01 0.02 0.00 : CPU Utilization= 84%
Dec 25 03:00:01 Jupiter nas_status.sh: loadavg= 0.01 0.03 0.00 : CPU Utilization= 84%
Dec 25 03:30:01 Jupiter nas_status.sh: loadavg= 0.00 0.00 0.00 : CPU Utilization= 84%
Dec 25 03:40:59 Jupiter linkstation: log saved to NAND(cron)
Dec 25 04:00:01 Jupiter nas_status.sh: loadavg= 0.10 0.11 0.14 : CPU Utilization= 84%
Dec 25 04:18:02 Jupiter S40B_update_notifications.sh: deleting old settings...
Dec 25 04:18:02 Jupiter last message repeated 2 times
Dec 25 04:18:02 Jupiter S40B_update_notifications.sh: checking and registering to cron...
Dec 25 04:18:02 Jupiter S40B_update_notifications.sh: old xml file found. deleting...
Dec 25 04:18:07 Jupiter S40B_update_notifications.sh: starting f/w cheking process ...
Dec 25 04:18:09 Jupiter S40B_update_notifications.sh: step1 result=0
Dec 25 04:18:09 Jupiter S40B_update_notifications.sh: version_is_latest
Dec 25 04:18:09 Jupiter S40B_update_notifications.sh: deleting from lcd...
Dec 25 04:30:01 Jupiter nas_status.sh: loadavg= 0.06 0.04 0.01 : CPU Utilization= 84%
Dec 25 05:00:02 Jupiter nas_status.sh: loadavg= 0.04 0.10 0.04 : CPU Utilization= 84%
Dec 25 05:30:01 Jupiter nas_status.sh: loadavg= 0.00 0.02 0.00 : CPU Utilization= 84%
Dec 25 06:00:01 Jupiter nas_status.sh: loadavg= 0.57 0.23 0.08 : CPU Utilization= 84%
Dec 25 06:30:01 Jupiter nas_status.sh: loadavg= 0.09 0.04 0.01 : CPU Utilization= 84%
Dec 25 06:31:33 Jupiter kernelmon: cmd=ioerr sda WRITE 11999873 1
Dec 25 06:31:33 Jupiter kernelmon: sda is not exist in devlink
Dec 25 06:31:33 Jupiter kernelmon: cmd=ioerr sdd WRITE 11999873 1
Dec 25 06:31:34 Jupiter kernelmon: sdd is not exist in devlink
Dec 25 06:31:34 Jupiter kernelmon: cmd=ioerr sdc WRITE 11999873 1
Dec 25 06:31:34 Jupiter kernelmon: sdc is not exist in devlink
Dec 25 06:31:35 Jupiter kernelmon: cmd=ioerr sdb WRITE 11999873 1
Dec 25 06:31:35 Jupiter kernelmon: sdb is not exist in devlink
Dec 25 06:31:36 Jupiter kernelmon: cmd=ioerr sda WRITE 11999873 2
Dec 25 06:31:36 Jupiter kernelmon: sda is not exist in devlink
Dec 25 06:31:36 Jupiter kernelmon: cmd=raiddegraded 1 8 2
Dec 25 06:31:36 Jupiter kernelmon: DISK_NO=1 : DISK_STATUS=array1 : RAID_LEVEL=raid5
Dec 25 06:31:38 Jupiter errormon[1898]: Information situation detected! OperationModeI12DEGRADE MODE
Dec 25 06:31:42 Jupiter kernelmon: lcd_error_man.sh array0_raid_error on diag_on(old error code)
Dec 25 06:31:42 Jupiter kernelmon: lcd_error_man.sh disk1_broken_error off(disk1_fault_error) on diag_on buzzer_long_on
Dec 25 06:31:42 Jupiter errormon[1898]: Error situation detected! HD1 Broken   E30Replace the DISK
Dec 25 06:31:54 Jupiter hdd_remove: Disk(disk1) removed : Status (array1)
Dec 25 06:32:04 Jupiter kernelmon: cmd=ioerr sdd WRITE 11999873 2
Dec 25 06:32:04 Jupiter kernelmon: sdd is not exist in devlink
Dec 25 06:32:05 Jupiter kernelmon: cmd=raiddegraded 1 8 50
Dec 25 06:32:05 Jupiter kernelmon: DISK_NO=4 : DISK_STATUS=array1 : RAID_LEVEL=raid5
Dec 25 06:32:09 Jupiter kernelmon: lcd_error_man.sh array0_raid_error on diag_on(old error code)
Dec 25 06:32:09 Jupiter kernelmon: lcd_error_man.sh disk4_broken_error off(disk4_fault_error) on diag_on buzzer_long_on
Dec 25 06:32:10 Jupiter errormon[1898]: Error situation detected! HD4 Broken   E30Replace the DISK
Dec 25 06:32:25 Jupiter linkstation: lcd_error_man.sh array1_raid_error on diag_on buzzer_on(old error code)
Dec 25 06:32:25 Jupiter linkstation: lcd_error_man.sh array2_raid_error off(old error code)
Dec 25 06:32:37 Jupiter hdd_remove: Disk(disk4) removed : Status (array1)
Dec 25 06:32:51 Jupiter kernelmon: cmd=ioerr sdc WRITE 11999873 2
Dec 25 06:32:52 Jupiter kernelmon: sdc is not exist in devlink
Dec 25 06:32:53 Jupiter kernelmon: cmd=raiddegraded 1 8 34
Dec 25 06:32:53 Jupiter kernelmon: DISK_NO=3 : DISK_STATUS=array1 : RAID_LEVEL=raid5
Dec 25 06:32:54 Jupiter kernelmon: lcd_error_man.sh array0_raid_error on diag_on(old error code)
Dec 25 06:32:54 Jupiter kernelmon: lcd_error_man.sh disk3_broken_error off(disk3_fault_error) on diag_on buzzer_long_on
Dec 25 06:32:55 Jupiter linkstation: lcd_error_man.sh disk2_broken_error(disk2_fault_error) off
Dec 25 06:33:01 Jupiter linkstation: lcd_error_man.sh array1_raid_error on diag_on buzzer_on(old error code)
Dec 25 06:33:01 Jupiter linkstation: lcd_error_man.sh array2_raid_error off(old error code)
Dec 25 06:33:01 Jupiter errormon[1898]: Error situation detected! HD3 Broken   E30Replace the DISK
Dec 25 06:33:12 Jupiter linkstation: log saved to NAND(raid_degrade)
Dec 25 06:33:14 Jupiter linkstation: lcd_error_man.sh disk3_broken_error(disk3_fault_error) off
Dec 25 06:33:21 Jupiter linkstation: lcd_error_man.sh disk2_broken_error(disk2_fault_error) off
Dec 25 06:33:21 Jupiter errormon[1898]: Error situation cleared! HD3 Broken   E30Replace the DISK
Dec 25 06:33:22 Jupiter hdd_remove: Disk(disk3) removed : Status (array1)
Dec 25 06:33:36 Jupiter kernelmon: cmd=ioerr sdb WRITE 11999873 2
Dec 25 06:33:36 Jupiter kernelmon: sdb is not exist in devlink
Dec 25 06:33:37 Jupiter kernelmon: cmd=raiddegraded 1 8 18
Dec 25 06:33:38 Jupiter kernelmon: DISK_NO=2 : DISK_STATUS=array1 : RAID_LEVEL=raid5
Dec 25 06:33:46 Jupiter linkstation: lcd_error_man.sh disk3_broken_error(disk3_fault_error) on
Dec 25 06:34:04 Jupiter kernelmon: lcd_error_man.sh array0_raid_error on diag_on(old error code)
Dec 25 06:34:04 Jupiter kernelmon: lcd_error_man.sh disk2_broken_error off(disk2_fault_error) on diag_on buzzer_long_on
Dec 25 06:34:04 Jupiter linkstation: lcd_error_man.sh array1_raid_error on diag_on buzzer_on(old error code)
Dec 25 06:34:04 Jupiter linkstation: lcd_error_man.sh array2_raid_error off(old error code)
Dec 25 06:34:05 Jupiter errormon[1898]: Error situation detected! HD3 Broken   E30Replace the DISK
Dec 25 06:34:15 Jupiter errormon[1898]: Error situation detected! HD2 Broken   E30Replace the DISK
Dec 25 06:34:18 Jupiter linkstation: log saved to NAND(raid_degrade)
Dec 25 06:34:26 Jupiter linkstation: lcd_error_man.sh disk2_broken_error(disk2_fault_error) off
Dec 25 06:34:26 Jupiter errormon[1898]: Error situation cleared! HD2 Broken   E30Replace the DISK
Dec 25 06:34:51 Jupiter hdd_remove: Disk(disk2) removed : Status (array1)
Dec 24 23:33:54 Jupiter u-boot: BUFFALO U-BOOT Start !!! 
Dec 24 23:33:54 Jupiter u-boot:  ** LOADER **
Dec 24 23:33:54 Jupiter u-boot:  ** BUFFALO BOARD: DB-MV78100-A-BP LE (CFG_ENV_ADDR=fff80000)
Dec 24 23:33:54 Jupiter u-boot: U-Boot 1.1.4 (Sep 29 2014 - 16:22:36) Marvell version: 3.3.5
Dec 24 23:33:54 Jupiter u-boot: Buffalo Version: 2.11-1.02

[davo]

Do you have a backup?

[paloaltoan]

This Terastation was the backup. Most of the data on it however was also backup to the cloud. But some was not and I don't want to lose that data.

[boltondc]

I have had this problem with my TeraStation XEL units.  The SysLogs are not always accurate and/or informative.  Open the unit door and check for RED lights for the drives.  If you have one drive with a red light, replace that drive and allow the unit to rebuild the RAID array.  If you have more than one red light then you have more than one failed drive, and the unit will not be able to rebuild the array.  I'm not sure what happens when you replace more than one drive but you will probably have to reformat/recreate the entire array.  There are instructions on the Buffalo site for replacing the hard drive. DO NOT replace any drives that have green lights; that leads to problems that you don't want to get in to.