To all concerned - finally back to normal. Only solution was (as suggested in this forum and by Buffalo tech support) to start from scratch:
remove all data to another server
delete array, reset to factory default settings
recreate RAID array
copy all data back to new RAID array
Did find out in the process that there were, in fact, many bad sectors in the original HD4 - hence the original error messages. Disappointing since drive is less than 3 years old. Once HD4 replaced with brand new drive, RAID array was built and checked without issues. I am now currently copying all data back to this new RAID array.
Many thanks to all, and in particular joma90, for your help.
Some comments (mostly regarding how unhappy I am with the firmware / software of these Buffalo terastations):
Be aware that the process of creating and checking a new RAID 5 array takes a long time! 12 TB configured as RAID 5 gives about 9 TB of available storage - this took more than 60 hours to create and resync the array! Don't really understand why since the disks were wiped clean - why does it take so long to resync a blank RAID 5 array!? And the Buffalo FW/SW simply does not give you any progress update - no way to tell how long the process will take or how far along you are. The only way I found was to use putty.exe to ssh into the root account and use the "cat /proc/mdstat" command. The output gives you a status update so that you can get a rough idea when the resync process will be complete. Surely the Buffalo software should be able to give you this info without you having to result to Linux commands!?
There should be a way to check sectors on a single drive within the array. That would have saved me from having to do the above process twice. I did it once with the old HD4 and waited 3 days until the sector errors started showing up in the syslog. I shouldn't have to off load all data, and reset RAID array just to find out (3 days later!) that a hard drive might have developed bad sectors.
And finally, there should be a way to manually clear error messages and/or re-run a RAID scan once they do appear. Once RAID errors appear, the button to modify RAID scanning settings became disabled. There is no way to get into that panel to re-run a scan on demand. I had to reset the date and time to fool the unit to doing a RAID scan at the next scheduled time. This would have also saved me some time. And even when I did that the scan finished so quickly I couldn't trust the result.
Sorry to say, I have used the Buffalo terastations for more than 10 years and when working fine they are very nice units, but when something does go wrong the you-know-what really hits the fan. And I find there is usually no graceful way of extracting yourself from disaster without offloading data, resetting everything, and reloading all data. When you are dealing with TBs of data this is simply not a desirable option. For my next NAS I will be looking at the competition - can't imagine myself going through this again in the future.
