Author Topic: Linkstation Quad "Modified Date" problems  (Read 17400 times)

Jimk

  • Calf
  • *
  • Posts: 14
Linkstation Quad "Modified Date" problems
« on: March 22, 2009, 02:28:54 PM »
   

(edited to fix a typo in my description)

 

I am experiencing what I believe to be a file timestamp error problem on two new LinkStation Quad 4TB products that I purchased to replace my overflowing Terastations.

Here is my scenario:

\\TS1 is my (old) Terastation 1TB, Model # HD-H1.0TGL/R5, running firmware 2.00
\\LS1 is a new Linkstation 4TB, Model #: LS-Q4.0TL/R5, running firmware 1.05
\\LS2 is a new Linkstation 4TB, Model #: LS-Q4.0TL/R5, running firmware 1.05
\\XP is my Windows XP computer (XP with SP2)
Time on all four systems were synched to my XP system before doing anything.

I copied all data from \\TS1 to \\LS1 using the following xcopy command on my XP system:

>xcopy /D /E /C /H /R /K /I \\TS1\Share \\LS1\Share

This took the better part of a day to complete, but it copied all of the files from \\TS1 to \\LS1.

Then, to ensure I picked up whatever files might have changed on \\TS1 during the lengthy copy period, I issued the same xcopy command again. I expected it to copy about 5 or 6 files. Unfortunately, it copied about 1/3rd of the files again (tens of thousands of files).

After some experimenting, I concluded that xcopy was being given incorrect "modified" date stamps for *some* of the files on \\LS1.


I have not yet been able to establish a pattern to which files it is receiving incorrect date stamps, but it is extremely reproducible and consistent.

Here are a series of tests I performed using a small folder containing 9 files:

Test #1: xcopy from \\TS1 to \\LS1 -
1) Start with the folder on \\TS1
2) XCOPY that folder to \\LS1 (the folder was created and all nine files were copied).
3) XCOPY the folder to \\LS1 again (3 of the nine files were copied AGAIN).

Test #2: xcopy from \\TS1 to \\TS1
1) Start with the folder on \\TS1
2) XCOPY that folder to another folder on \\TS1 (the folder was created and all nine files were copied).
3) XCOPY the folder to \\TS1 again (ZERO files were copied - as expected).

Test #3: xcopy from \\TS1 to My XP system
1) Start with the folder on \\TS1
2) XCOPY that folder to my XP system (the folder was created and all nine files were copied).
3) XCOPY the folder to my XP system again (ZERO files were copied - as expected).

Test #4: xcopy from my XP system to \\LS1
1) Start with the folder on my XP system
2) XCOPY that folder to \\LS1 (the folder was created and all nine files were copied).
3) XCOPY the folder to \\LS1 again (3 of the nine files were copied AGAIN).
Note: the same three files were copied here that were re-copied in Test 1 - very strange.

Test #5: xcopy from my XP system to \\LS2
1) Start with the folder on my XP system
2) XCOPY that folder to \\LS2 (the folder was created and all nine files were copied).
3) XCOPY the folder to \\LS2 again (3 of the nine files were copied AGAIN).
Note: the same three files were copied here that were re-copied in Test 1 & Test 4 - extremely strange.

For the following tests I deleted all files in the source folder except the three that seemed to have a problem
(that is, the three that get re-copied each time).
Test #6: xcopy only 3 files from my XP system to \\LS1
1) Start with the folder containing the 3 problem files on my XP system
2) XCOPY that folder to \\LS1 (the folder was created and all 3 files were copied).
3) XCOPY the folder to \\LS2 again (all 3 of the files were copied AGAIN).

Test #7: rename source files and xcopy from my XP system to \\LS1
1) Start with the folder containing the 3 problem files on my XP system
2) rename those three files to something else (including new file extensions).
3) XCOPY that folder to \\LS1 (the folder was created and all 3 files were copied).
4) XCOPY the folder to \\LS2 again (all 3 of the files were copied AGAIN).

Conclusions:
> The error occurs when copying to two different Linkstation Quad products (LS-Q4.0TL/R5) running firmware 1.05.
Not only that, but the same files are effected on \\LS1 and \\LS2.
> The error doesn't occur when copying to either an XP system or to a Terastation (HD-H1.0TGL/R5).
> The error is tied to contents of specific files.


Additional information ...
The LinkStations are configured as follows:
Ethernet Frame Size - 1,518 (Default)
AppleTalk - Disabled
FTP Server - Disabled
NTP Function - Enabled
No external USB drives connected
Link Speeds - \\LS1 is 1000Mbps, \\LS2 is 100Mbps

 

 

My Questions:

Is this is a known problem?

Is there a fix availble to correct this problem?

Message Edited by Jimk on 03-22-2009 12:54 PM
Message Edited by Jimk on 03-22-2009 12:55 PM
Message Edited by Jimk on 03-22-2009 12:56 PM
Message Edited by Jimk on 03-22-2009 12:57 PM

Jimk

  • Calf
  • *
  • Posts: 14
Re: Linkstation Quad "Modified Date" problems
« Reply #1 on: March 24, 2009, 10:09:04 AM »
   

Moderators? 

Any solutions or workarounds for this problem?

 

 


Jimk

  • Calf
  • *
  • Posts: 14
Re: Linkstation Quad "Modified Date" problems
« Reply #2 on: March 25, 2009, 06:21:59 PM »
   I have now written a program to determine that the Modified Date on some files is not being returned correctly by the Linkstation Quad.  I do not know if the date is actually stored incorrectly on disk, or if it is merely a transmission problem that is causing the low order bits of the time field to be corrupted when retrieved.  But something is definitely causing the wrong modified time value to be returned.

Jimk

  • Calf
  • *
  • Posts: 14
Modified Timestamp Corruptions
« Reply #3 on: March 26, 2009, 02:43:16 PM »
   

The table below shows data corruptions occurring in the low-order 32 bits of the "modified" timestamps on files copied to a LinkStation Quad server running firmware v1.05.

 

 

 

A total of 31 files (and one folder containing them) were copied using the command:

 

 

 

    xcopy /D /E /C /H /R /K /I h:\temp\d2 x:\temp\d2

 

 

 

In this command, drive h: is a local drive and drive x: is a network drive to the “share” folder of the Linkstation. Of these copies, 14 files and the folder itself had a corrupted (incorrect) timestamp when later retrieved.

 

 

 

Observations:

  • · The lowest 8 bits are always improperly zeroed.
  • · The upper 5 bits are always preserved.
  • · Bits 9 through 20 always seem to be corrupted.
  • · Bits 21 through 27 are often corrupted (but apparently not always).

 

 

 

The net effect is that any file with a corrupt timestamp (roughly 50% of the files in this test) will appear older than the file they are a copy of (by up to about 1 second).

 

 

 

Until Buffalo Technology provides a fix for this problem it is unlikely that any backup software that relies upon comparing modified timestamps will work.

 

 

Table Showing Timestamp Corruptions

 


buffalosupport

  • Calf
  • *
  • Posts: 6
Re: Modified Timestamp Corruptions
« Reply #4 on: March 27, 2009, 02:02:59 PM »

We are currently looking into this case, I will post more when we can recreate the issue.

 

 


Colin137

  • Big Bull
  • *****
  • Posts: 1125
Re: Linkstation Quad "Modified Date" problems
« Reply #5 on: March 28, 2009, 02:53:58 PM »
Sorry about the long wait. After much searching, I think I've found a solution for you. The issue appears to be that Windows sees the share (basically) as an NTFS filesystem, but Samba uses FAT-style timestamps.

The solution: use robocopy instead of xcopy. Robocopy has a /FFT switch that uses 2-second granularity for timestamps. You can download it from the MS Technet, in the package linked here:

http://www.microsoft.com/downloads/details.aspx?familyid=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en

Please reply to this thread to let us know if that works.

Jimk

  • Calf
  • *
  • Posts: 14
Re: Linkstation Quad "Modified Date" problems
« Reply #6 on: March 29, 2009, 12:54:37 AM »
   

Thanks Colin, but no thanks. You see, I am one of the original 14 Windows NT developers. My best friend developed the memory management and another good friend developed the kernel. All of us core NT architects and developers go back a long way together. So I know that xcopy was written long before NTFS existed. It was written for FAT and the SMB interface, and its timestamps are perfectly compatible with Samba. Furthermore, the timestamps that are being corrupted originated on a Terastation, not an NTFS file system.

 

 

 

 

Your reply makes it sound like this problem is "by design". But, if this was an xcopy incompatibility with Samba then you would also see it in your other products that utilize Samba, such as the Terastation. yet the Terastation does exactly the right thing when files are xcopy-ed to it. No, this is not an NTFS vs FAT vs Samba file system issue. This is a bug in the Linkstation that is OCCASIONALLY corrupting several bits in the Modified Time field of files.

 

 

 

If a file storage product contains a bug that corrupts data and the developers don't know why it is causing the corruption, then there is a good chance it will corrupt other pieces of data as well - such as the main data stream. I'm not about to put my faith in such a product, and I'm certainly not trusting all my data to such a product. I've given you plenty of detailed information to work from. It took me several hours to collect it all and present it for your analysis. When your developers find the bug, drop me an email. Until then I'm taking my two Linkstations back to the store tomorrow.

 

 

Jim Kelly


Jimk

  • Calf
  • *
  • Posts: 14
Re: Modified Timestamp Corruptions
« Reply #7 on: June 30, 2009, 09:16:17 AM »
   Has this problem been fixed yet?

JohnGray

  • Calf
  • *
  • Posts: 38
Re: Modified Timestamp Corruptions
« Reply #8 on: August 26, 2009, 12:04:14 PM »
   

I know this thread is several months old and is without a satisfactory conclusion, but it seems worth reporting on something I discovered today using ROBOCOPY, and then looked up on these forums to see if it had previously been encountered - and it had, by Jim Kelly.

 

Yesterday I had copied the contents of a Maxstor OneTouch II external drive, NTFS formatted, to my new LinkStation Pro LS-XHL (firmware is the UK v1.20), using ROBOCOPY (XP-010) with the /MIR (mirror) option.  This took about 12 hours.

 

Since the Maxtor gets updated daily with backup files from my PC's hard disk, I decided to copy any new files written on the Maxstor drive since yesterday to the LinkStation Pro,and found that far too many files were being copied across which ROBOCOPY thought were newer than those copied yesterday to the Linkstation Pro.  So I abandoned this run.

 

I then did two ROBOCOPY tests using the /L option (which does all the file timestamp comparisons, but doesn't do the actual copying, so that the Maxtor and the Linkstation are both left in the same state after as they were before).  I got the following results:


                Total      Copied     Skipped

    Files :    573566      184068      389498

    Bytes :   244.405 GB  161.705 GB   82.700 GB

 

This action had clearly 'copied' a huge number of files which were classed as Newer on the Maxtor, but which I was certain had not altered siince the previous copy.

 

So I used the ROBOCOPY /FFT option (whose meaning is to ignore file timestamp differences of up to 2 seconds) and the results were now:

 

                Total      Copied     Skipped

    Files :    573566       14382      559184

    Bytes :   244.405 GB   31.816 GB  212.589 GB

 

Inspecting the ROBOCOPY log showed that in the total of files that would have been copied across to the LinkStation Pro there were still a number of so-called 'newer' files which hadn't in fact been changed, but nowhere near as many as when the exact file timestamp comparison was made (previous test).

 

So I would allege that the file timestamp data that Windows is given by the Linkstation Pro (in this case) can be inaccurately low by as many as two seconds, and occasionally more.

 

Surely the LinkStation Pro's NTFS file system 'appearance' to Windows should be identical to that of a 'real' NTFS file system, and that includes file timestamp data?

 

I don't have the enthusiasm to examine these file timestamps to the 100 ns granularity offered by 'real' NTFS, nor whether consistent timestamp data is returned for the same file on different occasions, but probably some of the Buffalo techies should, to determine how this inaccuracy can be resolved.

 

Thanks!

 


gerit

  • Calf
  • *
  • Posts: 1
Re: Linkstation Quad "Modified Date" problems
« Reply #9 on: August 27, 2009, 08:27:04 AM »
   

Hello,

 

I found the same problem to with my new Linkstation Quad 2TB.

 

I tested it with xcopy, robocopy (without the switch /FFT), MS Richcopy and the windows binary port of rsync. All this programs copied to many files after the first copy process.

 

The roboycopy switch /FFT is solving the problem insufficiently, because this option was intended for a copy process btw FAT and NTFS file systems.

 

I think either there is a bug of the samba version or a bug of the XFS filesystem. In my opinion the samba is the problem.

 

Gerit


European

  • Tatanka
  • **
  • Posts: 62
Re: Linkstation Quad "Modified Date" problems
« Reply #10 on: October 17, 2009, 09:46:39 PM »
   

I have used Microsoft Richcopy to copy data from my PC to the Linkstation Pro LS-XHL (v1.10) with the condition to copy where modification dates are different. A second copy shouldn't any files. Instead it copies a lot if not all.

 

It sounds very similar to the above reported issues.

 

Does anyone have been successful in using Richcopy or other copy programs and if so what settings were used?

 


JohnGray

  • Calf
  • *
  • Posts: 38
Re: Linkstation Quad "Modified Date" problems
« Reply #11 on: October 18, 2009, 02:29:40 AM »
   

To be honest, I have found that RichCopy does not live up to the blurb about it, that it is the successor to RoboCopy!

 

Of course you will have found the RoboCopy switch:

/FFT :: assume FAT File Times (2-second granularity).

which is essential when copying from NTFS to NAS/FAT.

There was a small bug where some files got erroneously copied to a NAS/FAT system when they shouldn't have.

 

The Windows 7/ Windows Server 2008 update to RoboCopy now includes a switch

/DST :: compensate for one-hour DST time differences.

so the combination of these two should solve the NTFS to NAS/FAT problem after DST-> 'regular' time changeovers problem (the bug I mention above does not exist in the new version).

 

There's also a MultiThreaded switch in the newer version:

/MT[:n] :: Do multi-threaded copies with n threads (default 8).
                       n must be at least 1 and not greater than 128.

 

So, as always, 'the newer version fixes all known problems'...


KSullivan

  • Calf
  • *
  • Posts: 12
Re: Linkstation Quad "Modified Date" problems
« Reply #12 on: November 12, 2009, 01:37:52 PM »
   

HI all

I think this is related.. any  guidance.
I notice that between my Terastaation H1.0TGL/R5 and my  Drivestation... there is a now ONE HOUR difference in the Date Modified between backups.  I assum eits because of DST ...  but..

How does this happen ?
Robocopy (when run) is now recopying ALL from Terastation onto Drivestation and not just the items changed.  I use /MIR

 

Incidentally.. the DST option is in Vista version too... cos it works on my Robocopy. This seems to be sorting out the issue above.
But I remain cuious to understand HOW these dates got out of synch in the first place.. the fiels were mIrrored.

Can I set the System date on Drivestation ? 

ho hum !!!

 


Dingus

  • Calf
  • *
  • Posts: 1
Re: Linkstation Quad "Modified Date" problems
« Reply #13 on: December 06, 2009, 01:51:13 PM »
   

Ha ha. Actaully I don't know whether I should laugh or cry.  I think I just found the same problem wth my OLD OLD Linkstation HD-250LAN.

 

I have tried everything with robocopy.  The commands I am running from a batch file are:

 

net use Y: \\linkstation\backup

robocopy D:\Media Y:\Media /DCOPY:T /COPY:DAT /TIMFIX /MIR /Z /R:3 /V /FP /LOG+:C:\Users\Admin\sync.log /TEE
net use Y: /delete

 

All run in a batch file from the task scheduler on Windows 7.  Runs fine the first time, but every time after that continues to copy about 24 files EVEN THOUGH NOTHING HAS CHANGED BETWEEN Y:\Media and D:\Media because the time stamps are jacked up.

 

Unfortunately for me I can't take it back because I have owned it for a couple years at this point.  I will have to get something else and use this unit as a really odd sized and shaped paperweight until Buffalo Tech comes up with a fix.

 

I can't believe Buffalo Tech hasn't figured this out yet.  It seems this thread has been going on for a long time with multiple products and you would think it would be a high priority considering the chances for corruption  that original Windows NT developer cited. 

 

I would suggest that no one buy a linkstation product until they can verify Buffalo did something about this problem.


JohnGray

  • Calf
  • *
  • Posts: 38
Re: Linkstation Quad "Modified Date" problems
« Reply #14 on: December 07, 2009, 09:36:16 AM »
   

You are probably being a bit harsh.

 

There seem to be three problems involved in your post (these are common to all the NAS devices which use Samba):

 

1) the times on the PC and on the Linkstation must match - that is, the transition from daylight saving time to 'ordinary' time must have taken place on both devices, not just one.  Try resetting the Linkstation's time to match.

 

2) Samba on the Linkstation presents an NTFS-like interface to Windows, with the one relevant exception being that the timestamp granularity on Samba is (like FAT) 2 s, whereas that of NTFS is 100 ns.  Hence the need for the RoboCopy /FFT switch.

 

3) there is a minor RoboCopy bug, fixed in the Windows 7/Windows Server 2008 version where, even when using the /FFT switch, there is still 'a proportion' of 'identical' files copied from NTFS to Linkstation when they shouldn't be.  I've posted about this elsewhere on the forums (my figures for RoboCopy XP010 were that 101 files out of 86,400 files timestamped for 0, 1, 2, ...86399 seconds of the day were copied when they shouldn't have been).

 

That said, there is still the problem described by JimK, who knows more about the subject than almost anyone - but understanding and/or fixing the problems I've mentioned above are a prerequisite to getting a final solution from Buffalo, should one be forthcoming...