Buffalo Forums

Products => Storage => : Jimk March 22, 2009, 02:28:54 PM

: Linkstation Quad "Modified Date" problems
: Jimk March 22, 2009, 02:28:54 PM
   

(edited to fix a typo in my description)

 

I am experiencing what I believe to be a file timestamp error problem on two new LinkStation Quad 4TB products that I purchased to replace my overflowing Terastations.

Here is my scenario:

\\TS1 is my (old) Terastation 1TB, Model # HD-H1.0TGL/R5, running firmware 2.00
\\LS1 is a new Linkstation 4TB, Model #: LS-Q4.0TL/R5, running firmware 1.05
\\LS2 is a new Linkstation 4TB, Model #: LS-Q4.0TL/R5, running firmware 1.05
\\XP is my Windows XP computer (XP with SP2)
Time on all four systems were synched to my XP system before doing anything.

I copied all data from \\TS1 to \\LS1 using the following xcopy command on my XP system:

>xcopy /D /E /C /H /R /K /I \\TS1\Share \\LS1\Share

This took the better part of a day to complete, but it copied all of the files from \\TS1 to \\LS1.

Then, to ensure I picked up whatever files might have changed on \\TS1 during the lengthy copy period, I issued the same xcopy command again. I expected it to copy about 5 or 6 files. Unfortunately, it copied about 1/3rd of the files again (tens of thousands of files).

After some experimenting, I concluded that xcopy was being given incorrect "modified" date stamps for *some* of the files on \\LS1.


I have not yet been able to establish a pattern to which files it is receiving incorrect date stamps, but it is extremely reproducible and consistent.

Here are a series of tests I performed using a small folder containing 9 files:

Test #1: xcopy from \\TS1 to \\LS1 -
1) Start with the folder on \\TS1
2) XCOPY that folder to \\LS1 (the folder was created and all nine files were copied).
3) XCOPY the folder to \\LS1 again (3 of the nine files were copied AGAIN).

Test #2: xcopy from \\TS1 to \\TS1
1) Start with the folder on \\TS1
2) XCOPY that folder to another folder on \\TS1 (the folder was created and all nine files were copied).
3) XCOPY the folder to \\TS1 again (ZERO files were copied - as expected).

Test #3: xcopy from \\TS1 to My XP system
1) Start with the folder on \\TS1
2) XCOPY that folder to my XP system (the folder was created and all nine files were copied).
3) XCOPY the folder to my XP system again (ZERO files were copied - as expected).

Test #4: xcopy from my XP system to \\LS1
1) Start with the folder on my XP system
2) XCOPY that folder to \\LS1 (the folder was created and all nine files were copied).
3) XCOPY the folder to \\LS1 again (3 of the nine files were copied AGAIN).
Note: the same three files were copied here that were re-copied in Test 1 - very strange.

Test #5: xcopy from my XP system to \\LS2
1) Start with the folder on my XP system
2) XCOPY that folder to \\LS2 (the folder was created and all nine files were copied).
3) XCOPY the folder to \\LS2 again (3 of the nine files were copied AGAIN).
Note: the same three files were copied here that were re-copied in Test 1 & Test 4 - extremely strange.

For the following tests I deleted all files in the source folder except the three that seemed to have a problem
(that is, the three that get re-copied each time).
Test #6: xcopy only 3 files from my XP system to \\LS1
1) Start with the folder containing the 3 problem files on my XP system
2) XCOPY that folder to \\LS1 (the folder was created and all 3 files were copied).
3) XCOPY the folder to \\LS2 again (all 3 of the files were copied AGAIN).

Test #7: rename source files and xcopy from my XP system to \\LS1
1) Start with the folder containing the 3 problem files on my XP system
2) rename those three files to something else (including new file extensions).
3) XCOPY that folder to \\LS1 (the folder was created and all 3 files were copied).
4) XCOPY the folder to \\LS2 again (all 3 of the files were copied AGAIN).

Conclusions:
> The error occurs when copying to two different Linkstation Quad products (LS-Q4.0TL/R5) running firmware 1.05.
Not only that, but the same files are effected on \\LS1 and \\LS2.
> The error doesn't occur when copying to either an XP system or to a Terastation (HD-H1.0TGL/R5).
> The error is tied to contents of specific files.


Additional information ...
The LinkStations are configured as follows:
Ethernet Frame Size - 1,518 (Default)
AppleTalk - Disabled
FTP Server - Disabled
NTP Function - Enabled
No external USB drives connected
Link Speeds - \\LS1 is 1000Mbps, \\LS2 is 100Mbps

 

 

My Questions:

Is this is a known problem?

Is there a fix availble to correct this problem?

Message Edited by Jimk on 03-22-2009 12:54 PM
Message Edited by Jimk on 03-22-2009 12:55 PM
Message Edited by Jimk on 03-22-2009 12:56 PM
Message Edited by Jimk on 03-22-2009 12:57 PM
: Re: Linkstation Quad "Modified Date" problems
: Jimk March 24, 2009, 10:09:04 AM
   

Moderators? 

Any solutions or workarounds for this problem?

 

 

: Re: Linkstation Quad "Modified Date" problems
: Jimk March 25, 2009, 06:21:59 PM
   I have now written a program to determine that the Modified Date on some files is not being returned correctly by the Linkstation Quad.  I do not know if the date is actually stored incorrectly on disk, or if it is merely a transmission problem that is causing the low order bits of the time field to be corrupted when retrieved.  But something is definitely causing the wrong modified time value to be returned.
: Modified Timestamp Corruptions
: Jimk March 26, 2009, 02:43:16 PM
   

The table below shows data corruptions occurring in the low-order 32 bits of the "modified" timestamps on files copied to a LinkStation Quad server running firmware v1.05.

 

 

 

A total of 31 files (and one folder containing them) were copied using the command:

 

 

 

    xcopy /D /E /C /H /R /K /I h:\temp\d2 x:\temp\d2

 

 

 

In this command, drive h: is a local drive and drive x: is a network drive to the “share” folder of the Linkstation. Of these copies, 14 files and the folder itself had a corrupted (incorrect) timestamp when later retrieved.

 

 

 

Observations:

 

 

 

The net effect is that any file with a corrupt timestamp (roughly 50% of the files in this test) will appear older than the file they are a copy of (by up to about 1 second).

 

 

 

Until Buffalo Technology provides a fix for this problem it is unlikely that any backup software that relies upon comparing modified timestamps will work.

 

 

Table Showing Timestamp Corruptions

 

: Re: Modified Timestamp Corruptions
: buffalosupport March 27, 2009, 02:02:59 PM

We are currently looking into this case, I will post more when we can recreate the issue.

 

 

: Re: Linkstation Quad "Modified Date" problems
: Colin137 March 28, 2009, 02:53:58 PM
Sorry about the long wait. After much searching, I think I've found a solution for you. The issue appears to be that Windows sees the share (basically) as an NTFS filesystem, but Samba uses FAT-style timestamps.

The solution: use robocopy instead of xcopy. Robocopy has a /FFT switch that uses 2-second granularity for timestamps. You can download it from the MS Technet, in the package linked here:

http://www.microsoft.com/downloads/details.aspx?familyid=9d467a69-57ff-4ae7-96ee-b18c4790cffd&displaylang=en

Please reply to this thread to let us know if that works.
: Re: Linkstation Quad "Modified Date" problems
: Jimk March 29, 2009, 12:54:37 AM
   

Thanks Colin, but no thanks. You see, I am one of the original 14 Windows NT developers. My best friend developed the memory management and another good friend developed the kernel. All of us core NT architects and developers go back a long way together. So I know that xcopy was written long before NTFS existed. It was written for FAT and the SMB interface, and its timestamps are perfectly compatible with Samba. Furthermore, the timestamps that are being corrupted originated on a Terastation, not an NTFS file system.

 

 

 

 

Your reply makes it sound like this problem is "by design". But, if this was an xcopy incompatibility with Samba then you would also see it in your other products that utilize Samba, such as the Terastation. yet the Terastation does exactly the right thing when files are xcopy-ed to it. No, this is not an NTFS vs FAT vs Samba file system issue. This is a bug in the Linkstation that is OCCASIONALLY corrupting several bits in the Modified Time field of files.

 

 

 

If a file storage product contains a bug that corrupts data and the developers don't know why it is causing the corruption, then there is a good chance it will corrupt other pieces of data as well - such as the main data stream. I'm not about to put my faith in such a product, and I'm certainly not trusting all my data to such a product. I've given you plenty of detailed information to work from. It took me several hours to collect it all and present it for your analysis. When your developers find the bug, drop me an email. Until then I'm taking my two Linkstations back to the store tomorrow.

 

 

Jim Kelly

: Re: Modified Timestamp Corruptions
: Jimk June 30, 2009, 09:16:17 AM
   Has this problem been fixed yet?
: Re: Modified Timestamp Corruptions
: JohnGray August 26, 2009, 12:04:14 PM
   

I know this thread is several months old and is without a satisfactory conclusion, but it seems worth reporting on something I discovered today using ROBOCOPY, and then looked up on these forums to see if it had previously been encountered - and it had, by Jim Kelly.

 

Yesterday I had copied the contents of a Maxstor OneTouch II external drive, NTFS formatted, to my new LinkStation Pro LS-XHL (firmware is the UK v1.20), using ROBOCOPY (XP-010) with the /MIR (mirror) option.  This took about 12 hours.

 

Since the Maxtor gets updated daily with backup files from my PC's hard disk, I decided to copy any new files written on the Maxstor drive since yesterday to the LinkStation Pro,and found that far too many files were being copied across which ROBOCOPY thought were newer than those copied yesterday to the Linkstation Pro.  So I abandoned this run.

 

I then did two ROBOCOPY tests using the /L option (which does all the file timestamp comparisons, but doesn't do the actual copying, so that the Maxtor and the Linkstation are both left in the same state after as they were before).  I got the following results:


                Total      Copied     Skipped

    Files :    573566      184068      389498

    Bytes :   244.405 GB  161.705 GB   82.700 GB

 

This action had clearly 'copied' a huge number of files which were classed as Newer on the Maxtor, but which I was certain had not altered siince the previous copy.

 

So I used the ROBOCOPY /FFT option (whose meaning is to ignore file timestamp differences of up to 2 seconds) and the results were now:

 

                Total      Copied     Skipped

    Files :    573566       14382      559184

    Bytes :   244.405 GB   31.816 GB  212.589 GB

 

Inspecting the ROBOCOPY log showed that in the total of files that would have been copied across to the LinkStation Pro there were still a number of so-called 'newer' files which hadn't in fact been changed, but nowhere near as many as when the exact file timestamp comparison was made (previous test).

 

So I would allege that the file timestamp data that Windows is given by the Linkstation Pro (in this case) can be inaccurately low by as many as two seconds, and occasionally more.

 

Surely the LinkStation Pro's NTFS file system 'appearance' to Windows should be identical to that of a 'real' NTFS file system, and that includes file timestamp data?

 

I don't have the enthusiasm to examine these file timestamps to the 100 ns granularity offered by 'real' NTFS, nor whether consistent timestamp data is returned for the same file on different occasions, but probably some of the Buffalo techies should, to determine how this inaccuracy can be resolved.

 

Thanks!

 

: Re: Linkstation Quad "Modified Date" problems
: gerit August 27, 2009, 08:27:04 AM
   

Hello,

 

I found the same problem to with my new Linkstation Quad 2TB.

 

I tested it with xcopy, robocopy (without the switch /FFT), MS Richcopy and the windows binary port of rsync. All this programs copied to many files after the first copy process.

 

The roboycopy switch /FFT is solving the problem insufficiently, because this option was intended for a copy process btw FAT and NTFS file systems.

 

I think either there is a bug of the samba version or a bug of the XFS filesystem. In my opinion the samba is the problem.

 

Gerit

: Re: Linkstation Quad "Modified Date" problems
: European October 17, 2009, 09:46:39 PM
   

I have used Microsoft Richcopy to copy data from my PC to the Linkstation Pro LS-XHL (v1.10) with the condition to copy where modification dates are different. A second copy shouldn't any files. Instead it copies a lot if not all.

 

It sounds very similar to the above reported issues.

 

Does anyone have been successful in using Richcopy or other copy programs and if so what settings were used?

 

: Re: Linkstation Quad "Modified Date" problems
: JohnGray October 18, 2009, 02:29:40 AM
   

To be honest, I have found that RichCopy does not live up to the blurb about it, that it is the successor to RoboCopy!

 

Of course you will have found the RoboCopy switch:

/FFT :: assume FAT File Times (2-second granularity).

which is essential when copying from NTFS to NAS/FAT.

There was a small bug where some files got erroneously copied to a NAS/FAT system when they shouldn't have.

 

The Windows 7/ Windows Server 2008 update to RoboCopy now includes a switch

/DST :: compensate for one-hour DST time differences.

so the combination of these two should solve the NTFS to NAS/FAT problem after DST-> 'regular' time changeovers problem (the bug I mention above does not exist in the new version).

 

There's also a MultiThreaded switch in the newer version:

/MT[:n] :: Do multi-threaded copies with n threads (default 8).
                       n must be at least 1 and not greater than 128.

 

So, as always, 'the newer version fixes all known problems'...

: Re: Linkstation Quad "Modified Date" problems
: KSullivan November 12, 2009, 01:37:52 PM
   

HI all

I think this is related.. any  guidance.
I notice that between my Terastaation H1.0TGL/R5 and my  Drivestation... there is a now ONE HOUR difference in the Date Modified between backups.  I assum eits because of DST ...  but..

How does this happen ?
Robocopy (when run) is now recopying ALL from Terastation onto Drivestation and not just the items changed.  I use /MIR

 

Incidentally.. the DST option is in Vista version too... cos it works on my Robocopy. This seems to be sorting out the issue above.
But I remain cuious to understand HOW these dates got out of synch in the first place.. the fiels were mIrrored.

Can I set the System date on Drivestation ? 

ho hum !!!

 

: Re: Linkstation Quad "Modified Date" problems
: Dingus December 06, 2009, 01:51:13 PM
   

Ha ha. Actaully I don't know whether I should laugh or cry.  I think I just found the same problem wth my OLD OLD Linkstation HD-250LAN.

 

I have tried everything with robocopy.  The commands I am running from a batch file are:

 

net use Y: \\linkstation\backup

robocopy D:\Media Y:\Media /DCOPY:T /COPY:DAT /TIMFIX /MIR /Z /R:3 /V /FP /LOG+:C:\Users\Admin\sync.log /TEE
net use Y: /delete

 

All run in a batch file from the task scheduler on Windows 7.  Runs fine the first time, but every time after that continues to copy about 24 files EVEN THOUGH NOTHING HAS CHANGED BETWEEN Y:\Media and D:\Media because the time stamps are jacked up.

 

Unfortunately for me I can't take it back because I have owned it for a couple years at this point.  I will have to get something else and use this unit as a really odd sized and shaped paperweight until Buffalo Tech comes up with a fix.

 

I can't believe Buffalo Tech hasn't figured this out yet.  It seems this thread has been going on for a long time with multiple products and you would think it would be a high priority considering the chances for corruption  that original Windows NT developer cited. 

 

I would suggest that no one buy a linkstation product until they can verify Buffalo did something about this problem.

: Re: Linkstation Quad "Modified Date" problems
: JohnGray December 07, 2009, 09:36:16 AM
   

You are probably being a bit harsh.

 

There seem to be three problems involved in your post (these are common to all the NAS devices which use Samba):

 

1) the times on the PC and on the Linkstation must match - that is, the transition from daylight saving time to 'ordinary' time must have taken place on both devices, not just one.  Try resetting the Linkstation's time to match.

 

2) Samba on the Linkstation presents an NTFS-like interface to Windows, with the one relevant exception being that the timestamp granularity on Samba is (like FAT) 2 s, whereas that of NTFS is 100 ns.  Hence the need for the RoboCopy /FFT switch.

 

3) there is a minor RoboCopy bug, fixed in the Windows 7/Windows Server 2008 version where, even when using the /FFT switch, there is still 'a proportion' of 'identical' files copied from NTFS to Linkstation when they shouldn't be.  I've posted about this elsewhere on the forums (my figures for RoboCopy XP010 were that 101 files out of 86,400 files timestamped for 0, 1, 2, ...86399 seconds of the day were copied when they shouldn't have been).

 

That said, there is still the problem described by JimK, who knows more about the subject than almost anyone - but understanding and/or fixing the problems I've mentioned above are a prerequisite to getting a final solution from Buffalo, should one be forthcoming...

: Re: Linkstation Quad "Modified Date" problems
: Memoryman December 31, 2009, 11:18:58 AM
   

The Buffalotech Japan engineers explain the difference as due to the Linkstation and Terastation have different OS than Windows or Mac, therfore they are on a slightly different time schedule.

: Re: Linkstation Quad "Modified Date" problems
: JohnGray January 02, 2010, 05:28:22 AM
   

In the UK there is a word 'balderdash', which I take to mean a statement which explains absolutely nothing, and is devoid of factual meaning.   I need say no more...

: Re: Linkstation Quad "Modified Date" problems
: harald January 31, 2010, 08:04:20 AM
   

I'd use a stronger word or two - just to get their attention.

Worse problem with DriveStation FlexNet  HD-CELU2.

Drive looses about 6 to 7 seconds every minute.

Makes it pretty useless for backups or trying to find files by date as I just saw some files created two days ago have creation dates of over 10 days ago . Drive installed Jan 3, 2010.

At least the drive hasn't failed in the first 30 days - Guess I'll have to verify each of the 800GB of files to see if any have been corrupted.  What a waste of time.

: Re: Linkstation Quad "Modified Date" problems
: AndyCopper September 11, 2010, 03:52:21 AM

Hi, I'm having the same problems. we got this box to use a load of xcopy comands:

 

xcopy "F:\SQL_Backups\*.*" "A:\" /D /V /E

 

I agree that this should 'just work'. I have started rewriting loads o scripts using robocopy but this also doesn't seem to work.

 

I'm fairly fed up about this!

 

When is there going to be a fix????/

 

I may well have to return the box. Shame because my organisation prababy would have purchased a load of them if they had worked....

: Re: Linkstation Quad "Modified Date" problems
: lojk September 12, 2010, 04:32:46 PM

Yep, I too have a problem with my brand new Terastation 3 TS-XL 4TB (firm V1.40) :-( and i'm proper annoyed about it as like many other posters here i bought the box with the intention of using xcopy to automate backups.

 

At first I thought it was me being daft but i am reassured that others are experiencing the same problem - I tried xcopy with /s /e /d /c /h /r /y, using uppers and lowers also tried using windows synctoy and that was just as confused by the filestamp issue.

 

I personally feel the whole firmware feels pretty cheap and flaky and i am deeply disappointed - for not much more i could have bought a basic dell/hp server with a proper cpu and/or put my own drives in it with either win server or freenas on it and i'm already wishing (as is so often the case with these things) that i'd read this forum BEFORE shelling the £600 for the drive..

 

Well thats the bad news, the good news for the rest of the folks here is that I am a windows developer 'by trade' and i have already started work on an app to replace the functionality of xcopy but make allowances for these differences in filestamp and/or use other criteria for determining if the files have changed (thinking either a dat file like synctoy uses or a combination of archive bit setting/size analysis).

 

i know there are other apps out there but I'm writing this from scratch just for the purpose as I reckon the code monkeys at buffalo are unlikely to invest any effort into fixing this and i need to get this working quicker than they ever will -  i should be able to get this running over the next few days and i will release it for no charge when its ready.

 

Incidentally a quick fix is to use a container based backup solution. Windows Backup and Backup Exec are both happy locating the BackupToDisk location on the drive and they will be unaffected by this issue as they retain there own filestamp info inside that backup media although this doesnt really solve the problem if you use the drive as a destination in 'normal' file transactions.

 

I wonder if that is why buffalo are now offering noveo backup (or whatever it is called) free if purchased after a certain date.

 

I will post again with results or updates as soon as i can.

: Re: Linkstation Quad "Modified Date" problems
: AndyCopper September 15, 2010, 02:58:52 AM

Hi  lojk

 

thanks for that! I am planning to do the same thing using python (for widows) so I'll post my stuff up too).

 

It's a strange oversight on buffalos part and I'm surprised that a 'moderator' hasn't pick up on this.

 

I've read that you can improve things by changing the SAMBA settings but with this box you would have to hack into SSH and invalidate you warrantee in the process...

 

Cheers

: Re: Linkstation Quad "Modified Date" problems
: JoshC September 15, 2010, 06:56:40 AM

You cannot SSH or "hack" these units to change the SAMBA version. 

: Re: Linkstation Quad "Modified Date" problems
: AndyCopper September 15, 2010, 07:15:45 AM

I haven't yet tried it on this model but I have previously managed it on an old buffalo and very recent netgear NAS.

 

Here's a link to someone who claims to have managed:

 

 

http://www.trejan.com/projects/tera/

 

 

From what I've read you don't need to change the version of samba but change settings. I won't bother looking into this unless it's absolutely vital!

 

 

: Re: Linkstation Quad "Modified Date" problems
: AndyCopper September 15, 2010, 07:47:33 AM

Hi JoshC

 

I've just noticed that you're a moderator. Is there any way you can get this bug escalated so desperate users like myself don't consider anarchy!

 

This bug is a real pain because many people buy this device especially for xcopy, robocopy or rich copy use.

 

I'm going to have to solve this or return the unit within the next two weeks

 

A

 

And you're right, google is my SSH friend!

: Re: Linkstation Quad "Modified Date" problems
: lojk September 24, 2010, 02:56:21 PM

Current work schedules have held my plans for a custom app (i wanted to do it as a WPF test app) but to be honest i should be able to hack it out in a few hours in winforms/console if i could just find a mind-window wide enough..

 

The reason the impetus declined is as an interim measure i had to shift the main storage location back into my server (i.e. picked the good from the bad drives out of my previous array ) and im just using the terastation as a destination for a windows 7 backup (to disk) - mostly because my virginmedia online vault thing wont backup network locations and because the linkstation is not customisable (i.e. would love to put freenas on it) beyond the current firwmware - you cant even access the configuration files from a simple (Advanced User) config page that would solve most peoples problems....

 

its operation as a low power unit is great and works really well as a torrent downloader but its performance in a Raid5 is pretty weak and i had hoped i would be able to replace most of my server pc functions with it but sadly not - that still has to stay on for now ; for the 600£ i spent i could have just bought a load more disks and spares for my current array. I only recently found out that buffalo do have some windows 2003 based variants that i'd have bought if i'd known about them (and.or this problem).

 

Obviously the firmware on the box is a *nix variant - and all the linux distros that i know dont have this problem - how hard can it be to include the latest / correct build of the SAMBA or NTFS drivers (whicheer one is actually causing the problem) that does support the required date precision and fix this ridiculous problem once and for all - for now i'll keep improvising but i'm defiently not happy with this issue.

 

That said i have a Buffalo WZR-HP-G300NH wireless router that i can strongly receommend - its briliiance was the basis on which i took the gamble on the Terastation - how can you get one device so right and spoil another reasonably good (value for money) one with such a stupid problem.

 

I'm with you andy - these chaps are obivously reading these forums and are well aware of a long standing limitation in their devices that could  IMHO easily be argued makes them 'not fit for purpose' under uk trade law as xcopy is defintely a normal usage of a NAS box.

: Re: Linkstation Quad "Modified Date" problems
: primacomputer July 11, 2012, 01:07:32 AM
Any solution to this? I see this is an old thread but I'm having the exact same problem on my brand new linkstations and this was the first result on my Google search on the problem. I can confirm this isn't a samba/fat/ntfs issue as I've been using xcopy to backup to linux boxes for well over a decade. Mods, This is not the only serious defect I've come across in this product. Your responses (or lack there of) here and other places is pathetic. I'm going to take these pos back to the store and exchange it for something that works. Spread the word guys. Buffalo sells rubbish.
: Re: Linkstation Quad "Modified Date" problems
: AndyCopper July 11, 2012, 04:19:45 AM

Hi,

 

The short answer is NO. Because I didn't want to invalidate the warranty I didn't ssh in to sort out the problem, (this is a work item not a personal item). With the timestamp not working the get around is either to write a script that keeps it's own logs or to purchase a program that works in this way. After trying a few I ended up with http://www.superflexible.com/ which has worked on our network without issue for about a year now.

The fact the Bufferlo haven't resolved this is somewhat hilarious and I for one won't be purchasing a product from them either professionally or for personal use again!

 

Hope this helps.