Buggy NTFS drivers

[slobasso]

I have been using the command line on the mede8er quite a bit in the last few weeks.
I'm very comfortable in Linux at the command line, so I've been playing with the mede8er through telnet.

I compiled up a version of unrar and curl (wget on the mede8er doesn't support post-data)
I've also written an ash script to handle my downloads using the above programs.

Everything is working pretty well. The mede8er is handling the unrar process while video is playing without dropping frames.

Overall I'm very impressed by this machine, but I'm a bit concerned at what I've been seeing.
Some of my log files look corrupted.

At first I thought this was a bad hard drive, but after thinking about it, it doesn't seem likely. The information seems to point to a bug in the NTFS drivers.

On average I'm seeing a problem once or twice a day.

These log files are usually quite small ( < 4K ). The rar files are quite large ( ~200MB ).
The rar files are checksummed. If one were ever corrupted, it would be known. Considering that I'm dealing with ~2GB of rar files in a day and less than 1MB of logs. I have NEVER had a rar file corrupted, even though it should be 2000 times more likely and caught 100% of the time when it actually happens. The log file errors are only caught if I happen to look at the file.

All of this had me leaning to a busybox file descriptor bug, but a more recent experience drove home NTFS as the culprit.

foo.txt was corrupt. every time I cat the file it gave me the same garbage. The first half of the file looked like random garbage while the end half looked correct.

I wanted to keep a copy of the file so I executed: cp foo.txt save.foo.txt
save.foo.txt now had a copy of the garbage, but foo.txt was restored. It was now absolutely clean. I can't imagine a busybox bug that could accomplish this feat. This really leaves NTFS drivers or my internal memory as potential issues. If my memory were bad, I would probably see playback issues so I don't think memory is the issue.

After doing some google searches I found that nearly everyone says the same thing, don't use Linux to write to NTFS partitions if you care about your data.

Must that main partition be NTFS? Is there any way to get around this requirement?

[Maasbommel]

I have only noticed file corruption on my mede8er, when a file copy over my network stalled somehow and I had to abort this.

To check if you have some kind of HW problem, you could try to run dmesg on the telnet prompt when the corruption appears.
My guess is that you will see some kind of memory failure then.

Regarding to your question if NTFS for the main partition is a need, I am afraid it is, because most users want to easy connect to a windows system by USB connection.

Everything is working pretty well. The mede8er is handling the unrar process while video is playing without dropping frames.

Could it be that your are putting to much load on the Mede8er, so the processor is choking and file corruption is introduced?

[Insomniac]

Must that main partition be NTFS? Is there any way to get around this requirement?

At this point it would seem that the NTFS partition is a necessity - the issues start compounding when one has to cater for various different file formats, development time will increase, as will testing time...

The Mede8er aims to be a simple-to-use plug-n-play media player - with the emphasis on the 'media player'. While it would be great if the techs could have additional functionality available to them, this is unlikely to become a development focus - one may have to see what can be done in the future with respect to having a back-door for the 'tweakers' to play with the settings, as long as that same door doesn't slam on Mede8er's toes...
That back door may have its limitations too in respect of Realtek's proprietary code, but perhaps it takes shape as some form of facility for plug-ins as is already being experimented with by some users.
While they are nice to have for some techs, this is unlikely to become a core focus that would consume resources.

[slobasso]

I have only noticed file corruption on my mede8er, when a file copy over my network stalled somehow and I had to abort this.

I should hope you are not seeing corrupted files, but actually truncated files. I realize the difference is subtle to most people.

To check if you have some kind of HW problem, you could try to run dmesg on the telnet prompt when the corruption appears.
My guess is that you will see some kind of memory failure then.

No hardware problems on my mede8er. Not only that, but I connected an external formatted as an ext3 partition and I've been running the mede8er hard (loads > 6) for 2 days solid. No problems, but I haven't written anything to the NTFS partition.

This tells me that the problem lies in the NTFS drivers or the actual HDD. I will be testing the HDD in the next few days. I think the HDD will be good. It was fine before I put it in the mede8er.

Regarding to your question if NTFS for the main partition is a need, I am afraid it is, because most users want to easy connect to a windows system by USB connection.

Unfortunately I don't have a Windows machine to connect it to and I probably never will. This "feature" doesn't do much for me. I realize I am probably in the minority.

Could it be that your are putting to much load on the Mede8er, so the processor is choking and file corruption is introduced?

I don't believe that I was working it too hard. I've been working much harder than that for days without issue.

I notice in other topics that mede8er owners seem to be very unlucky when it comes to HDDs. AC Ryan owners are also starting to notice the very same problems (very similar HW and underlying software including the same NTFS drivers by Paragon).

[slobasso]

At this point it would seem that the NTFS partition is a necessity - the issues start compounding when one has to cater for various different file formats, development time will increase, as will testing time...

The Mede8er aims to be a simple-to-use plug-n-play media player - with the emphasis on the 'media player'. While it would be great if the techs could have additional functionality available to them, this is unlikely to become a development focus - one may have to see what can be done in the future with respect to having a back-door for the 'tweakers' to play with the settings, as long as that same door doesn't slam on Mede8er's toes...
That back door may have its limitations too in respect of Realtek's proprietary code, but perhaps it takes shape as some form of facility for plug-ins as is already being experimented with by some users.
While they are nice to have for some techs, this is unlikely to become a core focus that would consume resources.

My problem here is that I was seeing problems before I started tweaking things. Just like many people on this board, I was seeing corrupt files from clean sources, files and even whole directories disappearing. The problem is that the corruption is always happening, but at a slow rate. My scripts merely increased that rate. As far as I'm concerned that's a good thing. I'd rather know about this with only 400GB on the drive rather than 1.5TB.

I'm guessing it will take 6-12 months for everyone to realize that there is a problem and that it's in the NTFS drivers and then convince Paragon that they need to fix it and then to actually fix it. Maybe longer now that I think of it. In the meantime I'm going to be working on a workaround.

[Insomniac]

My problem here is that I was seeing problems before I started tweaking things. Just like many people on this board, I was seeing corrupt files from clean sources, files and even whole directories disappearing. The problem is that the corruption is always happening, but at a slow rate. My scripts merely increased that rate. As far as I'm concerned that's a good thing. I'd rather know about this with only 400GB on the drive rather than 1.5TB.

I'm guessing it will take 6-12 months for everyone to realize that there is a problem and that it's in the NTFS drivers and then convince Paragon that they need to fix it and then to actually fix it. Maybe longer now that I think of it. In the meantime I'm going to be working on a workaround.

Perhaps you could provide some hard evidence and build a solid case that could be provided to the devs and Realtek that would clearly and concisely substantiate your claims - That would certainly go a long way to assisting with this issue so that the culprit can be determined and fixed.

Right now it is considered that SAMBA is responsible for the corruption during file transfer and this is the area that Realtek is striving to improve.

[slobasso]

I'm planning on doing a little bit of experimenting, but my efforts will be limited. I'm only willing to try and corrupt my file system so many times.

My guess is I can write a trivial script that will show the corruption. It would be a pain to go through that effort only to hear someone say that I'm using the mede8er incorrectly. I'm simply not willing to play with the user interface copying files until it corrupts and then trying to repeat those steps.

The one problem with file system corruption is that it tends to grow geometrically. A clean file system tends to stay clean until something kicks it just right. Then it tends to give you more corruption even while doing things that didn't cause problems before.

As far as SAMBA vs NTFS goes. SAMBA may be at fault for dropped connections and truncated files, but that's it. Lower network layers may be at fault also, but it's really hard to tell. The NTFS drivers handle the actual writing of contents to disk and even more importantly it should be the only code that has the ability corrupt directories and make files undeleteable. NTFS is a journaled file system, even power outages during writes shouldn't cause these kinds of corruptions. You may see incomplete files, but this is different from file system corruptions.

So what I'm saying is that there are probably SAMBA bugs and it would be nice to see them fixed, but the NTFS problems are are destroying data even after it was successfully copied. This is much worse because I don't know it's happening until it disappears or fails to play. Whereas a failed copy is irritating and annoying, but at least I know it failed and can generally try again.

From what I've seen I can tell you what is probably happening:
In memory data referring to the file system is getting corrupted. This may be a race condition having to do with some re-entrancy issue. Most of the time this corrupt data is just cache and may turn into a bad read. Sometimes this data is flushed to disk, probably because the corrupt data is believed to be modified. Once flushed to disk, the corruption is permanent.

This would explain my previous experience where a non-modifying cp cleaned the original.

[slobasso]

Here is a simple script. When I run this script within an NTFS directory, it eventually corrupts the file, usually within seconds and definitely in less than a minute. When I run it again against an ext3 partition, I can run 10 instances simultaneously and indefinitely without any sign of corruption.

Directions for repeating:
login via telnet to the mede8er
make a directory on your NTFS partition. I named mine "try"
   mkdir /tmp/hdd/volumes/HDD1/try
   cd /tmp/hdd/volumes/HDD1/try
put the following script into a file in that directory. I named my file "try.sh"
run the script
   ./try.sh &

This will run the script in the background. At this point you should have a prompt.
List the directory contents.
   ls
You should see a file named x.#### where the #### represent a number. Ignore any x.####.tmp files you may see.
look at the file contents substituting the correct file name.
   cat x.####

Repeat until corruption seen.
   cat x.####

To slightly increase the load on the file system you can run another.
   ./try.sh &

This will create a second file, same name with a different number.

At some point the beginning of the file will become corrupted.

To stop the processes.
   fg
Then hit:  Ctrl C
You will need to do this a few times if you created more than 1 instance.

WARNING, DO NOT RUN THIS UNLESS YOU ARE WILLING TO POTENTIALLY CORRUPT YOUR HDD!!!

Code: [Select]

#!/bin/ash

tryfile="x.$$"
touch "${tryfile}"

trynum=0
while true ; do
    cp "${tryfile}" "${tryfile}.tmp"
    echo "this is some text" $trynum >>"${tryfile}.tmp"
    mv "${tryfile}.tmp" "${tryfile}"
    trynum=$(( ${trynum} + 1))
   
    sleep 1
done


[slobasso]

Well, I've verified that my HDD is good. This really leaves the Paragon 7.x NTFS drivers as the only possible issue. If you want more proof, I can show you links to many other forums where other devices using this same SDK have run into this same problem.

[phoenix73]

This is a very serious problem if the NTFS support on meder (Realtek SDK in fact), is not stable.

We all can't afford to backup Gb of datas on external hard-disk ;(

BTW, I tried to use the meder  as an external USB hard-disk on my Mac Book Pro, using NTFS mouter and noticed that all the files copied on it seems to disapears.

I switch back to copy via the telnet/console and will use SAMBA to copy future files.

NTFS stability on Realtek SDK is a very serious and dangerous issue, can we have more informations about it ?

Here are the current free memory on my meder, while copying about 250Gb from an external USB drive via cp :

/tmp/hdd/volumes/HDD1/Series # free
              total         used         free       shared      buffers
  Mem:       120080       118636         1444            0          120
 Swap:       160672         1216       159456
Total:       280752       119852       160900

Load is low :

/tmp/hdd/volumes/HDD1/Series # more /proc/loadavg
2.41 2.31 2.28 4/62 3040

[slobasso]

From the research I've done It looks like USB Slave mode is probably the only safe way to modify the main NTFS partition. Any other modifying access is suspect. Slave mode bypasses the Paragon NTFS drivers.

Access modes that are suspect:
FTP
NAS
Any copy, rename, move or delete using the GUI
Pretty much any modifying access that is not USB slave mode

Probably safe modes:
USB slave connected to Windows

Read-only operations in the player should be safe.

Modifying content on an external formatted as anything other than NTFS, ie FAT16, FAT32, ext2, ext3, should be safe.

[phoenix73]

>FTP
>NAS
>Any copy, rename, move or delete using the GUI
>Pretty much any modifying access that is not USB slave mode

So all operations on NTFS using the Linux NTFS support is dangerous ?

[slobasso]

Yes, but you may get lucky. Also if you do regular chkdsk operations on the HDD it may minimize the damage.

I would suggest you do your bulk copy from Windows in slave mode. Keep your backup and minimize changes done from the GUI, NAS, or FTP. Occasionally connect it to your PC and run chkdsk.

Video files are fairly resilient and can take small amounts of damage and still be usable.

I'm currently using an external formatted as ext3. The ext3 partition runs solid. Not a single instance of corruption, ever! I still have an internal formatted with the mede8er, but I am currently only using the NTFS partition for testing purposes.

I happen to have a few HDDs to swap out (a luxury most users don't have). Every single one of them will run perfectly until they are used as an internal for the mede8er. If you use them enough the corruption starts to show up. To verify this I have compiled a version of md5deep, an md5 checksum utility. I can easily copy some content onto the internal and remove my external. Since I have no data on the internal I can run operations that I'm pretty sure will corrupt the data reformat the HDD without worry.

Keeping my data on an external formatted as ext3 has made me immune to the corruption, but it is a bit of a pain to use 2 HDDs for what I was hoping to use 1.

As I said before this will probably take a long time to fix. Paragon is at fault, but Realtek supplies the driver as part of the SDK (I presume) and Mede8er is stuck in the middle with users wondering why their data is corrupting.

[phoenix73]

What's the problem with Paragon ?

Did it break things on the NTFS partition and then make Linux ntfs support unhappy or not stable ?

Since many people Will probably never use the meder as an external disk, I strongly advice meder/realtek to let people choose the partition type for the HD, ext3 or ntfs.

What do you think about ?

[slobasso]

NTFS is just very complicated and difficult to get correct. There are two products out there for NTFS support, the Paragon drivers and the NTFS-3g drivers. The NTFS-3g drivers are probably better, but may still have issues. I would have trouble trusting either, especially since I can't run chkdsk because I don't have any computers running Windows. So I get no value from having any NTFS partitions. Also wired network speeds seemed acceptable compared to its USB speed.

I completely agree with giving the option of other partition types, but the opinion is that connecting to Windows is a huge selling point and supporting another usage model complicates testing and reliability issues. Even though no amount of testing will improve the current reliability issues.

Lose a feature and gain incredible stability, it was an easy decision for me.