About every x-th file I can experience a CRC error when I use windows explorer and copy and paste a file from the vault to a directory outside the vault (decrypt the file). I can’t narrow down the problem at the moment, but a comparison of the files shows significant differences in the file content. I’ll add to the post if I manage to reproduce the problem as well as possible…
For the test I used a vault that was created with version 1.6.17 and I copied the file to the “Desktop”.
It’s reproducible. Copying files from the vault using copy and paste results in an unknown number of corrupted files. But I still don’t have an overview of how many files are defective and whether this can only be done via the explorer or in another way.
Update: Problem also occurs when I use “total commander” or cmd (window) with copy command. Next, I will now test on another machine…
Update: Problem reproduced on another machine (windows 10, cryptomator 1.71, winFSP, FUSE). This time the vault was created directly with version 1.71. When copying files from the vault, some are broken, crc-error.
I classify this behavior as critical, so I will now downgrade all machines. Then I can see if the error is gone.
last update: everything fine with version 1.6.17, no crc errors, all files are decoded correctly. (I have not tested encryption on crc). The behavior of the current version is a very, very big problem, since using it leads to data loss!
total commander and the crc-check from 7zip. if i compare a file with wrong crc with the original in the vault (as shown in the screenshot) you can also clearly see that the files are not the same.
I tested with large files (50 MB) and smaller files (1 MB) and found the problem in each case, the occurrence was different sometimes about 1 per 100, sometimes about 5 per 100. Overall it seems rather random … computer 1 has an ssd, computer 2 conventional hdd. I also did an hd test to rule out a defective drive.
if no one else can reproduce the problem, i would create a screen video documenting the entire test. (but that takes some time).
Rainer did you only see this with cryptomator 1.7.x? I saw that you mentioned version 1.6.17 worked without CRC errors but what did you mean you didn’t test encryption on CRC?
Also, in windows explorer does this specifically happen only when using copy-paste (Ctrl-C / Ctrl-V) or does it also happen when drag and dropping files out of the vault (e.g. to desktop), either to copy or to move? Or is the method of copy/move irrelevant since it also happens via total commander?
Finally I wanted to ask, is your installed winfsp version the same as the one mentioned by Michael above or did you have a much older version (i.e. did you uninstall the previous one before installing the new one via cryptomator installation?
Thanks for the info. I am still on version 1.6.17 but I will try to test this on a different computer in case I can reproduce this.
I only noticed it with version 1.7.1. (not tested with 1.7.0) I noticed the problem with decryption and so far I’ve only tested this. I still have to test whether it also occurs when encrypting. I’ve only tested copy, it seems to be independent of which way it’s copied, it also occurs when working with the copy command in the cmd window. I always use the winfsp version that is included in the .exe installer, before each (new) installation of cryptomator I first delete the installed winfsp version.
little screencast to show the problem (without sound):
sometimes it is difficult to reproduce the error, sometimes it is possible to copy 10,000 files without there being a problem, on the other hand it can also occur frequently. I can reproduce it on different machines, but I can’t pinpoint when and why it happens.
Update:
It’s a decryption problem, there are no errors when encrypting. When decrypting, it doesn’t matter whether you work with copying or moving. It also doesn’t matter if the cmd copy or move command is used or the file explorer or another tool.
The problem does not occur with smaller files (e.g. 1 kb).
Based on your video, I again tried to reproduce that.
At first with 100 files (2,9 MB) copied from inside the vault to a destination outside the vault. Used total commander 10.52 64 bit. And enabled the checkbox for “verifying”.
Result: no errors during copy.
I then created 10.000 txt files with lorem ipsum text (again ~2,9 MB each), copied them into the vault (with file explorer) and then did the same as above with total commander to copy them back outside the vault.
That took some time to copy (because I was on a slow HDD), but at the end the result was the same.
Still not able to reproduce this on my machine.
So I made an other set, this time on my SSD.
Set was:
100 MOV films (each ~158 MB)
500 HEIC Pictures (each ~5 MB)
1.000 PDF (each 100 pages text, ~0,6 MB)
1.000 TXT files (each ~2,9 MB)
1.000 DOCX files (each 900 pages text, ~ 0,9 MB)
That’s a total of 3.600 files and ~22 GB.
All in one folder.
And then I did the Total Commander copy/verifying again from inside the vault to a place on the same SSD outside the vault.
And here we go. I got a difference notification nearly immediately for the 2nd MOV file, and the 5th, and then I aborted. I started again and this time it was MOV file #41,#55,#64,#66. All other files were fine.I clicked “ignore” every time an error came up and waited til the copy process was done.
I checked the difference between the files in and outside the vault of the mentioned MOV files with a MD5 checksum tool (Checksum Control Portable)
All checks showed identical MD5 hashes.
And all file pairs had the exact same amount of Bytes
I then did a copy/verifying of all these files with no cryptomator interaction at all, just from one place on the SSD to an other: no errors
Then I switched the volume type from WINFSP to WinFSP (local drive): Total Commander again reported copy errors for some other MOV files (all others ok), but again the MD5 hashes and amount of bytes were identical between the files.
So from my point of view, I can reproduce the behavior of Total Commander together with WinFSP and larger MOV files, but I can not reproduce any loss of data. Looks to me like a false error of Total Commander. Unfortunately I do not have a clue what might cause there error messages.
I can also generate the error with the copy command, there is no connection here with the total comander. But the files are really different. The difference is that a larger contiguous area within the file is different, i.e. no individual characters but a contiguous block. I can’t generate the error with small files (1 kb … 500 kb). It occurs most frequently with medium-sized files (1 MB … 50 MB). On the other hand, it is observed less frequently with larger files. As already reported, I can reproduce the error on two different computers, the files with a CRC error also turn out to be definitely different when compared.
Add:
examples of deviating data areas in a 1 mb files:
no idea if the info helps. The common feature is the address of the deviating area ending in “…FF8”.
Add:
Now I understand what happened. I created files with increasing numbers. All files with a CRC error show that they are put together incorrectly, recognizable by a jump in the ascending numbering.
mysterious. My two win10 machines must have something that is causing this error. Maybe it happens with older processors? In general, my test systems are completely different, different mainboards, one has an hdd, one has an ssd, different processors, but both are getting a bit old. No external virus scanner, only Microsoft’s own engine.
Error can be reliably reproduced at any time. But only winfsp networkdrive > decrypt, encrypt no problem, winfsp localdrive no problems either, whether decrypt or encrypt.
Since I’m a little helpless. It’s not a problem for me because I don’t have to use the network drive.
It’s just a problem for all users who have a similar configuration and don’t discover the CRC errors, they destroy part of the data without realizing it…
Also wenn man die Diagnose aktiviert und sich die Logdateien ansieht stellt man bei den “Threads” zwischen dem Local und Netzwerkmodus eine unterschiedliche Strategie fest. Wenn man Dateien kopiert wird im Local-Drive Modus eine Datei offenbar überwiegend! (gibt wohl Ausnahmen) von jeweils einem Thread eingelesen, (ein anderer Thread liest dann eine andere Datei ein, usw.), während im Netzwerk-Modus offenbar anders verfahren wird, hier scheinen immer mehrere Threads eine Datei einzulesen. Genau an der Stelle scheint etwas durcheinander zu geraten, da das Fehlerbild der defekten Dateien auch so aussieht dass diese falsch zusammengesetzt wird. An dieser Stelle komme ich aber nicht weiter, die Logs enthalten ja auch nur die read und nicht die write Anweisungen und welcher Thread nun genau und warum Datei-Fragmente vertauscht könnte man vermutlich sowieso nicht ohne weiteres erkennen …
Add:
Mein Verdacht hat sich bestätigt: Ich habe im Bios sowohl “Multicore” als auch “Hyperthreading” abgestellt und das Problem ist verschwunden. Gegen-Check im Log: Es ist auch wirklich nur ein Thread aktiv.
Es gibt also beim (Standard) Netzwerk-Drive und bestimmten CPUs (vermutlich älteren) den Effekt das die Threads Datei-Blöcke nicht korrekt in die Datei schreiben.
Nun ist die Frage ob die CPU an sich das Problem verursacht oder ob es an der Rechengeschwindigkeit liegt … Mal sehen ob sich das noch weiter eingrenzen lässt …
I have reproduced this data corruption with version 1.7.1 (on a Windows 10 Pro laptop). I have NOT been able to do the same with version 1.7.2, but I still believe this critical issue is important enough to look into in order to understand what causes it with version 1.7.1 so that we can avoid it in the future.
Here are my steps to reproduce:
A) Identical test file creation:
Created an empty .txt file
Pasted in it a lorem-ipsum text paragraph multiple times so that the file slightly exceeds 1MB size (was about ~4.5MB)
copied the file(s) multiple times 1 → 2 → 4 → 8-> … → 200+ so that I got 200+ identical files
selected all files (Ctrl-A) and right-clicked and selected rename, typed a file name and pressed enter to auto-rename all files incrementally
windows auto renamed the files like this: file (1).txt, file (2).txt up to file (200).txt (OP said incremental naming MIGHT play a role, I haven’t tested with random file names)
selected all 200 files and 7zipped them (right click → add to filename.7zip) in order to verify that no data corruption has occurred up to this point
All CRC match as expected
B) Cryptomator:
Installed cryptomator 1.7.1 along with its winfsp on Windows 10 Pro
Chose winfsp in cryptomator settings and set application to english and restarted laptop
Created a vault and unlocked it
Copied in it the 200 identical text files that had been created previously
Again, selected all 200 files (in the open vault this time) and 7zipped them (right click → Add to filename.7zip) to verify that no data corruption has occurred yet
All CRC match here too as expected, no problems yet
C) Data corruption:
Selected all 200 text files from the vault and copied them to a new folder on desktop (by right-click dragging)
Selected all 200 text files that were just copied to desktop and 7zipped them by right-click → Add to filename.zip
Opened 7zip to check CRC codes: around 15-20 files have been unexpectedly altered during copy
Edit: More than 30 files we altered, screenshots only show approx half the files.
Reminder: Data alteration / corruption was only reproduced with version 1.7.1, not 1.7.2. But since this is reproducible please look into it to avoid this critical issue in the future.
While i do not doubt your results, i still cannot reproduce this. This time i tried 2000 files (of 6MB each) and copied with the regular Windows Explorer.