1.7.x defective decrypted files (fixed in 1.7.3)

About every x-th file I can experience a CRC error when I use windows explorer and copy and paste a file from the vault to a directory outside the vault (decrypt the file). I can’t narrow down the problem at the moment, but a comparison of the files shows significant differences in the file content. I’ll add to the post if I manage to reproduce the problem as well as possible…

For the test I used a vault that was created with version 1.6.17 and I copied the file to the “Desktop”.

Example with 1 MB .txt files (CRC listing):

lorem-ipsum - Kopie (30) - Kopie.txt 4C4F6BB4
lorem-ipsum - Kopie (31) - Kopie.txt 4C4F6BB4
lorem-ipsum - Kopie (32) - Kopie.txt 4C4F6BB4
lorem-ipsum - Kopie (33) - Kopie.txt 4C4F6BB4
lorem-ipsum - Kopie (34) - Kopie.txt 4C4F6BB4
lorem-ipsum - Kopie (35) - Kopie.txt 4C4F6BB4
lorem-ipsum - Kopie (36) - Kopie.txt 4C4F6BB4
lorem-ipsum - Kopie (37) - Kopie.txt 60BFB1B5
lorem-ipsum - Kopie (38) - Kopie.txt 4C4F6BB4
lorem-ipsum - Kopie (39) - Kopie.txt 4C4F6BB4

It’s reproducible. Copying files from the vault using copy and paste results in an unknown number of corrupted files. But I still don’t have an overview of how many files are defective and whether this can only be done via the explorer or in another way.

Update: Problem also occurs when I use “total commander” or cmd (window) with copy command. Next, I will now test on another machine…

Update: Problem reproduced on another machine (windows 10, cryptomator 1.71, winFSP, FUSE). This time the vault was created directly with version 1.71. When copying files from the vault, some are broken, crc-error.

I classify this behavior as critical, so I will now downgrade all machines. Then I can see if the error is gone.

Hi Rainer. Which volume type are you using?

winfsp FUSE, with comes with the .exe installer (i have win10 64). (every test on local drive, no sync to cloud etc …).

last update: everything fine with version 1.6.17, no crc errors, all files are decoded correctly. (I have not tested encryption on crc). The behavior of the current version is a very, very big problem, since using it leads to data loss!

Hi. Thank you for your report.
I tried to reproduce that.
Windows 11 Pro 10.0.22621
Cryptomator 1.7.1
WinFSP 2022.02 (1.12.22339)

What I did:

  1. created 100 text files with lorem ipsum content, each ~3MB
  2. created a new vault
  3. copied the 100 original files in the vault
  4. copied the 100 files from the vault to a new destination on my hdd
  5. did a compare between the original files and the ones i have copied out of the vault with 2 different tools

Result: both tools reported no differences
Which tool do you use to check the crc?

total commander and the crc-check from 7zip. if i compare a file with wrong crc with the original in the vault (as shown in the screenshot) you can also clearly see that the files are not the same.

I tested with large files (50 MB) and smaller files (1 MB) and found the problem in each case, the occurrence was different sometimes about 1 per 100, sometimes about 5 per 100. Overall it seems rather random … computer 1 has an ssd, computer 2 conventional hdd. I also did an hd test to rule out a defective drive.

if no one else can reproduce the problem, i would create a screen video documenting the entire test. (but that takes some time).

Rainer did you only see this with cryptomator 1.7.x? I saw that you mentioned version 1.6.17 worked without CRC errors but what did you mean you didn’t test encryption on CRC?

Also, in windows explorer does this specifically happen only when using copy-paste (Ctrl-C / Ctrl-V) or does it also happen when drag and dropping files out of the vault (e.g. to desktop), either to copy or to move? Or is the method of copy/move irrelevant since it also happens via total commander?

Finally I wanted to ask, is your installed winfsp version the same as the one mentioned by Michael above or did you have a much older version (i.e. did you uninstall the previous one before installing the new one via cryptomator installation?

Thanks for the info. I am still on version 1.6.17 but I will try to test this on a different computer in case I can reproduce this.

I only noticed it with version 1.7.1. (not tested with 1.7.0) I noticed the problem with decryption and so far I’ve only tested this. I still have to test whether it also occurs when encrypting. I’ve only tested copy, it seems to be independent of which way it’s copied, it also occurs when working with the copy command in the cmd window. I always use the winfsp version that is included in the .exe installer, before each (new) installation of cryptomator I first delete the installed winfsp version.

1 Like

little screencast to show the problem (without sound):

sometimes it is difficult to reproduce the error, sometimes it is possible to copy 10,000 files without there being a problem, on the other hand it can also occur frequently. I can reproduce it on different machines, but I can’t pinpoint when and why it happens.

Update:
It’s a decryption problem, there are no errors when encrypting. When decrypting, it doesn’t matter whether you work with copying or moving. It also doesn’t matter if the cmd copy or move command is used or the file explorer or another tool.

The problem does not occur with smaller files (e.g. 1 kb).

Based on your video, I again tried to reproduce that.
At first with 100 files (2,9 MB) copied from inside the vault to a destination outside the vault. Used total commander 10.52 64 bit. And enabled the checkbox for “verifying”.
Result: no errors during copy.

I then created 10.000 txt files with lorem ipsum text (again ~2,9 MB each), copied them into the vault (with file explorer) and then did the same as above with total commander to copy them back outside the vault.
That took some time to copy (because I was on a slow HDD), but at the end the result was the same.
Still not able to reproduce this on my machine.

So I made an other set, this time on my SSD.
Set was:

  • 100 MOV films (each ~158 MB)
  • 500 HEIC Pictures (each ~5 MB)
  • 1.000 PDF (each 100 pages text, ~0,6 MB)
  • 1.000 TXT files (each ~2,9 MB)
  • 1.000 DOCX files (each 900 pages text, ~ 0,9 MB)

That’s a total of 3.600 files and ~22 GB.
All in one folder.
And then I did the Total Commander copy/verifying again from inside the vault to a place on the same SSD outside the vault.
And here we go. I got a difference notification nearly immediately for the 2nd MOV file, and the 5th, and then I aborted. I started again and this time it was MOV file #41,#55,#64,#66. All other files were fine.I clicked “ignore” every time an error came up and waited til the copy process was done.
I checked the difference between the files in and outside the vault of the mentioned MOV files with a MD5 checksum tool (Checksum Control Portable)
All checks showed identical MD5 hashes.
And all file pairs had the exact same amount of Bytes

I then did a copy/verifying of all these files with no cryptomator interaction at all, just from one place on the SSD to an other: no errors
Then I switched the volume type from WINFSP to WinFSP (local drive): Total Commander again reported copy errors for some other MOV files (all others ok), but again the MD5 hashes and amount of bytes were identical between the files.

So from my point of view, I can reproduce the behavior of Total Commander together with WinFSP and larger MOV files, but I can not reproduce any loss of data. Looks to me like a false error of Total Commander. Unfortunately I do not have a clue what might cause there error messages.

I can also generate the error with the copy command, there is no connection here with the total comander. But the files are really different. The difference is that a larger contiguous area within the file is different, i.e. no individual characters but a contiguous block. I can’t generate the error with small files (1 kb … 500 kb). It occurs most frequently with medium-sized files (1 MB … 50 MB). On the other hand, it is observed less frequently with larger files. As already reported, I can reproduce the error on two different computers, the files with a CRC error also turn out to be definitely different when compared.

Add:
examples of deviating data areas in a 1 mb files:

file 1: 080000 … 087FF8
file 2: 0C0000 … 0C7FF8
file 3: 00C010 … 00FFF8
file 4: 0A0010 … 0A7FF8
file 5: 080010 … 087FF8
file 6: 008010 … 00FFF8

no idea if the info helps. The common feature is the address of the deviating area ending in “…FF8”.

Add:

Now I understand what happened. I created files with increasing numbers. All files with a CRC error show that they are put together incorrectly, recognizable by a jump in the ascending numbering.

The file size remains exactly the same. However, a block/area of the file is written twice and overwrites the original content.

Important update:
The behavior described only occurs with the winfsp network drive. With winfsp local drive or webdav there are no bad files.

Add: Problem exist also with vers. 1.7.2.

I looked into this issue.

I used Cryptomator 1.7.2, volume type WinFsp, installed WinFsp version is 1.12.22339

At first I created 1000 identical 2MB files and copied those into the vault and out of the vault again. Copy operation was performed with copy /v.

Afterwards I compared the files bitwise between orginal <-> insideVault, original<->outsideVault and insideVault<->outsideVault using fc /b.

No difference was reported on my computer.

Short Update: Also with files of size 6MB (copied 600 hundered of those) fc could not detect any difference.

mysterious. My two win10 machines must have something that is causing this error. Maybe it happens with older processors? In general, my test systems are completely different, different mainboards, one has an hdd, one has an ssd, different processors, but both are getting a bit old. No external virus scanner, only Microsoft’s own engine.

Error can be reliably reproduced at any time. But only winfsp networkdrive > decrypt, encrypt no problem, winfsp localdrive no problems either, whether decrypt or encrypt.

System 1:

Prozessor Intel(R) Core™ i5 CPU 650 @ 3.20GHz, 3193 MHz, 2 Kern(e), 4 logische(r) Prozessor(en)

System 2:

Prozessor Intel(R) Core™2 Quad CPU Q9300 @ 2.50GHz, 2494 MHz, 4 Kern(e), 4 logische(r) Prozessor(en)

Since I’m a little helpless. It’s not a problem for me because I don’t have to use the network drive.

It’s just a problem for all users who have a similar configuration and don’t discover the CRC errors, they destroy part of the data without realizing it…

@Rainer What version of WinFsp is installed on those systems?

2022.2
I ALWAYS use the winfsp contained in the exe and delete the previous one before a new Cryptomator installation.

Also wenn man die Diagnose aktiviert und sich die Logdateien ansieht stellt man bei den “Threads” zwischen dem Local und Netzwerkmodus eine unterschiedliche Strategie fest. Wenn man Dateien kopiert wird im Local-Drive Modus eine Datei offenbar überwiegend! (gibt wohl Ausnahmen) von jeweils einem Thread eingelesen, (ein anderer Thread liest dann eine andere Datei ein, usw.), während im Netzwerk-Modus offenbar anders verfahren wird, hier scheinen immer mehrere Threads eine Datei einzulesen. Genau an der Stelle scheint etwas durcheinander zu geraten, da das Fehlerbild der defekten Dateien auch so aussieht dass diese falsch zusammengesetzt wird. An dieser Stelle komme ich aber nicht weiter, die Logs enthalten ja auch nur die read und nicht die write Anweisungen und welcher Thread nun genau und warum Datei-Fragmente vertauscht könnte man vermutlich sowieso nicht ohne weiteres erkennen …

Add:
Mein Verdacht hat sich bestätigt: Ich habe im Bios sowohl “Multicore” als auch “Hyperthreading” abgestellt und das Problem ist verschwunden. Gegen-Check im Log: Es ist auch wirklich nur ein Thread aktiv.

Es gibt also beim (Standard) Netzwerk-Drive und bestimmten CPUs (vermutlich älteren) den Effekt das die Threads Datei-Blöcke nicht korrekt in die Datei schreiben.

Nun ist die Frage ob die CPU an sich das Problem verursacht oder ob es an der Rechengeschwindigkeit liegt … Mal sehen ob sich das noch weiter eingrenzen lässt …

2 Likes

I have reproduced this data corruption with version 1.7.1 (on a Windows 10 Pro laptop). I have NOT been able to do the same with version 1.7.2, but I still believe this critical issue is important enough to look into in order to understand what causes it with version 1.7.1 so that we can avoid it in the future.

Here are my steps to reproduce:

A) Identical test file creation:

  • Created an empty .txt file
  • Pasted in it a lorem-ipsum text paragraph multiple times so that the file slightly exceeds 1MB size (was about ~4.5MB)
  • copied the file(s) multiple times 1 → 2 → 4 → 8-> … → 200+ so that I got 200+ identical files
  • selected all files (Ctrl-A) and right-clicked and selected rename, typed a file name and pressed enter to auto-rename all files incrementally
  • windows auto renamed the files like this: file (1).txt, file (2).txt up to file (200).txt (OP said incremental naming MIGHT play a role, I haven’t tested with random file names)
  • selected all 200 files and 7zipped them (right click → add to filename.7zip) in order to verify that no data corruption has occurred up to this point
  • All CRC match as expected

B) Cryptomator:

  • Installed cryptomator 1.7.1 along with its winfsp on Windows 10 Pro
  • Chose winfsp in cryptomator settings and set application to english and restarted laptop
  • Created a vault and unlocked it
  • Copied in it the 200 identical text files that had been created previously
  • Again, selected all 200 files (in the open vault this time) and 7zipped them (right click → Add to filename.7zip) to verify that no data corruption has occurred yet
  • All CRC match here too as expected, no problems yet

C) Data corruption:

  • Selected all 200 text files from the vault and copied them to a new folder on desktop (by right-click dragging)
  • Selected all 200 text files that were just copied to desktop and 7zipped them by right-click → Add to filename.zip
  • Opened 7zip to check CRC codes: around 15-20 files have been unexpectedly altered during copy



Edit: More than 30 files we altered, screenshots only show approx half the files.
Reminder: Data alteration / corruption was only reproduced with version 1.7.1, not 1.7.2. But since this is reproducible please look into it to avoid this critical issue in the future.

1 Like

While i do not doubt your results, i still cannot reproduce this. This time i tried 2000 files (of 6MB each) and copied with the regular Windows Explorer.

@Rcmtr @Rainer Can you two please provide your Windows Edition, Version and Build number?