Missing sync-target folder within vault after big sync

chr15t0ph · March 6, 2022, 8:05pm

Hi! I need help:

Under Windows 10, I am using Cryptomator (1.6.5) over pCloud (3.11.8) using Dokany (1.3, via chocolatey).

My pCloud 2TB lifetime share is mounted as P:
I have a single vault P:\cryptomator-cpn that cryptomator mounts to Q:
Within the vault I created directory Q:\sync-silo\LW-U\

Then I used FreeFileSync to mirror my U: drive of 92 GB drive to Q:\sync-silo\LW-U
(With ADSL, i.e. 4000kilobit/s = 500KB/s upload.)
About 58000 directories and about 220000 files.
FreeFileSync terminated after about half a day.
Then the cryptomator Vault statistics showed idle for read and write.
However, for pCloud, it took another two days to report complete cloud file sync.
I unmounted/locked the cryptomator vault and waited the two days before doing anything else with the vault.

(On C: drive, for pCloud temp-space, there is and was enough amount space for the 92 GB of data that I added to my encrypted vault, it did not exceed the free space on the system partition.)

After pCloud sync, I wanted to look into the vault, but the directory Q:\sync-silo was empty.
No Q:\sync-silo\LW-U any more.

Still in P:\cryptomator-cpn\d\ there are 1024 directories worth of 92GB of stuff.

Health check via crytomator returend without fault.
Restarting the computer and reconnecting/remounting did not help.

(I would attach the health report. I would attach a debug log where I mounted the vault and accessed Q:\sync-silo\ with the explorer. But I get an error “Sorry, new users can not upload attachments.”)

Specific questions:
What went wrong? Can I do something to fix it?

General questions:
Is this an unusual set-up that I tried?
Am I overhelming the crytpo-cloud stack with 92 GB / 58000 directories / 220000 files?
Shouldn’t I use pCloud?
Should I sync less than 5 GB per Sync-Session (i.e. the pCloud cache size)?
Any advice on best practice for such drive mirrors would be appreciated!

Kind regards
Christoph

chr15t0ph · March 11, 2022, 9:37am

Okay. Interim Report:

I upgraded Dokany to latest 1.5.x version, manually, just to be on the safe side.
I created a new cryptomator volume.
And I resynced via FreeFileSync from scratch.
(“Did you try to turn it off and on again?”)

For the first 40GB I did it in disjunct 8 or 9 syncs, each with less than 5GB (below pCloud cache size).
The directory was filled after each sync.
However, there occurred FreeFileSync errors about read priviliges: I ignored them in FFS until later.

For the last 50 GB: Getting brave, I did them in one big sync.
It worked also without folders missing. (Still other FFS errors, ignored for the time beeing.)
The pCloud cache size seems to be irrelevant for adding big file sets.

With my bandwith that took another several days, as it is expected.
So I changed nothing but the Dokany version, but this time the target file system structure was not corrupted. Must not necessarily be the Dokany version, bad luck (or a Heisen-bug in the other components) might still be an explanation for the corrupted first try.

Now about the FreeFileSync errors:

Please be aware, that I sync the same file set for years with FreeFileSync to several different discs, also crypted ones by VeraCrypt. The FFS errors that we talk about here do not appear with on-prem disc targets. I know FFS to be very stable and trustworthy.

I did a complete FFS comparison for an overview of the unsynced (read permission error) files.

The FreeFileSync comparison now takes VEEERY long: for 92 GB / 58000 directories / 220000 files => greater than 40/50 minutes!

To make this a feasible scenario, either Cryptomator filesystem-organization or pCloud Sync-Cache, must take the next step. My internet bandwidth and RTT is quite good for a private person (we talk here about my download with 100 MBit/s, not about upload rates any more). In 40 minutes with 100 MBit/s, there is time for 30 GB download volume! Demanding “faster network” would be unrealistic for solving this cloud sync impediment. FreeFileSync is used in standard configuration, comparing just by change date and file size (not by file contents). I assume this filesystem meta data access pattern could be optimized by Cryptomator filessystem organization.

The errors about several files remain. With pushing the “mirror” button several times (brute force human behaviour), some read permission errors dissolve spontaneously and the files get synced. Some files remain in error state. Its no coherent error behavior.
After three complete resyncs the sync set did not become empty, but not necessarily due to the errorneous files: There popped up several files in “file changed, need sync” state that a moment before had already be synced. wtf

I will stop the experiment here.

Final Conclusion: At the moment, March 2022, you cannot achieve stable Cloud backups of 100GB discs via Cryptomator (1.6.5) over pCloud (3.11.8) using Dokany (1.5).

(Again, best practice comments, showing me the way or my errors, would still be appreciated. E.g., has anyone exercised a similar setup under Linux with stable results?)

Kind regards
Christoph