Split encrypted folders for the shake of actually... working?

I am probably deprecating my use of Cryptomator.
Very interesting project and as per this thead:

WinFSP actually made it a bit of a workable solution for my needs.
Problem with me is: HUGE list of files in a folder and this over a network.
Really really really having a hard time adding things to those folders and everything not freeze (even for minutes).
Also killed it for me that Android app was payware (for something not mature yet). But this only as a secondary reason.

To my issue:

Since I am phasing out Crypromator use (will keep it active on the side to check on development and use it for small folders with very few files), I am trying to move files out of those aforementioned huge folders.
This is a VERY tough job.
Even when they start actually moving files, Cryptomator seems to periodically tax my 3600X CPU (so it is not a network bottlenet) and things go slooow (between each file, not the actual transfer). Antivirus is disabled for those folders (source and destination).

So I am thinking and please tell me if this is a viable solution:
Is there a way to split the ENCRYPTED folders (along with any needed metadata files), to more smaller ones? So that transfer becomes easier, as smaller folders are faster to read and tax the system way less?

In my encrypted tree structure, I see a folder called “d” and in it are 695 folders containing 2 letter folders (which is bad already, as systems seems to have filled most combinations) and inside them are (single?) encoded folders with .c9r files in them.

Can I temporarily move away (“hide”) most 2 letter folders to let system thing the contents are much less and then little by little reveal the rest of the folders?
Will this work or destroy my contents?

Note, I am on 1.6.0a1 (new vault version?). (EDIT: Just updated to a2)

EDIT #2:

Seems things are not so easy. I mover all non-2x folders away and tried to unlock. It failed (the error was related to the vault not finding its root folder).
Then it came to me to move back all the folders with older dates, thinking that maybe the folder names (since they are from day one) are in there.
This made it successfully mount the vault BUT empty (no folder shown).
It even recreated some 2-digit folders (but empty AFAIK).
So I moved everything back, pending better ideas.

So… nobody can help?

In any case, very painful experience.
I unlocked that folder on another PC (8 core, 16 thread) trying to simply move a few folders on another folder in that “disk”… which is practically a RENAME (no real traffic, no real decoding)…
Well it must be 2 hours with my Cryptomator at 80% of my CPU (!!!), with the move frozen around the middle (up to the middle it moved fast).

So there is definitely an issue there and I am sad just knowning how long moving the remaining 50k files will take…

Photo above is just trying to read the folder, no actual file handling. I’d call that pathetic.

Cryptomator never claimed to give the optimal performance. In our community we have users which manage some terabytes with it, but when you write “HUGE” directory and “over the network”, you might hit limits. If you need a performant filesystem, consider using Cyberduck or gocryptfs. Additionally, in my humble opinion, instead of moping, you should also consider to pay money for buisness/enterprise solutions, which also give some perks like priority support. (Your post is not two days old and you are already pushing it)

No. To understand, you should know how the vault internal structure looks like by reading Security Architecture — Cryptomator 1.5.0 documentation. Your problem is not listing all directories, it is loading the content of a single one with a lot of files. To move files out of it, you should avoid listing the content of the directory at all. Windows provides a CLI tool called robocopy to copy/move files. By using it and the terminal and not the Window Explorer, the files in a directory are not prefetched and only touched if targeted.

This does not atter, its just the first two chars of a hash. Look into the security architecture docs for more info on why.

2 Likes

Thanks for the reply.

  1. I never expected enterprise support, I know what I got into. I did expect more replies to be honest, but that is not for me to criticize, just something I noticed.

  2. I would not pay for something like this, that has so many free alternatives, for my home server. I definitely don’t decide with the same variables that affect what I use at work and clients.

  3. I never pushed (like “where is support when you need it?” or something), I only updated with new info, that might help people help me (since I didn’t get any help before your reply).

  4. Indeed, I didn’t go deep into the solution’s documentation. I should.

  5. I am aware that big part of the issue is the actual listing, but when the listing is there already (in Explorer or DOpus window), I didn’t expect it is still an issue. I can definitely use robocopy, it was in my plans already, still trying things. If I use robocopy I should probably limit it to a single thread?

  6. I mentioned “over network” myself as possible part of the problem, YET high CPU usage (on two different computers) doesn’t show problem with network (I mean the CPU is working with something supposedly and Cryptomator process is hitting it hard).

My next attempt, please advice if this would work, is this:

  • Copy the WHOLE structure ENCRYPTED as is, in the destination disk. This will eliminate the “over network” part and also make things more portable (the destination is a USB disk)

  • Mount the vault from the new destination and start moving from vault to outside of the vault. Since this will be “self contained” on a USB, I can put the whole process on the backburner in some secondary machine and let it work for days.

Will this work? I suspect it will.

Thanks again.

© 2021 Skymatic GmbH • Privacy PolicyImpressum