Split encrypted folders for the shake of actually... working?

[NLS]

I am probably deprecating my use of Cryptomator.
Very interesting project and as per this thead:

WebDAV vs Dokany, timeouts and other questions (please?)

I’ll keep updating this thread, to not mess the whole forum, with data that others may find interesting. Speed seems to improve over time (!?)… I haven’t tried yet to connect to the same encrypted folder from two places at the same time or using different methods. It seems to work ok with the transfers, I have moved several thousand files (!). A single folder (which is the last I am encodings), has over 40000 files and 30000 have already been encrypted (I try to make this in chunks beca…

WinFSP actually made it a bit of a workable solution for my needs.
Problem with me is: HUGE list of files in a folder and this over a network.
Really really really having a hard time adding things to those folders and everything not freeze (even for minutes).
Also killed it for me that Android app was payware (for something not mature yet). But this only as a secondary reason.

To my issue:

Since I am phasing out Crypromator use (will keep it active on the side to check on development and use it for small folders with very few files), I am trying to move files out of those aforementioned huge folders.
This is a VERY tough job.
Even when they start actually moving files, Cryptomator seems to periodically tax my 3600X CPU (so it is not a network bottlenet) and things go slooow (between each file, not the actual transfer). Antivirus is disabled for those folders (source and destination).

So I am thinking and please tell me if this is a viable solution:
Is there a way to split the ENCRYPTED folders (along with any needed metadata files), to more smaller ones? So that transfer becomes easier, as smaller folders are faster to read and tax the system way less?

In my encrypted tree structure, I see a folder called “d” and in it are 695 folders containing 2 letter folders (which is bad already, as systems seems to have filled most combinations) and inside them are (single?) encoded folders with .c9r files in them.

Can I temporarily move away (“hide”) most 2 letter folders to let system thing the contents are much less and then little by little reveal the rest of the folders?
Will this work or destroy my contents?

Note, I am on 1.6.0a1 (new vault version?). (EDIT: Just updated to a2)

EDIT #2:

Seems things are not so easy. I mover all non-2x folders away and tried to unlock. It failed (the error was related to the vault not finding its root folder).
Then it came to me to move back all the folders with older dates, thinking that maybe the folder names (since they are from day one) are in there.
This made it successfully mount the vault BUT empty (no folder shown).
It even recreated some 2-digit folders (but empty AFAIK).
So I moved everything back, pending better ideas.

[NLS]

So… nobody can help?

In any case, very painful experience.
I unlocked that folder on another PC (8 core, 16 thread) trying to simply move a few folders on another folder in that “disk”… which is practically a RENAME (no real traffic, no real decoding)…
Well it must be 2 hours with my Cryptomator at 80% of my CPU (!!!), with the move frozen around the middle (up to the middle it moved fast).

So there is definitely an issue there and I am sad just knowning how long moving the remaining 50k files will take…

Photo above is just trying to read the folder, no actual file handling. I’d call that pathetic.

[infeo]

@NLS

Problem with me is: HUGE list of files in a folder and this over a network.

@NLS

I’d call that pathetic.

Cryptomator never claimed to give the optimal performance. In our community we have users which manage some terabytes with it, but when you write “HUGE” directory and “over the network”, you might hit limits. If you need a performant filesystem, consider using Cyberduck or gocryptfs. Additionally, in my humble opinion, instead of moping, you should also consider to pay money for buisness/enterprise solutions, which also give some perks like priority support. (Your post is not two days old and you are already pushing it)

@NLS

Is there a way to split the ENCRYPTED folders (along with any needed metadata files), to more smaller ones?

No. To understand, you should know how the vault internal structure looks like by reading Security Architecture — Cryptomator 1.7.0 documentation. Your problem is not listing all directories, it is loading the content of a single one with a lot of files. To move files out of it, you should avoid listing the content of the directory at all. Windows provides a CLI tool called robocopy to copy/move files. By using it and the terminal and not the Window Explorer, the files in a directory are not prefetched and only touched if targeted.

@NLS

which is bad already, as systems seems to have filled most combinations

This does not atter, its just the first two chars of a hash. Look into the security architecture docs for more info on why.

[NLS]

Thanks for the reply.

1. I never expected enterprise support, I know what I got into. I did expect more replies to be honest, but that is not for me to criticize, just something I noticed.

2. I would not pay for something like this, that has so many free alternatives, for my home server. I definitely don’t decide with the same variables that affect what I use at work and clients.

3. I never pushed (like “where is support when you need it?” or something), I only updated with new info, that might help people help me (since I didn’t get any help before your reply).

4. Indeed, I didn’t go deep into the solution’s documentation. I should.

5. I am aware that big part of the issue is the actual listing, but when the listing is there already (in Explorer or DOpus window), I didn’t expect it is still an issue. I can definitely use robocopy, it was in my plans already, still trying things. If I use robocopy I should probably limit it to a single thread?

6. I mentioned “over network” myself as possible part of the problem, YET high CPU usage (on two different computers) doesn’t show problem with network (I mean the CPU is working with something supposedly and Cryptomator process is hitting it hard).

My next attempt, please advice if this would work, is this:

• Copy the WHOLE structure ENCRYPTED as is, in the destination disk. This will eliminate the “over network” part and also make things more portable (the destination is a USB disk)

• Mount the vault from the new destination and start moving from vault to outside of the vault. Since this will be “self contained” on a USB, I can put the whole process on the backburner in some secondary machine and let it work for days.

Will this work? I suspect it will.

Thanks again.

[infeo]

@NLS

If I use robocopy I should probably limit it to a single thread?

From my point of view, this is a little bit over cautious, but you need to balance between speed and safety. Maybe start with 4 threads and if an error appears, reduce, otherwise think about a thread increase. If you only copy your data, there will be no data loss on error.

Your next attempt with the external drive sounds solid. We would be happy, if you report back if it worked!

[NLS]

I can give you a preliminary report.

Since I didn’t have any feedback before your post, I let it “all in”, left the default of 8 threads that robocopy has.

I copied the encoded (master) folder so everything (errors included) happens on my copy.
From an I/O point of view, I should have used TWO 4TB disks (source and destination), but although I have those, cable and management restrictions made me do it on the same USB disk.

This adds a complication (not a matter of cryptomator), that this disk is veracrypted. So I actually copied the encrypted data on an ENCRYPTED partition.
So I first have to mount the veracrypt volume and THEN point cryptomator to the “cryptomatored” folder.

Weirdly enough, this (mostly) works.
I do this in steps, I have moved all smaller folders already (with no errors apparently), but for the huge one, I do it in steps. I move a few GB of data to a separate folder within the vault and move them away to the final “unencrypted” (but actually veracrypted) destination and keep doing this until it ends.

So the ACTUAL (preliminary) results/observations:

• It is way faster than before, so network DID have an impact (weirdly enough not seen in saturating the actual network, but the CPU… any idea why?).

• Although I tested some data ok, I am not 100% sure I have no corruption, mostly because I see a disk “consumption” difference (the end result being a tiny bit smaller), but I am not 100% sure yet and will know after I complete the move AND maybe do some (painful because of speed and freezes) comparison with the original data (original vault).

• It is not totally problem free even as a process. I have a few hundred files (final count pending as transfer goes on) that seem to have a issue FROM cryptomator side.
  As I said I move a few hundred files to a smaller folder (still inside the vault) and then THOSE are getting unencrypted and moved out of the vault. This process has left weird residue. If I open them, they open fine, but (a) some remain in the original folder (like they didn’t move), (b) some in the destination folder (inside vault, before decryption) show “error 0” or even an unknown error in windows explorer when trying to move them out of the vault.

Not sure what happened to those files and I am not sure if cryptomator has any tools to actually check those data (it is not a disk error, I 've done several checkdisks) or I just delete them and do some kind of comparison with original vault (before copying vault to USB).

So when transfer finishes, I will need to do some sampling (cannot check all files, they seem to be several) and see:

• Are they actually transferred? (some seem to be, even with the error)

• Those transferred, work?

So, in less words: Faster, looks better, but definitely not 100% transparent or 100% safe. I am glad I did a copy, not a move.