5

Parallel File System Scans


sabin1001

Make it so Space Observer can do multithreaded parallel scans of large file systems. The goal is to make a scan of a 20TB filesystem not take multiple days.

A

Activity Newest / Oldest

N

Nathan

Really looking forward to this, we have a NAS location with well over 1 billion files, to do an initial scan would take months, so we had to split up the location into 100's of separate scans and deploy multiple Space Ob Servers just so we can do scans in parallel.

I would say the following things need to change:

1. Allow a single Space Ob Server to execute multiple parallel scans, having to deploy multiple VM's/ Space Ob Server's to do this is a waste of resources in our case.

2. Multithread the initial scans, something like a dir listing at the root and then multiple threads can split off to scan each subdirectory and so on.

3. Increase multithreading of scans after initial, I think the current limit is 32


Team SpaceObServer

Thanks for your post. We already plan to launch this feature with one of our next releases.


  • N