I've been thinking about your tool ant the time it takes.. I have a mechanism in place to copy my software repository between deployment servers at different sites. Doing a comparison over WAN links was painful, so I came up with this method:

1. Create a reference file somewhere with the time the sync was performed. The file is created BEFORE the sync so it is on all slave systems.
2. Use your favorite tool to sync the folders between master and slave. I use Unison, as it is WAN-friendly, unlike XCopy or RoboCopy.
3. Once an hour (or whatever schedule you prefer) scan the master folder structure, comparing file timestamps against the reference file timestamp.
4. If there are any newer files/folders detected, update the reference file (delete/recreate, or write to an INI format file) so the timestamp changes.

My slave servers simply compare their copy of the reference file with the master and initiate a sync if the timestamps are different. They perform this comparison about 10 minutes after the above process

My code originally scanned only folder timestamps, since in a software distribution model, we either add, rename, or delete files and folders. These actions update the parent folder timestamp, which is enough to detect a change that requires replication. I can scan my 16 Gig folder structure with 3688 folders in 85 seconds. Working on a smaller scale, I checked the WinXP structure, which includes all of the XP install files, service packs, and hotfixes - about 850 Meg worth. I can scan/compare all 165 folders there in 7-8 seconds, and every one of the 7039 files in 16 seconds.

Clearly, the difference here is not performing the file to file comparison, but simply detecting if something has changed since the last sync. I have Kix tools for file compare/copy using MD5 Checksum comparison, and while they are useful for small (file deployment) processes, I would not use them for large folder synchronizations. This is one of the reasons I like Unison - it generates a checksum index locally, then spins up a copy of itself on the remote system to do the same - it can then exchange this small file of differences to determine exactly which files need to be copied, with minimal network overhead.

My UDF returns a "0" if nothing changed since the last check, or a "1", followed by the list of changed files/folders if changes were detected. This lets you copy the specific files that were changed.

Anyway - just something to consider in the design of your tool to improve the performance.

Glenn
_________________________
Actually I am a Rocket Scientist! \:D