Well, there are a couple of assumptions, and I DO NOT compare the timestamps of each file in two locations!

BIG ASSUMPTION #1
The two directories are supposed to be identical. Differences are permitted only between scan/sync processes.

BIG ASSUMPTION #2
This is a MASTER/slave relationship. Files can be modified only in one location, and all other sites are effectively READ ONLY.

Given these assumptions, when a file changes at the master site, I can detect it quite easily. I read the timestamp of the LASTSYNC.INI file in the root of the data folder. I compare that timestamp with all other timestamps in the directory structure. My scan never leaves the file system, much less the server, so it's very fast. Since I use Unison to sync the files, I can stop scanning after the first detected change (knowing a sync is required), although I don't. I log all changed files, then write the current time to LASTSYNC.INI. This assures that this file has a later time stamp than any modified file within the structure.

The slave servers simply compare the timestamp of their local copy of LASTSYNC.INI with that of the same file on the MASTER server. If it's different by more than 10 seconds, it launches a Unison command to sync the folders. The 10-second window allows small time drift between poorly connected sites, possibly without accurate NTP services.

I use Unison instead of RoboCopy/XCopy because it's fast, reliable, and WAN-friendly. Unison is freeware. You install a copy on the master server running as a service, using SrvAny. The secondary server invokes a Unison command line - something like
 Code:
Unison.exe E:\Data socket://master:1479/E:\data -batch -force socket://remote:1479/E:\data

This command, run from a slave system after detecting a newer copy of \\master\Data\LASTSYNC.INI, does the following:
  • Opens a socket connection to the MASTER server on port 1479, passing the path.
  • MASTER builds a timestamp/checksum report and sends it to the slave. Meanwhile, the slave is creating its own report.
  • Slave compares its local report with the one returned by the master. It identifies the differences, and sends a list of file requests to the MASTER, who begins the transfer process.

The -batch argument allows Unison to run in unattended mode, and the -Force defines the MASTER root path. This process is great for WAN connections because all the time/checksum processing is done locally to each server.

On a side note regarding Unison - I wrote a Kix-based tool to do deep folder scans and file by file checksum comparisons of images on our web servers. It insured that production servers had the latest images from the deployment server, and it ran every 2 hours during the day. Using pure kix code, it took about 45 minutes to process the 20,000+ images, even if nothing was copied (an IN-SYNC condition). Using Unison with two UNC root definitions doing regular NetBIOS communications, it dropped to 90 seconds - the team was exstatic. After I installed Unison as a service on the slave systems (this process is configured as a MASTER PUSH, not a slave-pull), I reduced the total process time to about 7 seconds for an IN-SYNC condition.

BTW - Unison is capable of bidirectional synchronizations, although it's a bit tricky, and may require some manual intervention if a file is modified in both roots.

Glenn
_________________________
Actually I am a Rocket Scientist! \:D