#184257 - 2007-12-27 04:12 AM
Monitoring Changes on a Drive
|
buckla1
Fresh Scripter
Registered: 2007-12-27
Posts: 7
|
Hi Folks
This is my first ever post and i guess i just wanted to offer up some code that i have been working on for a long time.
Description: The Code Starts by taking a scan of a drive you select and puts the Size of the directory, the size of the files including file names and Directory names, including the owner of the file into TXT files. you can then run the program again in 10 seconds or when ever, six months later and it will run a second snapshot the same as the first, and compares the differences. Eg what files have been deleted, changed size, are new files, and who owns the files.
The idea came up in a practical sence when i was doing some helpdesk work for a company and they had very low disk space on there server and over the weekend the server hard drive was completely filled and the server locked up and crashed ready for the users on Monday morning. The manager said what changed and who did it....
I hope you don't think of me as a leach as i have borrow some code from different examples as i am only a newby and have been learning how to write KiXtart script as i write this code.
I know you guys will see that this can be written smarter, faster and i'm hoping that you can offer me alittle constructive criticism, so i can learn how i could have written it better. This code is working code so feel free to try it out.
At the moment it must be run from C:\Program Files\DiskChange
it also requires a few Command Prompt Commands
deltree.exe
treeinfo.exe
xcacls.exe
you will also need to be an Administrator of the Drive you are running this script against.
The Reason i used these commands was because i had no idea how to perform the same task using KiXtart Code.
As for the Adding the Domain\admin Username to Folders that normally are unreadable like the System Volume Information Folder this is because the Treeinfo command can't read it and therefor gives me an unreadable output. So i just made it so it added read attributes to the folder so that it can run correctly.
Once again please forgive my ignorance regarding borrowing Code. GetExtFileProperties is Allen's off this site and it was perfect. So why mess with it. As For any other similarities to other peoples code i'm not sure who as this has been a pet project over the last 18 months. Most of that time was spent trying to make it faster.
Sorry Left out about the Deleting all file excpet part.
You need to create 1 file
[Filesnottodelete.sto]
treeinfo.exe
xcacls.exe
deltree.exe
Filesnottodelete.sto
IDSuite
(also put in the name you called the script.kix file or else it will be deleted after it runs)
Just put in the list the files in the c:\Program Files\Diskchange\ Folder you don't want to delete.
Edited by buckla1 (2007-12-28 03:53 AM)
|
|
Top
|
|
|
|
|
#184258 - 2007-12-27 04:23 AM
Re: Monitoring Changes on a Drive
[Re: buckla1]
|
buckla1
Fresh Scripter
Registered: 2007-12-27
Posts: 7
|
Removed Code
Edited by buckla1 (2007-12-28 03:16 AM)
|
|
Top
|
|
|
|
|
#184259 - 2007-12-27 04:32 AM
Re: Monitoring Changes on a Drive
[Re: buckla1]
|
buckla1
Fresh Scripter
Registered: 2007-12-27
Posts: 7
|
Sorry Left out about the Deleting all file excpet part.
You need to create 1 file
[Filesnottodelete.sto]
treeinfo.exe
xcacls.exe
deltree.exe
Filesnottodelete.sto
IDSuite
(also put in the name you called the script.kix file or else it will be deleted after it runs)
Just put in the list the files in the c:\Program Files\Diskchange\ Folder you don't want to delete.
|
|
Top
|
|
|
|
|
#184263 - 2007-12-27 02:01 PM
Re: Monitoring Changes on a Drive
[Re: buckla1]
|
Benny69
Moderator
Registered: 2003-10-29
Posts: 1036
Loc: Lincoln, Ne
|
Hi buckla1, and welcome to the board.
The reason that your code was cut off was because it was too long, if you would please, go back and edit your other posts, remove the code,
zip up your script and then attach it to your post. You can do this by using the 'File Manager' located on the post edit screen.
Attachments
Description:
|
|
Top
|
|
|
|
|
#184272 - 2007-12-27 08:18 PM
Re: Monitoring Changes on a Drive
[Re: Glenn Barnas]
|
NTDOC
Administrator
Registered: 2000-07-28
Posts: 11625
Loc: CA
|
This FAQ explains most of this, and the good part is it works on MOST bulletin boards on the Internet.
Learning how to use this type of formatting will improve your posting skills on just about every board you visit.
The Post/Reply Formatting Box and How to use it
|
|
Top
|
|
|
|
|
#184295 - 2007-12-28 02:46 AM
Re: Monitoring Changes on a Drive
[Re: NTDOC]
|
buckla1
Fresh Scripter
Registered: 2007-12-27
Posts: 7
|
Thanks Guys for the pointers and the welcome.
I have Zipped and attached the Script and the needed text file.
Hope thats ok.
Attachments
DiskChange.zip (553 downloads)
Description:
|
|
Top
|
|
|
|
|
#184298 - 2007-12-28 03:59 AM
Re: Monitoring Changes on a Drive
[Re: NTDOC]
|
buckla1
Fresh Scripter
Registered: 2007-12-27
Posts: 7
|
Thanks NTDOC is that OK?
One of the problems is the scan takes about 20 minutes each time, and about the same for the comparison. So all up about 40 minutes to get your answer. Much quicker than doing it manually. But If i remove the section on storing and getting the owner name it reduces down to about 12 minutes, i'm just not sure how i can approch it to make it faster.
I had thought about approching it using Functions but to be honest i just don't think i use them properly yet. But i don't think that is going to change the speed.
What about the way that i'm scaning to the txt files. I thought about reading the txt files into Arrays and but it is still having to read the txt file so then i'm just hover handeling.. So here i stand.
Any help would be teriffic.
|
|
Top
|
|
|
|
|
#184356 - 2008-01-02 04:11 AM
Re: Monitoring Changes on a Drive
[Re: buckla1]
|
buckla1
Fresh Scripter
Registered: 2007-12-27
Posts: 7
|
Hi Everyone
I found a couple of problems with the Delete Section not deleting the *.L2G files which is now fixed. and i have added a reporting section to the Main Window, Also regarding the exemption of files related to the working of this program i had missed 3 and that is also now fixed.
Hope this fixes some of the problems that some of you may have found.
Attachments
DiskChange.zip (523 downloads)
Description:
|
|
Top
|
|
|
|
|
#184372 - 2008-01-02 05:05 PM
Re: Monitoring Changes on a Drive
[Re: buckla1]
|
Glenn Barnas
KiX Supporter
Registered: 2003-01-28
Posts: 4401
Loc: New Jersey
|
I've been thinking about your tool ant the time it takes.. I have a mechanism in place to copy my software repository between deployment servers at different sites. Doing a comparison over WAN links was painful, so I came up with this method:
1. Create a reference file somewhere with the time the sync was performed. The file is created BEFORE the sync so it is on all slave systems.
2. Use your favorite tool to sync the folders between master and slave. I use Unison, as it is WAN-friendly, unlike XCopy or RoboCopy.
3. Once an hour (or whatever schedule you prefer) scan the master folder structure, comparing file timestamps against the reference file timestamp.
4. If there are any newer files/folders detected, update the reference file (delete/recreate, or write to an INI format file) so the timestamp changes.
My slave servers simply compare their copy of the reference file with the master and initiate a sync if the timestamps are different. They perform this comparison about 10 minutes after the above process
My code originally scanned only folder timestamps, since in a software distribution model, we either add, rename, or delete files and folders. These actions update the parent folder timestamp, which is enough to detect a change that requires replication. I can scan my 16 Gig folder structure with 3688 folders in 85 seconds. Working on a smaller scale, I checked the WinXP structure, which includes all of the XP install files, service packs, and hotfixes - about 850 Meg worth. I can scan/compare all 165 folders there in 7-8 seconds, and every one of the 7039 files in 16 seconds.
Clearly, the difference here is not performing the file to file comparison, but simply detecting if something has changed since the last sync. I have Kix tools for file compare/copy using MD5 Checksum comparison, and while they are useful for small (file deployment) processes, I would not use them for large folder synchronizations. This is one of the reasons I like Unison - it generates a checksum index locally, then spins up a copy of itself on the remote system to do the same - it can then exchange this small file of differences to determine exactly which files need to be copied, with minimal network overhead.
My UDF returns a "0" if nothing changed since the last check, or a "1", followed by the list of changed files/folders if changes were detected. This lets you copy the specific files that were changed.
Anyway - just something to consider in the design of your tool to improve the performance.
Glenn
_________________________
Actually I am a Rocket Scientist! 
|
|
Top
|
|
|
|
|
#184451 - 2008-01-08 01:31 AM
Re: Monitoring Changes on a Drive
[Re: Glenn Barnas]
|
buckla1
Fresh Scripter
Registered: 2007-12-27
Posts: 7
|
Thanks for the Reply Glenn
That’s incredibly fast.
So can you say how you are able to search that many files and folders retrieve the timestamps for each in that few seconds? What command are you using?
Wouldn't it slow you down having TextfileA with 6000 with filenames and directories and timestamp then running scan two with 6001 entries with filenames and directories and timestamp and comparing the two, or is there some type of array and doing a type of ascan against the array?
I'm intrigued...
Edited by buckla1 (2008-01-08 01:44 AM)
|
|
Top
|
|
|
|
|
#184452 - 2008-01-08 04:56 AM
Re: Monitoring Changes on a Drive
[Re: buckla1]
|
Glenn Barnas
KiX Supporter
Registered: 2003-01-28
Posts: 4401
Loc: New Jersey
|
Well, there are a couple of assumptions, and I DO NOT compare the timestamps of each file in two locations!
BIG ASSUMPTION #1
The two directories are supposed to be identical. Differences are permitted only between scan/sync processes.
BIG ASSUMPTION #2
This is a MASTER/slave relationship. Files can be modified only in one location, and all other sites are effectively READ ONLY.
Given these assumptions, when a file changes at the master site, I can detect it quite easily. I read the timestamp of the LASTSYNC.INI file in the root of the data folder. I compare that timestamp with all other timestamps in the directory structure. My scan never leaves the file system, much less the server, so it's very fast. Since I use Unison to sync the files, I can stop scanning after the first detected change (knowing a sync is required), although I don't. I log all changed files, then write the current time to LASTSYNC.INI. This assures that this file has a later time stamp than any modified file within the structure.
The slave servers simply compare the timestamp of their local copy of LASTSYNC.INI with that of the same file on the MASTER server. If it's different by more than 10 seconds, it launches a Unison command to sync the folders. The 10-second window allows small time drift between poorly connected sites, possibly without accurate NTP services.
I use Unison instead of RoboCopy/XCopy because it's fast, reliable, and WAN-friendly. Unison is freeware. You install a copy on the master server running as a service, using SrvAny. The secondary server invokes a Unison command line - something like
Unison.exe E:\Data socket://master:1479/E:\data -batch -force socket://remote:1479/E:\data
This command, run from a slave system after detecting a newer copy of \\master\Data\LASTSYNC.INI, does the following:
- Opens a socket connection to the MASTER server on port 1479, passing the path.
- MASTER builds a timestamp/checksum report and sends it to the slave. Meanwhile, the slave is creating its own report.
- Slave compares its local report with the one returned by the master. It identifies the differences, and sends a list of file requests to the MASTER, who begins the transfer process.
The -batch argument allows Unison to run in unattended mode, and the -Force defines the MASTER root path. This process is great for WAN connections because all the time/checksum processing is done locally to each server.
On a side note regarding Unison - I wrote a Kix-based tool to do deep folder scans and file by file checksum comparisons of images on our web servers. It insured that production servers had the latest images from the deployment server, and it ran every 2 hours during the day. Using pure kix code, it took about 45 minutes to process the 20,000+ images, even if nothing was copied (an IN-SYNC condition). Using Unison with two UNC root definitions doing regular NetBIOS communications, it dropped to 90 seconds - the team was exstatic. After I installed Unison as a service on the slave systems (this process is configured as a MASTER PUSH, not a slave-pull), I reduced the total process time to about 7 seconds for an IN-SYNC condition.
BTW - Unison is capable of bidirectional synchronizations, although it's a bit tricky, and may require some manual intervention if a file is modified in both roots.
Glenn
_________________________
Actually I am a Rocket Scientist!
|
|
Top
|
|
|
|
|
#208217 - 2013-12-30 08:07 AM
Re: Monitoring Changes on a Drive
[Re: Glenn Barnas]
|
basad
Fresh Scripter
Registered: 2004-12-18
Posts: 11
|
Dear Glen,
Do you mind sharing your Code/Script?.
I would like to sync 2 folders across the WAN using Unison.
Thanks and Regards,
basad
|
|
Top
|
|
|
|
|
#208219 - 2013-12-30 01:50 PM
Re: Monitoring Changes on a Drive
[Re: basad]
|
Glenn Barnas
KiX Supporter
Registered: 2003-01-28
Posts: 4401
Loc: New Jersey
|
Basad,
The actual code is part of one of our commercial apps, so I can't post it here directly. It's also pretty specific to our app.
The process is not difficult to establish as long as you have a master/slave relationship between your servers.
On each of the servers that you want to sync, you install Unison - this is basically putting Unison.exe somewhere in the PATH and defining the UNISON environment variable to point to a folder where the Unison profiles will be stored. A profile defines the two root paths to sync, as well as any optional parameters. Unison also stores the replica data files there. (small files with information about both sides of the replicas.)
On the Master server, you configure Unison to run as a service using SrvAny. (from the Win2K Resource Kit) The command to run is simply "unison -socket 1479", which starts Unison listening for connections on port 1479. (You can use any free port you wish.)
Then, on the master server, you create a script that runs via the task scheduler on a defined schedule. The frequency will depend on the sync time, but 2-4 hours has generally worked for us. This script uses the FolderModCheck() UDF to quickly determine if any folders in the source path have been modified. (Download the UDF from the Kix Library on my web site.) This leverages the fact that modifying (adding/removing/changing) any file in a folder also updates the folder timestamp. Thus, you only need to compare the folder timestamps and not every file time stamp. This script is something like:$LastTS = GetFileTime('\root\LastChange.ini')
If FolderModCheck('\root', $LastTS) ; did any file change?
$ = WriteProfileString('\root\LastChange.ini', 'COMMON', 'LastChange', $LastTS)
EndIf
On the client server(s), on a regular cycle, you run a script that compares the timestamp of the local copy of the LastChange.ini file with the remote copy. Remember, this file is part of the replication and will exist on both sides. If the timestamps match, then nothing needs to be done. If they don't match, then a sync needs to be initiated. You can SHELL or RUN the command illustrated in my previous post, changing the "E:\Data" path to the correct path on your system. The "master" and "remote" are the hostnames of the master and slave servers.
What's nice is that you can put the copy of Unison on two PCs, start the "service" by entering the service command in a command prompt on one computer, and manually enter the slave's sync command on the other computer. You should be able to test the connections and refine the profile settings before putting it into production on your servers. By playing with Unison on two PCs, I was able to work out all the necessary options in just a few hours of tinkering. The Unison manual is only about 30 pages, and the parameters you want to focus on in your profiles are "-root", "-force", "-batch", "-path", and those related to defining log paths.
You can sync multiple slaves. You should be able to make multiple connections to a single listener, but I prefer not to. You can create a unique listener for each slave by creating multiple SrvAny services, with each one listening on a different port. Our commercial product has a single Kix-based socket listener that requests a connection. The sync manager allows up to 10 sessions, and either responds with a port number to use for sync after dynamically starting a Unison listener, or a NAK to reject the request. That's probably overkill for most environments, but we've deployed SWDIST into environments with as many as 400 slave servers hitting a dozen intermediate hosts that in turn sync with a single master.
Glenn
_________________________
Actually I am a Rocket Scientist!
|
|
Top
|
|
|
|
|
#208222 - 2013-12-30 05:45 PM
Re: Monitoring Changes on a Drive
[Re: Glenn Barnas]
|
basad
Fresh Scripter
Registered: 2004-12-18
Posts: 11
|
Thanks Glenn, I will test it and will come back with the results.
Regards
Basad
|
|
Top
|
|
|
|
|
#208224 - 2013-12-30 06:44 PM
Re: Monitoring Changes on a Drive
[Re: Glenn Barnas]
|
Glenn Barnas
KiX Supporter
Registered: 2003-01-28
Posts: 4401
Loc: New Jersey
|
They threw us out early for the holiday.. 
The following is a Unison Profile that is placed on the external Media HDD, along with a copy of UNISON. Both are in a \UNISON folder. The profile name is "MEDIASYNC.PRF". This allows you to invoke Unison with just the profile name.
"MServer" is the name of the media server where Unison is already running. Specify the path to the root of the media folder in the "root" and "force" arguments. The "force" insures a one-way replication to the forced path. The second "root" defines the path on the external HDD on the local computer. If you want 2-way sync, it's good to define a preference - replace "force" with "prefer". If the same file has been modified on both sides, the "preferred" side will win out and replace the other file.
log, batch, and times are options that enable logging, run in batch mode (no user prompting), and maintain modification times of the files during replication.
The "path" statements define the folders in the ROOT paths to sync. If these are omitted, then all subfolders are synced.
For a production sync, this configuration would be done on a slave server.root = socket://MServer:1758/D:\Media\Multimedia\
force = socket://MServer:1758/D:\Media\
root = e:\
log = true
batch = true
times = true
path = Music
path = Cartoons
path = AudioBooks
path = Movies\Classics
The following is a SYNC.BAT file that is in the root of the external Media HDD. Note that if the local HDD on the target system is not "E:" then both the batch file and the MediaSync.Prf files will need to be updated.@Echo Off
REM - this script assumes that the external drive is "E:"
REM - Data will be synced from the media server to the external HDD.
Set LCLHDD=E:
REM Define the UNISON home folder
Set UNISON=%LCLHDD%\UNISON
Run the Unison command and specify the MediaSync.prf profile
%LCLHDD%\Unison MediaSync
Installing Unison as a service is simple if you use SrvAny from the Win2K Resource Kit. This can be downloaded from MS. The package consists of 2 executable files - InstSrv.exe and SrvAny.exe. The first is used to set up the service, the second is the service file itself.
On the (master) server, create a UNISON folder. (C:\Progra~2\Unison)
Create a SYSTEM environment variable called UNISON that references that path. (UNISON=C:\Progra~2\Unison)
Place the Unison.exe on the server, preferrably in the Unison folder.
Copy the SrvAny.exe to the server. It should be placed in a folder in the System PATH. C:\Windows is OK, but I prefer a dedicated folder for add-on server tools like this. Your choice.
You don't need to copy the InstSrv.exe file to the server.. just run "InstSrv.exe Unison d:\path\to\SrvAny.exe" to define the service.
Open RegEdit and locate the "HKLM\SYSTEM\CurrentControlSet\Services\UNISON" path.
Create a SubKey called "Parameters", and then, in the Parameters key, create two REG_SZ values:Application d:\path\to\unison.exe
AppParameters -socket 1758 -logfile d:\path\to\unison.log
Run NET START UNISON or start the Unison service from the service control panel. The server is ready!
Start simple - create a folder on the server root, put a couple of files in it and run the UNISON MEDIASYNC command on the client. (for a data sync, you can rename the bat and profile files to something meaningful.)
Verify that the folders are synced.. check the logs on the server.
With this setup, it took about 4 hours to complete an initial sync of my library to the external HDD over a Gig-E link (about 650G of music and audio books). Later sync's take just minutes.
Glenn
_________________________
Actually I am a Rocket Scientist!
|
|
Top
|
|
|
|
|
#208232 - 2013-12-31 06:58 AM
Re: Monitoring Changes on a Drive
[Re: Glenn Barnas]
|
basad
Fresh Scripter
Registered: 2004-12-18
Posts: 11
|
Your valuable input Glenn is highly appreciated, I’m sure it will take me sometime to digest this, but I’m sure it will be ok.
Thanks again
Basad
|
|
Top
|
|
|
|
|
#208240 - 2014-01-02 06:57 AM
Re: Monitoring Changes on a Drive
[Re: basad]
|
basad
Fresh Scripter
Registered: 2004-12-18
Posts: 11
|
First of all happy new year to all members.
Thanks Glenn; the test scenario worked perfectly between 2 PCs (both windows 7). However; I could not get unison to create the log file on the Master, is it becuase I ran the service on Windows 7?.
I'm using the latest stable version of Unison (2.4)
Thanks
|
|
Top
|
|
|
|
Moderator: Jochen, Allen, Radimus, Glenn Barnas, ShaneEP, Ruud van Velsen, Arend_, Mart
|
0 registered
and 229 anonymous users online.
|
|
|
