Help wanted to create an improved backup program... - KiXtart.org - official site

You are not logged in. [Log In] KiXtart.org website » Forums » KiXtart » Lounge » Help wanted to create an improved backup program...

Page 1 of 1

1

Topic Options

#102641 - 2003-07-08 02:25 PM Help wanted to create an improved backup program...
Fernando Madruga Fernando Madruga Starting to like KiXtart Registered: 2002-08-21 Posts: 149 Loc: Coimbra.Portugal.Europe.Earth....	Hi. I've been away from this forum (time's a killer!) but I decided to come back with a challenge for everyone that wants to enter. I have an idea that I want to expand and improve with your help... I'm the only programmer in my company's limited IT staff, so I have no one else to talk this one with... Purpose: Creating an "intelligent" backup program that would allow users to backup their data in a compressed, easily retrievable, central location. It would *not* create duplicate backup files for the same data, even among users. Ideally it should run at logoff, but can run instead in the logon script. How: Something along these lines: Using the WinNotify calls to detect modified files in a given sub-folder(s) (My Documents, Favorites, Desktop, Outlook personal folders and settings, Office Settings / templates, etc...) "Brand" user's files and folders with a "unique" ID using an MD5 checksum, to compensate for file renaming/moving Could use NTFS streams for that (store one in "My Documents" with a list of files/folders/IDs/CRCs) Provide graphical feedback on the whole copy process and estimated time for completion Allow for system and user specified exclusions (folders NOT to be copied over to the server) Allow for system and user specified INCLUSIONS (folders to be copied over to the server) Use "logged in user" to do the backup Advise AGAINST doing a backup when home server is on a different subnet (for laptop users) Compress files individually (.gz ?) Keep updated CRC file list in the server's backup and use that one if user deletes hidden backup config Allow for "Complete Restore" to compensate for user moving to a different machine and/or having it's PC's HD reformatted Only do one of the following if user's PC is on the same subnet as user's backup server: Create a FULL initial backup, generating CRCs and creating configuration files if no backup present on server Do a FULL restore if backup present and no files present after asking user for permission Optionaly, but this will complicate things further, allow for "incremental/diferential" backups and/or multiple backup versions Some mechanism will be needed to create a new file ID whenever a user modifies a file that has been saved if more users have saved that same file This is just a starting sketch: it has a long way to go, but when we're done, I'm sure we can all take advantage of this! NOTE: This is NOT* to start coding! First we'll elaborate on the idea, problems, solutions and only THEN will we start coding... _________________________ Later, [b]Mad[/b]ruga
Top

#102642 - 2003-07-08 03:14 PM Re: Help wanted to create an improved backup program...
Shawn Shawn Administrator Registered: 1999-08-13 Posts: 8611	Question ... understand the Win*Notify calls but to what end ? Will this "app" run as background process and detect changed files on the fly, or will it run once (either during logon or logoff) and reconcile then. If the latter, why would you need real-time notifications. -Shawn [ 08. July 2003, 15:15: Message edited by: Shawn ]
Top

#102643 - 2003-07-08 03:24 PM Re: Help wanted to create an improved backup program...
Sealeopard Sealeopard KiX Master Registered: 2001-04-25 Posts: 11165 Loc: Boston, MA, USA	Can you utilize the NTBACKUP application? you can dynamically create the backup task folders, to e.g. only perform incremental/differential backups to the server. This would remove a lot of the logic and you can still put a unique ID into the backup set. The disadvantage is that the resore operation is not scriptable with NTBACKUP, as far as I know. _________________________ There are two types of vessels, submarines and targets.
Top

#102644 - 2003-07-08 03:49 PM Re: Help wanted to create an improved backup program...
Fernando Madruga Fernando Madruga Starting to like KiXtart Registered: 2002-08-21 Posts: 149 Loc: Coimbra.Portugal.Europe.Earth....	To Shawn: Good point! I'd thought about backup programs for a few times now, and this post was a "copy/paste" of several diferent ones... Indeed, if we are going to "KiX" it, we'd not need the WinNotify... That's for when I thought of making a perl service to install on the machines... To Sealeopard: Nope. One of the main goals of this thing is avoiding to store 2 or more copies of a file, simply because 20 users received that file by email and decided to store it in their personal folders. NTBackup will simply store them all, because it's being run by diferent users on diferent PCs and would not know of the existence of the others... In fact, NTBackup is so "dumb" that it will store a file twice or more if the same file appears on two or more folders in one job... _________________________ Later, [b]Mad[/b]ruga
Top

#102645 - 2003-07-08 03:52 PM Re: Help wanted to create an improved backup program...
Sealeopard Sealeopard KiX Master Registered: 2001-04-25 Posts: 11165 Loc: Boston, MA, USA	Diskspace is cheap. Get a couple of 250MB (S)ATAs, a ATA-RAID card and do RAID 0/1/5/10/50. Then you don't have to worry about duplicate files. _________________________ There are two types of vessels, submarines and targets.
Top

#102646 - 2003-07-08 04:25 PM Re: Help wanted to create an improved backup program...
Fernando Madruga Fernando Madruga Starting to like KiXtart Registered: 2002-08-21 Posts: 149 Loc: Coimbra.Portugal.Europe.Earth....	Sure, it is cheap. However, this is just the starting point for something else, namely, a program that would replicate whole directories of files among servers. And yes, I know that I can use rsync, but if I move a 2 GB folder elsewhere in the server, it will try to re-copy those 2 GB... Considering that we use many links with just 32 or 64 kbps... We also use TONS of catalogues, and some of the most "stupid" ones, change the file names on each monthly update! Some sort of "random" file names... Instead of copying TONS of GB each month, my idea was to take advantage of file properties (MD5/file size/type) and save a lot of bandwith updating 20+ diferent sites... So, even if we can use a blunt aproach such as "add more HD space", a more intelligent aproach would give us some added beneficts... Hope it's a bit more clear what can be achieved with this kind of tool, should one create it... _________________________ Later, [b]Mad[/b]ruga
Top

Page 1 of 1

1

Previous Topic

Previous Topic View All Topics

View All Topics

Index Next Topic

Next Topic

Moderator: Arend_, Allen, Jochen, Radimus, Glenn Barnas, ShaneEP, Ruud van Velsen, Mart

Hop to:

Shout Box

Who's Online

0 registered and 987 anonymous users online.

Newest Members

StuTheCoder, M_Moore, BeeEm, min_seow, Audio
17884 Registered Users

Generated in 0.054 seconds in which 0.024 seconds were spent on a total of 12 queries. Zlib compression enabled.

click tracking