brewdude6
(Hey THIS is FUN)
2006-12-04 06:26 AM
Dirplus to gather files

I have an archive directory that I'm going to use dirplus and 7zip to create a zip file for every day of the year. This is a huge directory without subdirectories containing pdf's, csv's and tabs with this name format (RPT710.D.00001.ALL.20061203.0000001.CSV). What would be the best way to increment through every day (about two years worth of files) with a dirplus command? I've thought about creating a text file of dates and looping through that array and dirplus each element, but not sure if that would be very efficient.

Gargoyle
(MM club member)
2006-12-04 06:53 AM
Re: Dirplus to gather files

If the date is always in the same place and in the same format, you can Split() the array that is returned by DirPlus, and then use that information to move forward.

Code:
$Counter2 = 0
For $Counter = 0 to UBound($DirPlus_Array)
  $Split_Name = Split($DirPlus_Array[$Counter],".")
  If $SPlit_Name[3] = $Specified_Date
    Redim Preserve $Build_Array[$Counter2]
    $Build_Array[$Counter2] = $DirPlus_Array[$Counter]
    $Counter2 = $Counter2 + 1
  EndIf
Next


brewdude6
(Hey THIS is FUN)
2006-12-04 04:15 PM
Re: Dirplus to gather files

There are 1.5 million files in this directory. Should I break up the arrays created somehow?

Gargoyle
(MM club member)
2006-12-04 07:07 PM
Re: Dirplus to gather files

I do not know what the limitations of KiX is in that regards, while it may take time to search an array with 1.5 million entries, it is not like you are going to be doing this on every users workstation. It may be inconvienent for the person that ends up running the script.

Others may be able to answer if there is a limitation with KiX in regards to the size of the array's that you would be working with.

NTDOCAdministrator
(KiX Master)
2006-12-04 07:15 PM
Re: Dirplus to gather files

The memory of the machine and string length are the only two I can think of.

A few people have run some large array scripts similar and didn't seem to have an issue. Now if you're trying to put the array back together as a string then concatenated then yes you're going to bump into a string lenght issue pretty quick on that size.

But processing each element should be okay as long as there is enough memory in the system.

brewdude6
(Hey THIS is FUN)
2006-12-05 12:06 AM
Re: Dirplus to gather files

How would I find the $Specified_Date variable? Would I need a separate array with those in them?

LonkeroAdministrator
(KiX Master Guru)
2006-12-05 12:32 AM
Re: Dirplus to gather files

hmm...
think you could go the other way.

Code:
for $year=2004 to 2006
 for $month=1 to 12
  for $day=1 to 31
   $file=dir("folder\RPT*.ALL."+$year+right("0"+$month,2)+right("0"+$day,2)+".*.CSV")
   $files=""
   while $file and @error=0
    $files=$files+"\"+$file
    $file=dir()
   loop
   if 2<len($files)
    ? "make zip for " $year " " $month " " $day " consisting off:" ?
    for each $f in split($files,"\")
     "  " $f ?
    next
   endif
  next
 next
next


or something similar...

brewdude6
(Hey THIS is FUN)
2006-12-05 04:19 AM
Re: Dirplus to gather files

Thanks for the suggestion. If I try this it simply adds all the files to the zip file for the date fields present. I know I've got some variables in strings...just experimenting.

Code:

For $year=2004 to 2006
 For $month=1 to 12
  For $day=1 to 31
   $file=Dir("c:\admin1\*.*.*.*."+$year+Right("0"+$month,2)+Right("0"+$day,2)+".*.*")
   $files=""
   While $file And @error=0
    $files=$files+"\"+$file
    $file=Dir()
   Loop
   If 2<Len($files)
    ;? "make zip for " $year$month$day " consisting off:" ?
    For Each $f in Split($files,"\")
     $f ?
	Shell "%comspec% /c 7z.exe u -tzip c:\admin2\RPT$year$month$day c:\admin1\$f"
    Next
   EndIf
  Next
 Next
Next

LonkeroAdministrator
(KiX Master Guru)
2006-12-05 06:55 AM
Re: Dirplus to gather files

you should use the right() trick I used for the days so all dates have the same length and you can know what is RPT2004111
currently, it can be eleventh of jan 2004 or first of nov 2004.

but, back to your post.
when you say "if I try this, it simply..."
does that mean that it somehow doesn't work or that it is simply brilliant?
as adding the files to daily archives was the task, right?

brewdude6
(Hey THIS is FUN)
2006-12-05 01:47 PM
Re: Dirplus to gather files

It was simply brilliant...pardon me for not clarifying that.

brewdude6
(Hey THIS is FUN)
2006-12-05 04:09 PM
Re: Dirplus to gather files

Are you adding the "\" character to the $files variable so you'll have a delimiter for the split command?

What about the If 2<Len($files) line? I'm not quite sure what checking the length is doing for you here.

Would I need to use the "right" command to pull the string for the file name with the "$files" variable and not the "$file"?

I know this is all over the map, but I'm trying to make sure I understand what is going on.

Thanks for your help.

LonkeroAdministrator
(KiX Master Guru)
2006-12-05 09:00 PM
Re: Dirplus to gather files

yes, \ is delimiter.
and right(,2) is just for checking if there is files for that day. could have simply used len()

don't use right or anything, you get the filenames if you split with the delimiter like shown in my example.

I also see you removed all the matching characters from the dir:
$file=Dir("c:\admin1\*.*.*.*."+$year+Right("0"+$month,2)+Right("0"+$day,2)+".*.*")

even though, it now requires to have enough dots in the name, but at least leaving the constant values, if there are any should be considered.
if there is none, then it's fine as it is.

brewdude6
(Hey THIS is FUN)
2006-12-06 05:56 AM
Re: Dirplus to gather files

I need to create a file for each day ie: RPT20061204.zip with all the files with *.20061204.* in the file name added to that days zip file. Maybe I should just move the files to a directory for each day first to keep things simple, and then zip the folders later.

LonkeroAdministrator
(KiX Master Guru)
2006-12-06 03:31 PM
Re: Dirplus to gather files

what's wrong with the add method you used earlier?

brewdude6
(Hey THIS is FUN)
2006-12-08 09:56 PM
Re: Dirplus to gather files

I don't know how to correct it, but the method that I posted creates a file correctly, but adds every file in the directory to every zip file. I'd also like to pad a "0" to the single digit days of the month.

LonkeroAdministrator
(KiX Master Guru)
2006-12-08 10:49 PM
Re: Dirplus to gather files

well, then you need to use the right() trick I used with my example.

hmm...
if all files are added to all zips, does all of the files also show in "$f ?" listing?
if yes, sounds like kixtart does not support multiple * wildcards.
but that need to be studied before saying anything sure.

brewdude6
(Hey THIS is FUN)
2006-12-09 12:35 AM
Re: Dirplus to gather files

Thanks..the right() trick does work for the file name. The $f? does return the expected files. I'm sure it's the way I'm shelling out to 7zip. For some reason, every file is added to each zip. If I replace the shell line to echo the $f variable I get the correct list of files for each zip.

NTDOCAdministrator
(KiX Master)
2006-12-09 12:55 AM
Re: Dirplus to gather files

Not sure if it will help, but you can take a look at the script I ran here using 7Zip.

http://www.kixtart.org/forums/ubbthreads.php?ubb=showflat&Number=135983=#post135984

.

LonkeroAdministrator
(KiX Master Guru)
2006-12-09 01:55 AM
Re: Dirplus to gather files

weird.
there is nothing special in the shell line that would make it behave like that.
well, except the lack of 2-digit month and day, like already said some days ago.
but, if you still with the right() hack experience this... well, guess, either the script needs to be made a lot more comp...
k, as I was typing I realised it.

change this:
For Each $f in Split($files,"\")

to this:
For Each $f in Split(substr($files,2),"\")

and the symptoms are gone.

brewdude6
(Hey THIS is FUN)
2006-12-09 02:06 AM
Re: Dirplus to gather files

SWEET! Thank you very much! I obviously would of never came up with that fix.

brewdude6
(Hey THIS is FUN)
2006-12-09 04:05 AM
Re: Dirplus to gather files

Any guesses on how long it will take a 4way with 4gb of ram to compress 1,476,361 files taking up 56.7gb of space?

Sealeopard
(KiX Master)
2006-12-09 04:12 AM
Re: Dirplus to gather files

Always too long.

Sorry, but the processing speed would depend on so many factors that it's basically meaningless. Maybe a couple of hours at most. Sounds like a nice over-night task, in wich case you wouldn't worry about processing speed anyway.

brewdude6
(Hey THIS is FUN)
2006-12-09 05:54 AM
Re: Dirplus to gather files

I know it's meaningless...that's why I said "guesses". Thanks for the insight.