Page 1 of 1 1
Topic Options
#135803 - 2005-03-16 09:25 PM how to scrub a file?
annie Offline
Fresh Scripter

Registered: 2002-02-12
Posts: 10
I need to 'scrub' a log file that's being placed on my server and will grow continually unless I purge records > say 72 hours old. Anyone know a decent way to do this? help??
Top
#135804 - 2005-03-16 09:26 PM Re: how to scrub a file?
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11616
Loc: CA
Try searching here on the board using different keywords.

Plenty of code and UDF stuff to do what you want.

Sorry don't have time to do the search for you right now.

Top
#135805 - 2005-03-16 09:33 PM Re: how to scrub a file?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Normally with log files you rotate them rather than delete records.

Pick a period (hourly, daily) and when it is triggered MOVE the file to a new name - keep a few generations.

The only tricky thing is that the process writing to the file needs to understand what is happening when the file moves. You may need to stop the process, move the file and then re-start the process.

For example, assuming your file is called "mylogfile.txt" and you want to keep a weeks worth then at midnight each day you would do the following in a batch file:
Code:
move mylogfile.006 mylogfile.007
move mylogfile.005 mylogfile.006
move mylogfile.004 mylogfile.005
move mylogfile.003 mylogfile.004
move mylogfile.002 mylogfile.003
move mylogfile.001 mylogfile.001
move mylogfile.txt mylogfile.001


Top
#135806 - 2005-03-18 07:11 PM Re: how to scrub a file?
annie Offline
Fresh Scripter

Registered: 2002-02-12
Posts: 10
I did search, and didn't find anything that removed records, only things that removed files or folders. I don't really wnat to copy the file, as my goal is to reduce the amount of space the logs are taking up, not increase by generating multiple copies. all I want to do is shave the oldest records off the bottom.
Top
#135807 - 2005-03-18 07:32 PM Re: how to scrub a file?
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22344
Loc: OK
so, why not just use readfile() and then writefile()

nah, even easier, readline() + drop-the-last-lines-code + writeline()
_________________________
!

download KiXnet

Top
#135808 - 2005-03-18 08:40 PM Re: how to scrub a file?
Bryce Offline
KiX Supporter
*****

Registered: 2000-02-29
Posts: 3166
Loc: Houston TX
here is some code, using FSO to return a files creation date, and then compairing the age of that date vs the current date and time.

i ma using the UDF's FlipCTime() and TimeConvert()

Code:

$f = 'logfile.log'
$f = CreateObject("Scripting.FileSystemObject").GetFile($f)

$date = Split($f.DateCreated)[0]
$date = Split($date,"/")
$date = $date[2]+"/"+$date[0]+"/"+$date[1]

$time = timeconvert(Split($f.DateCreated)[1] +" "+ Split($f.DateCreated)[2])

$72h = 259200

If flipctime(@DATE,@TIME) - flipctime($date,$time) > $72h
? 'the file ' + $f.name + ' was created 72+ hours ago'
EndIf



Bryce

Top
#135809 - 2005-03-18 09:23 PM Re: how to scrub a file?
Les Offline
KiX Master
*****

Registered: 2001-06-11
Posts: 12734
Loc: fortfrances.on.ca
You guys are missing the req's. It is just one file. Within this file are dated records and the ojective is to purge "older than" records. AFAIK, since KiX cannot edit files in place, the only option is to open the existing log AND create/open a new file, recursively parse and skip or copy to new until EOF, close both files and delete the original log, keeping the new smaller log.
_________________________
Give a man a fish and he will be back for more. Slap him with a fish and he will go away forever.

Top
#135810 - 2005-03-18 09:28 PM Re: how to scrub a file?
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11616
Loc: CA
Questions:

1. How many lines in the log file or how large in MB
2. Do you want to purge from the TOP or BOTTOM of log?
3. Is this log file open by another process, any issues there?
4. Entries older then x How are old entries determined?

Richard's solution would trim down the file size for you automatically. If you don't want to keep the older files, then delete them instead of saving them.


Edited by NTDOC (2005-03-18 10:24 PM)

Top
#135811 - 2005-03-18 10:12 PM Re: how to scrub a file?
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22344
Loc: OK
isn't that just what les and someone else already said...
_________________________
!

download KiXnet

Top
#135812 - 2005-03-18 10:19 PM Re: how to scrub a file?
Les Offline
KiX Master
*****

Registered: 2001-06-11
Posts: 12734
Loc: fortfrances.on.ca
While I don't think Annie is ready to accept Richard's suggestion yet, I believe it is the most logical approach. Rather than let hte log file grow too large, move it to another file on a regular schedule and purge the old files once they reach a certain age.

Also, this topic should have been called "How to prune a file?"
_________________________
Give a man a fish and he will be back for more. Slap him with a fish and he will go away forever.

Top
#135813 - 2005-03-18 10:40 PM Re: how to scrub a file?
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11616
Loc: CA
Here are some UDFs that might help if you do want to edit the file

UDFs from Waltz
ReadFileHead() - Delivers the first #n lines of a file

ReadFileTail() - Delivers the last #n lines of a file

UDF from Bryce
Tail() - return the last Line or Lines of a file


Top
#135814 - 2005-03-21 09:29 AM Re: how to scrub a file?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Annie,

You are not keeping multiple copies by rotating the file. You set the rotation period so that you keep the same number of records, but simply split up into several files. The rotation automatically truncates the file.

Let's say you want to keep no more than 100 megabytes of log file data. In this case check the file every so often and when it reaches 10MB rotate it. Keep 9 instance of the rotated files.

You will never have much more than 100MB of log files stored providing you set the check interval correctly.

Top
#135815 - 2005-03-21 10:18 AM Re: how to scrub a file?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
There is an extensive log maintenance "kit" here: http://www.kixtart.org/ubbthreads/showflat.php?Cat=&Number=75703&page=&view=&sb=5&o=&vc=1

However, I think that this is a bit more fully featured than you need.

Here is a simple example of how you could do the size limit thing in KiXtart:
Code:
Function udfRotateLog($sPath,$iSize,Optional $iGenerations,Optional $bInitLog)


Dim $sMask,$i,$fd

If Not CInt($iGenerations) $iGenerations=1 EndIf
If Not CInt($bInitLog) $bInitLog=0 EndIf
$iSize=CDBL($iSize)

While Len($sMask)<Len($iGenerations) $sMask=CSTR($sMask)+"0" Loop

If GetFileSize($sPath)>=$iSize
; Sanity check - Ignore
; 1 Read Only
; 4 System
; 16 Direcroty
; 256 Temporary
; 4096 Offline
If (1+4+16+256+4096) & GetFileAttr($sPath) Exit 1 EndIf
If @ERROR Exit @ERROR EndIf
; Remove oldest generation
Del $sPath+"."+$iGenerations
; Rotate existing log files
For $i=$iGenerations To 2 Step -1
Move $sPath+"."+Right($sMask+($i-1),Len($iGenerations)) $sPath+"."+Right($sMask+$i,Len($iGenerations))
; Ignore "file not found"
If Not(@ERROR=0 OR @ERROR=2) Exit @ERROR EndIf
Next
; Now move primary log file
Move $sPath $sPath+"."+Right($sMask+1,Len($iGenerations))
If @ERROR Exit @ERROR EndIf
; Finally, create an empty log file if needed
If $bInitLog
$fd=FreeFileHandle()
If @ERROR Exit @ERROR EndIf
If Open($fd,$sPath,1+4) Exit @ERROR EndIf
$=Close($fd)
EndIf
EndIf
Exit 0
EndFunction



Parameters are:
  • $sPath - Path to your log file.
  • $iSize - When file reaches this size (in bytes) it is rotated. If you set this to zero, the file is always rotated.
  • $iGenerations - How many generations of the file to keep
  • $bInitLog - Set to true if you want to create an empty file once the log file has been rotated.


So to keep 50 MB of history, and create an empty file after the rotation:
Code:
$sLogFile="c:\foo\mylogfile.txt"

$iMaxSize=10*1024*1024 ; 10 Megabytes
$iMaxHistory=5

udfRotateLog($sLogFile,$sMaxSize,$iMaxHistory,1)


Top
Page 1 of 1 1


Moderator:  Arend_, Allen, Jochen, Radimus, Glenn Barnas, ShaneEP, Ruud van Velsen, Mart 
Hop to:
Shout Box

Who's Online
2 registered (Robdutoit, MacduMela) and 244 anonymous users online.
Newest Members
MacduMela, karimKix, NicoK, Kelly001, MichaelK72
17776 Registered Users

Generated in 0.06 seconds in which 0.022 seconds were spent on a total of 12 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org