Page 1 of 1 1
Topic Options
#165290 - 2006-08-01 06:03 PM Maximum filesize for Open()
peaps Offline
Fresh Scripter

Registered: 2005-04-12
Posts: 25
Quick question, -is there a maximum filesize that open() can cope with?

I am trying to open and readline() from a 66MB file and the first time I readline() it's coming back with an @ERROR code of -1 meaning end of file reached.

this is my code:



Code:
 BREAK ON

$x = open(3, "\\skynet\Safe\_index\gd.txt")
? "Open file " + $x
$line = readline(3)
? "First read: " + @ERROR
while @ERROR = 0
? $line
$line = readline(3)
loop
? "Done"
$x = close(3)


Top
#165291 - 2006-08-01 06:22 PM Re: Maximum filesize for Open()
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
check the error of open as well.
and the len() of $line.

Top
#165292 - 2006-08-02 05:06 AM Re: Maximum filesize for Open()
Sealeopard Offline
KiX Master
*****

Registered: 2001-04-25
Posts: 11164
Loc: Boston, MA, USA
Is tha a single line?

READLINE read the complete line of text up to the first @CRLF. If your .TXT file does not contain a @CRLF, then READLNE tries to read 66MB into the variable.
_________________________
There are two types of vessels, submarines and targets.

Top
#165293 - 2006-08-02 09:25 AM Re: Maximum filesize for Open()
peaps Offline
Fresh Scripter

Registered: 2005-04-12
Posts: 25
No the first line is about 20 characters long. I have checked for the @ERROR on open() but it's 0. So really not quite sure what's going on here.

The text file was created by running "DIR /B/S > gd.txt" on the command line, and this definately appends each line with @CRLF.

Top
#165294 - 2006-08-02 09:38 AM Re: Maximum filesize for Open()
peaps Offline
Fresh Scripter

Registered: 2005-04-12
Posts: 25
Hmm!! very interesting..

I changed the code replacing the UNC path with a local path:

Code:
 	$x = open(3, ".\gd.txt")
? "Open file " + $x
$line = readline(3)
? "First read: " + @ERROR
while @ERROR = 0
? $line
$line = readline(3)
loop
? "Done"
$x = close(3)



... and that works!

It obviously doesn't like BIG files over UNC.. Unless there's something wrong with my system. I have tried a smaller text file over UNC and it displayed it no problem.

Any ideas anyone?

Top
#165295 - 2006-08-02 09:45 AM Re: Maximum filesize for Open()
Mart Moderator Offline
KiX Supporter
*****

Registered: 2002-03-27
Posts: 4672
Loc: The Netherlands
I know the mode parameter on open is optional but the manuals states there are two default values (0 and 2). Typo I guess.

What happens if you change your open line to this:

Code:

$x = open(3, "\\skynet\Safe\_index\gd.txt", 2)




Quote:


Mode
Optional parameter that indicates what should happen if the file does not exist. This parameter can have the following values:

0 - If the file does not exist, OPEN fails with return code 2 (default).
1 - If the file does not exist, OPEN will create a new file.
2 - Opens the file for read access (default).
4 - Opens the file for write access.





BTW: A server called skynet? Reminds me of the Terminator movies.
http://en.wikipedia.org/wiki/Skynet


Edited by Mart (2006-08-02 09:47 AM)
_________________________
Mart

- Chuck Norris once sold ebay to ebay on ebay.

Top
#165296 - 2006-08-02 10:14 AM Re: Maximum filesize for Open()
peaps Offline
Fresh Scripter

Registered: 2005-04-12
Posts: 25
Tried the ,2 on the open() line, but that made no difference.. There must be more things at play here.

And yeah Skynet was actually stolen from the Terminator film. Sounds quite good! And if I remember it tried to take over the world. That's kinda like what I'm trying to do with lots of kix scripts...


Edited by peaps (2006-08-02 10:17 AM)

Top
#165297 - 2006-08-02 10:51 AM Re: Maximum filesize for Open()
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
dir...
you doing that dir how?
with the kix script before you call open?

with run perhaps?

Top
#165298 - 2006-08-02 11:18 AM Re: Maximum filesize for Open()
peaps Offline
Fresh Scripter

Registered: 2005-04-12
Posts: 25
No, I already created the dir output earlier, just by hand. It's basically an index file for rapid searching of a massive drive of documents. I have an AT job that runs an index every 24 hours over night, and the kix script just scans through the text file line by line looking for text matches.
Top
#165299 - 2006-08-02 12:04 PM Re: Maximum filesize for Open()
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
k, just asked.
you could have had the creation still on the way when kixtart opened the file, thus the file was empty at the moment of the open()

but if that's not the case...

Top
#165300 - 2006-08-02 02:46 PM Re: Maximum filesize for Open()
peaps Offline
Fresh Scripter

Registered: 2005-04-12
Posts: 25
Do you think that maybe this is one for Ruud? Would anyone else be prepared to try this out, i.e get a massive text file and see if you can open it over UNC. It doesn't need to be a separate physical machine, doing it on the same machine causes this problem.
Top
#165301 - 2006-08-02 09:34 PM Re: Maximum filesize for Open()
Mart Moderator Offline
KiX Supporter
*****

Registered: 2002-03-27
Posts: 4672
Loc: The Netherlands
Can I get the file somewhere? FTP or something?
Just did a little test on a 1mb "empty" file and it was still going strong when it reached 700.000 lines. This is not working as I hoped so if possible can you put the file temporary on a FTP server or something?
_________________________
Mart

- Chuck Norris once sold ebay to ebay on ebay.

Top
#165302 - 2006-08-02 09:38 PM Re: Maximum filesize for Open()
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11624
Loc: CA
Can upload here or similar sites.

http://www.rapidshare.de

Top
#165303 - 2006-08-02 11:42 PM Re: Maximum filesize for Open() when using UNC path
Witto Offline
MM club member
*****

Registered: 2004-09-29
Posts: 1828
Loc: Belgium
peaps,

Just created a dir file on my computer. it was about 10 MB. Your script works fine.

Created a big file like this:
Code:

copy dir1.txt+dir1.txt+dir1.txt+dir1.txt+dir1.txt+dir1.txt+dir1.txt dir.txt


Now I have a file of about 70 MB.
Your script with UNC path returns
Code:

Open file 0
First read: -1
Done


Scripts like I modified them:
This one is erratic
Code:

Break On
$x = Open(3,"\\"+@WKSTA+"\c$\dir.txt")
? "Open file " + $x
$line = ReadLine(3)
? "First read: " + @ERROR
While @ERROR = 0
? $line
$line = ReadLine(3)
Loop
? "Done"
$x = Close(3)


This one is OK
Code:

Break On
$x = Open(3,"c:\dir.txt")
? "Open file " + $x
$line = ReadLine(3)
? "First read: " + @ERROR
While @ERROR = 0
? $line
$line = ReadLine(3)
Loop
? "Done"
$x = Close(3)


I tested this with kix32.exe 4.52 and 4.53 Beta 1
IMHO this is an issue that should be moved to the "Beta forum".


Edited by Witto (2006-08-03 09:43 AM)

Top
#165304 - 2006-08-03 10:13 AM Re: Maximum filesize for Open() when using UNC path
peaps Offline
Fresh Scripter

Registered: 2005-04-12
Posts: 25
Ok. So now two of us have reproduced this problem, it's not just me or my PC! I'm glad.. What's the next step then?
Top
#165305 - 2006-08-03 11:47 AM Re: Maximum filesize for Open() when using UNC path
Mart Moderator Offline
KiX Supporter
*****

Registered: 2002-03-27
Posts: 4672
Loc: The Netherlands
I just tried to open a 45MB file from a remote disk and this works ok. Second test with an 85MB file failed.

The best thing to do (imho) is let the script that gathers the info copy the file to one central point and run the script that query’s the files on the system the files are stored on.

Opening large files over the network causes lots of traffic. Copying causes the same traffic but the copy process only has to run once for each info gathering action. If possible you can schedule this at a time that network traffic is low. When the copy action is done you can query the file as much as you want without causing huge amounts of network traffic and slowing down all other regular network traffic.


Edited by Mart (2006-08-03 11:48 AM)
_________________________
Mart

- Chuck Norris once sold ebay to ebay on ebay.

Top
#165306 - 2006-08-03 12:28 PM Re: Maximum filesize for Open() when using UNC path
peaps Offline
Fresh Scripter

Registered: 2005-04-12
Posts: 25
Well, that is certainly a possiblity. Especially if I zip the file first. It goes from 66MB to 4.5MB, then unzip (pkunzip) and keep it as a local cache.

But I was more getting at the apparent problem of opening large files over UNC paths; how should this be addressed?

Top
#165307 - 2006-08-03 01:37 PM Re: Maximum filesize for Open() when using UNC path
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
like you can see, I already moved the thread to beta.
I also mailed to ruud about the issue

Top
#165308 - 2006-08-03 03:21 PM Re: Maximum filesize for Open() when using UNC path
Ruud van Velsen Moderator Offline
Developer
*****

Registered: 1999-05-06
Posts: 391
Loc: Amsterdam, The Netherlands
Thanks for the excellent 'home-work' :-)

This is actually a limitation in the Win32 API used by Readline. Turns out that reading files this large (64MB+) across the network can run into system resource limitations. I will investigate to see if there is a way around this. If there is, the fix will be included in 4.53.

Regards,

Ruud

Top
Page 1 of 1 1


Moderator:  ShaneEP, Arend_, Jochen, Radimus, Glenn Barnas, Allen, Ruud van Velsen, Mart 
Hop to:
Shout Box

Who's Online
0 registered and 394 anonymous users online.
Newest Members
SERoyalty, mytar, Gabriel, Alex_Evos, Dansen
17869 Registered Users

Generated in 0.077 seconds in which 0.027 seconds were spent on a total of 14 queries. Zlib compression enabled.