#120709 - 2004-06-07 04:07 PM
KiXbin - kixtart binary access library (cRFC)
|
Lonkero
KiX Master Guru
   
Registered: 2001-06-05
Posts: 22346
Loc: OK
|
as has derived from: http://www.kixtart.org/ubbthreads/showflat.php?Cat=&Number=119472&page=0&view=collapsed&sb=5&o=2&fpart=1
the current dll version is 1.0.4
it now supports also text files (ie, write array of characters and it writes you an textfile)
if someone of you can reach ruud, might want to consider asking about this. I checked the scriptlogic forum too (where he has been seen the last, afaik) but no life there either.
for file-functions I consider adding yet truncading and appending for write-modes. and for read more partial read as if you've been pointed to wrong file it can take a while to read 5G of data before script gets it's hand to it to realize it was wrong file.
about richards suggestion on seek() not sure what he wants to seek. particular byte or byte frequence or even text-style matching (which indeed produces huge overheat)?
k, for file-operations that's about it. 44k dll.
then I must ask is that all? like, we can access other devices than normal FS-IO. we can burn CD's or we can do tape-backups. we can stream data to network (using sockets or not) and so on.
I mean, I had 2 things in mind why I had to code this. 1) get my ntbackup script working without selecting the data in ntbackup, so unprofessional  2) email with attachments directly from kixtart (with help of this and hoby's net::telnet impl.)
so, now I can do my backup stuff with KF and only do the actual writing with ntbackup. and I can email myself screenshot from client wksta when they run into trouble during logon (ok, ok, I need to implement clipboard code for that, but you get my point)
anyways, the options are limited only by my lack of coding talents (which is limitation I intend to remove) and the lack of need. so, if you always wanted to do something but you couldn't as kixtart has it's limits on binary. shoot it.
_________________________
!download KiXnet
|
|
Top
|
|
|
|
#120711 - 2004-06-07 04:46 PM
Re: KiXbin - kixtart binary access library (cRFC)
|
Richard H.
Administrator
   
Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
|
Seek
Seek (as used in 'C') sets the file "cursor", which is where the next read and/or write will happen.
The two most important file postioning functions are:
- ftell(filehandle)
which returns the current cursor position in the file stream.
- fseek(filehandle,position,origin)
which moves to the specific position. "position" may be +/- "origin" may be:
- "from the start of file" (SEEK_SET)
- "from the current position" (SEEK_CUR)
- "from the end of file" (SEEK_END)
Positioning the cursor past the end of file is valid, and will write zeros to the intermediate characters. Positioning the cursor before the start of the file is just plain daft
|
|
Top
|
|
|
|
#120716 - 2004-06-08 10:37 AM
Re: KiXbin - kixtart binary access library (cRFC)
|
Richard H.
Administrator
   
Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
|
Quote:
but with current bulk methods it seems rather out there.
Not too sure what you mean. Reading and writing blocks is unaffected. You simply read or write the blocks at the current position in the file which is a more natural way of coding.
If you are working with binary files then at some point you are going to want to handle specific data types, like longs and decimals. You could convert these and write small blocks, but it would make more sense to do something like: Code:
; Create binary file handler $oKiXbinFile=createobject("kixbin.file") ; Open student data file. $oFile=$oKiXbinFile.FileOpen("c:\data\students.dat",2+4) ; Open file for both read (2) and write (4). $iRecordSize=512 ; Student records are 512 bytes long $iStudentNo=56 ; Unique student number $iRecordOffset=($iStudentNo-1)*$iRecordSize $lYear=2004 $iAge=16 $dAverageScore=73.28 $sClass="Physics L6"
; Move file cursor to student record $oFile.Seek($iRecordOffset,SEEK_SET) ; Update student record $oFile.Write($lYear) $oFile.Write($iAge) $oFile.Write($dAverageScore) $oFile.Write($sClass) ; Flush changes to disk and close stream $oFile.Close()
Unfortunately, KiXtart does not support all basic types, so it will probably require coercing: Code:
$oFile.WriteLong($lYear) $oFile.WriteShort($iAge) $oFile.WriteDouble($dAverageScore) $oFile.WriteString($sClass)
Or, taking the values from VarType: Code:
$oFile.Write($lYear, 2) ; Integer $oFile.Write($iAge,17) ; Byte $oFile.Write($dAverageScore,5) ; Double $oFile.Write($sClass) ; String (default)
Blocks of course will work similarly: Code:
$oFile.ReadBlock(256) ; Read up to 256 bytes and return as an array, array will be smaller on EOF. $oFile.WriteBlock($aiData) ; Write array. $oFile.WriteBlock($aiData,256) ; Write 256 bytes from $aiData, pad with zeros if $aiData is shorter.
Quote:
reading the docs, the normal trunc just sets the EOF which ain't truly truncing the data.
Hmm. Maybe it's a windows thing. truncate()/ftruncate() will (normally) set the file length, truncating the file and releasing unused frames.
Quote:
...I need to instruct ppl to check the file with getfilesize...
Or: Code:
$oFile.seek(0,SEEK_END) /* go to end of file */ $iFileSize=$oFile.Tell() /* Get cursor position, which == file size in bytes */
|
|
Top
|
|
|
|
#120717 - 2004-06-08 10:43 AM
Re: KiXbin - kixtart binary access library (cRFC)
|
Lonkero
KiX Master Guru
   
Registered: 2001-06-05
Posts: 22346
Loc: OK
|
ja, that's the method I use. anyways, reading and writing blocks of data on stream is same as writing the whole file. the data goes in, one by one, in BYTES. thus, there is no other vartype basically available than ints ([for kixtart, LONG] with the current requested method) but, as variant supports other types too, it can take string input (in form of character array)
anyways, the data is always to be byte ordered and if not huge overheat wanted, the array way is the only way to do it efficiently.
_________________________
!download KiXnet
|
|
Top
|
|
|
|
#120718 - 2004-06-08 11:29 AM
Re: KiXbin - kixtart binary access library (cRFC)
|
Lonkero
KiX Master Guru
   
Registered: 2001-06-05
Posts: 22346
Loc: OK
|
k, reading again after some thinking, you are actually wishing to get all the low-level functions exported. mode flags etc need to be translated though (or restricted), but anyways... that's an option, too. question arises, who is gonna use it then. I know I would, but as an example, it would take week or two for our mods to study how to use it  talk about normal users then...
anyways, it indeed sounds better to have these methods. but for normal read&write operations, it's zillion times faster to use (I say the word again) bulk access. it takes around 80ms to access a method via com interface. so, taking an example and writing 1k file with say, 256 byte blocks takes a heck of a time. suddenly, even though the idea sounds good, it's lot better and faster to load even 10M file to kixtart array and manipulate that than direct manipulation.
_________________________
!download KiXnet
|
|
Top
|
|
|
|
#120719 - 2004-06-08 11:46 AM
Re: KiXbin - kixtart binary access library (cRFC)
|
Richard H.
Administrator
   
Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
|
Quote:
ja, that's the method I use. anyways, reading and writing blocks of data on stream is same as writing the whole file. the data goes in, one by one, in BYTES. thus, there is no other vartype basically available than ints ([for kixtart, LONG] with the current requested method) but, as variant supports other types too, it can take string input (in form of character array)
anyways, the data is always to be byte ordered and if not huge overheat wanted, the array way is the only way to do it efficiently.
Not quite right. File IO is cached and buffered, and written efficiently. You *can* make it inefficient by flushing the io stream after every write, or setting the file io to unbuffered.
There is a small overhead in system calls, but it will be far less that the overhead required to use KiXtart to convert data between types and manage arrays.
Data in files is as you say unstructured. You have to impose a structure on it. You could do this by reading in the bytes and then write KiXtart UDFs to convert it. Bytes, ints, uints and strings are a doddle to convert, anything with a mantissa is a bugger though. Much easier to read the bytes directly into the correct variable type, which "automatically" coverts the data.
I'm not trying to teach you to suck eggs here, but I don't know how familiar you are with file io so apologies in advance if you already know this stuff.
As a quick example, to write a single integer "i" to an open file "fpFile": Code:
fwrite(&i,sizeof i,1,fpFile);
To read it back again: Code:
fread(&i,sizeof i,1,fpFile);
There is no conversion done - the contents of the variable are simply read and written.
You need the additional data types when reading and writing as the file you are accessing may be used by other applications. If you limit the types of data that may be written you also limit the files that you can access.
There is no reason not to accept variant data and cast/convert to the correct type, using scanf/printf for example.
|
|
Top
|
|
|
|
#120721 - 2004-06-08 12:23 PM
Re: KiXbin - kixtart binary access library (cRFC)
|
Lonkero
KiX Master Guru
   
Registered: 2001-06-05
Posts: 22346
Loc: OK
|
ja... I've done speed tests only one small files (less than 1M) and on those, passing any control to kixtart instead of the whole file makes the delays with Idispatch grow huge.
what comes to data type, not sure what you mean by actually anything you said before. we both know that kixtart can basically handle only longs and strings and arrays of those.
having the ability to write or read anything else is waste as it can't be used anyways. as an addition, kixtart already has text-I/O routines, so only binary (hence the dll) is lacking. thus, writing strings with the dll is not on option. well, could do binaryString but as already realized, it takes seconds to read even small files.
so, all I'm left with is bytes. and taking account kixtart only handles longs and strings, I can export and import the data only with these vartypes. again, we know that kixtart has trouble with, say chr(0) and afaik, with everything that goes above 256. so, strings as export is not an option. conclusion, all that is left is long.
to import, I have already enabled all that kixtart can handle, bytes as strings (one char long) or longs. both input and output on fileIO is done with INT (fwrite,fread) and trunslated to I4 or from I4 just for kix.
that about covers it. so, asking where does the additional data-types come into action? and, if you access the files as series of bytes, how you can limit your access in any way? heck, it's bytes, no matter what vartype you see externally.
_________________________
!download KiXnet
|
|
Top
|
|
|
|
#120722 - 2004-06-08 02:11 PM
Re: KiXbin - kixtart binary access library (cRFC)
|
Richard H.
Administrator
   
Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
|
Quote:
we both know that kixtart can basically handle only longs and strings and arrays of those.
You forgot doubles.
Quote:
having the ability to write or read anything else is waste as it can't be used anyways.
No. Although KiXtart can only deal with them using one of the three variant classes, the file may of course be for an application which can use many more types.
Quote:
kixtart already has text-I/O routines, so only binary (hence the dll) is lacking
But you cannot mix'n'match text IO and binary IO.
Quote:
heck, it's bytes, no matter what vartype you see externally.
Of course, but have you tried converting a stored decimal
Let me give you an example. This is a small file from my application, as you would see it from the KiXbin file read: Code:
$aiFileData=255,6,115,116,114,105,110,103,174,71,225,122,12,36,254,192,249,255,255,255
This data comprises:
- An unsigned char
- An unsigned char which is the length of the following string
- The string (not zero terminated)
- A double
- An signed integer
Now, if you wanted to read and update my file in KiXtart you could do it, but it would be pretty tricky. Especially that double.
|
|
Top
|
|
|
|
#120724 - 2004-06-08 03:44 PM
Re: KiXbin - kixtart binary access library (cRFC)
|
Richard H.
Administrator
   
Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
|
Quote:
yes, it's tricky, but there is no easier way. if there is, please show me.
Show me yours first 
From what I remember when I was playing with COM automation, you can determine the data type that is being passed, and convert where necessary.
You should be able to return all the types, cast to longs, doubles and strings.
It doesn't matter a whole lot that in the file it is an unsigned integer, and when it arrives in KiXtart it is a double. When you write "123.45" as a double in the file it doesn't matter that in KiXtart it is a string - you just need to handle the type and convert to the correct form.
I don't know C++, so bear with me while I explain in C.
If you have a function which is receiving a string and needs to write it out as a double there are a couple of ways. Here's one: Code:
/* Create a simple output file */
#include <stdio.h>
void fnWriteStringAsBYTE(FILE *fp,char *s) { char c; c=(char)atoi(s); fwrite(&c,sizeof c,1,fp); }
void fnWriteStringAsINT(FILE *fp,char *s) { int i; i=atoi(s); fwrite(&i,sizeof i,1,fp); }
void fnWriteStringAsLONG(FILE *fp,char *s) { long l; l=atol(s); fwrite(&l,sizeof l,1,fp); }
void fnWriteStringAsDOUBLE(FILE *fp,char *s) { double d; d=atof(s); fwrite(&d,sizeof d,1,fp); }
int main() { fnWriteStringAsBYTE(stdout,"123"); fnWriteStringAsINT(stdout,"123"); fnWriteStringAsLONG(stdout,"123456"); fnWriteStringAsDOUBLE(stdout,"1234.56"); return 0; }
When reading it is even easier. Load the variable into the correct type, then cast it to the type that you are going to return to KiXtart. The type you return just needs to be big enough to handle the full range of the original type.
|
|
Top
|
|
|
|
#120727 - 2004-06-09 11:10 AM
Re: KiXbin - kixtart binary access library (cRFC)
|
Richard H.
Administrator
   
Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
|
Quote:
fwrite() does right with int, so not sure how would I write double with it.
Not quite right. fwrite() takes a *void* pointer, and writes the memory that the pointer points to.
I think I see your confusion. You need to understand how variables are allocated space and are referenced. It is one of the more complicated areas, but opens up a whole new world of programming.
I'll give a short explanation, but you really need to get a good book. I still use my original K&R 'C' book - it is invaluable.
Ok, here's how it works. Variables are simply contiguous memory locations. You can think of them in exactly the same way as the data in your file. Different types of data require different numbers of contiguous bytes.
(I'm assuming 1 memory word=1 byte=8 bits for this explanation) char=1 byte unsigned char=1 byte short int=2 bytes long int=4 bytes double presicion float=8 bytes Note, different architectures will have differ requirements.
Now normally you access the variable by it's name. This is in fact (resolved to) a pointer to the location in memory. The compiler knows how big the variable is, so knows how much memory to read or write when the variable is accessed.
When your code says: Code:
int i i=10;
The first line declares a variable "i" which reserves 4 bytes of memory and sets "i" to point to that memory. The second line writes the value 10 in byte order to the memory location.
Now the tricky bit. You can get the address of the variable by using the "&" operator. This means that you can access the memory *directly*.
Here is a very simple example: Code:
#include <stdio.h> int main() { int i; char c[sizeof(int)]; /* c is a pointer to an array of chars (a string) the same length as an int */ i=7827319; memcpy(&c,&i,sizeof(i)); /* copy bytes from int to string */ printf("%d=%s\n",i,c); return 0; }
The "sizeof" operator returns the size (in bytes) of the data type.
The example code declares i as an int, then c as an array of chars, the same size as an int.
We assign a value to the int, and then use memcpy() to copy the memory pointed to by i (&i) to that pointed to by c (&c). "sizeof" is used to determine how many bytes to copy.
The result is a very short message:
Quote:
7827319=wow
You can do exactly the same thing will all variable types.
This means that you can use fread/fwrite to read and write any type of data without losing information.
If I take one of my earlier examples: Code:
void fnWriteStringAsDOUBLE(FILE *fp,char *s) { double d; d=atof(s); fwrite(&d,sizeof d,1,fp); }
Here, fwrite() is passed the *address* of the double (&d). It is also told how big the data is (sizeof d), and that we are writing one instance.
This will cause the memory containing the value of "d" to be written to disk, in this case 8 bytes.
fread() does exactly the same thing in reverse.
BTW, where was the quote from?
|
|
Top
|
|
|
|
#120728 - 2004-06-09 12:46 PM
Re: KiXbin - kixtart binary access library (cRFC)
|
Richard H.
Administrator
   
Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
|
With your golfing hat on you can probably see that the "wow" example can be reduced to: Code:
#include <stdio.h> int main() { int i=7827319; printf("%d=%s\n",i,&i); return 0; }
|
|
Top
|
|
|
|
Moderator: Shawn, ShaneEP, Ruud van Velsen, Arend_, Jochen, Radimus, Glenn Barnas, Allen, Mart
|
0 registered
and 665 anonymous users online.
|
|
|