LonkeroAdministrator
(KiX Master Guru)
2004-06-07 04:07 PM
KiXbin - kixtart binary access library (cRFC)

as has derived from:
http://www.kixtart.org/ubbthreads/showflat.php?Cat=&Number=119472&page=0&view=collapsed&sb=5&o=2&fpart=1

the current dll version is 1.0.4

it now supports also text files (ie, write array of characters and it writes you an textfile)

if someone of you can reach ruud, might want to consider asking about this.
I checked the scriptlogic forum too (where he has been seen the last, afaik) but no life there either.

for file-functions I consider adding yet truncading and appending for write-modes.
and for read more partial read as if you've been pointed to wrong file it can take a while to read 5G of data before script gets it's hand to it to realize it was wrong file.

about richards suggestion on seek()
not sure what he wants to seek.
particular byte or byte frequence or even text-style matching (which indeed produces huge overheat)?

k, for file-operations that's about it.
44k dll.

then I must ask is that all?
like, we can access other devices than normal FS-IO.
we can burn CD's or we can do tape-backups.
we can stream data to network (using sockets or not) and so on.

I mean, I had 2 things in mind why I had to code this.
1) get my ntbackup script working without selecting the data in ntbackup, so unprofessional
2) email with attachments directly from kixtart (with help of this and hoby's net::telnet impl.)

so, now I can do my backup stuff with KF and only do the actual writing with ntbackup.
and I can email myself screenshot from client wksta when they run into trouble during logon (ok, ok, I need to implement clipboard code for that, but you get my point)


anyways, the options are limited only by my lack of coding talents (which is limitation I intend to remove) and the lack of need.
so, if you always wanted to do something but you couldn't as kixtart has it's limits on binary.
shoot it.


LonkeroAdministrator
(KiX Master Guru)
2004-06-07 04:15 PM
Re: KiXbin - kixtart binary access library (cRFC)

oh, and the file is at www.gwspikval.com/jooel/files/KiXbin

Richard H.Administrator
(KiX Supporter)
2004-06-07 04:46 PM
Re: KiXbin - kixtart binary access library (cRFC)

Seek

Seek (as used in 'C') sets the file "cursor", which is where the next read and/or write will happen.

The two most important file postioning functions are:

  1. ftell(filehandle)
    which returns the current cursor position in the file stream.

  2. fseek(filehandle,position,origin)
    which moves to the specific position.
    "position" may be +/-
    "origin" may be:
    • "from the start of file" (SEEK_SET)
    • "from the current position" (SEEK_CUR)
    • "from the end of file" (SEEK_END)

    Positioning the cursor past the end of file is valid, and will write zeros to the intermediate characters.
    Positioning the cursor before the start of the file is just plain daft


LonkeroAdministrator
(KiX Master Guru)
2004-06-07 05:00 PM
Re: KiXbin - kixtart binary access library (cRFC)

yup, me using that fseek already
but for what reason should I export such method?


LonkeroAdministrator
(KiX Master Guru)
2004-06-07 05:08 PM
Re: KiXbin - kixtart binary access library (cRFC)

hmm...
I would understand if I did provide really low level file handling.
starting from create() open() and so on.

but with current bulk methods it seems rather out there.

instead of exporting these directly, why not have:
ReadEx() - read the specified block (specify amount and start)
WriteEx() - write the specified block (specify amount and start)

hmm...
that would remove the need for append() too.
specifying writeEx("file",$data,"SEEK_END",0)

that is...


LonkeroAdministrator
(KiX Master Guru)
2004-06-07 10:23 PM
Re: KiXbin - kixtart binary access library (cRFC)

thinking on the syntax...
maybe better go with:
readEx(file,startposition,amount)
writeEx(file,startposition,datatowrite)

but with these, either I need to instruct ppl to check the file with getfilesize or export:
fílesize()

and also:
uncompressedSize()


LonkeroAdministrator
(KiX Master Guru)
2004-06-08 12:37 AM
Re: KiXbin - kixtart binary access library (cRFC)

k, little further the road:

just added append there.
still got to work out replace() and trunc()
reading the docs, the normal trunc just sets the EOF which ain't truly truncing the data.


Richard H.Administrator
(KiX Supporter)
2004-06-08 10:37 AM
Re: KiXbin - kixtart binary access library (cRFC)

Quote:

but with current bulk methods it seems rather out there.



Not too sure what you mean. Reading and writing blocks is unaffected. You simply read or write the blocks at the current position in the file which is a more natural way of coding.

If you are working with binary files then at some point you are going to want to handle specific data types, like longs and decimals. You could convert these and write small blocks, but it would make more sense to do something like:
Code:

; Create binary file handler
$oKiXbinFile=createobject("kixbin.file")

; Open student data file.
$oFile=$oKiXbinFile.FileOpen("c:\data\students.dat",2+4) ; Open file for both read (2) and write (4).

$iRecordSize=512 ; Student records are 512 bytes long
$iStudentNo=56 ; Unique student number
$iRecordOffset=($iStudentNo-1)*$iRecordSize

$lYear=2004
$iAge=16
$dAverageScore=73.28
$sClass="Physics L6"

; Move file cursor to student record
$oFile.Seek($iRecordOffset,SEEK_SET)

; Update student record
$oFile.Write($lYear)
$oFile.Write($iAge)
$oFile.Write($dAverageScore)
$oFile.Write($sClass)

; Flush changes to disk and close stream
$oFile.Close()



Unfortunately, KiXtart does not support all basic types, so it will probably require coercing:
Code:
$oFile.WriteLong($lYear)
$oFile.WriteShort($iAge)
$oFile.WriteDouble($dAverageScore)
$oFile.WriteString($sClass)



Or, taking the values from VarType:
Code:
$oFile.Write($lYear, 2)         ; Integer
$oFile.Write($iAge,17) ; Byte
$oFile.Write($dAverageScore,5) ; Double
$oFile.Write($sClass) ; String (default)



Blocks of course will work similarly:
Code:
$oFile.ReadBlock(256) ; Read up to 256 bytes and return as an array, array will be smaller on EOF.
$oFile.WriteBlock($aiData) ; Write array.
$oFile.WriteBlock($aiData,256) ; Write 256 bytes from $aiData, pad with zeros if $aiData is shorter.




Quote:

reading the docs, the normal trunc just sets the EOF which ain't truly truncing the data.




Hmm. Maybe it's a windows thing. truncate()/ftruncate() will (normally) set the file length, truncating the file and releasing unused frames.

Quote:

...I need to instruct ppl to check the file with getfilesize...




Or:
Code:
$oFile.seek(0,SEEK_END) /* go to end of file */
$iFileSize=$oFile.Tell() /* Get cursor position, which == file size in bytes */



LonkeroAdministrator
(KiX Master Guru)
2004-06-08 10:43 AM
Re: KiXbin - kixtart binary access library (cRFC)

ja, that's the method I use.
anyways, reading and writing blocks of data on stream is same as writing the whole file.
the data goes in, one by one, in BYTES.
thus, there is no other vartype basically available than ints ([for kixtart, LONG] with the current requested method)
but, as variant supports other types too, it can take string input (in form of character array)

anyways, the data is always to be byte ordered and if not huge overheat wanted, the array way is the only way to do it efficiently.


LonkeroAdministrator
(KiX Master Guru)
2004-06-08 11:29 AM
Re: KiXbin - kixtart binary access library (cRFC)

k, reading again after some thinking, you are actually wishing to get all the low-level functions exported.
mode flags etc need to be translated though (or restricted), but anyways...
that's an option, too.
question arises, who is gonna use it then.
I know I would, but as an example, it would take week or two for our mods to study how to use it
talk about normal users then...

anyways, it indeed sounds better to have these methods.
but for normal read&write operations, it's zillion times faster to use (I say the word again) bulk access.
it takes around 80ms to access a method via com interface.
so, taking an example and writing 1k file with say, 256 byte blocks takes a heck of a time.
suddenly, even though the idea sounds good, it's lot better and faster to load even 10M file to kixtart array and manipulate that than direct manipulation.


Richard H.Administrator
(KiX Supporter)
2004-06-08 11:46 AM
Re: KiXbin - kixtart binary access library (cRFC)

Quote:

ja, that's the method I use.
anyways, reading and writing blocks of data on stream is same as writing the whole file.
the data goes in, one by one, in BYTES.
thus, there is no other vartype basically available than ints ([for kixtart, LONG] with the current requested method)
but, as variant supports other types too, it can take string input (in form of character array)

anyways, the data is always to be byte ordered and if not huge overheat wanted, the array way is the only way to do it efficiently.




Not quite right. File IO is cached and buffered, and written efficiently. You *can* make it inefficient by flushing the io stream after every write, or setting the file io to unbuffered.

There is a small overhead in system calls, but it will be far less that the overhead required to use KiXtart to convert data between types and manage arrays.

Data in files is as you say unstructured. You have to impose a structure on it. You could do this by reading in the bytes and then write KiXtart UDFs to convert it. Bytes, ints, uints and strings are a doddle to convert, anything with a mantissa is a bugger though. Much easier to read the bytes directly into the correct variable type, which "automatically" coverts the data.

I'm not trying to teach you to suck eggs here, but I don't know how familiar you are with file io so apologies in advance if you already know this stuff.

As a quick example, to write a single integer "i" to an open file "fpFile":
Code:
fwrite(&i,sizeof i,1,fpFile);


To read it back again:
Code:
fread(&i,sizeof i,1,fpFile);



There is no conversion done - the contents of the variable are simply read and written.

You need the additional data types when reading and writing as the file you are accessing may be used by other applications. If you limit the types of data that may be written you also limit the files that you can access.

There is no reason not to accept variant data and cast/convert to the correct type, using scanf/printf for example.


Richard H.Administrator
(KiX Supporter)
2004-06-08 11:57 AM
Re: KiXbin - kixtart binary access library (cRFC)

I've just read your next post, I see that it is the overhead of the COM call which you think will be the problem rather than file IO.

In my earlier examples, I included "ReadBlock()" to read a block of data into an array - this deals with large unstructured reads, though reading a 10MB file in one go might slow things down a bit!

The other caveat with large arrays is that if you hand them off to UDFs to interpret they are copied unless you make them GLOBAL, which will create it's own overhead in memory and CPU.


LonkeroAdministrator
(KiX Master Guru)
2004-06-08 12:23 PM
Re: KiXbin - kixtart binary access library (cRFC)

ja...
I've done speed tests only one small files (less than 1M) and on those, passing any control to kixtart instead of the whole file makes the delays with Idispatch grow huge.

what comes to data type, not sure what you mean by actually anything you said before.
we both know that kixtart can basically handle only longs and strings and arrays of those.

having the ability to write or read anything else is waste as it can't be used anyways.
as an addition, kixtart already has text-I/O routines, so only binary (hence the dll) is lacking.
thus, writing strings with the dll is not on option.
well, could do binaryString but as already realized, it takes seconds to read even small files.

so, all I'm left with is bytes.
and taking account kixtart only handles longs and strings, I can export and import the data only with these vartypes.
again, we know that kixtart has trouble with, say chr(0) and afaik, with everything that goes above 256.
so, strings as export is not an option.
conclusion, all that is left is long.

to import, I have already enabled all that kixtart can handle, bytes as strings (one char long) or longs.
both input and output on fileIO is done with INT (fwrite,fread) and trunslated to I4 or from I4 just for kix.

that about covers it.
so, asking where does the additional data-types come into action?
and, if you access the files as series of bytes, how you can limit your access in any way?
heck, it's bytes, no matter what vartype you see externally.


Richard H.Administrator
(KiX Supporter)
2004-06-08 02:11 PM
Re: KiXbin - kixtart binary access library (cRFC)

Quote:

we both know that kixtart can basically handle only longs and strings and arrays of those.



You forgot doubles.

Quote:

having the ability to write or read anything else is waste as it can't be used anyways.



No. Although KiXtart can only deal with them using one of the three variant classes, the file may of course be for an application which can use many more types.

Quote:

kixtart already has text-I/O routines, so only binary (hence the dll) is lacking



But you cannot mix'n'match text IO and binary IO.

Quote:

heck, it's bytes, no matter what vartype you see externally.



Of course, but have you tried converting a stored decimal

Let me give you an example. This is a small file from my application, as you would see it from the KiXbin file read:
Code:
$aiFileData=255,6,115,116,114,105,110,103,174,71,225,122,12,36,254,192,249,255,255,255



This data comprises:
  1. An unsigned char
  2. An unsigned char which is the length of the following string
  3. The string (not zero terminated)
  4. A double
  5. An signed integer


Now, if you wanted to read and update my file in KiXtart you could do it, but it would be pretty tricky. Especially that double.


LonkeroAdministrator
(KiX Master Guru)
2004-06-08 03:12 PM
Re: KiXbin - kixtart binary access library (cRFC)

yes, it's tricky, but there is no easier way.
if there is, please show me.


Richard H.Administrator
(KiX Supporter)
2004-06-08 03:44 PM
Re: KiXbin - kixtart binary access library (cRFC)

Quote:

yes, it's tricky, but there is no easier way.
if there is, please show me.



Show me yours first

From what I remember when I was playing with COM automation, you can determine the data type that is being passed, and convert where necessary.

You should be able to return all the types, cast to longs, doubles and strings.

It doesn't matter a whole lot that in the file it is an unsigned integer, and when it arrives in KiXtart it is a double. When you write "123.45" as a double in the file it doesn't matter that in KiXtart it is a string - you just need to handle the type and convert to the correct form.

I don't know C++, so bear with me while I explain in C.

If you have a function which is receiving a string and needs to write it out as a double there are a couple of ways. Here's one:
Code:
/* Create a simple output file */

#include <stdio.h>

void fnWriteStringAsBYTE(FILE *fp,char *s)
{
char c;
c=(char)atoi(s);
fwrite(&c,sizeof c,1,fp);
}

void fnWriteStringAsINT(FILE *fp,char *s)
{
int i;
i=atoi(s);
fwrite(&i,sizeof i,1,fp);
}

void fnWriteStringAsLONG(FILE *fp,char *s)
{
long l;
l=atol(s);
fwrite(&l,sizeof l,1,fp);
}

void fnWriteStringAsDOUBLE(FILE *fp,char *s)
{
double d;
d=atof(s);
fwrite(&d,sizeof d,1,fp);
}

int main()
{
fnWriteStringAsBYTE(stdout,"123");
fnWriteStringAsINT(stdout,"123");
fnWriteStringAsLONG(stdout,"123456");
fnWriteStringAsDOUBLE(stdout,"1234.56");
return 0;
}



When reading it is even easier.
Load the variable into the correct type, then cast it to the type that you are going to return to KiXtart. The type you return just needs to be big enough to handle the full range of the original type.


LonkeroAdministrator
(KiX Master Guru)
2004-06-08 09:49 PM
Re: KiXbin - kixtart binary access library (cRFC)

still can't really understand.
fwrite() does right with int, so not sure how would I write double with it.

so say I might get around that (which is tricky already), you want to be able to write data, then write data in different type and then write additional data (if that ain't tricky enough) just to be able to go around the forming of the proper value in kix.


LonkeroAdministrator
(KiX Master Guru)
2004-06-08 11:56 PM
Re: KiXbin - kixtart binary access library (cRFC)

and back to the double case:
Quote:

When an object of floating type is converted to an integral type, the fractional part is truncated. No rounding takes place in the conversion process. Truncation means that a number like 1.3 is converted to 1, and –1.3 is converted to –1.




now, isn't that the standard way?
so, if one passes a numeric double (not array of chars) to variant and I use variants change type, I loose everything after decimal point.

so, unless I start doing data type conversions and mangles and gangles inside the dll, pure double is no option, right?


Richard H.Administrator
(KiX Supporter)
2004-06-09 11:10 AM
Re: KiXbin - kixtart binary access library (cRFC)

Quote:

fwrite() does right with int, so not sure how would I write double with it.




Not quite right. fwrite() takes a *void* pointer, and writes the memory that the pointer points to.

I think I see your confusion. You need to understand how variables are allocated space and are referenced. It is one of the more complicated areas, but opens up a whole new world of programming.

I'll give a short explanation, but you really need to get a good book. I still use my original K&R 'C' book - it is invaluable.

Ok, here's how it works. Variables are simply contiguous memory locations. You can think of them in exactly the same way as the data in your file. Different types of data require different numbers of contiguous bytes.

(I'm assuming 1 memory word=1 byte=8 bits for this explanation)
char=1 byte
unsigned char=1 byte
short int=2 bytes
long int=4 bytes
double presicion float=8 bytes
Note, different architectures will have differ requirements.

Now normally you access the variable by it's name. This is in fact (resolved to) a pointer to the location in memory. The compiler knows how big the variable is, so knows how much memory to read or write when the variable is accessed.

When your code says:
Code:
int i
i=10;



The first line declares a variable "i" which reserves 4 bytes of memory and sets "i" to point to that memory. The second line writes the value 10 in byte order to the memory location.

Now the tricky bit. You can get the address of the variable by using the "&" operator. This means that you can access the memory *directly*.

Here is a very simple example:
Code:
#include <stdio.h>

int main()
{
int i;
char c[sizeof(int)]; /* c is a pointer to an array of chars (a string) the same length as an int */

i=7827319;

memcpy(&c,&i,sizeof(i)); /* copy bytes from int to string */

printf("%d=%s\n",i,c);
return 0;
}



The "sizeof" operator returns the size (in bytes) of the data type.

The example code declares i as an int, then c as an array of chars, the same size as an int.

We assign a value to the int, and then use memcpy() to copy the memory pointed to by i (&i) to that pointed to by c (&c). "sizeof" is used to determine how many bytes to copy.

The result is a very short message:
Quote:

7827319=wow




You can do exactly the same thing will all variable types.

This means that you can use fread/fwrite to read and write any type of data without losing information.

If I take one of my earlier examples:
Code:
void fnWriteStringAsDOUBLE(FILE *fp,char *s)
{
double d;
d=atof(s);
fwrite(&d,sizeof d,1,fp);
}


Here, fwrite() is passed the *address* of the double (&d). It is also told how big the data is (sizeof d), and that we are writing one instance.

This will cause the memory containing the value of "d" to be written to disk, in this case 8 bytes.

fread() does exactly the same thing in reverse.

BTW, where was the quote from?


Richard H.Administrator
(KiX Supporter)
2004-06-09 12:46 PM
Re: KiXbin - kixtart binary access library (cRFC)

With your golfing hat on you can probably see that the "wow" example can be reduced to:
Code:
#include <stdio.h>

int main()
{
int i=7827319;

printf("%d=%s\n",i,&i);
return 0;
}



LonkeroAdministrator
(KiX Master Guru)
2004-06-09 01:35 PM
Re: KiXbin - kixtart binary access library (cRFC)

ja ja.
feel like being in school again

these are the little things why I shouldn't have tropped it even though they forced me to study swedish

good lesson on simplyfying things.
so, to my next q.
well, I need to think that over...
should I remove the reading and writing of arrays and fall back to something.
I know I can't write array and somehow recognice where it is when reading.
and, I can't read multiple bytes of different types and give them to kixtart without array.
eh, I need some time to mix this up in mind but if you have a opinion, say it.
will take some time before I get back to drawing board.


LonkeroAdministrator
(KiX Master Guru)
2004-06-09 03:18 PM
Re: KiXbin - kixtart binary access library (cRFC)

just thinking structs here...
although kixtart does not support those, arrays of different vartypes kinda is like one.
wonder, should there be switch for how the data should be aligned... and is that possible.


Richard H.Administrator
(KiX Supporter)
2004-06-09 04:21 PM
Re: KiXbin - kixtart binary access library (cRFC)

Well, structs are lovely things, and are great for reading and writing fixed length records with mixed data types.

However I think that they may be one step too far though.

I suppose you could do something like:
Code:
$aRecordFormat=	"i",		; Unique Pupil Number
"c20", ; Student forename
"c", ; Student middle initial
"c20", ; Student family name
"d" ; Student average score as percentage

$aRecordData= "100",
"Richard",
"M",
"Howarth",
"99.99"

$obj.WriteRecord($aRecordFormat,$aRecordData)



I would have thought at this point you would be looking at ODBC drivers


LonkeroAdministrator
(KiX Master Guru)
2004-06-09 04:22 PM
Re: KiXbin - kixtart binary access library (cRFC)

haven't I told you I hate DB's?

k, back to thinking.