|
|
|||||||
as has derived from: http://www.kixtart.org/ubbthreads/showflat.php?Cat=&Number=119472&page=0&view=collapsed&sb=5&o=2&fpart=1 the current dll version is 1.0.4 it now supports also text files (ie, write array of characters and it writes you an textfile) if someone of you can reach ruud, might want to consider asking about this. I checked the scriptlogic forum too (where he has been seen the last, afaik) but no life there either. for file-functions I consider adding yet truncading and appending for write-modes. and for read more partial read as if you've been pointed to wrong file it can take a while to read 5G of data before script gets it's hand to it to realize it was wrong file. about richards suggestion on seek() not sure what he wants to seek. particular byte or byte frequence or even text-style matching (which indeed produces huge overheat)? k, for file-operations that's about it. 44k dll. then I must ask is that all? like, we can access other devices than normal FS-IO. we can burn CD's or we can do tape-backups. we can stream data to network (using sockets or not) and so on. I mean, I had 2 things in mind why I had to code this. 1) get my ntbackup script working without selecting the data in ntbackup, so unprofessional 2) email with attachments directly from kixtart (with help of this and hoby's net::telnet impl.) so, now I can do my backup stuff with KF and only do the actual writing with ntbackup. and I can email myself screenshot from client wksta when they run into trouble during logon (ok, ok, I need to implement clipboard code for that, but you get my point) anyways, the options are limited only by my lack of coding talents (which is limitation I intend to remove) and the lack of need. so, if you always wanted to do something but you couldn't as kixtart has it's limits on binary. shoot it. |
||||||||
|
|
|||||||
oh, and the file is at www.gwspikval.com/jooel/files/KiXbin |
||||||||
|
|
|||||||
Seek Seek (as used in 'C') sets the file "cursor", which is where the next read and/or write will happen. The two most important file postioning functions are:
|
||||||||
|
|
|||||||
yup, me using that fseek already but for what reason should I export such method? |
||||||||
|
|
|||||||
hmm... I would understand if I did provide really low level file handling. starting from create() open() and so on. but with current bulk methods it seems rather out there. instead of exporting these directly, why not have: ReadEx() - read the specified block (specify amount and start) WriteEx() - write the specified block (specify amount and start) hmm... that would remove the need for append() too. specifying writeEx("file",$data,"SEEK_END",0) that is... |
||||||||
|
|
|||||||
thinking on the syntax... maybe better go with: readEx(file,startposition,amount) writeEx(file,startposition,datatowrite) but with these, either I need to instruct ppl to check the file with getfilesize or export: fílesize() and also: uncompressedSize() |
||||||||
|
|
|||||||
k, little further the road: just added append there. still got to work out replace() and trunc() reading the docs, the normal trunc just sets the EOF which ain't truly truncing the data. |
||||||||
|
|
|||||||
Quote: Not too sure what you mean. Reading and writing blocks is unaffected. You simply read or write the blocks at the current position in the file which is a more natural way of coding. If you are working with binary files then at some point you are going to want to handle specific data types, like longs and decimals. You could convert these and write small blocks, but it would make more sense to do something like: Code:
Unfortunately, KiXtart does not support all basic types, so it will probably require coercing: Code: $oFile.WriteLong($lYear) Or, taking the values from VarType: Code: $oFile.Write($lYear, 2) ; Integer Blocks of course will work similarly: Code: $oFile.ReadBlock(256) ; Read up to 256 bytes and return as an array, array will be smaller on EOF. Quote: Hmm. Maybe it's a windows thing. truncate()/ftruncate() will (normally) set the file length, truncating the file and releasing unused frames. Quote: Or: Code: $oFile.seek(0,SEEK_END) /* go to end of file */ |
||||||||
|
|
|||||||
ja, that's the method I use. anyways, reading and writing blocks of data on stream is same as writing the whole file. the data goes in, one by one, in BYTES. thus, there is no other vartype basically available than ints ([for kixtart, LONG] with the current requested method) but, as variant supports other types too, it can take string input (in form of character array) anyways, the data is always to be byte ordered and if not huge overheat wanted, the array way is the only way to do it efficiently. |
||||||||
|
|
|||||||
k, reading again after some thinking, you are actually wishing to get all the low-level functions exported. mode flags etc need to be translated though (or restricted), but anyways... that's an option, too. question arises, who is gonna use it then. I know I would, but as an example, it would take week or two for our mods to study how to use it talk about normal users then... anyways, it indeed sounds better to have these methods. but for normal read&write operations, it's zillion times faster to use (I say the word again) bulk access. it takes around 80ms to access a method via com interface. so, taking an example and writing 1k file with say, 256 byte blocks takes a heck of a time. suddenly, even though the idea sounds good, it's lot better and faster to load even 10M file to kixtart array and manipulate that than direct manipulation. |
||||||||
|
|
|||||||
Quote: Not quite right. File IO is cached and buffered, and written efficiently. You *can* make it inefficient by flushing the io stream after every write, or setting the file io to unbuffered. There is a small overhead in system calls, but it will be far less that the overhead required to use KiXtart to convert data between types and manage arrays. Data in files is as you say unstructured. You have to impose a structure on it. You could do this by reading in the bytes and then write KiXtart UDFs to convert it. Bytes, ints, uints and strings are a doddle to convert, anything with a mantissa is a bugger though. Much easier to read the bytes directly into the correct variable type, which "automatically" coverts the data. I'm not trying to teach you to suck eggs here, but I don't know how familiar you are with file io so apologies in advance if you already know this stuff. As a quick example, to write a single integer "i" to an open file "fpFile": Code: fwrite(&i,sizeof i,1,fpFile); To read it back again: Code: fread(&i,sizeof i,1,fpFile); There is no conversion done - the contents of the variable are simply read and written. You need the additional data types when reading and writing as the file you are accessing may be used by other applications. If you limit the types of data that may be written you also limit the files that you can access. There is no reason not to accept variant data and cast/convert to the correct type, using scanf/printf for example. |
||||||||
|
|
|||||||
I've just read your next post, I see that it is the overhead of the COM call which you think will be the problem rather than file IO. In my earlier examples, I included "ReadBlock()" to read a block of data into an array - this deals with large unstructured reads, though reading a 10MB file in one go might slow things down a bit! The other caveat with large arrays is that if you hand them off to UDFs to interpret they are copied unless you make them GLOBAL, which will create it's own overhead in memory and CPU. |
||||||||
|
|
|||||||
ja... I've done speed tests only one small files (less than 1M) and on those, passing any control to kixtart instead of the whole file makes the delays with Idispatch grow huge. what comes to data type, not sure what you mean by actually anything you said before. we both know that kixtart can basically handle only longs and strings and arrays of those. having the ability to write or read anything else is waste as it can't be used anyways. as an addition, kixtart already has text-I/O routines, so only binary (hence the dll) is lacking. thus, writing strings with the dll is not on option. well, could do binaryString but as already realized, it takes seconds to read even small files. so, all I'm left with is bytes. and taking account kixtart only handles longs and strings, I can export and import the data only with these vartypes. again, we know that kixtart has trouble with, say chr(0) and afaik, with everything that goes above 256. so, strings as export is not an option. conclusion, all that is left is long. to import, I have already enabled all that kixtart can handle, bytes as strings (one char long) or longs. both input and output on fileIO is done with INT (fwrite,fread) and trunslated to I4 or from I4 just for kix. that about covers it. so, asking where does the additional data-types come into action? and, if you access the files as series of bytes, how you can limit your access in any way? heck, it's bytes, no matter what vartype you see externally. |
||||||||
|
|
|||||||
Quote: You forgot doubles. Quote: No. Although KiXtart can only deal with them using one of the three variant classes, the file may of course be for an application which can use many more types. Quote: But you cannot mix'n'match text IO and binary IO. Quote: Of course, but have you tried converting a stored decimal Let me give you an example. This is a small file from my application, as you would see it from the KiXbin file read: Code: $aiFileData=255,6,115,116,114,105,110,103,174,71,225,122,12,36,254,192,249,255,255,255 This data comprises:
Now, if you wanted to read and update my file in KiXtart you could do it, but it would be pretty tricky. Especially that double. |
||||||||
|
|
|||||||
yes, it's tricky, but there is no easier way. if there is, please show me. |
||||||||
|
|
|||||||
Quote: Show me yours first From what I remember when I was playing with COM automation, you can determine the data type that is being passed, and convert where necessary. You should be able to return all the types, cast to longs, doubles and strings. It doesn't matter a whole lot that in the file it is an unsigned integer, and when it arrives in KiXtart it is a double. When you write "123.45" as a double in the file it doesn't matter that in KiXtart it is a string - you just need to handle the type and convert to the correct form. I don't know C++, so bear with me while I explain in C. If you have a function which is receiving a string and needs to write it out as a double there are a couple of ways. Here's one: Code: /* Create a simple output file */ When reading it is even easier. Load the variable into the correct type, then cast it to the type that you are going to return to KiXtart. The type you return just needs to be big enough to handle the full range of the original type. |
||||||||
|
|
|||||||
still can't really understand. fwrite() does right with int, so not sure how would I write double with it. so say I might get around that (which is tricky already), you want to be able to write data, then write data in different type and then write additional data (if that ain't tricky enough) just to be able to go around the forming of the proper value in kix. |
||||||||
|
|
|||||||
and back to the double case: Quote: now, isn't that the standard way? so, if one passes a numeric double (not array of chars) to variant and I use variants change type, I loose everything after decimal point. so, unless I start doing data type conversions and mangles and gangles inside the dll, pure double is no option, right? |
||||||||
|
|
|||||||
Quote: Not quite right. fwrite() takes a *void* pointer, and writes the memory that the pointer points to. I think I see your confusion. You need to understand how variables are allocated space and are referenced. It is one of the more complicated areas, but opens up a whole new world of programming. I'll give a short explanation, but you really need to get a good book. I still use my original K&R 'C' book - it is invaluable. Ok, here's how it works. Variables are simply contiguous memory locations. You can think of them in exactly the same way as the data in your file. Different types of data require different numbers of contiguous bytes. (I'm assuming 1 memory word=1 byte=8 bits for this explanation) char=1 byte unsigned char=1 byte short int=2 bytes long int=4 bytes double presicion float=8 bytes Note, different architectures will have differ requirements. Now normally you access the variable by it's name. This is in fact (resolved to) a pointer to the location in memory. The compiler knows how big the variable is, so knows how much memory to read or write when the variable is accessed. When your code says: Code: int i The first line declares a variable "i" which reserves 4 bytes of memory and sets "i" to point to that memory. The second line writes the value 10 in byte order to the memory location. Now the tricky bit. You can get the address of the variable by using the "&" operator. This means that you can access the memory *directly*. Here is a very simple example: Code: #include <stdio.h> The "sizeof" operator returns the size (in bytes) of the data type. The example code declares i as an int, then c as an array of chars, the same size as an int. We assign a value to the int, and then use memcpy() to copy the memory pointed to by i (&i) to that pointed to by c (&c). "sizeof" is used to determine how many bytes to copy. The result is a very short message: Quote: You can do exactly the same thing will all variable types. This means that you can use fread/fwrite to read and write any type of data without losing information. If I take one of my earlier examples: Code: void fnWriteStringAsDOUBLE(FILE *fp,char *s) Here, fwrite() is passed the *address* of the double (&d). It is also told how big the data is (sizeof d), and that we are writing one instance. This will cause the memory containing the value of "d" to be written to disk, in this case 8 bytes. fread() does exactly the same thing in reverse. BTW, where was the quote from? |
||||||||
|
|
|||||||
With your golfing hat on you can probably see that the "wow" example can be reduced to: Code: #include <stdio.h> |
||||||||
|
|
|||||||
ja ja. feel like being in school again these are the little things why I shouldn't have tropped it even though they forced me to study swedish good lesson on simplyfying things. so, to my next q. well, I need to think that over... should I remove the reading and writing of arrays and fall back to something. I know I can't write array and somehow recognice where it is when reading. and, I can't read multiple bytes of different types and give them to kixtart without array. eh, I need some time to mix this up in mind but if you have a opinion, say it. will take some time before I get back to drawing board. |
||||||||
|
|
|||||||
just thinking structs here... although kixtart does not support those, arrays of different vartypes kinda is like one. wonder, should there be switch for how the data should be aligned... and is that possible. |
||||||||
|
|
|||||||
Well, structs are lovely things, and are great for reading and writing fixed length records with mixed data types. However I think that they may be one step too far though. I suppose you could do something like: Code: $aRecordFormat= "i", ; Unique Pupil Number I would have thought at this point you would be looking at ODBC drivers |
||||||||
|
|
|||||||
haven't I told you I hate DB's? k, back to thinking. |