| Mart |
| (KiX Supporter) |
| 2008-01-03 11:43 AM |
|
Get file encoding (.txt or .csv files)
|
Guys,
Might be a simple one but I've been looking at this for some time now and cant' figure out how to do this.
Is there a way to get the encoding of a .txt or .csv file (UTF8, ANSI, etc...) with kix?
Background info:
We are replacing an old custom build app with a new one done in kix with a GUI in KF.NET.
Steps:
- Import a tab separated file, do some checks and math on each line and spit out a comma separated file.
- Import the csv file into the app used by the finance department.
- Print and send out the invoices.
All input and output files for both the old and new app are exactly the same when I open them in editplus or notepad but the output from the new app cannot be imported to create the invoices. It fails on +/- 30% of the lines. No clear error messages just a general error.
Did some extensive searching and fixed some bugs in the new app but still no luck. I'm desperately looking for things that could cause this. All tests up until now crashed and burned. There is one last thing that I can think of that might be the cause and that is file encoding. Both files might have different encoding types.
| Benny69 |
| (MM club member) |
| 2008-01-03 03:08 PM |
|
|
Re: Get file encoding (.txt or .csv files)
|
only a guess but the files being imported are to large of txt files (too many lines). Break them up into smaller files to see if they can be imported.
| Mart |
| (KiX Supporter) |
| 2008-01-03 04:13 PM |
|
|
|
Re: Get file encoding (.txt or .csv files)
|
That was my first guess but the import in and export from the kix app works great (somewhere around 100 lines). The errors are thrown by the application our financial department uses. The file being imported in this app works if it is created from the old app but not if it is created from my new KF.NET app. Both files are 100%, totally, definitely, absolutely, exactly the same so my guess is that the file encoding might be different and the file from my KF.NET app has the wrong encoding.
| DrillSergeant |
| (MM club member) |
| 2008-01-03 04:30 PM |
|
|
Re: Get file encoding (.txt or .csv files)
|
Don't you see any differences with a hex editor?
http://www.chmaas.handshake.de/delphi/freeware/xvi32/xvi32.htm
Richard H.
|
| (KiX Supporter) |
| 2008-01-03 04:59 PM |
|
Re: Get file encoding (.txt or .csv files)
|
If the files are simple text ("ASCII", not unicode) then they don't have an encoding as such - the encoding is how you are dealing with the characters rather than an attribute of the data stream.
When you say that the files are the same, are you doing a binary compare?
| Mart |
| (KiX Supporter) |
| 2008-01-03 05:10 PM |
|
|
|
Re: Get file encoding (.txt or .csv files)
|
Ok. Did some more testing, importing and exporting and it gives the same results no matter what kind of encoding is used. All give the same results
I think we'll be calling in a consultant from the manufacturer to come visit us and tell us what we are doing wrong. We have been at this for several days now and looked at all kinds of options one by one and they all give the exact same results.Maybe I'll get some kind of genius idea tonight. Who knows, let’s wait and see.
| Mart |
| (KiX Supporter) |
| 2008-01-03 05:15 PM |
|
|
|
Re: Get file encoding (.txt or .csv files)
|
Originally Posted By: Richard H.
If the files are simple text ("ASCII", not unicode) then they don't have an encoding as such - the encoding is how you are dealing with the characters rather than an attribute of the data stream.
When you say that the files are the same, are you doing a binary compare?
When you say that the files are the same, are you doing a binary compare?
Nope. I did not do a binary compare.
Going home now. I'll do some more digging around tommorow and keep you posted.
| Arend_ |
| (MM club member) |
| 2008-01-03 05:29 PM |
|
|
Re: Get file encoding (.txt or .csv files)
|
Mart, Could you show the pieces of code of both the old and the new app on how they create the output file and how the data is written to the output file ?
Other then that my guess would be the calculation is doing something wrong on some values and generates wrongfull output. maybe even hard spaces or something similar. Try having the new program spit out the input file only from tab delimiters to comma delimiters and see if that can be imported.
Also, to check if the output files from both the old and new app are the same, trying matching them line by line using FSO.
| Mart |
| (KiX Supporter) |
| 2008-01-03 07:33 PM |
|
|
|
Re: Get file encoding (.txt or .csv files)
|
Well the code from the old app is a humongous monster. It is a script in a filemaker DB. This script is more then 80 A4 sheets long and makes plenty use of goto kinda commands so it jumps from page 1 to 75 and back to 6 before going to 25 etc.... It's almost impossible to follow but we managed to figure out what it does exactly so we know how the output should look.
I did a line by line caparison (placed corresponding lines beneath each other in editplus) and found a few bugs in the new app. Fixed them but still no luck.
I'll FTP the stuff home and do a line by line check using FSO and not a text editor cause (in my experience) they sometimes do not show some things as they actually are.
| Mart |
| (KiX Supporter) |
| 2008-01-03 09:36 PM |
|
|
|
Re: Get file encoding (.txt or .csv files)
|
Ok now we're getting somewhere.
After all it looks like it had nothing to do with the encoding of the file. Just like Richard already posted. It all comes down to a limitation of the financial app we bought and have been using for several years now.
Got everything ftp-ed to my home computer. Ran some tests, found that the old app only takes two separate files for the two publication (one is a print version and the other an online version) we need to send invoices for. The new app had this all build into one. Much easier and faster that way. Did the same tests at work today but somehow missed the two files thing. Maybe the Chinese food I just picked up kixtarted my brain or something. I ripped apart the two publications and processed them one at the time with the old and new app. Both apps spit out an output that was 100% exactly the same. Did several checks line by line and they all told me each line was the same in both files. I imported the output from the new app and the errors are gone. It seems like the financial app does not like two different publications at the same time. It would be nice if some kind of message told me so instead of saying than an unknown method of an unknown object encountered unknown errors. Even the manufacturer's support desk people couldn't help me.
I guess I learned something today. God d#mn those freaky error messages!
Ok tomorrow it's back to the drawing board to rewrite parts of the app I modified with some workarounds and quick and dirty ways of commenting and adding extra lines to split the two publications and see if it will work as it should after the rewrite.
| Mart |
| (KiX Supporter) |
| 2008-01-08 10:24 PM |
|
|
|
Re: Get file encoding (.txt or .csv files)
|
Ok, got stuff going tonight.
Did a big rewrite of some parts to remove the workarounds and quick fixes. I had to slap in some major extra if and select statements but also rip out some other stuff so the scripts did not grow that much. Somehow this has increased the speed a lot. It will even take both (or more if there are more) publications at once. Have to say that it was not just kix (actually me telling kix to do stuff) screwing things up by using wrong logic that seemed ok but we also had to do some changes to the financial app and it's SQL DB and the app that creates the export to be processed.
All is working fine now.