Page 1 of 1 1
Topic Options
#184385 - 2008-01-03 11:43 AM Get file encoding (.txt or .csv files)
Mart Moderator Offline
KiX Supporter
*****

Registered: 2002-03-27
Posts: 4672
Loc: The Netherlands
Guys,

Might be a simple one but I've been looking at this for some time now and cant' figure out how to do this.

Is there a way to get the encoding of a .txt or .csv file (UTF8, ANSI, etc...) with kix?

Background info:
We are replacing an old custom build app with a new one done in kix with a GUI in KF.NET.
Steps:
- Import a tab separated file, do some checks and math on each line and spit out a comma separated file.
- Import the csv file into the app used by the finance department.
- Print and send out the invoices.

All input and output files for both the old and new app are exactly the same when I open them in editplus or notepad but the output from the new app cannot be imported to create the invoices. It fails on +/- 30% of the lines. No clear error messages just a general error.

Did some extensive searching and fixed some bugs in the new app but still no luck. I'm desperately looking for things that could cause this. All tests up until now crashed and burned. There is one last thing that I can think of that might be the cause and that is file encoding. Both files might have different encoding types.
_________________________
Mart

- Chuck Norris once sold ebay to ebay on ebay.

Top
#184390 - 2008-01-03 03:08 PM Re: Get file encoding (.txt or .csv files) [Re: Mart]
Benny69 Offline
Moderator
*****

Registered: 2003-10-29
Posts: 1036
Loc: Lincoln, Ne
only a guess but the files being imported are to large of txt files (too many lines). Break them up into smaller files to see if they can be imported.
_________________________
Wait don't order yet,... get KiXforms Designer .NET 2.0 (Beta)
KiXforms Designer .NET 2.0 (Beta)

Top
#184392 - 2008-01-03 04:13 PM Re: Get file encoding (.txt or .csv files) [Re: Benny69]
Mart Moderator Offline
KiX Supporter
*****

Registered: 2002-03-27
Posts: 4672
Loc: The Netherlands
That was my first guess but the import in and export from the kix app works great (somewhere around 100 lines). The errors are thrown by the application our financial department uses. The file being imported in this app works if it is created from the old app but not if it is created from my new KF.NET app. Both files are 100%, totally, definitely, absolutely, exactly the same so my guess is that the file encoding might be different and the file from my KF.NET app has the wrong encoding.

Edited by Mart (2008-01-03 04:17 PM)
Edit Reason: typo
_________________________
Mart

- Chuck Norris once sold ebay to ebay on ebay.

Top
#184393 - 2008-01-03 04:30 PM Re: Get file encoding (.txt or .csv files) [Re: Mart]
DrillSergeant Offline
MM club member
*****

Registered: 2004-07-09
Posts: 1164
Loc: Eijsden, the Netherlands
Don't you see any differences with a hex editor?

http://www.chmaas.handshake.de/delphi/freeware/xvi32/xvi32.htm
_________________________
The Code is out there

Top
#184394 - 2008-01-03 04:59 PM Re: Get file encoding (.txt or .csv files) [Re: Mart]
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
If the files are simple text ("ASCII", not unicode) then they don't have an encoding as such - the encoding is how you are dealing with the characters rather than an attribute of the data stream.

When you say that the files are the same, are you doing a binary compare?

Top
#184396 - 2008-01-03 05:10 PM Re: Get file encoding (.txt or .csv files) [Re: DrillSergeant]
Mart Moderator Offline
KiX Supporter
*****

Registered: 2002-03-27
Posts: 4672
Loc: The Netherlands
Ok. Did some more testing, importing and exporting and it gives the same results no matter what kind of encoding is used. All give the same results I think we'll be calling in a consultant from the manufacturer to come visit us and tell us what we are doing wrong. We have been at this for several days now and looked at all kinds of options one by one and they all give the exact same results.

Maybe I'll get some kind of genius idea tonight. Who knows, let’s wait and see.
_________________________
Mart

- Chuck Norris once sold ebay to ebay on ebay.

Top
#184397 - 2008-01-03 05:15 PM Re: Get file encoding (.txt or .csv files) [Re: Richard H.]
Mart Moderator Offline
KiX Supporter
*****

Registered: 2002-03-27
Posts: 4672
Loc: The Netherlands
 Originally Posted By: Richard H.
If the files are simple text ("ASCII", not unicode) then they don't have an encoding as such - the encoding is how you are dealing with the characters rather than an attribute of the data stream.

When you say that the files are the same, are you doing a binary compare?


Nope. I did not do a binary compare.
Going home now. I'll do some more digging around tommorow and keep you posted.
_________________________
Mart

- Chuck Norris once sold ebay to ebay on ebay.

Top
#184399 - 2008-01-03 05:29 PM Re: Get file encoding (.txt or .csv files) [Re: Mart]
Arend_ Moderator Offline
MM club member
*****

Registered: 2005-01-17
Posts: 1894
Loc: Hilversum, The Netherlands
Mart, Could you show the pieces of code of both the old and the new app on how they create the output file and how the data is written to the output file ?
Other then that my guess would be the calculation is doing something wrong on some values and generates wrongfull output. maybe even hard spaces or something similar. Try having the new program spit out the input file only from tab delimiters to comma delimiters and see if that can be imported.

Also, to check if the output files from both the old and new app are the same, trying matching them line by line using FSO.

Top
#184401 - 2008-01-03 07:33 PM Re: Get file encoding (.txt or .csv files) [Re: Arend_]
Mart Moderator Offline
KiX Supporter
*****

Registered: 2002-03-27
Posts: 4672
Loc: The Netherlands
Well the code from the old app is a humongous monster. It is a script in a filemaker DB. This script is more then 80 A4 sheets long and makes plenty use of goto kinda commands so it jumps from page 1 to 75 and back to 6 before going to 25 etc.... It's almost impossible to follow but we managed to figure out what it does exactly so we know how the output should look.

I did a line by line caparison (placed corresponding lines beneath each other in editplus) and found a few bugs in the new app. Fixed them but still no luck.

I'll FTP the stuff home and do a line by line check using FSO and not a text editor cause (in my experience) they sometimes do not show some things as they actually are.
_________________________
Mart

- Chuck Norris once sold ebay to ebay on ebay.

Top
#184406 - 2008-01-03 09:36 PM Re: Get file encoding (.txt or .csv files) [Re: Mart]
Mart Moderator Offline
KiX Supporter
*****

Registered: 2002-03-27
Posts: 4672
Loc: The Netherlands
Ok now we're getting somewhere.

After all it looks like it had nothing to do with the encoding of the file. Just like Richard already posted. It all comes down to a limitation of the financial app we bought and have been using for several years now.

Got everything ftp-ed to my home computer. Ran some tests, found that the old app only takes two separate files for the two publication (one is a print version and the other an online version) we need to send invoices for. The new app had this all build into one. Much easier and faster that way. Did the same tests at work today but somehow missed the two files thing. Maybe the Chinese food I just picked up kixtarted my brain or something. I ripped apart the two publications and processed them one at the time with the old and new app. Both apps spit out an output that was 100% exactly the same. Did several checks line by line and they all told me each line was the same in both files. I imported the output from the new app and the errors are gone. It seems like the financial app does not like two different publications at the same time. It would be nice if some kind of message told me so instead of saying than an unknown method of an unknown object encountered unknown errors. Even the manufacturer's support desk people couldn't help me.

I guess I learned something today. God d#mn those freaky error messages!

Ok tomorrow it's back to the drawing board to rewrite parts of the app I modified with some workarounds and quick and dirty ways of commenting and adding extra lines to split the two publications and see if it will work as it should after the rewrite.
_________________________
Mart

- Chuck Norris once sold ebay to ebay on ebay.

Top
#184471 - 2008-01-08 10:24 PM Re: Get file encoding (.txt or .csv files) [Re: Mart]
Mart Moderator Offline
KiX Supporter
*****

Registered: 2002-03-27
Posts: 4672
Loc: The Netherlands
Ok, got stuff going tonight.
Did a big rewrite of some parts to remove the workarounds and quick fixes. I had to slap in some major extra if and select statements but also rip out some other stuff so the scripts did not grow that much. Somehow this has increased the speed a lot. It will even take both (or more if there are more) publications at once. Have to say that it was not just kix (actually me telling kix to do stuff) screwing things up by using wrong logic that seemed ok but we also had to do some changes to the financial app and it's SQL DB and the app that creates the export to be processed.

All is working fine now.
_________________________
Mart

- Chuck Norris once sold ebay to ebay on ebay.

Top
Page 1 of 1 1


Moderator:  Jochen, Allen, Radimus, Glenn Barnas, ShaneEP, Ruud van Velsen, Arend_, Mart 
Hop to:
Shout Box

Who's Online
0 registered and 507 anonymous users online.
Newest Members
gespanntleuchten, DaveatAdvanced, Paulo_Alves, UsTaaa, xxJJxx
17864 Registered Users

Generated in 0.062 seconds in which 0.024 seconds were spent on a total of 13 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org