iffy
(Starting to like KiXtart)
2006-01-13 07:09 PM
bug (?) when passing variable on the commandline

Consider the following:

Code:

test.KiX contains just a single line of code
? $var

A few 'tests' give the following results:

D:\>KIX32.EXE test.KiX $var="abc"
abc

D:\>KIX32.EXE test.KiX $var="a b c"
a b c

D:\>KIX32.EXE test.KiX $var="\abc"
\abc

D:\>KIX32.EXE test.KiX $var="\\abc"
\\abc

D:\>KIX32.EXE test.KiX $var="a\b\c"
a\b\c

D:\>KIX32.EXE test.KiX $var="abc\"
abc"

D:\>KIX32.EXE test.KiX $var="abc\\"
abc\

D:\>KIX32.EXE test.KiX $var="abc\\\"
abc\"

D:\>KIX32.EXE test.KiX $var="abc\\\\"
abc\\

D:\>KIX32.EXE test.KiX $var="\abc\\\\"
\abc\\

D:\>KIX32.EXE test.KiX $var="\\abc\\\\"
\\abc\\

D:\>KIX32.EXE test.KiX $var="\\\a\b\c\\\\"
\\\a\b\c\\



Apperently an odd number of trailing backslashes produces incorrect results.

Btw.. tested with v4.50 on WinXP/SP2 and W2k3/SP1


LonkeroAdministrator
(KiX Master Guru)
2006-01-13 07:22 PM
*not a bug*

instead of commandline, use startmenu run.

see the difference?

this is normal dos thing.
backslash is a escape character and thus the results are obvious, right?


iffy
(Starting to like KiXtart)
2006-01-13 07:26 PM
Re: *not a bug*

Read more carefully... First it only does it with training backslashes, not for backslashes in front or in the middle as my examples clearly show. Secondly why do I also get a double quote?

Les
(KiX Master)
2006-01-13 07:44 PM
Re: *not a bug*

On the commandline, the hack \ has special meaning. It is the *escape* character.

iffy
(Starting to like KiXtart)
2006-01-13 07:52 PM
Re: *not a bug*

Hmm.. interesting, then why have I been using a carot for years as the escape character in DOS? Try echo \> and echo ^>
Again, escape character or not, the behaviour is not consistent and therefor a bug. I'd like Ruud's thoughts on this one.


LonkeroAdministrator
(KiX Master Guru)
2006-01-13 08:07 PM
Re: *not a bug*

echo is proven to be non-standard among the commandline tools.
anyway using double backslashes donates that there should be a backslash.
using 1,3,5 or so on, escapes the next character.
that's ok behavior in my mind.

the issue I do see here is that the leading backslashes are not escaped.

another issue I do see that I can't reproduce your issue
just tried out and my Wkix32.exe 4.52 beta2 does not do this.
nor did 4.51


LonkeroAdministrator
(KiX Master Guru)
2006-01-13 08:18 PM
Re: bug (?) when passing variable on the commandline

ok, after some testing I can confirm that the behavior of kix32.exe and wkix32.exe do differ.

Les
(KiX Master)
2006-01-13 08:30 PM
Re: *not a bug*

Quote:

Try echo \> and echo ^>



As Jooel said, ECHO is not the same as commandline. Shawn will attest to that as we recently discussed it.


LonkeroAdministrator
(KiX Master Guru)
2006-01-13 08:44 PM
Re: *not a bug*

k, I made a simple test app with c++

and by posthing this, I even made it "open source"
Code:

int _tmain(int argc, _TCHAR* argv[])
{
for (int i = 0;i<argc;i++) {
int o = printf("%d -> %s\n",i, argv[i]);
}
return 0;
}



you can get it from HERE

with this, it's proven that it's not a kixtart issue.


Les
(KiX Master)
2006-01-13 08:50 PM
Re: *not a bug*

Here is Shawn's topic:
http://www.kixtart.org/ubbthreads/showflat.php?Cat=0&Number=153766


iffy
(Starting to like KiXtart)
2006-01-13 09:08 PM
Re: *not a bug*

The issue is not just leading backslashes, also backslashes in the middle are not escaped. Or the other way around, only a trailing backslash is escaped.

Another example

Code:

D:\>KIX32.EXE test.KiX $var="\\aa\\bb\\cc\\"
\\aa\\bb\\cc\ <- only the trailing backslash is escaped, I would have expected either \\aa\\bb\\cc\\ or \aa\bb\cc\ as result

D:\>KIX32.EXE test.KiX $var="\aa\bb\cc\"
\aa\bb\cc" <- only the trailing backslash is escaped



LonkeroAdministrator
(KiX Master Guru)
2006-01-13 09:40 PM
Re: *not a bug*

iffy, try out that exe.
it outputs what you give it as arguments to the console.
and my exe has nothing to do with kixtart.
but is build with the normal exe headers but otherwise is just that main I posted.


iffy
(Starting to like KiXtart)
2006-01-13 10:50 PM
Re: *not a bug*

Ok, tried it out, the odd behaviour is apparently a "feature" of the C++ compiler of VS2003. Very curious how this is processed with a Watson compiler or on a linux box. I haven't done C or C++ since the early 90's so I've no easy test environment myself. Personally I consider this a bug of kixtart, oddities of a compiler should not be imposed on the user, sort of like your tagline Jooel But on former occasions I've learned that the board has some strict view about what is a kixtart or exterior problem so I guess I'll have to live with it.

Edit: found the ancient Borland C v3.1 on my system... not to my surprise it produces the imho correct expected results, that is the trailing backslash in NOT escaped. <--- not true.. see edit2 below

Edit2: I stand corrected, forgot to doublequote the arguments, BC3 behaves the same, guess it's a printf oddity then. Also tried VB6, vbscript and a DOS batch file, they do all produce the 'correct' result.


ShawnAdministrator
(KiX Supporter)
2006-01-13 11:00 PM
Re: *not a bug*

I just compiled and ran Jooel's code under Visual Studio 6 ... same results.

AllenAdministrator
(KiX Supporter)
2006-01-14 12:26 AM
Re: *not a bug*

I confirmed Iffy results on a .bat and .vbs file returning the "expected" results. While this is may not be a bug in kixtart, I can certainly relate with Iffy that this would be unexpected behavior.

LonkeroAdministrator
(KiX Master Guru)
2006-01-14 12:37 AM
Re: *not a bug*

allen, how you can confirm this with bat?
ofcourse same question goes for anyone saying they can get "correct" result with bat.
remember, echo is not the answer here.

hmm...
set neither changes it.
and that's about it.
not sure how would more test it.
I shall try changing my exe a bit...


AllenAdministrator
(KiX Supporter)
2006-01-14 01:03 AM
Re: *not a bug*

I used echo and set as tests.

LonkeroAdministrator
(KiX Master Guru)
2006-01-14 01:07 AM
Re: *not a bug*

oh, and if someone thinks printf() has an issue, here is a highschool version of the test code:
Code:

#include <iostream>
using namespace std;

int _tmain(int argc, _TCHAR* argv[])
{
for (int i = 0;i<argc;i++) {
cout << i << " -> " << argv[i] << "\n";
}
return 0;
}



and the results are the same.
the last backslashes are escaped.

also, if I place some backslashes before the first quote (right before the arg that is), they also are escaped.

so, again, my VS has nothing to do with kix (except that kixtart was probably build with VS too) and still my programs have the same "issue"
that is, the issue is not kixtart related.


iffy
(Starting to like KiXtart)
2006-01-14 10:16 AM
Re: *not a bug*

My only point is that it is unexpected behaviour and I had to add extra code in my KiX script to fix the input.

Off topic rant & rave:
As a user I don't really care if C/C++ has imho weird behaviour, I want simple consistent results. As I said before, why impose this on the end user of a product, in this case me. Why do I need to know and keep in mind in what language my compiler/interpreter was written? As a developer it was interesting to dig into it a little tho, understanding the 'deeper' cause makes for better code in the end... But all in all, I still wouldn't mind if the input was fixed and that KiX would produce the 'expected' results.


LonkeroAdministrator
(KiX Master Guru)
2006-01-14 11:08 AM
Re: *not a bug*

yes.
I hear you.
the point I was trying to make here is that fixing this behaviour isn't that easy.

anyway, until ruud comments on this, you can always use wkix32.exe as workaround.


Les
(KiX Master)
2006-01-14 02:53 PM
Re: *not a bug*

Quote:

My only point is that it is unexpected behaviour...



Then what do you make of the weird behaviour when you are trying to pass an arguement that starts with a number? I had to prefix the var (used # in my case) and then strip it off after.
Code:
  If Left($Recipient,1) = "#"
$Recipient = Right($Recipient,Len($Recipient)-1)
endif



LonkeroAdministrator
(KiX Master Guru)
2006-01-14 05:03 PM
a bug?

oh...
was talking to a friend and decided to test once more and was able to produce a code that does not escape the characters:
Code:

#include "windows.h"
#include <iostream>
int main() {
printf("%s",GetCommandLine());
}



so, using ANSI argument parsing does introduce the problem.
or using the cout from std.

so, I'm starting to think this might be called a bug after all.


ShawnAdministrator
(KiX Supporter)
2006-01-14 05:27 PM
Re: a bug?

a bug where ?

LonkeroAdministrator
(KiX Master Guru)
2006-01-14 05:57 PM
Re: a bug?

a bug in handling of the input argument.

ShawnAdministrator
(KiX Supporter)
2006-01-14 06:16 PM
Re: a bug?

in where ? Windows ?

LonkeroAdministrator
(KiX Master Guru)
2006-01-14 10:25 PM
Re: a bug?

kixtart.

ShawnAdministrator
(KiX Supporter)
2006-01-14 11:15 PM
Re: a bug?

sorry you lost me. Did kix32 process the command-line any different than any other console app ?

-Shawn


LonkeroAdministrator
(KiX Master Guru)
2006-01-15 12:02 AM
Re: a bug?

that does not matter.
and yes it did.
anyway, it don't matter.

kix32.exe is said to function in some manner and when that does not apply, it's a bug.
also, as wkix32.exe and my last code with printf() and GetCommandLine() show, the correct working is quite doable.
again, that does not matter.
even if it wasn't, it should be documented.
and as it is not, it qualifies as a bug.

imho.


Ruud van Velsen
(Hey THIS is FUN)
2006-01-17 05:39 PM
Re: *not a bug*

Wow, excellent test/debug work there.

The behaviour you have found is actually by design of the C/C++ commandline argument handler. Details on this (including examples) can be found in:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccelng/htm/progs_12.asp

Hope this clarifies the issue.

Regards,

Ruud


Richard H.Administrator
(KiX Supporter)
2006-01-18 10:03 AM
Re: *not a bug*

So to clarify, is there no way you will be able to fix this because it is a feature of the startup code?

FYI I get the same problem compiling with a port of the GCC compiler, possibly it's using the same libraries for the start-up code.


LonkeroAdministrator
(KiX Master Guru)
2006-01-18 10:52 AM
Re: *not a bug*

richie, there is a way.
using different input method removes the problem.


Ruud van Velsen
(Hey THIS is FUN)
2006-01-18 12:48 PM
Re: *not a bug*

If I were to implement a new macro called @COMMANDLINE, which would return the entire, un-processed, commandline would that help at all?

This request is on the todo-list anyway, so if this resolves this particular issue as well, I'll be glad to try and get this into 4.52.

Regards,

Ruud


Richard H.Administrator
(KiX Supporter)
2006-01-18 01:00 PM
Re: *not a bug*

If you can pull in the command line in an unmolested format, can you not then parse it in a more sensible way than the startup code?

I don't think it would help much to have to use two different methods to get command line variable assignments.

If creating a bespoke argument parser is going to cause a lot of work or delay I guess that in the short term we could probably create a "command line parser" UDF to do the job - at least we could be sure of the accuracy of the data.


AllenAdministrator
(KiX Supporter)
2006-01-18 01:09 PM
Re: *not a bug*

If it's not a lot of trouble to add, I give a vote for the macro.

LonkeroAdministrator
(KiX Master Guru)
2006-01-18 03:06 PM
Re: *not a bug*

just like richie said, it does not remove the problem.

after adding the macro, would it be possible to use extra arguments on commandline?
iirc, that was the reason for it. so one could extend the kix commandline have other parameter switches like all the exe's have...
oh, and ofcourse the path to kix.exe


iffy
(Starting to like KiXtart)
2006-01-18 08:46 PM
Re: *not a bug*

Are there more known 'irregularities' than the one that started this topic?

If not, isn't 'correcting' just this one 'odd' behaviour relatively simple without introducing new stuff?

If there are more issues than the trailing escape character issue than I'd go for leaving the current commandline parsing as it is and introduce the new @commandline macro as suggested by Ruud.

PS.. just curious: is the problem in C/C++ in printf/cout or in argv?


LonkeroAdministrator
(KiX Master Guru)
2006-01-18 10:31 PM
Re: *not a bug*

iffy, it seems to be a problem of cout as well as argv.

Richard H.Administrator
(KiX Supporter)
2006-01-19 10:03 AM
Re: *not a bug*

Quote:

Are there more known 'irregularities' than the one that started this topic?




Yup, check the link that Ruud provided to see what the problem is. Basically, before your program starts there is a bit of startup/setup code that is run. It does a number of things, one of which is to present argc / argv / envp as parameters for Main() (at least in the C world).

MS has changed command line handling a few times, and each time it has veered away from doing anything you would consider sensible. Obviously they think programmers appreciate being able to work in idiosyncratic and inconsistent environments.

I'd be interested if anyone can think of a good reason for this behaviour.

Quote:

If not, isn't 'correcting' just this one 'odd' behaviour relatively simple without introducing new stuff?




No, it's probably not that simple. As the startup code is executed before you get you hands on it in the program there is no way to affect it's action. Because you cannot see the original data you cannot determine if it's been screwed about.

The best you can do is to write your own command line parser and parse the original command line. It's not a particularly complicated bit of code to write, but it is new code which is not already in the KiXtart interpreter.

Quote:

If there are more issues than the trailing escape character issue than I'd go for leaving the current commandline parsing as it is and introduce the new @commandline macro as suggested by Ruud.




The @COMMANDLINE macro is going to be there anyway. The problem is that if you don't fix this anomoly in some way you are going to be in the position of having to say that sometimes assigning variable values on the command line will work in KiXtart and sometimes it won't, depending on the data. That's not a good place to be.

Quote:

PS.. just curious: is the problem in C/C++ in printf/cout or in argv?




This issue specifically is in the handling of the command line arguments. The handling of strings by library functions like printf() is standardised and is not an issue (unless they are badly broken ).


LonkeroAdministrator
(KiX Master Guru)
2006-01-19 11:16 AM
Re: *not a bug*

again.
fixing this issue is rather simple.
instead of using the args parsed for you, parse them yourself from the raw commandline.

and richie, not sure I understand your flaming of MS here.
you just posted a while ago that same issue is there with gcc.
last time I checked, it's not MS compiler.


Richard H.Administrator
(KiX Supporter)
2006-01-19 01:45 PM
Re: *not a bug*

Quote:

again.
fixing this issue is rather simple.
instead of using the args parsed for you, parse them yourself from the raw commandline.




It is a work-around, but I don't think that it's a good long term solution to say 'using $var="VALUE" on the command line doesn't always work as you'd expect so you're better off parsing the line yourself'. Much better to fix '$var="VALUE"' so that it always works as expected.

Quote:

and richie, not sure I understand your flaming of MS here.



Hah! Having had to put up with doubling speech marks, using ^ as an escape and various other unspeakable and changing kludges over the years, it's no surprise that the goal-posts have been moved again. I appreciate that some of this nastiness is down to the command line interpreter.

Quote:

you just posted a while ago that same issue is there with gcc.
last time I checked, it's not MS compiler.



Correct. However it will of course use standard Windows code and libraries including start-up code, and that is where the problem lies.


LonkeroAdministrator
(KiX Master Guru)
2006-01-19 02:12 PM
Re: *not a bug*

Quote:


It is a work-around, but I don't think that it's a good long term solution to say 'using $var="VALUE" on the command line doesn't always work as you'd expect so you're better off parsing the line yourself'. Much better to fix '$var="VALUE"' so that it always works as expected.





again, you are not reading me.
Ruud can easily fix this inbuild if he skips using argv and starts using getCommandLine();
he doesn't have to do anyhing more but to add a strtok(); (or what the c-split was...)

when he gets to using the getCommandLine, he could also add the whole string in to the macro he is promising.
but lets not mistake that I ever said user should parse the commandline.
it was ruud and some other folks.


ShawnAdministrator
(KiX Supporter)
2006-01-19 02:24 PM
Re: *not a bug*

So if Ruud "fixes" this issue (i still maintain that nothing is broke) ... how many scripts you figure might break because of it ?

LonkeroAdministrator
(KiX Master Guru)
2006-01-19 02:48 PM
Re: *not a bug*

none.
btw, searched for some time and found:
http://msdn.microsoft.com/library/en-us/...e_Arguments.asp

so, like I remembered, you can even disable the flawed argument parsing routines thus saving space.
didn't try though how this does affect the working ones.


Richard H.Administrator
(KiX Supporter)
2006-01-19 03:08 PM
Re: *not a bug*

Quote:

but lets not mistake that I ever said user should parse the commandline.
it was ruud and some other folks.



Ahhh. You said "instead of using the args parsed for you, parse them yourself from the raw commandline" and I misunderstood.

Quote:

how many scripts you figure might break because of it ?




Agree with Jooel - it shouldn't break anything, unless someone is relying on "\\" at the end of an argument being reduced to "\".


ShawnAdministrator
(KiX Supporter)
2006-01-19 03:29 PM
Re: *not a bug*

my two cents - I don't think Ruud should be changing the documented behavior of the Windows runtime library.

iffy
(Starting to like KiXtart)
2006-01-19 08:02 PM
Re: *not a bug*

Well, personally I think the average KiX programmer could care less if it's documented behavior or not, or if the problem is in the programming language or the OS.

I think that the average KiX programmer wants simple consistent ways of working with KiX and not having to think and be aware of, to quote Richard, idiosyncratic and inconsistent environments.

I still vote for fixing the commandline parsing of KiX but if not please let's document this odd behaviour very clearly with one or more good examples.

Slightly offtopic rant follows...
I'm still curious to know of other specific examples where this issue if manifested. If you try dir "c:\config.sys\" for example it is not interpreted as dir "c:\config.sys", clearly MS 'fixed' the issue themselves. I spend hours every day doing system administration from the commandline, either with MS or 3th party tools in a variety of scripting languages but I've never in all these years come across this. I did learn to live with the annoying double quote thingy, the carot I don't see as an oddity, it just is the defacto escape character on the (nt) commandline.


ShawnAdministrator
(KiX Supporter)
2006-01-19 08:21 PM
Re: *not a bug*

Quote:


I still vote for fixing the commandline parsing of KiX but if not please let's document this odd behaviour very clearly with one or more good examples.





Can't argue with that, no way and I agree. Should be documented. I've gone through hell trying to sort this stuff out too. The big reason I'm suggesting (and this is my two cents) that it not be fixed is because you know how these "lets fix something at our end that is broke at their end" things go ... what happens is that they fix things at their end then your end breaks.


LonkeroAdministrator
(KiX Master Guru)
2006-01-19 09:50 PM
Re: *not a bug*

eh?
whose end?
kixtart is single developer thing.
if he fixes it, there is no other end to screw it up.


ShawnAdministrator
(KiX Supporter)
2006-01-19 10:04 PM
Re: *not a bug*

So what are you suggesting ? That Kixtart use GetCommandLine() to get the raw command line, then parse it using new rules that aren't so odd ?

LonkeroAdministrator
(KiX Master Guru)
2006-01-19 10:16 PM
Re: *not a bug*

new rules?
parse it with the old rules.
just like the manual dictates.
that's not so hard.
see wkix32.exe, it already does work properly.
old rules, fixed functioning.


Les
(KiX Master)
2006-01-19 10:22 PM
Re: *not a bug*

Leave KiX32.exe the way it is and use WKiX32.exe for the new school way and nothing breaks.
Why can't you just let it go and get on with your life?


LonkeroAdministrator
(KiX Master Guru)
2006-01-19 10:34 PM
Re: *not a bug*

new school?
again, the syntax of kixtart commandline has been same for looooong time.
some switches have been added but otherwise, it has remained the same.

so, what does "new school" mean?
are you saying that if someone finds a bug long time after software is published, it should not be fixed in future versions?


ShawnAdministrator
(KiX Supporter)
2006-01-19 10:41 PM
Re: *not a bug*

I'm taking your advise Les and letting it go. Suggest you do same.

Chris S.
(MM club member)
2006-01-20 02:19 AM
Re: *not a bug*

I don't know, I kinda have to take sides with Lonk on this one. If it behaves one way with KiX32.exe it should behave exactly the same way with WKiX32.exe. Otherwise you'll have to have two scripts (one for WKiX and one for KiX) to properly parse the command line.

Les
(KiX Master)
2006-01-20 03:16 PM
Re: *not a bug*

I have only one word to toss in here and then will take my own advice as suggested by Shawn.

LEGACY


Ruud van Velsen
(Hey THIS is FUN)
2006-04-17 12:07 PM
Re: *not a bug*

RC1 will have the new @COMMANDLINE macro. I hope that resolves most issues without breaking any of the existing scripts.

Ruud