ciper
(Fresh Scripter)
2004-11-04 03:22 AM
Detokenize as an option in new version

When using kix32 with the /t argument to create a tokenized script an additional argument should be available to enable/disable extraction of the original script.

In other words, make it possible to create a file that cannot be extracted if thats needed (to hide contents) or a file that can be extracted at a later time (for faster/smaller scripts without a concern for hiding content)



Richard H.Administrator
(KiX Supporter)
2004-11-04 10:06 AM
Re: Detokenize as an option in new version

Tokenisation is a one-way process. More than simply replacing KiXtart keyword with tokens it compiles the code to KiXtart runtime byte-code. This will include any optimisations to the code structure that can be made at parse-time.

A stand-alone exe could reverse engineer the tokenised file and produce a working script, but it would not be the original script.

I'm struggling to think of an application - what would you use a reverse engineering facility for?


JochenAdministrator
(KiX Supporter)
2004-11-04 11:46 AM
Re: Detokenize as an option in new version

Most probably the need to apply version changes after tokenization (to the tokenized script) ... because one accidentally deleted the source scripts... been there once, but luckily found one copy on one of my other boxes

Richard H.Administrator
(KiX Supporter)
2004-11-04 12:22 PM
Re: Detokenize as an option in new version

That's not a good argument for reverse engineering. It's a good argument for proper source control or data archival.

JochenAdministrator
(KiX Supporter)
2004-11-04 02:11 PM
Re: Detokenize as an option in new version

Sure ... didn't meant that as an argument pro reverse engineering. Just tried to think out loud on possible reasons.
IMO this option should never make it to the release. Sorry if you misunderstood


ShawnAdministrator
(KiX Supporter)
2004-11-04 02:12 PM
Re: Detokenize as an option in new version

Richard/Cappy ... you guys got any comments to add on this thread, whats your views on "the size of kix32.exe" ?

http://www.kixtart.org/ubbthreads/showflat.php?Cat=&Board=UBB4&Number=128013



JochenAdministrator
(KiX Supporter)
2004-11-04 02:17 PM
Re: Detokenize as an option in new version

Size in real world doesn't matter at all, it's all about whatcha do with it.
Go back to school if you disagree


Richard H.Administrator
(KiX Supporter)
2004-11-04 02:36 PM
Re: Detokenize as an option in new version

Quote:

Sure ... didn't meant that as an argument pro reverse engineering. Just tried to think out loud on possible reasons.
IMO this option should never make it to the release. Sorry if you misunderstood




Not at all. I understand entirely. It was the only thing I could think of as well and (as we agree) it's a misuse.

I was just waiting for someone to suggest it so I could jump on it


ciper
(Fresh Scripter)
2004-11-04 10:25 PM
Re: Detokenize as an option in new version

It would help when you need to sit at the machine not working properly and you want to debug the script. If I want to run the script manually and step through the commands using the debugger how will this work with a tokenized file?

Sounds like many of you suggest using an archived copy (which should be the same as on the system but you cant be entirely sure) and running it on the machine in detokenized format. That means either the script must be modified to run from the users system (instead of the shared location it was originally on) or I have to store the detokenized version on the central location and point the machine to it. I wont have access to the same resources at a users desk as I would at my workstation so I have to go back to retrieve this archived copy.

What if the problem with the machine was that he didnt get the latest version of the script. Preventing me from detokenizing and comparing agaisnt the standard copy makes troubleshooting a little harder. I could do a CRC check against the two to verify this but this again takes another application I may not have access to at a users desk.

All of this adds up to more time troubleshooting. If I have access to the tokenized file and a kix executable I should be able to extract then debug the script at the location.

I do have a use for a script that is unreadable. Engineers love to circumvent our standard software installations, virus scanners for example. They will reverse engineer the method to detect virus scan installation and create dummy positives. If I could hide this script from them it would be very hard. This data would be stored in a second script with the "do not allow detokenize" enabled.


Sealeopard
(KiX Master)
2004-11-06 12:25 AM
Re: Detokenize as an option in new version

It's all about trade-offs. If you expect the need for troubleshooting your scripts then using the tokenizer might not be a good option. As Newton already discovered: "For every convenience there's also an inconvenience".

Richard H.Administrator
(KiX Supporter)
2004-11-08 10:11 AM
Re: Detokenize as an option in new version

Quote:

It would help when you need to sit at the machine not working properly and you want to debug the script. If I want to run the script manually and step through the commands using the debugger how will this work with a tokenized file?




Not very well

Quote:

Sounds like many of you suggest using an archived copy (which should be the same as on the system but you cant be entirely sure) and running it on the machine in detokenized format. That means either the script must be modified to run from the users system (instead of the shared location it was originally on) or I have to store the detokenized version on the central location and point the machine to it. I wont have access to the same resources at a users desk as I would at my workstation so I have to go back to retrieve this archived copy.




I don't understand this at all. You say you are going to run the original script manually - what does it matter where you get it from? You surely aren't suggesting that you want to leave a "detokenised" copy unprotected on the netlogon share? There is of course no reason not to have a restricted folder on the netlogon share which contains the original script so it is easy for you to retrieve. You will update the two in parallel when you release a new version.

Quote:

What if the problem with the machine was that he didnt get the latest version of the script. Preventing me from detokenizing and comparing agaisnt the standard copy makes troubleshooting a little harder.




Not at all. Whenever I make an amendment to my login script I update the version number and the last changed date. The script has an internal record of all changes. When a user logs on a hidden log file is written to their home directory containing the version number of all scripts, where they log in from, which logon server they are authenticating against, all the process stages that occur and and errors. An error causes the log to be mailed to me.

Identifying the script that has been run is therefore trivial.

If you don't want to go to this much effort, add a handler which on seeing the $VERSION flag on the command line just dumps the version info and exits.

Quote:

I could do a CRC check against the two to verify this but this again takes another application I may not have access to at a users desk.




Simple command line utilities like this do not require installation and can be executed from any available share.

Quote:

All of this adds up to more time troubleshooting. If I have access to the tokenized file and a kix executable I should be able to extract then debug the script at the location.



You have access to the original file. Running it at the user location is no more time consuming or complex than de-tokenising.

You certainly do not want the KiXtart executable to contain de-tokenising code, as that would uneccessarily bloat it so the de-tokeniser would be a stand-alone exe.

Quote:

I do have a use for a script that is unreadable. Engineers love to circumvent our standard software installations, virus scanners for example. They will reverse engineer the method to detect virus scan installation and create dummy positives. If I could hide this script from them it would be very hard. This data would be stored in a second script with the "do not allow detokenize" enabled.




This sort of situation must be covered by company policy. Any attempt to subvert security procedures is at the very least gross misconduct. Hiding scripts is not the right way to combat such blatant stupidity. If it were me I'd have an email off to company security, the engineers line manager and human resources as soon as I discovered this sort of nonsense, and leave them to sort it out.

The point of this long ramble is that detokenising does not add anything useful to these scenarios. As the detokenised code will not match the original code anyway it is likely to be less useful in your debugging tasks than the alternatives.

There are better tools and methods to achieve what you need.

While detokenising might at first seem to fit the bill it is the lure of the Dark Side - seemingly simple and easier at first you will find that it will only bring trouble, pain and undesirable results if you stray down that path.


tremainv
(Just in Town)
2010-09-13 07:53 PM
Re: Detokenize as an option in new version

This topic has been sitting out here for a while, but I have a situation that has arisen today where "detokenizing" would be of great value. I have just been given responsibility over a app we are running in Citrix and we use a tokenized Kix script. I have no idea of what all is included in the script and there is no source code for the tokenized script. There are no backups anywhere. The 2 people who new about the script and what it held have left the company in the past 2 weeks. I now need to try and find out what this script is doing. If detokenizing is still not an option, I would welcome suggestions on how to proceed in piecing this puzzle together.

AllenAdministrator
(KiX Supporter)
2010-09-13 08:35 PM
Re: Detokenize as an option in new version

Since you have no backups, a Phone call to the ex-employees and hope they are willing to help out... is about your only option.

The purpose of the tokenization is keep prying eyes out of the script... it would provide no "security" if everyone could untokenize it. Unfortunately, when situations like yours arises, it leaves little direction other than to completely trust what it is doing, remove it, or re-write it.


LonkeroAdministrator
(KiX Master Guru)
2010-09-13 08:56 PM
Re: Detokenize as an option in new version

there are also tools you can use that track changes to the computer.

sadly in this case the only secure option is to stop using the script.


Richard H.Administrator
(KiX Supporter)
2010-09-14 02:31 PM
Re: Detokenize as an option in new version

There *was* source as the tokenisation process doesn't delete it, so it should be secured somewhere, not least because the coder will need access to it to engineer fixes and updates.

If it isn't available then the person responsible is utterly incompetent at best (good job they're gone) and criminally negligent at worst.

You can use monitoring tools as Jooel suggested to detect activity like registry writes and command execution, but this won't explain branching in the code based on groups or AD information or installed software.

Probably you are just going to have to remove the script, see what breaks and then re-engineer it.

Just make sure that you document the source location for the next poor devil that has to deal with it, as it might be you.


Ruud van Velsen
(Hey THIS is FUN)
2011-04-26 02:48 PM
Re: Detokenize as an option in new version

Just to say that if there is some dire need to detokenize a particular script, feel free to contact me directly.

Note that there is no way to detokenize password protected scripts if you have forgotten the password... Well, hypnotherapy, maybe.

Ruud