thepip3r
(Hey THIS is FUN)
2006-02-13 04:42 AM
Multiple threads in KiX?

Is there a way to kick off multiple threads? For those of you who helped me with my Dir() problem. I was writing that to try and figure out which orgs were the biggest offenders of wasted space and it's been running for 2 days on about 1.5TB. Is there a way I can make each loop it's own thread to speed up the process? Or any other way in general?? All thoughts/recommendations are greatly appreciated.

LonkeroAdministrator
(KiX Master Guru)
2006-02-13 07:31 AM
Re: Multiple threads in KiX?

you would need to spawn a second kixtart with run to accomplish 2 threads.

NTDOCAdministrator
(KiX Master)
2006-02-13 07:39 AM
Re: Multiple threads in KiX?

Well if you really want to know what is stored where this type of sotware will run circles around any of your home grown scripts.

http://wm.quest.com/products/storagemanagerwindows/

http://www.emc.com/products/software/visual_srm/visual_srm.jsp
 

thepip3r
(Hey THIS is FUN)
2006-02-13 05:26 PM
Re: Multiple threads in KiX?

Jooel: Is there a max number of threads you can spawn or could I potentially spawn 10+ threads if resources allowed?

NTDOC: While I appreciate the link, I'm all about doing the most work with the least amount of money and those programs would cost a small fortune. There is a program on SourceForge in beta called WinDirStat that doesn't something not as robust and doesn't have a reporting tool but is fast and gives a nice output to visually represent where your biggest file sizes exist.

Richard H.Administrator
(KiX Supporter)
2006-02-13 05:38 PM
Re: Multiple threads in KiX?

You can spawn as many processes as memory will allow.

You will need to pass any variables that you need to preserve on the command line.

I remember doing some sort of "matrix" type of thing using multiple threads a while back - I'll see if I can dig it out...

thepip3r
(Hey THIS is FUN)
2006-02-13 05:44 PM
Re: Multiple threads in KiX?

Thanx Richard, an example to look at would help a lot.

Richard H.Administrator
(KiX Supporter)
2006-02-13 05:50 PM
Re: Multiple threads in KiX?

Oops! I remembered wrong. It was threaded, but within the one script which is not particularly useful for you (but it is pretty ) http://www.kixtart.org/ubbthreads/showth...page=0&vc=1

This thread shows a script which calls child scripts and passes data via INI files. It's a bit big for a demo, but you may be able to pick some details out of it:
http://www.kixtart.org/ubbthreads/showflat.php?Cat=0&Number=76089&an=&page=0&vc=1

Richard H.Administrator
(KiX Supporter)
2006-02-13 06:07 PM
Re: Multiple threads in KiX?

Here is a simple example of multiple processes.

The parent spawns copies of itself, passing $iChildID as a parameter on the command line.

The child processes communicate to the parent via the INI file - in this case they just increment a simple counter:

Code:
Break ON
 
$sINI=".\Thread.ini"
$iChildMax=10
 
If $iChildID
	Srnd(@TICKS*$iChildID)
	$iSleepTime=RND(10)+2
	"This is child # "+$iChildID+" sleeping for "+$iSleepTime+" seconds..."+@CRLF
	Sleep $iSleepTime
	$=WriteProfileString($sINI,"MAIN","COUNTER",1+CInt(ReadProfileString($sINI,"MAIN","COUNTER")))
	"Child #"+$iChildID+" exiting."+@CRLF
	Exit 0
EndIf
 
$=WriteProfileString($sINI,"MAIN","COUNTER",0)

For $i = 1 To $iChildMax
	"Starting child #"+$i+@CRLF
	RUN "kix32.exe "+@SCRIPTNAME+" $$iChildID="+$i
Next
 
$iChildCount=CInt(ReadProfileString($sINI,"MAIN","COUNTER"))
While $iChildCount<$iChildMax
	"Waiting for children to exit, count is: "+$iChildCount+@CRLF
	Sleep 1
	$iChildCount=CInt(ReadProfileString($sINI,"MAIN","COUNTER"))
Loop
 
"All child processes exited, parent closing."+@CRLF
 
Exit 0

thepip3r
(Hey THIS is FUN)
2006-02-13 07:05 PM
Re: Multiple threads in KiX?

The Matrix/KiX things is awesome but I have a couple of questions on how to get each thread to run what I want it to. First off, this is the piece of code in question that spawns new threads correct?
Code:
            If $avServers[3,$i]<0.0001 AND $avServers[3,$i]>-0.0001
               $gNull=WriteProfileString($gINI,$avServers[0,$i],"Status","9Poll Request")
               Run $gCOMSPEC+" "+$gKIX+' "'+@SCRIPTDIR+"\"+@SCRIPTNAME+'" $$sServer="'+$avServers[0,$i]+'"'
            Else


If that's the case, I see how you're kicking off a new thread but how to you tell that thread to perform what you want it to? Like for mine, I'd probably kick off a new inst of 10 folders but how does that new KiX file know what commands to run?

Edit: Sorry for the post. I started writing this like an hour ago and JUST posted it. I didn't see you post before and it will probably answer some of my questions.

thepip3r
(Hey THIS is FUN)
2006-02-13 07:21 PM
Re: Multiple threads in KiX?

When I run your script, this is what I get, is this what it's supposed to say??

Quote:

Starting child #1
Starting child #2
Starting child #3
Starting child #4
Starting child #5
Starting child #6
Starting child #7
Starting child #8
Starting child #9
Starting child #10
Waiting for children to exit, count is: 0

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!
Waiting for children to exit, count is: 0

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!

Richard H.Administrator
(KiX Supporter)
2006-02-13 08:04 PM
Re: Multiple threads in KiX?

Errr..no.

How are you running the script?

It calls itself, but I didn't include any path information in the RUN statement so it assumes that your current working directory is the directory that the script is in.

You could chuck @SCRIPTDIR in if you want to make it bullet proof.

Change the "RUN" line to read:
Code:
RUN 'kix32.exe "'+@SCRIPTDIR+'\'+@SCRIPTNAME+'" $$iChildID='+$i


The output should actually look something like this:
Quote:

C:\Temp>kix32 thread.kix
Starting child #1
Starting child #2
Starting child #3
Starting child #4
Starting child #5
Starting child #6
Starting child #7
Starting child #8
Starting child #9
Starting child #10
Waiting for children to exit, count is: 0
This is child # 10 sleeping for 11 seconds...
This is child # 2 sleeping for 8 seconds...
This is child # 5 sleeping for 12 seconds...
This is child # 6 sleeping for 10 seconds...
This is child # 7 sleeping for 8 seconds...
This is child # 8 sleeping for 6 seconds...
This is child # 3 sleeping for 6 seconds...
This is child # 1 sleeping for 10 seconds...
This is child # 4 sleeping for 4 seconds...
This is child # 9 sleeping for 5 seconds...
Waiting for children to exit, count is: 0
Waiting for children to exit, count is: 0
Waiting for children to exit, count is: 0
Waiting for children to exit, count is: 0
Child #4 exiting.
Waiting for children to exit, count is: 1
Child #9 exiting.
Waiting for children to exit, count is: 2
Child #8 exiting.
Child #3 exiting.
Waiting for children to exit, count is: 4
Waiting for children to exit, count is: 4
Child #2 exiting.
Child #7 exiting.
Waiting for children to exit, count is: 6
Waiting for children to exit, count is: 6
Child #6 exiting.
Child #1 exiting.
Waiting for children to exit, count is: 8
Child #10 exiting.
Waiting for children to exit, count is: 9
Child #5 exiting.
All child processes exited, parent closing.

C:\Temp>

thepip3r
(Hey THIS is FUN)
2006-02-22 02:21 AM
Re: Multiple threads in KiX?

Is there a way to pass a variable to one of the child scripts though? It would seem like there should be a way to do this. The example I'm trying to think of is say you have a function that takes 4 seconds to return results. But you have to run that function on 500 objects so is there a way to throw that function into like a completely seperate kix file and then call that file but pass the current object name to it so you'd theoretically have 500 windows open all running the function on the same computer name?

Sealeopard
(KiX Master)
2006-02-22 06:26 AM
Re: Multiple threads in KiX?

Yes, just add another variable to thescript call, just like the ChildID variable.

Les
(KiX Master)
2006-02-22 04:20 PM
Re: Multiple threads in KiX?

That would just be a one-way trasfer. You would need to use an intermediary like an INI file or reg key/value.

Richard H.Administrator
(KiX Supporter)
2006-02-23 11:13 AM
Re: Multiple threads in KiX?

Quote:

Is there a way to pass a variable to one of the child scripts though?



Yes, the example that I provided you passes a variable on the command line. Any simple variable (number, string) can be passed this way.

You *cannot* pass a complex variable like an array or an object. You would need to pass a string reference to the object so that the child process can create a new local copy. for itself. Arrays would have to be converted to a string form and passed on the command line, by file or registry.

Quote:

The example I'm trying to think of is say you have a function that takes 4 seconds to return results. But you have to run that function on 500 objects so is there a way to throw that function into like a completely seperate kix file and then call that file but pass the current object name to it so you'd theoretically have 500 windows open all running the function on the same computer name?



Yes, in the simple example above all the threads are running in parallel. Once they have started they communicate their state with the parent process by using the INI file. It would be trivial to expand the example to make this two-way communication.

You could also communicate via the registry if you didn't want to use INI files.

Glenn BarnasAdministrator
(KiX Supporter)
2006-02-27 09:32 PM
Re: Multiple threads in KiX?

I have a system healthcheck tool that uses script threading.. it assembles a list of target systems to query - something in the order of 3-400. It then reads that array, and for each target system, does a Run command -
Code:
 
Run 'Kix32 ./child.kix $HOST=' + $HostName

It's in a loop, with a counter - I delay 60 seconds every time the counter hits 25, just to throttle things somewhat. The Child.kix script performs several remote data collections from the target system and writes the collected data to a system-specific INI file.

In this fashion, I am able to collect data from several hundred servers in short order, although I do run the CPU utilization to 100% for about 15 minutes when the processes are first launching. Not an issue since this runs in the middle of the night.

By comparison, using a SHELL command to run the processes serially took many (9+) hours, since some of the processes dump event log data over a WAN connection. Those few WAN stragglers take about an hour to complete, and my entire collection completes in about an hour.

Glenn

NTDOCAdministrator
(KiX Master)
2006-02-27 10:15 PM
Re: Multiple threads in KiX?

Just as a suggestion Glenn, maybe add some WMI code to your script to check the current CPU/Memory usage and if needed throttle back, recheck, ramp backup, repeat, etc...

That way it could be used on any system and given a threshold of CPU/Memory to not exceed.

Glenn BarnasAdministrator
(KiX Supporter)
2006-02-28 03:38 PM
Re: Multiple threads in KiX?

Not a bad idea, but for this app, the server where it runs is dedicated to network management, and it was my intent to saturate the system.