Page 1 of 1 1
Topic Options
#157078 - 2006-02-13 04:42 AM Multiple threads in KiX?
thepip3r Offline
Hey THIS is FUN
*****

Registered: 2005-03-02
Posts: 350
Is there a way to kick off multiple threads? For those of you who helped me with my Dir() problem. I was writing that to try and figure out which orgs were the biggest offenders of wasted space and it's been running for 2 days on about 1.5TB. Is there a way I can make each loop it's own thread to speed up the process? Or any other way in general?? All thoughts/recommendations are greatly appreciated.
Top
#157079 - 2006-02-13 07:31 AM Re: Multiple threads in KiX?
Lonkero Administrator Offline
KiX Master Guru
*****

Registered: 2001-06-05
Posts: 22346
Loc: OK
you would need to spawn a second kixtart with run to accomplish 2 threads.
_________________________
!

download KiXnet

Top
#157080 - 2006-02-13 07:39 AM Re: Multiple threads in KiX?
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11623
Loc: CA
Well if you really want to know what is stored where this type of sotware will run circles around any of your home grown scripts.

http://wm.quest.com/products/storagemanagerwindows/

http://www.emc.com/products/software/visual_srm/visual_srm.jsp
 

Top
#157081 - 2006-02-13 05:26 PM Re: Multiple threads in KiX?
thepip3r Offline
Hey THIS is FUN
*****

Registered: 2005-03-02
Posts: 350
Jooel: Is there a max number of threads you can spawn or could I potentially spawn 10+ threads if resources allowed?

NTDOC: While I appreciate the link, I'm all about doing the most work with the least amount of money and those programs would cost a small fortune. There is a program on SourceForge in beta called WinDirStat that doesn't something not as robust and doesn't have a reporting tool but is fast and gives a nice output to visually represent where your biggest file sizes exist.



Edited by thepip3r (2006-02-13 05:29 PM)

Top
#157082 - 2006-02-13 05:38 PM Re: Multiple threads in KiX?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
You can spawn as many processes as memory will allow.

You will need to pass any variables that you need to preserve on the command line.

I remember doing some sort of "matrix" type of thing using multiple threads a while back - I'll see if I can dig it out...

Top
#157083 - 2006-02-13 05:44 PM Re: Multiple threads in KiX?
thepip3r Offline
Hey THIS is FUN
*****

Registered: 2005-03-02
Posts: 350
Thanx Richard, an example to look at would help a lot.
Top
#157084 - 2006-02-13 05:50 PM Re: Multiple threads in KiX?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Oops! I remembered wrong. It was threaded, but within the one script which is not particularly useful for you (but it is pretty ) http://www.kixtart.org/ubbthreads/showth...page=0&vc=1

This thread shows a script which calls child scripts and passes data via INI files. It's a bit big for a demo, but you may be able to pick some details out of it:
http://www.kixtart.org/ubbthreads/showflat.php?Cat=0&Number=76089&an=&page=0&vc=1

Top
#157085 - 2006-02-13 06:07 PM Re: Multiple threads in KiX?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Here is a simple example of multiple processes.

The parent spawns copies of itself, passing $iChildID as a parameter on the command line.

The child processes communicate to the parent via the INI file - in this case they just increment a simple counter:

Code:
Break ON

$sINI=".\Thread.ini"
$iChildMax=10

If $iChildID
Srnd(@TICKS*$iChildID)
$iSleepTime=RND(10)+2
"This is child # "+$iChildID+" sleeping for "+$iSleepTime+" seconds..."+@CRLF
Sleep $iSleepTime
$=WriteProfileString($sINI,"MAIN","COUNTER",1+CInt(ReadProfileString($sINI,"MAIN","COUNTER")))
"Child #"+$iChildID+" exiting."+@CRLF
Exit 0
EndIf

$=WriteProfileString($sINI,"MAIN","COUNTER",0)

For $i = 1 To $iChildMax
"Starting child #"+$i+@CRLF
RUN "kix32.exe "+@SCRIPTNAME+" $$iChildID="+$i
Next

$iChildCount=CInt(ReadProfileString($sINI,"MAIN","COUNTER"))
While $iChildCount<$iChildMax
"Waiting for children to exit, count is: "+$iChildCount+@CRLF
Sleep 1
$iChildCount=CInt(ReadProfileString($sINI,"MAIN","COUNTER"))
Loop

"All child processes exited, parent closing."+@CRLF

Exit 0


Top
#157086 - 2006-02-13 07:05 PM Re: Multiple threads in KiX?
thepip3r Offline
Hey THIS is FUN
*****

Registered: 2005-03-02
Posts: 350
The Matrix/KiX things is awesome but I have a couple of questions on how to get each thread to run what I want it to. First off, this is the piece of code in question that spawns new threads correct?
Code:
            If $avServers[3,$i]<0.0001 AND $avServers[3,$i]>-0.0001
$gNull=WriteProfileString($gINI,$avServers[0,$i],"Status","9Poll Request")
Run $gCOMSPEC+" "+$gKIX+' "'+@SCRIPTDIR+"\"+@SCRIPTNAME+'" $$sServer="'+$avServers[0,$i]+'"'
Else



If that's the case, I see how you're kicking off a new thread but how to you tell that thread to perform what you want it to? Like for mine, I'd probably kick off a new inst of 10 folders but how does that new KiX file know what commands to run?

Edit: Sorry for the post. I started writing this like an hour ago and JUST posted it. I didn't see you post before and it will probably answer some of my questions.


Edited by thepip3r (2006-02-13 07:07 PM)

Top
#157087 - 2006-02-13 07:21 PM Re: Multiple threads in KiX?
thepip3r Offline
Hey THIS is FUN
*****

Registered: 2005-03-02
Posts: 350
When I run your script, this is what I get, is this what it's supposed to say??

Quote:

Starting child #1
Starting child #2
Starting child #3
Starting child #4
Starting child #5
Starting child #6
Starting child #7
Starting child #8
Starting child #9
Starting child #10
Waiting for children to exit, count is: 0

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!
Waiting for children to exit, count is: 0

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!

ERROR : failed to find/open script [test.kix]!



Top
#157088 - 2006-02-13 08:04 PM Re: Multiple threads in KiX?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Errr..no.

How are you running the script?

It calls itself, but I didn't include any path information in the RUN statement so it assumes that your current working directory is the directory that the script is in.

You could chuck @SCRIPTDIR in if you want to make it bullet proof.

Change the "RUN" line to read:
Code:
RUN 'kix32.exe "'+@SCRIPTDIR+'\'+@SCRIPTNAME+'" $$iChildID='+$i



The output should actually look something like this:
Quote:

C:\Temp>kix32 thread.kix
Starting child #1
Starting child #2
Starting child #3
Starting child #4
Starting child #5
Starting child #6
Starting child #7
Starting child #8
Starting child #9
Starting child #10
Waiting for children to exit, count is: 0
This is child # 10 sleeping for 11 seconds...
This is child # 2 sleeping for 8 seconds...
This is child # 5 sleeping for 12 seconds...
This is child # 6 sleeping for 10 seconds...
This is child # 7 sleeping for 8 seconds...
This is child # 8 sleeping for 6 seconds...
This is child # 3 sleeping for 6 seconds...
This is child # 1 sleeping for 10 seconds...
This is child # 4 sleeping for 4 seconds...
This is child # 9 sleeping for 5 seconds...
Waiting for children to exit, count is: 0
Waiting for children to exit, count is: 0
Waiting for children to exit, count is: 0
Waiting for children to exit, count is: 0
Child #4 exiting.
Waiting for children to exit, count is: 1
Child #9 exiting.
Waiting for children to exit, count is: 2
Child #8 exiting.
Child #3 exiting.
Waiting for children to exit, count is: 4
Waiting for children to exit, count is: 4
Child #2 exiting.
Child #7 exiting.
Waiting for children to exit, count is: 6
Waiting for children to exit, count is: 6
Child #6 exiting.
Child #1 exiting.
Waiting for children to exit, count is: 8
Child #10 exiting.
Waiting for children to exit, count is: 9
Child #5 exiting.
All child processes exited, parent closing.

C:\Temp>



Top
#157089 - 2006-02-22 02:21 AM Re: Multiple threads in KiX?
thepip3r Offline
Hey THIS is FUN
*****

Registered: 2005-03-02
Posts: 350
Is there a way to pass a variable to one of the child scripts though? It would seem like there should be a way to do this. The example I'm trying to think of is say you have a function that takes 4 seconds to return results. But you have to run that function on 500 objects so is there a way to throw that function into like a completely seperate kix file and then call that file but pass the current object name to it so you'd theoretically have 500 windows open all running the function on the same computer name?
Top
#157090 - 2006-02-22 06:26 AM Re: Multiple threads in KiX?
Sealeopard Offline
KiX Master
*****

Registered: 2001-04-25
Posts: 11164
Loc: Boston, MA, USA
Yes, just add another variable to thescript call, just like the ChildID variable.
_________________________
There are two types of vessels, submarines and targets.

Top
#157091 - 2006-02-22 04:20 PM Re: Multiple threads in KiX?
Les Offline
KiX Master
*****

Registered: 2001-06-11
Posts: 12734
Loc: fortfrances.on.ca
That would just be a one-way trasfer. You would need to use an intermediary like an INI file or reg key/value.
_________________________
Give a man a fish and he will be back for more. Slap him with a fish and he will go away forever.

Top
#157092 - 2006-02-23 11:13 AM Re: Multiple threads in KiX?
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Quote:

Is there a way to pass a variable to one of the child scripts though?




Yes, the example that I provided you passes a variable on the command line. Any simple variable (number, string) can be passed this way.

You *cannot* pass a complex variable like an array or an object. You would need to pass a string reference to the object so that the child process can create a new local copy. for itself. Arrays would have to be converted to a string form and passed on the command line, by file or registry.

Quote:

The example I'm trying to think of is say you have a function that takes 4 seconds to return results. But you have to run that function on 500 objects so is there a way to throw that function into like a completely seperate kix file and then call that file but pass the current object name to it so you'd theoretically have 500 windows open all running the function on the same computer name?




Yes, in the simple example above all the threads are running in parallel. Once they have started they communicate their state with the parent process by using the INI file. It would be trivial to expand the example to make this two-way communication.

You could also communicate via the registry if you didn't want to use INI files.

Top
#157093 - 2006-02-27 09:32 PM Re: Multiple threads in KiX?
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4396
Loc: New Jersey
I have a system healthcheck tool that uses script threading.. it assembles a list of target systems to query - something in the order of 3-400. It then reads that array, and for each target system, does a Run command -
Code:
 
Run 'Kix32 ./child.kix $HOST=' + $HostName


It's in a loop, with a counter - I delay 60 seconds every time the counter hits 25, just to throttle things somewhat. The Child.kix script performs several remote data collections from the target system and writes the collected data to a system-specific INI file.

In this fashion, I am able to collect data from several hundred servers in short order, although I do run the CPU utilization to 100% for about 15 minutes when the processes are first launching. Not an issue since this runs in the middle of the night.

By comparison, using a SHELL command to run the processes serially took many (9+) hours, since some of the processes dump event log data over a WAN connection. Those few WAN stragglers take about an hour to complete, and my entire collection completes in about an hour.

Glenn
_________________________
Actually I am a Rocket Scientist! \:D

Top
#157094 - 2006-02-27 10:15 PM Re: Multiple threads in KiX?
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11623
Loc: CA
Just as a suggestion Glenn, maybe add some WMI code to your script to check the current CPU/Memory usage and if needed throttle back, recheck, ramp backup, repeat, etc...

That way it could be used on any system and given a threshold of CPU/Memory to not exceed.

Top
#157095 - 2006-02-28 03:38 PM Re: Multiple threads in KiX?
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4396
Loc: New Jersey
Not a bad idea, but for this app, the server where it runs is dedicated to network management, and it was my intent to saturate the system.
_________________________
Actually I am a Rocket Scientist! \:D

Top
Page 1 of 1 1


Moderator:  Jochen, Allen, Radimus, Glenn Barnas, ShaneEP, Ruud van Velsen, Arend_, Mart 
Hop to:
Shout Box

Who's Online
1 registered (Allen) and 466 anonymous users online.
Newest Members
gespanntleuchten, DaveatAdvanced, Paulo_Alves, UsTaaa, xxJJxx
17864 Registered Users

Generated in 0.103 seconds in which 0.056 seconds were spent on a total of 12 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org