I was trying to create a script that could multitask. I would like to perform some tasks using psexec but psexec executes every command for every pc one by one. So I started with the follwing testscript:
rem.kix
Code:
break on
$x = 15
global $t
for $y = 0 TO $x STEP 1
? "y = $y"
$t = 0
while $t < 6 AND $y =< $x
call tst.kix
$t = $t + 1
? "t = $t"
loop
sleep 5
next
tst.kix
Code:
break on
sleep 3
$t = $t - 1
exit
but since call executes and then returnes to the script, the counter doesn't work.
You would have to RUN a seperate instance of kix32.exe to achieve true multi-tasking, like maybe:
RUN "kix32.exe sub.kix"
Here's the challenge though - your instances have no easy way of comunicating back to your main thread, like with using $vars etc. What I've seen done, is to use the registry for this purpose... example:
1) Create your own little subkey somewhere 2) Have each child report a status back to the main thread, using its own value under this subkey. 3) Each child would need some kind of unique identifier so that it will create it own status value (PID?) 4) What happens when the main script dies. The child scripts should terminate if it looses contact with the parent.
#182055 - 2007-10-2904:27 PMRe: multi threading
[Re: Shawn]
Glenn BarnasGlenn Barnas KiX Supporter
Registered: 2003-01-28
Posts: 4402
Loc: New Jersey
I have a process that kicks off about 50 Kix32 processes via Run. Each process is passed an argument that it uses to create an INI file with it's status & results. (that arg becomes the INI file name.) The child processes create the INI file in the TEMP folder, updating it as they process their task. When the process is done, the last thing id does is to move the INI file to a folder where the parent process is watching for the files to appear. It reads the INI file, updates the common database, and then deletes the file. If there are more systems to process, it spawns off another child task.
I limit the number of child processes to 50, which (in my environment) ramps the CPU utilization to 100%. I process data collection from about 300 servers in about 15 minutes this way. There are a few older systems that take longer, but the bulk are completed quickly.
FWIW - my "parent" process is actually a kix script configured as a system service using SrvAny.
Glenn
_________________________ Actually I am a Rocket Scientist!
#182062 - 2007-10-2909:36 PMRe: multi threading
[Re: vroedie]
Glenn BarnasGlenn Barnas KiX Supporter
Registered: 2003-01-28
Posts: 4402
Loc: New Jersey
OK - here's a sample of "parent.kix" and "child.kix" that illustrates one way to do this - first, the PARENT script:
And the CHILD script:
The CHILD waits 0-4 seconds, then creates a .TMP file, waits an additional 0-4 seconds (to simulate work communicating to a remote system) and then renames the TMP file to INI before exiting.
PARENT quickly spawns off 10 children, then waits for INI files to appear. Each INI that appears reduces the worker count, allowing more children to be run, until all 15 have completed. The status line shows the count grow from 1-15, and the workers increase to 10, then decrease/increase until all 15 tasks have been initiated, and finally workers decreases to zero.
Again, it's a simple, silly example that illustrates the concept, but should be able to form the building block of what you need. You should probably incorporate a max time limit that you wait for workers to complete, or use some type of process interrogation so you don't wait for children that die before completing their tasks. Unlikely, but possible.
Hope this gets you going in the right direction! I tested this, and it should work to illustrate the concept just fine - using Kix 4.60 here..
Glenn
_________________________ Actually I am a Rocket Scientist!
Glenn, should the children worry about the parent dying ? Or should they just carry-on until done. Guess it depends on what whats being done. Never tried but do processes started with RUN terminate when the main script ends ? My guess would be ... not.
#182070 - 2007-10-3002:26 AMRe: multi threading
[Re: Shawn]
Glenn BarnasGlenn Barnas KiX Supporter
Registered: 2003-01-28
Posts: 4402
Loc: New Jersey
I've always written these scripts such that the children are independent of the parent. They carry on even if the parent dies. I don't think it's like Unix, where the child processes don't survive the parent unless started with Exec.
In my example, the children will collect the data even if the parent dies. The next cycle, the parent will most likely simply clean up after the prior process if anything remained. Of course, the new parent could also assume that the prior parent didn't succeed, and could process the files present before creating its own offspring.
Since we don't have the luxury of signals and IPC (that I know of, at least!) within Kix, I just write simple processes. Most are non-critical, so a parent dying would result in no data collection that day - not a crisis. My primary use of this technology is the nightly query of about 300 servers - we gather disk space info, summarize event log warnings/errors, verify scheduled tasks, and a couple of other "server health" items. All gets collected with one collection script per server, run 50 at a time, scheduled similarly to what I illustrated above. 90% of the collection runs in 15 minutes, but we have a few ancient systems that take about 30-40 minutes to complete.
My Kix-based software deployment system works in a similar fashion - a KF GUI is used to stage a web content release - moving files from the dev fileserver to a deployment server (code freeze). Another person runs the KF GUI that initiates the deployment - it takes the GUI input, generates a (temp) INI file with the instructions, and then drops the INII file into a queue folder. The person doing the deploy has A-D rights to run the deploy to QA or PROD, but no logon or share access to the web servers.
The service running on the deploy server sees the new INI file in the queue, determines which environment it is being deployed to, and creates a scheduled event (task, but no trigger) using the appropriate credentials. This is a modification of the "run" concept above, since we need to use an alternate ID. (tried RunNas, but no joy) The service then triggers the task with a RunNow action, starting the actual deploy process. I can kick off about 3 deployments per minute - different releases, different "products", and different environments.
The Staging process is self-contained, but the Deploy GUI, scheduling service, and deploy-job processor all communicate with each other via the job INI file - even to the point of passing interactive status messages back to the GUI. The end-result is then passed back to a status file that the Staging tool can interrogate, displaying the status (pending, queued, success, or failure) of the staged job. As complex as this is, it's tolerant of any component failure - if the service fails, the jobs hold in the queue. If the deploy process fails, the job can be requeued. If the deployment itself fails, the modified files are restored from the ZIP backup.
I'd really like to play with some type of IPC, so two independent processes could more readily commnuicate - in this case, the Deploy Console and the deploy service, then the deploy service and the deploy-job processor. KF sockets, maybe?
G-
PS - aren't ya glad you asked a simple question?
_________________________ Actually I am a Rocket Scientist!
one question so far... the parent starts the child processes hidden. Is there a way to have the parent start new windows so I can see the progress of the child processes? I've tried setting the SETCONSOLE("SHOW") option, but no luck.
I've tried to combine some scripts, so I could read a file with hosts, put those host in an array and then start a child process for each host in the file. Now the script seems to run the script for just the first entry in the file.
Code:
; PARENT.KIX ======================================================
Break On
; Will spawn off 10 child processes. Each child will take two arguments. The first arg will be
; the child's ID, the second represents something to do..
; The child will perform a task, write the answer to a temporary INI file, then rename the ini
; file before exiting (simulating lots of work being done by the child, writing much data to the ini file)
; The parent will monitor the folder for the INI files, read them when they appear, and
; consolidate the data. The INI files will be deleted by the parent process when complete.
$ = SetOption('NoVarsInStrings', 'On')
Del '.\Summary.txt' ; start with a fresh "central database" for logging the results
If Not IsDeclared($Arg)
"You must supply a file containing a list of hosts"+@CRLF
" Usage: kix32 "+@SCRIPTNAME+" $$Arg=file (in same directory)"+@CRLF
Exit 0
EndIf
DIM $list[0]
DIM $ctr, $ctr2
$ctr = 0
IF Open(3, ".\" + $Arg) = 0
Dim $x
$x = ReadLine(3)
WHILE @ERROR = 0
redim preserve $list[$ctr]
$list[$ctr] = $x
$ctr = $ctr + 1
$x = ReadLine(3)
LOOP
Else
"Cannot find file " + $Arg + @CRLF
ENDIF
;$Data = ' AaAbAcAdAeBaBbBcBdBeCaCbCcCdCeDaDbDcDdDe'
$Count = 1 ; count of tasks to process
$Workers = 0 ; count of active workers
$ctr2 = $ctr + 1
For Each $Entry in $list
? $Entry + @CRLF
Next
For Each $Entry in $list
While $Count < $ctr2 ; max of 15
; limit number of child processes
If $Workers < 10
; fewer than 10 workers - start one!
; just a silly command here to pass the count and two characters from the data string
; in real world, A1 would be an ID (as it is here) and A2 might be the name of the remote system
; that the child should interrogate
;? "Line read: [" + $Entry + "]"
$Command = '%comspec% /C START /SEPARATE kix32.exe .\child.kix $A1=' + $Count + ' $A2="' + $Entry + '"'
Run $Command ; run asynchronously
$Workers = $Workers + 1 ; increment the worker count
$Count = $Count + 1 ; increment the process count
EndIF
; Process completed work, allowing more children to run
; any INI files found?
$File = Dir('*.ini')
While $File <> "" and @ERROR = 0
; Yes!
$Workers = $Workers - 1 ; worker completed, reduce the active count
; write the data from the child's INI file to the "central database"
$ = RedirectOutput('.\summary.txt')
$File ', '
ReadProfileString('.\' + $File, 'Common', 'Arg') ?
$ = RedirectOutput('')
; remove the worker file
Del '.\' + $File
; check for more as long as we're here..
$File = Dir() ; retrieve next file
Loop
; display status
' Count: ' $Count ', Workers: ' $Workers ' ' Chr(13)
Loop
; no more children to run, need to process remaining files that are waiting for completion
While $Workers > 0
$File = Dir('*.ini')
While $File <> "" and @ERROR = 0
; found a file - that means a worker completed it's task
$Workers = $Workers - 1
; update the "central database"
$ = RedirectOutput('.\summary.txt')
$File ', '
ReadProfileString('.\' + $File, 'Common', 'Arg') ?
$ = RedirectOutput('')
; delete the worker file
Del '.\' + $File
; Status display
' Count: ' $Count ', Workers: ' $Workers ' ' Chr(13)
Sleep 0.2
; any more files ready?
$File = Dir() ; retrieve next file
Loop
Loop
Next
?
' Displaying results:' ?
Display 'Summary.txt' ?
quit 0
#182092 - 2007-10-3012:53 PMRe: multi threading
[Re: vroedie]
Glenn BarnasGlenn Barnas KiX Supporter
Registered: 2003-01-28
Posts: 4402
Loc: New Jersey
Not tested, but try this - simplify! I'll leave it up to you to add some diagnostic messages so you can see what's going on.
Glenn
Code:
; PARENT.KIX ======================================================
Break On
; Will spawn off 10 child processes. Each child will take two arguments. The first arg will be
; the child's ID, the second represents something to do..
; The child will perform a task, write the answer to a temporary INI file, then rename the ini
; file before exiting (simulating lots of work being done by the child, writing much data to the ini file)
; The parent will monitor the folder for the INI files, read them when they appear, and
; consolidate the data. The INI files will be deleted by the parent process when complete.
$ = SetOption('NoVarsInStrings', 'On')
Del '.\Summary.txt' ; start with a fresh "central database" for logging the results
If Not IsDeclared($Arg)
"You must supply a file containing a list of hosts"+@CRLF
" Usage: kix32 "+@SCRIPTNAME+" $$Arg=file (in same directory)"+@CRLF
Exit 0
EndIf
DIM $list[0]
DIM $ctr, $ctr2
$ctr = 0
$Count = 1 ; count of tasks to process
$Workers = 0 ; count of active workers
$ctr2 = $ctr + 1
IF Open(3, ".\" + $Arg) = 0
$Entry = ReadLine(3)
WHILE Not @ERROR
; limit number of child processes
If $Workers < 10
; fewer than 10 workers - start one!
; just a silly command here to pass the count and two characters from the data string
; in real world, A1 would be an ID (as it is here) and A2 might be the name of the remote system
; that the child should interrogate
;? "Line read: [" + $Entry + "]"
$Command = '%comspec% /C START /SEPARATE kix32.exe .\child.kix $A1=' + $Count + ' $A2="' + $Entry + '"'
Run $Command ; run asynchronously
$Workers = $Workers + 1 ; increment the worker count
$Count = $Count + 1 ; increment the process count
EndIF
; Process completed work, allowing more children to run
; any INI files found?
$File = Dir('*.ini')
While $File <> "" and @ERROR = 0
; Yes!
$Workers = $Workers - 1 ; worker completed, reduce the active count
; write the data from the child's INI file to the "central database"
$ = RedirectOutput('.\summary.txt')
$File ', '
ReadProfileString('.\' + $File, 'Common', 'Arg') ?
$ = RedirectOutput('')
; remove the worker file
Del '.\' + $File
; check for more as long as we're here..
$File = Dir() ; retrieve next file
Loop
; display status
' Count: ' $Count ', Workers: ' $Workers ' ' Chr(13)
$Entry = ReadLine(3)
LOOP
Else
"Cannot find file " + $Arg + @CRLF
ENDIF
; no more children to run, need to process remaining files that are waiting for completion
While $Workers > 0
$File = Dir('*.ini')
While $File <> "" and @ERROR = 0
; found a file - that means a worker completed it's task
$Workers = $Workers - 1
; update the "central database"
$ = RedirectOutput('.\summary.txt')
$File ', '
ReadProfileString('.\' + $File, 'Common', 'Arg') ?
$ = RedirectOutput('')
; delete the worker file
Del '.\' + $File
; Status display
' Count: ' $Count ', Workers: ' $Workers ' ' Chr(13)
Sleep 0.2
; any more files ready?
$File = Dir() ; retrieve next file
Loop
Loop
Next
?
' Displaying results:' ?
Display 'Summary.txt' ?
quit 0
_________________________ Actually I am a Rocket Scientist!
#182095 - 2007-10-3002:23 PMRe: multi threading
[Re: vroedie]
Glenn BarnasGlenn Barnas KiX Supporter
Registered: 2003-01-28
Posts: 4402
Loc: New Jersey
You don't want a limit of 15 actions, do you? You want to process every element. In my example, I had 15 elements - it was easier to hard-code the elements than to try and provide a data file for you to read, too..
The line If $Workers < 10 is what controls the number of child worker threads. Change it to If $Workers < $MaxWorkers and Dim/define $MaxWorkers at the beginning to limit the number of concurrent child processes running. Like I said, I have it set to 50, which runs the server at 100% CPU load for 10-12 minutes. It's trial and error - My server is dedicated to this task at the time it runs, so, I fully load it to get the job done. If your server is shared, you can keep increasing the MaxWorkers value until you hit 50% load or so .
Bottom line - my "first" outer loop has been modified to run once for each value read from the file, which is what you need in order to process a random number of elements. The middle loop keeps reading the data, and the innermost loop simply keeps the number of child processes at 10 (or, better yet, $MaxWorkers).
Glenn
_________________________ Actually I am a Rocket Scientist!
Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
Here is another multi-threading example (now with max child control Glenn ;))
This example has the parent and child code in the same script, and takes a list of hosts from the command line and/or a file.
The child code just does a ping on each host and returns the ping time to the parent. The parent displays the colour-coded ping time.
The parent monitors the child and will detect when one takes too long to start (10 seconds) - I haven't included any checks to determine if a child has exited unexpectedly, though you can use the child PID as a lookup in the process table if that is important to you.
Break ON $=SetOption("Explicit","ON")
; YAPP.kix - Yet Another Parallel Pinger ; ; Pass a comma seperated lists of hostsnames / IP addresses or a list in a file. ; The hosts will be pinged asynchronously, and the results displayed.
; $iMaxChild may also be set on the command line to change the limit of child ; processes which may be active at any time.
; Amendment History ; ----------------- ; 20071030.13:00 R. Howarth <rhowarth@harsco.com> ; Added code to limit active child processes. ; Allow debug state to be set on command line. ; Allow maxchild to be set on command line.
IfNot IsDeclared($DEBUG) GLOBAL $DEBUG $DEBUG=0 EndIf IfNot IsDeclared($DEBUG) Global $iMaxChild $iMaxChild=20 EndIf
Dim $sINI,$sHost,$iChildActive,$vChildPID,$vResponseTime,$vStatusCode,$iStartTicks Dim $iMaxTicks $iMaxTicks=5000 Dim $iInterval $iInterval="0.1" Dim $iLastHost Dim $sActiveList Dim $sNewList
Dim $sErrorColour, $sNormColour, $sColour
Dim $aiTime $aiTime= Split("-1,100,250,500",",") Dim $aiColour $aiColour=Split("g+/n,y+/n,y/n,r+/n",",")
Dim $iTimeout $iTimeout=10000 ; Wait 10 seconds for a response
$sErrorColour="r+/n" $sNormColour="w/n"
Color $sNormColour
If IsDeclared($iParentPID) ; ------------------------------------------------- ; CHILD PAYLOAD - code executed for child process ; ------------------------------------------------- $sHost=$sHostList myDEBUG("Child started for "+$sHostList) $sINI=@SCRIPTDIR+"\YAPP_"+$iParentPID+".ini" myDEBUG("Child started for "+$sHostList+", INI="+$sINI) $=WriteProfileString($sINI,$sHost,"PID",@PID) $vResponseTime=wmiPing($sHostList,$iTimeout) $vStatusCode=@ERROR If $vStatusCode $vStatusCode=$vResponseTime $vResponseTime=(-1) EndIf $=WriteProfileString($sINI,$sHost,"ResponseTime",$vResponseTime) $=WriteProfileString($sINI,$sHost,"StatusCode",$vStatusCode) Else ; ------------------------------------------------- ; PARENT PAYLOAD - code executed for parent process ; ------------------------------------------------- $sINI=@SCRIPTDIR+"\YAPP_"+@PID+".ini" Del $sINI IfNot IsDeclared($sHostList) ANDNot IsDeclared($sFile) "You must supply a comma seperated list of hosts to ping or a file containing a list"+@CRLF " Usage: kix32 "+@SCRIPTNAME+" [$$DEBUG=0|1] [$$iMaxChild=20] $$sHostList=hostname[,hostname...] $$sFile=path_to_file"+@CRLF Exit 0 EndIf IfNot IsDeclared($sHostLIst) Global $sHostList EndIf If IsDeclared($sFile) Dim $fhFile $fhFile=FreeFileHandle() If Open($fhFile,$sFile) "Cannot open "+$sFile+" for reading"+@CRLF Exit 1 EndIf $sHost=ReadLine($fhFile) WhileNot @ERROR $sHostList=$sHostList+","+$sHost $sHost=ReadLine($fhFile) Loop EndIf $sHostList=Split($sHostList,",") While $iChildActive OR $iLastHost<UBound($sHostList) If ($iChildActive<$iMaxChild) AND $iLastHost<UBound($sHostList) $iLastHost=$iLastHost+1 $sHost=$sHostList[$iLastHost] If $sHost myDEBUG("Starting "+$sHost) $=WriteProfileString($sINI,$sHost,"PID","Pending") $=WriteProfileString($sINI,$sHost,"StartTicks",@TICKS) RUN @SCRIPTEXE+" "+@SCRIPTNAME+" $$iParentPID="+@PID+" $$sHostList="+$sHost $iChildActive=$iChildActive+1 $sActiveList=$sActiveList+","+$sHost EndIf EndIf myDEBUG("Active list: "+$sActiveList) $sNewList="" For Each $sHost in Split(SubStr($sActiveList,2),",") If $sHost $vStatusCode=ReadProfileString($sINI,$sHost,"StatusCode") $vResponseTime=ReadProfileString($sINI,$sHost,"ResponseTime") $vChildPID=ReadProfileString($sINI,$sHost,"PID") $iStartTicks=ReadProfileString($sINI,$sHost,"StartTicks") myDEBUG($sHost+": Status "+$vStatusCode) myDEBUG($sHost+": Response "+$vResponseTime) myDEBUG($sHost+": PID "+$vChildPID) Select Case $vChildPID="Completed" ; No action for completed processes Case $vChildPID="Pending" AND (@TICKS-$iStartTicks)>$iMaxTicks Color $sErrorColour "Child process for "+$sHost+" has failed to start in time" Color $sNormColour @CRLF $iChildActive=$iChildActive-1 $=WriteProfileString($sINI,$sHost,"PID","Completed") Case $vChildPID<>"0" AND $vStatusCode<>"" $=WriteProfileString($sINI,$sHost,"PID","Completed") $iChildActive=$iChildActive-1 If $vStatusCode="0" $sColour=$aiColour[0] For $ = 0 To UBound($aiTime) If CINT($vResponseTime)>=$aiTime[$] $sColour=$aiColour[$] EndIf Next Color $sColour $sHost+" responded in "+$vResponseTime+" ms" Color $sNormColour @CRLF Else Color $sErrorColour $sHost+" failed to respond, error code is "+$vStatusCode Color $sNormColour @CRLF EndIf Case "StillRunningOrPending" $sNewList=$sNewList+","+$sHost EndSelect EndIf Next $sActiveList=$sNewList If $iChildActive Sleep($iInterval) EndIf Loop Del $sINI EndIf
Exit 0
Function wmiPing($sHost,Optional $Timeout) Dim $sQuery,$oWMI,$oItem,$cItems $sQuery = "Select ResponseTime,StatusCode From Win32_PingStatus where Address='" + $sHost + "'" If $Timeout $sQuery = $sQuery + " And TimeOut=" + $Timeout EndIf $oWMI = GetObject("winmgmts:root\cimv2") $cItems = $oWMI.ExecQuery($sQuery) For Each $oItem In $cItems If (VarTypeName($oItem.StatusCode) = 'Null') $wmiPing=1 Exit 1 EndIf If $oItem.StatusCode $wmiPing=$oItem.StatusCode Exit 1 EndIf $wmiPing = $oItem.ResponseTime Next Exit 0 EndFunction
Function myDEBUG($s) If $DEBUG "DEBUG: "+$s+@CRLF EndIf EndFunction ; vim: sw=4 ts=4
ok, I thought the "While $Count < 16 ; max of 15 " part was for the max number of processes. the problem I'm having, when I remove this part, is that only 10 processes run. even when the other processes stop, no new processes are started.
