Page 1 of 1 1
Topic Options
#197237 - 2009-12-30 04:47 PM For Allen ;)
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4396
Loc: New Jersey
Since Allen is bored and my brain is starting to hurt, I'll post this as a challenge...

I have a Kix script that is running as a service on 160+ servers across the enterprise. This script runs in two modes:
1. called without args, it assumes the service role as a scheduler/dispatcher, running in an infinite loop and dispatching commands after specific time periods elapse. At the appointed time, the script calls a copy of itself via RUN, with an argument that identifies one of 8 specific tasks to perform.
2. called with an arg, it performs one of the specific duties of the service and exits. The result/status is checked by the service.

This method allows all of the code to reside in a single script, making maintenance easy. It also eliminates a common problem with schedule/dispatch logic where a sub-task can run long enough to prevent another subtask from running it its timeslot. The dispatcher invokes subtasks on 2-minute Odd, 2-minute Even, 6-minute, hourly on each of four 15-minute periods, and 4-hour periods. Subtasks take from sub-second to multi-hours to complete, and logic is present to prevent multiple copies of sub-tasks from running as well as terminate tasks that run more than 6 hours. (so far, ~90 minutes has been the longest-running sub-task, so this time-limit value may be reduced.)

So - here's what I'm struggling with.. I want the script to update itself.

The service runs constantly, but generally has little work to do overnight. It synchronizes a directory structure over the WAN from an upstream peer determined by DNS SRV records. It also consolidates individual local logs and then pushes the consolidation logs upstream, where they ultimately arrive at the master server.

The script can check its peer to determine if a newer version of the script is available. It can copy that script over itself while it is running. The child processes will immediately begin using the new copy. To avoid any compatability issues with the child process returning data to the parent, I want the scheduler to restart itself as soon as it overwrites the script file. If this were Unix, I'd simply Exec the new copy, replacing the current running process with the latest script.

Restarting the service would accomplish this, but I have two challenges. If there are child processes running, they will die.. so I need to perform the "update & restart" only when no sub processes are active. This is easy enough to detect. The one I'm struggling with is how to restart the service without some external dependency. RUN "XNET RESTART KixService" would work, but would place the dependency on the XNET.EXE command. The UDFs can start/stop the service, but once I stop, I can't restart. \:o None of the UDFs seem to support a restart.

So - in a nutshell - I'm looking for some ideas on how I can restart an actively running script via the script itself, thus loading the newest version.

Thanks!

Glenn
_________________________
Actually I am a Rocket Scientist! \:D

Top
#197238 - 2009-12-30 05:00 PM Re: For Allen ;) [Re: Glenn Barnas]
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
When I've done this in the past I've designed the script so that it writes it's state to an INI file, so if (for example) the process is killed off when it next starts it loads it's state information from the INI file and continues.

Another technique is to simple dump the variables / state information out to a KiXtart mini-script that can then be CALLed to reinitialise the process. This works well as long as you don't have really complex variables like objects.

Looking at what you've got I'd be inclined to simply start a second dispatcher process which manages new tasks and leaves the old one managing extant tasks. When the old tasks all complete the original dispatcher can either exit or just go into hibernation.

Another, maybe simpler solution is to have your initially launched script process to be *really* dumb. The only thing it does is to launch the dispatcher and then wait. When the dispatcher exits, your dumb process relaunches it. This will allow you to keep the dumb bootstrap process so simple that you shouldn't ever need to change it, while allowing you to refresh the running dispatcher process with new updates.

Top
#197240 - 2009-12-30 07:40 PM Re: For Allen ;) [Re: Richard H.]
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4396
Loc: New Jersey
Fake Allen (Richard), ;\)

I maintain state of all child processes, so picking a good time to update (when no children are present) is not a problem. The scheduler can set the "GoodUpdateTime" flag based on knowing that there is a period where no events are needed. Another scheduler process detects the presence of a new version and sets the UpdatePending flag. All I need to do is
 Code:
If Not $ACTIVE And $GoodUpdateTime And $UpdatePending
  Del 'kixservice.kix'
  Move 'kixservice.new' 'kixservice.kix'
  Run 'XNET RESTART kixservice'
EndIf

This works, but I'm trying to remove the XNET dependency. If I spawn a child script, it usually dies as soon as the service stops.

Your final "dumb" solution would work pretty effectively for this - the only downside is adding the extra script to the package. This could be as simple as
 Code:
While 1
  Shell 'kix32.exe kixservice.kix'
Loop
which would launch and wait for the service to exit and just restart it if it ever ended. I'd have to test this to insure that the child process stops when the parent is terminated by a service stop command. This might be as simple as replacing the run command above with EXIT 0.

I'm also going to investigate enabling "Recovery" on the service. Thus, if the service "fails" the O/S will simply restart it after 1 minute. I don't have anything so critical that being stopped for a minute would be an issue.

Thanks,

Glenn
_________________________
Actually I am a Rocket Scientist! \:D

Top
#197241 - 2009-12-30 09:33 PM Re: For Allen ;) [Re: Glenn Barnas]
Allen Administrator Online   shocked
KiX Supporter
*****

Registered: 2003-04-19
Posts: 4545
Loc: USA
Real Allen Here... just getting back.... dang it, missed another one.
Top
#197242 - 2009-12-30 09:46 PM Re: For Allen ;) [Re: Allen]
Allen Administrator Online   shocked
KiX Supporter
*****

Registered: 2003-04-19
Posts: 4545
Loc: USA
I've got an idea, but need to check into it. The other more obvious way would be to create the script on the fly, ie a temp.kix file and then run it... not much different than what you have above though.

And just for my own clarification, when the service stops, its killing the kix32.exe processes, which means the script stops, and then you can't restart the process?

Top
#197248 - 2009-12-31 12:59 AM Re: For Allen ;) [Re: Allen]
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4396
Loc: New Jersey
Hey - ya snooze, ya looze! ;\)

I'm using SrvAny, as I documented in the Script Vault article "Running Kix as a Service". SrvAny.exe controls the process that is defined. WKix32.exe is the application to run, and the script (and any args) is the parameter. Stopping SrvAny (the "KixService" service) terminates WKix32, which stops the script itself. That's why I was feeling challenged about restarting the service / script.

I'm going to test Fake Allen's solution tonight - I'm pretty sure that stopping it will stop the child, which is actually the main script. If that works, I'm home-free for a self-updating service.

The service itself is fairly simple, but performs many different scheduled tasks. All times are relative to service start and not time of day, with two exceptions.
  • Midnight - the log file name is changed, effectively resulting in a daily log rotation, and any log more than 30 days old is removed.
  • 5am - a system-wide health check is performed by the Master Cylind - er - server. All SRV records are retrieved from DNS, each subordinate server is queried to insure it is responding, logging, and has a current replication of the data folder.
  • Every hour:
    :15 after startup - If Primary, check for changes and set MODIFIED flag
    :30 after startup - Update the local product index
    :45 after startup - If Primary, validate the content, remove any unauthorised files
    :60 after startup - If not primary, determine if a Folder Sync is needed
  • Every 6 minutes - Perform log consolodation, forwarding logs upstream to master server
  • Every 2 minutes - If Primary or Secondary, check for sync requests and queue up to 20 sync processes.
With this in place, we can make a change to the primary server in the HQ location, and within 2 hours, that change is replicated to 160+ remote sites. Any process performed at the remote site is logged, and the logs are pushed upstream and consolidated/sorted on the primary server, again within a few hours. The sync time depends on the hierarchy - secondry and slaves that use the primary as their peer sync up within no more than 2 hours, while servers that sync with a secondary server can take up to 3 hours to sync. For our application, this is very fast, since the design spec was complete propogation within 24 hours.

Every server is listed in DNS with a SRV record, which identifies the hostname, host IP, subnet served, Class (primary/secondary/slave) and service communication port. With a simple DNS query, I can identify every server that provides this service, the server that provides service to my subnet, and - if the script is running on a server, what class of service it should provide.

I'll let you know how the "dumb" process works out, but am still open to ideas.

Thanks

Glenn
_________________________
Actually I am a Rocket Scientist! \:D

Top
#197249 - 2009-12-31 01:47 AM Re: For Allen ;) [Re: Glenn Barnas]
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4396
Loc: New Jersey
OK - some joy, but some additional complexity.

I can keep the sub-process running as illulstrated above, allowing the sub-process to detect and perform a self-update and then simply exit.

The problem is that when the parent (dumb) process is terminated, the child (smart) process continues to run (which seems unusual to me). I modified the concept by passing the PID of the parent to the child. During each iteration of the loop, it checks to see if its parent is still running, and if it isn't, it dies.

Here's the code I'm using to test:
Dumb Service Manager (SVC.KIX):
 Code:
Break On
While 1
  Shell 'WKix32.exe service.kix $PID=' + @PID
Loop
"Smart" Service Code (SERVICE.KIX):
 Code:
Break On

If Not IsDeclared($PID)
  Global $PID
  'NoPid - cannot run!' ? 
  Quit 87
EndIf


While 1		; start of main service loop

  ; insure that our parent process is running - we die if it isn't
  $Rc = WmiProcessList($PID)
  If @ERROR Exit 0 EndIf


  ; test for service code update
  If Exist(@SCRIPTDIR + '\service.new')
    If Exist(@SCRIPTDIR + '\service.old')
      Del @SCRIPTDIR + '\service.old'
    EndIf
    Move @SCRIPTDIR + '\service.kix' @SCRIPTDIR + '\service.old'
    Move @SCRIPTDIR + '\service.new' @SCRIPTDIR + '\service.kix'
    'Updated!' ?
    Exit 0
  EndIf

  ; Identify who we are
  'Version 1' ?

  Sleep 2	; pretend we're doing serious work here...

Loop		; end of main service loop
without the WMIProcessList and TimeDiff UDFs.

To test the service self-update, create a copy of service.kix (service.txt) and edit the Version number.

First test - run SVC.KIX - second window should open with "VERSION 1" message repeating. Terminate SVC.KIX and the second window should close.

Second test - run SVC.KIX - second window should open with "VERSION 1" message repeating. Copy SERVICE.XXX to SERVICE.NEW. "Updating" message should flash, window should close, new window should open with "VERSION 2" message repeating. Folder should contain a SERVICE.OLD file.

The SERVICE.KIX needs the WMIProcessList and TimeDiff UDFs from my web site to run.



Glenn
_________________________
Actually I am a Rocket Scientist! \:D

Top
#197250 - 2009-12-31 02:38 AM Re: For Allen ;) [Re: Glenn Barnas]
Sealeopard Offline
KiX Master
*****

Registered: 2001-04-25
Posts: 11164
Loc: Boston, MA, USA
Why don't you schedule a restart via Task Scheduler as part of the upgrade process?
_________________________
There are two types of vessels, submarines and targets.

Top
#197252 - 2009-12-31 09:34 AM Re: For Allen ;) [Re: Sealeopard]
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
 Quote:
Your final "dumb" solution would work pretty effectively for this - the only downside is adding the extra script to the package


No, you don't need an extra script.

You can keep all of it in the same script - you just need to nail down the bootstrap code so that it is so simple that you will never need to change it.

KiXtart does not lock the script file so overwriting it is not going to cause any problems.

The fact that the process executing the bootstrap code may have an out of date copy of the script in memory is not relevant - the bootstrap code is never going to change, so the process can continue to run with the stale copy.

If you want to make it *really* bullet proof you could make your bootstrap process regularly check the registry for updated code to Execute() instead of it's local in-memory copy of the code. Then, when a new version of the script is deployed it checks the version of the running parent when the next child is invoked and if the parent is out of date it writes the code patch to the registry so that the bootstrap process reads it in the next iteration. The bootstrap script removes the code from the registry at startup of course. I don't think that this additional complexity is needed if you get a good design for the bootstrap code.

Top
#197257 - 2009-12-31 04:05 PM Re: For Allen ;) [Re: Richard H.]
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4396
Loc: New Jersey
I'm clearly missing something with your idea of bootstrap code embedded in the main script. The service code is 3400+ lines with 23 UDFs - any piece could be updated.

What I see using embedded code is the same problem I've had all along - how do I get the service to restart itself without using external commands? This is the only challenge I have - the detectionof new code and ultimate replacement of the code works fine.

Using a small bootloader to maintain the primary service works well, and does not need any more complexity than verifying that the parent PID is still active. The WMIProcessList UDF is even already part of the service core code. Of course, if the child processes would die when the parent died, it would eliminate the need to monitor the parent bootstrap process.

Glenn
_________________________
Actually I am a Rocket Scientist! \:D

Top
#197261 - 2009-12-31 07:18 PM Re: For Allen ;) [Re: Glenn Barnas]
NTDOC Administrator Offline
Administrator
*****

Registered: 2000-07-28
Posts: 11623
Loc: CA
What I think can be done is just add some code to your main script. In that script it looks for an entry in the registry, or reads a .ini file for a change. Say maybe once an hour. Then IF it finds an entry of 1 for NEW CODE - it then creates a scheduled task that will run in five minutes. That scheduled task then restarts the process and resets the registry or .ini back to 0 and deletes the newly created task.

You of course externally change either the registry, ini file or some shared ini file across the network.

Top
#197262 - 2009-12-31 07:48 PM Re: For Allen ;) [Re: NTDOC]
Allen Administrator Online   shocked
KiX Supporter
*****

Registered: 2003-04-19
Posts: 4545
Loc: USA
Here is what I was looking for... old thread but might have some value here: GuardDog() - http://www.kixtart.org/forums/ubbthreads...true#Post129944

I wish I had the the time to write the code I was thinking but I just don't have time right now. The gist was, you would fire up your service and it would run the script and another kix process using the guarddog monitoring it. If the script/pid ended, it would autorestart it.



Edited by Allen (2009-12-31 07:49 PM)

Top
#197265 - 2010-01-04 03:20 PM Re: For Allen ;) [Re: Glenn Barnas]
Richard H. Administrator Offline
Administrator
*****

Registered: 2000-01-24
Posts: 4946
Loc: Leatherhead, Surrey, UK
 Quote:
I'm clearly missing something with your idea of bootstrap code embedded in the main script. The service code is 3400+ lines with 23 UDFs - any piece could be updated.


Maybe I'm not explaining myself very well. Here is a short example:
 Code:
BREAK ON

; Variable definition
$sKey_BootStrap="HKLM\foo\bar"
$sVal_BootStrap="BootStrapCode"
$sBootStrapCode='Shell @@SCRIPTEXE+" "+@@SCRIPTDIR+"/"+@@SCRIPTNAME $$PPID=Dispatcher"'

; Write bootstrap code
; Every instance of this script will re-write the bootstrap code.
; This means that when a new version of the script is release the
; bootstrap process will automatically be updated with new code.

$=WriteValue($sKey_BootStrap,$sVal_BootStrap,$sBootStrapCode))

; BOOTSTRAP SECTION START-------------------------------------------
; If this is the boostrap instance of the script
; just keep running the bootstrap code.

If IsDeclared($PPID)=0
	While TRUE
		$=Execute(ReadValue($sKey_BootStrap,$sVal_BootStrap))
	Loop
	Exit 0 /* NotReached */
EndIf
; BOOTSTRAP SECTION END---------------------------------------------

;
; Main script.
; If any of the script below is changed the bootstrap process will be unaffected.

If $PPID="Dispatcher"
	; This is a primary dispatcher process which will create the workers
Else
	; This is a worker process
EndIf

Exit 0


The process works like this:
  • The primary (boostrap instance) of the script is started.
  • It writes the bootstrap payload code to the registry.
  • As the variable $PPID is not set it starts an infinite loop which reads the payload code from the registry and then executes it.
  • The payload code just starts another instance of the script setting variable $PPID to "Dispatcher"
  • The "Dispatcher" instance of the script starts creating the real worker processes.


So to update running code all that you need to do is:
  1. Overwrite the script with the new version.
  2. Wait for the worker processes to complete.
  3. Optionally kick off a dummy no-op worker.
  4. Exit the dispatcher script


When the dispatcher script exits it is immediately restarted by the bootstrap prcess, only now of course it will pick up the new version of the file so your new code is activated.

The reason for triggering a dummy worker before exiting the dispatcher is that the dummy worker will update the bootstrap payload code in the registry with whatever is in your new file. This means that even your initial boostrap process will get the updated code, the *only* thing you cannot change is the basic While Loop construct.

Because the bootstrap code is so simple and self-updating it isn't affected by you changing anything else in the file. The bootstrap process will continue to run with the stale version loaded in memory, but that doesn't matter because it doesn't use any of the changeable code. Any new instance of KiXtart which is launched will pick up the new file and UDF's contained.

Top
#197433 - 2010-01-19 04:01 PM Re: For Allen ;) [Re: Richard H.]
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4396
Loc: New Jersey
OK - here's the basis of what I wound up doing:
 Code:
Break On

; Example code to run a script as a service with the following features:
;  * Manager process insures that the dispatch process keeps running
;  * Manager process is controlled as a service by SrvAny.exe
;  * Manager is invoked when the script is called without a $PID argument
;  * Dispatcher process invokes a Processor to run specific tasks
;  * Dispatcher runs in an infinite loop, tracks the PID of the manager
;    and terminates if the Manager process terminates.
;  * Dispatcher checks for a new version of the script and copies/exits for
;    a self-updating feature. The Manager then restarts using the new script
;  * Processor has a $PID of 0 and $ARG defining the specific task to perform
;  * Processor performs the task and exits
;  * Processor is invoked via RUN to run independently, allowing many Procesor
;    events to occur. The code can limit the number of similar Processor tasks
;    by tracking the PIDs.

Dim $Rc


; Set desired program options - NoVarsInStrings is required
$Rc = SetOption('NoVarsInStrings', 'On')
$Rc = SetOption('Explicit', 'On')



; If $PID is not specified, run as a front-end process
If Not IsDeclared($PID)
  Global $PID
  'Service front-end started. PID=' @PID ?

  ; Loop forever until terminated by external triggger (Service stop)
  ; no application-specific initialization is required.
  ; run the script with a PID arg so it runs as the back-end dispatcher
  ; It will terminate itself if the front-end PID dies. It can also 
  ; terminate due to finding a newer version - it will self-update and exit,
  ; and this front-end will restart it.
  While 1
    Shell '%COMSPEC% /c Kix32.exe ' + @SCRIPTNAME + ' $PID=' + @PID
  Loop
EndIf



; perform application initialization common to the Dispatch service and all Processor sub-tasks
Dim $A						; random arg value
Dim $I						; counter
'common init' ?
SRnd(@TICKS)



; if $ARG is declared, then this is a PROCESSOR sub-task running independent of the service
; Run the specific task and exit.
If IsDeclared($ARG)
  ; Use a Select/Case to decide what to do
  ; for now, pretend we're doing something useful...
  'Specific subtask: ' $ARG ?
  Sleep 5
  Exit 0
EndIf



; This is the service back-end, which performs additional init, overall management, and
; scheduled dispatching of sub-tasks. It monitors the front-end process PID and terminates
; if it dies. It also checks for a newer version, copies it over the current version and
; exits. The front-end service will restart this, using the new code.
'Starting service back-end. PID=' @PID ?
$I = 0
While 1		; start of main service loop

  ; insure that our parent process is running - we die if it isn't
  $Rc = WmiProcessList($PID)
  If @ERROR 'Terminate' ? Exit 0 EndIf

  ; test for service code update
  If Exist(@SCRIPTDIR + '\service.new')
    If Exist(@SCRIPTDIR + '\service.old')
      Del @SCRIPTDIR + '\service.old'
    EndIf
    Move @SCRIPTDIR + '\service.kix' @SCRIPTDIR + '\service.old'
    Move @SCRIPTDIR + '\service.new' @SCRIPTDIR + '\service.kix'
    'Updated!' ?
    Exit 0
  EndIf

  ; Identify who we are
  ; change this to "version 2" and save as "service.new" to test the auto-update feature
  'Version 1' ?

  Sleep 0.5	; pretend we're doing serious work here...
  $I = $I + 1
  ; trigger a sub-task - would normally be done on a timer of some sort, with different
  ; Processor events occurring at different times. For now, just provide a random character
  ; as the $ARG argument
  If $I = 20	; run a subtask every 10 secondss
    $I = 0
    $A = Int(Rnd(8)) + 1
    Run '%COMSPEC% /c wKix32.exe ' + @SCRIPTNAME + ' $PID=' + 0 + ' $ARG="' + Chr(64 + $A) + '"'
  EndIf

Loop		; end of main service loop
This is just a simple example, but it does everything I need - one script, runs as the manager, dispatcher, and processor depending on how it is invoked. Stopping the service terminates the manager. The Dispatcher sees the manager PID disappear and it terminates. If the Dispatcher sees a code update, it copies the new file over the current file and exits, where the manager simply restarts it using the new code. Any child Processor tasks from that point will use the new code version.

Thanks for the feedback!

Glenn
_________________________
Actually I am a Rocket Scientist! \:D

Top
#197517 - 2010-01-24 05:57 PM Re: For Allen ;) [Re: Glenn Barnas]
Arend_ Moderator Offline
MM club member
*****

Registered: 2005-01-17
Posts: 1894
Loc: Hilversum, The Netherlands
 Originally Posted By: Glenn Barnas
The UDFs can start/stop the service, but once I stop, I can't restart. \:o None of the UDFs seem to support a restart.


I've run into the same problem using a "Live Update" feature of one of my scripts. My Solution was simple. I let it Write a batch file that contains two lines:
 Code:
Net stop Service
Net Start Service

And let the Live Update invoke it and as soon as the scripts starts it checks for that batch file and deletes it.

Top
#197519 - 2010-01-24 10:25 PM Re: For Allen ;) [Re: Arend_]
Glenn Barnas Administrator Offline
KiX Supporter
*****

Registered: 2003-01-28
Posts: 4396
Loc: New Jersey
Considered something like that, but since this is distributed across a few hundred servers, I wanted to try and keep it in one script. The extra code was minimal, since I already had the UDFs to get the process list.

Glenn
_________________________
Actually I am a Rocket Scientist! \:D

Top
Page 1 of 1 1


Moderator:  Jochen, Allen, Radimus, Glenn Barnas, ShaneEP, Ruud van Velsen, Arend_, Mart 
Hop to:
Shout Box

Who's Online
1 registered (Allen) and 466 anonymous users online.
Newest Members
gespanntleuchten, DaveatAdvanced, Paulo_Alves, UsTaaa, xxJJxx
17864 Registered Users

Generated in 0.05 seconds in which 0.017 seconds were spent on a total of 13 queries. Zlib compression enabled.

Search the board with:
superb Board Search
or try with google:
Google
Web kixtart.org