Danski
(Fresh Scripter)
2014-05-05 01:02 AM
KiXtart stalling GPClient for 10 minutes on logon

Hello,

I work for an enterprise (USA based) at a branch office (New Zealand based, ~280ms ping) that is having an intermittent logon problem only at our site.
I've never written a KiXtart script and only found out about kix when it was called up by a group policy boot script.

When the command to run kix is called a 600 second delay occurs before the gpclient calls timeout. The problem only happens when a foreign domain controller is used to serve up the logon scripts.

Logon Script that calls kix:
 Code:
adlogon.cmd

set HERE=%0\..
set ADLOGFN=%TEMP%\adlogon_%USERNAME%
set ADLOGEXT=.log
set KIXLOG=%ADLOGFN%_kix%ADLOGEXT%

...
echo  %HERE%\kix32.exe "%HERE%\adlogon.kix" $adlcdir="%HERE%" "$logfile=%KIXLOG%"
%HERE%\kix32.exe "%HERE%\adlogon.kix" $adlcdir="%HERE%" "$logfile=%KIXLOG%"
...



We have put an echo above the line that calls kix32 with the exact same command and it outputs the following:
\\<LOCALDC>\netlogon\ADLogon\adlogon.cmd\..\kix32.exe "\\<LOCALDC>\netlogon\ADLogon\adlogon.cmd\..\adlogon.kix" $adlcdir="\\<LOCALDC>\netlogon\ADLogon\adlogon.cmd\.." "$logfile=C:\Temp\adlogon_<USERNAME>_kix.log"

As you can see the kix program, script and run dir are all on the DC.
The kix script that runs starts off with:

 Code:
adlogon.kix

;::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
;This is the logon script for the Active Directory <DOMAIN> domain
;::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
IF ISDECLARED($logfile)
  IF REDIRECTOUTPUT($logfile,1) <>0
    REDIRECTOUTPUT()
  ENDIF
ENDIF
? "@DATE @TIME,ADLogon Kixtart Started"
BREAK ON
$RC = LogEvent( 4 , 5901 , "AD Logon script started: "+@LDOMAIN+"\"+@USERID+" at site: "+@SITE+" by "+@LSERVER,"","KIX32")
...


The 3rd last line is output to the log file and the last line is output to the windows event viewer.
When a foreign DC is used these two lines are not run but the echo to output the command to call kix is run from the adlogon.cmd.

If you have any idea or suggested problem solving tips as to how or why kix doesn't run or stops please let me know.

Version: KiXtart 2001 4.22
Windows 7 AMD64 Enterprise SP1

Occurs on all Win7 pcs on this site.

Thank you.


AllenAdministrator
(KiX Supporter)
2014-05-05 03:06 AM
Re: KiXtart stalling GPClient for 10 minutes on logon

Are you saying the hang is happening in those 10 or so lines of the kix script? Aside from having macros in one of your strings, I really don't see anything jumping out as a problem. You will likely have to post the entire script, or troubleshoot down to the line causing the issue.

Also, in your batch file, why do all the paths have \..\ in them?

Have you considered upgrading the kix32.exe. The current version is 4.64, and 4.22 is like 10 years old.


Danski
(Fresh Scripter)
2014-05-05 03:58 AM
Re: KiXtart stalling GPClient for 10 minutes on logon

Yes, the hang is happening between the echo out of the command to run kix32.exe and the ? "@DATE @TIME,ADLogon Kixtart Started" line.
This has been established due to the echo line being present in the log file as the last entry on a failure.

I don't know why the paths have \..\ in them. The local env var is set to HERE=%0\.. where %0\ picks up the directory that the calling script is located.

I must stress that this is not my script/domain/deployment. It's just become my problem as the local IT guy. These scripts were probably written when version 4.22 was the latest and they work well on 30k computers a day but when we try to fetch and run them from a reasonably slow WAN link on a foreign DC with 2-300ms of ping there becomes a problem.

My best bet is on the network failing to download the kix32 and adlogon.kix but I wanted to know if the community had experienced anything like this before.


AllenAdministrator
(KiX Supporter)
2014-05-05 05:17 AM
Re: KiXtart stalling GPClient for 10 minutes on logon

I don't think Windows 7 has a C:\Temp directory by default, instead it is stored in %appdata%. It's possible this folder is not there and causing some weird error with the redirect.

Can you modify any of the scripts?




LonkeroAdministrator
(KiX Master Guru)
2014-05-05 05:32 AM
Re: KiXtart stalling GPClient for 10 minutes on logon

lol.
Allen, nobody is calling c:\temp.

Danski, your issue indeed seems to be exremely slow network connection. clearly if in 10 minutes, you are unable to download some 300kB worth of data, there is something really badly wrong with the network.

although, I would also check if your AV is set to scan network drives. some AV's are so stupid that they will actually scan the entire folder from which you are trying to launch something. so, instead of you just loading kix32.exe, you are loading the whole netlogon folder for the AV to scan and that's what's causing the time out.

there are workarounds for this slow behavior and lot of examples done in the 200x era, when slow/costly wan links were the norm. but before those workarounds, you should check if netlogon share is excluded from antivirus scanning.


AllenAdministrator
(KiX Supporter)
2014-05-05 12:12 PM
Re: KiXtart stalling GPClient for 10 minutes on logon

See $logfile lonk

LonkeroAdministrator
(KiX Master Guru)
2014-05-05 01:38 PM
Re: KiXtart stalling GPClient for 10 minutes on logon

I am not blind enough to get c:\temp from %temp% ;\)
but I was blind enough to not see that in his echo output he has c:\temp. The script is correctly calling for %temp%.


AllenAdministrator
(KiX Supporter)
2014-05-05 03:54 PM
Re: KiXtart stalling GPClient for 10 minutes on logon

Please post your kix script if you want anything more than guesses.

LonkeroAdministrator
(KiX Master Guru)
2014-05-05 04:28 PM
Re: KiXtart stalling GPClient for 10 minutes on logon

I wonder who did the scripts. the batch looks professional at first glance but if you think about it, it is wasteful and counter-productive in design. it's like someone tried to make as many points of failure as possible and slow things down as much as possible without actually introducing sleeps or pauses.


ShaneEP
(MM club member)
2014-05-05 06:49 PM
Re: KiXtart stalling GPClient for 10 minutes on logon

Im confused by the echo output...Why is the command file name adlogon.cmd in the path of kix32.exe? Is there a subfolder named that?

"\\LOCALDC\netlogon\ADLogon\adlogon.cmd\..\kix32.exe"

Or is that why the \..\ is there, to drop it back a level?....I think I'm too young for cmd files.


LonkeroAdministrator
(KiX Master Guru)
2014-05-05 07:42 PM
Re: KiXtart stalling GPClient for 10 minutes on logon

lol. indeed %0 tells the commandline used to launch the process (minus arguments). %0\..\ will give you the path where the executable resides.

Danski
(Fresh Scripter)
2014-05-05 11:45 PM
Re: KiXtart stalling GPClient for 10 minutes on logon

\\~DC~\NETLOGON\ADLogon directory has all of the files that are required.
adlogon.cmd
adlogon.kix
KIX32.exe

Due to this being an enterprise deployment I do have to be careful of what I release here. I don't think pasting the script in its entirety would be ok or given that it doesn't hit the 6th line in the script, necessarily helpful.

I am investigating the antivirus being the problem. There seems to be no explicit exclusion of the NETLOGON folder. The product is Symantec Endpoint Protection V.12

I just want to reemphasize that the script works fine and runs in less than 1 second from the local DC.

Thank you all for your help so far.


ChristopheM
(Hey THIS is FUN)
2014-05-06 12:08 AM
Re: KiXtart stalling GPClient for 10 minutes on logon

1) to get the script path and the script name, i use the following code
set scriptdir=%~dp0
set scriptname=%~n0

%0 is the name of the current script
~ remove " at beginning and end of the name
d extracts drive
p extracts path
n extracts the name of the script without extension
if needed, x extraxt the extension

so %~dpnx0 gives the full script name.

2) have you tried to copy kix32.exe locally ? this may reduce time to launch adlogon.kix because it prevents from copying 300 KB on a WAN link (except the first time for each user of a workstation) and the AV scans locally !!!

3) i think the first parameter $adlcdir is not useful because it is the name of the kixtart script directory. in the kix script, you can use $adlcdir = @scriptdir.

to debug, this gives a script like that for adlogon.cmd :
 Code:
@echo off
setlocal
  set scriptdir=%~dp0
  set scriptname=%~n0
  
  set AdLogFilename=%temp%\%scriptname%_%username%
  set AdLogExt=.log
  set KIXLOG=%ADLOGFN%_kix%ADLOGEXT%

  if not exist "%temp%\kix32.exe" (
    rem -- copy kix32 locally if not exist --
    copy "%scriptdir%kix32.exe" "%temp%\"
  ) else (
    rem -- update kix32 locally if needed --
    replace "%scriptdir%kix32.exe" "%temp%\" /u
  )
...
  set command="%temp%\kix32.exe" "%scriptdir%%scriptname%.kix" $logfile="%kixlog%"
  echo command : %command%
  echo before kix : %date% %time%
  %command%
endlocal

for the beginning of kix script :
 Code:
"beginning kix : " @Date " " @Time ?

break off
$adlcdir = @scriptdir
...
When the code is debugged, you can remove comment and echo lines.


ShaneEP
(MM club member)
2014-05-06 12:16 AM
Re: KiXtart stalling GPClient for 10 minutes on logon

Should this line...
 Code:
set KIXLOG=%ADLOGFN%_kix%ADLOGEXT%
be...
 Code:
set KIXLOG=%AdLogFilename%_kix%ADLOGEXT%
?


LonkeroAdministrator
(KiX Master Guru)
2014-05-06 05:06 AM
Re: KiXtart stalling GPClient for 10 minutes on logon

nope. %adlogfilename% is not defined. %adlogfn% is.

the author might correct me on this, but I am rather confident that he has no rights to modify the scripts. as such, the only thing he can do is try to troubleshoot why it is acting funny.

And in any case, asking him to modify perfectly working code on a whim is far from best practice. especially when he just admitted he has not even heard of kixtart before seeing that script.

Christophe, when I mentioned earlier about workarounds, the local copy is exactly what I meant. and since most of the time the local DC is working and up, there is most likely never going to be kixtart executable download happening over the wan link. But, if my guess is right about his rights, the only thing he can do is ask the corporate to consider changes at least for his site.


ChristopheM
(Hey THIS is FUN)
2014-05-06 09:07 PM
Re: KiXtart stalling GPClient for 10 minutes on logon

you are rigth.
in my example, i forget to change this string.


Danski
(Fresh Scripter)
2014-05-09 01:37 AM
Re: KiXtart stalling GPClient for 10 minutes on logon

I have made some of the suggested changes to the adlogon.cmd

 Code:
...
set HERE=%~dp0
...
set RUNDIR=%TEMP%\ADLogon
...
:: KIX drive mappings
  echo Starting AD logon script for %USERDNSDOMAIN%...
  echo Copying KIX32/KIX Script/Mapping file to %TEMP%\ADLogon...

  :: Copy to local machine to prevent running from DC (AV issues etc)
  xcopy %HERE%* %RUNDIR%\ /Y /E /C

  echo  %RUNDIR%\kix32.exe "%RUNDIR%\adlogon.kix" $adlcdir="%RUNDIR%" "$logfile=%KIXLOG%"
  %RUNDIR%\kix32.exe "%RUNDIR%\adlogon.kix" $adlcdir="%RUNDIR%" "$logfile=%KIXLOG%"


I've emailed this script to a domain administrator that I've been working with on this issue. Now to wait and see what he comes back with.

Thank you everyone so far.


LonkeroAdministrator
(KiX Master Guru)
2014-05-09 04:13 AM
Re: KiXtart stalling GPClient for 10 minutes on logon

instead of always copying, which would not prevent the issue of slowness, you could simply copy only if the file isn't there yet.

ChristopheM
(Hey THIS is FUN)
2014-05-09 07:12 PM
Re: KiXtart stalling GPClient for 10 minutes on logon

that is exactly what i did in my code.
but a better solution is to use robocopy :
 Code:
if not exist "%rundir%\robocopy.exe" copy "%here%robocopy.exe" "%rundir%\"

"%rundir%\robocopy.exe" "%here%." "%rundir%" /s /xo

So just newer or missing files are copied locally.


Danski
(Fresh Scripter)
2014-05-11 09:56 PM
Re: KiXtart stalling GPClient for 10 minutes on logon

Following the lead of other logon scripts for the enterprise I have specifically not used /D so the scripts cannot be modified while on the local drive.

As for robocopy, using /MIR would probably be the best solution but since we still have a few XP machines lingering on the network this isn't an option.

The one problem I don't have a solution for is if someone accidentally/unwittingly dumps a large file in the \\DC\ADLOGON folder as it will copy down. The other option would be to explicitly call xcopy only on the files kix needs but this would then make the script a hidden dependency if the files were to change in future.

It's tough writing code for an enterprise.


Mart
(KiX Supporter)
2014-05-11 10:47 PM
Re: KiXtart stalling GPClient for 10 minutes on logon

 Originally Posted By: Danski

....

It's tough writing code for an enterprise.


Amen to that. There are a lot of things to take into consideration. Been there done that......uhmmm still doing it sometimes.


NTDOCAdministrator
(KiX Master)
2014-05-13 08:36 PM
Re: KiXtart stalling GPClient for 10 minutes on logon

Easier when the other admins don't know how to script so you're the only hand in the cookie jar :-)