I think from what i remember, ADSI does not support nested groups. I haven't had a chance to test it but have you tried just using LDAP for the query

(pseudo code)
Code:

if Getobject(AdsPathOfTheGroup).IsMember(AdsPathOfTheMachine)
  ; Exclude from SMS
else
  ; Install SMS
endif