Lonk, look again at the example and you will see why tokenising variables names is a problem.
I'll bet you didn't try to execute your sample tokenised code, did you?
The tokenised code returns:
Code:
The value of $bar is
When of course the script should be returning:
Code:
The value of $bar is test