parse tree vs just tokenizing

[Skaperen]

i originally thought of tokenizing in a different way, more like what command would need. i wrote some code to do that back in my C days and also way back in my assembler days. so i was thinking in those terms. i haven't needed anything like that, yet, so i had no reason to look at the tokenize module. i knew it existed but hadn't had a reason to read up on it.

well, now i have, and not only is it a solution to my immediate need, but it looks like something i can do many things with, including a Python oriented editor where i can have string substitution be applied to specific language parts. for example, change "foo" to "bar" in names, not in string literals.

i'm playing around, tonight.