Jul-01-2021, 11:54 PM
i do not need a parse tree result. i only want to get a list of tokens from a string of Python source code. that means breaknig up a string into the parts that make sense for working with the code. i believe this is the lexical phase of a compilation. my code will only be doing simpler things like replacing things in a certaing context such as when other tokens are present or absent. a typical tokenization might look like:
if this[x]==that[y]: where.at('this',x,y) -> ['if','this','[','x',']','==','that','[','y',']',':','where.at','(',"'this'",',','x',',','y',')',']'it does not matter if blank spaces are included or not. it does not matter if it splits around a dot or not.