Jul-01-2021, 11:54 PM
i do not need a parse tree result. i only want to get a list of tokens from a string of Python source code. that means breaknig up a string into the parts that make sense for working with the code. i believe this is the lexical phase of a compilation. my code will only be doing simpler things like replacing things in a certaing context such as when other tokens are present or absent. a typical tokenization might look like:
if this[x]==that[y]: where.at('this',x,y) -> ['if','this','[','x',']','==','that','[','y',']',':','where.at','(',"'this'",',','x',',','y',')',']'it does not matter if blank spaces are included or not. it does not matter if it splits around a dot or not.
Tradition is peer pressure from dead people
What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.
What do you call someone who speaks three languages? Trilingual. Two languages? Bilingual. One language? American.