Python Forum
i woule a way to parse a line python source like split
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
i woule a way to parse a line python source like split
#2
You can use the tokenize module to get a list of tokens from python code:
>>> f = io.BytesIO(b"a[0]=int(ab[16])+2")
>>> tokens = list(tokenize.tokenize(f.readline))
>>> pp(tokens)
[
    tokenize.TokenInfo(
        type=57,
        string='utf-8',
        start=(0, 0),
        end=(0, 0),
        line=''
    ),
    tokenize.TokenInfo(
        type=1,
        string='a',
        start=(1, 0),
        end=(1, 1),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=53,
        string='[',
        start=(1, 1),
        end=(1, 2),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=2,
        string='0',
        start=(1, 2),
        end=(1, 3),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=53,
        string=']',
        start=(1, 3),
        end=(1, 4),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=53,
        string='=',
        start=(1, 4),
        end=(1, 5),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=1,
        string='int',
        start=(1, 5),
        end=(1, 8),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=53,
        string='(',
        start=(1, 8),
        end=(1, 9),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=1,
        string='ab',
        start=(1, 9),
        end=(1, 11),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=53,
        string='[',
        start=(1, 11),
        end=(1, 12),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=2,
        string='16',
        start=(1, 12),
        end=(1, 14),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=53,
        string=']',
        start=(1, 14),
        end=(1, 15),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=53,
        string=')',
        start=(1, 15),
        end=(1, 16),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=53,
        string='+',
        start=(1, 16),
        end=(1, 17),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=2,
        string='2',
        start=(1, 17),
        end=(1, 18),
        line='a[0]=int(ab[16])+2'
    ),
    tokenize.TokenInfo(
        type=0,
        string='',
        start=(2, 0),
        end=(2, 0),
        line=''
    )
]
>>> [t.string for t in tokens]
['utf-8', 'a', '[', '0', ']', '=', 'int', '(', 'ab', '[', '16', ']', ')', '+', '2', '']
However, if you're looking to modify the code, using ast with a custom NodeTransformer might be simpler. ast will get rid of comments though, so that might be a problem.
Reply


Messages In This Thread
RE: i woule a way to parse a line python source like split - by stranac - Nov-11-2018, 06:38 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Algorithm for extracting comments from Python source code Pavel1982 6 719 Feb-28-2024, 09:52 PM
Last Post: Pavel1982
  [split] Parse Nested JSON String in Python mmm07 4 1,647 Mar-28-2023, 06:07 PM
Last Post: snippsat
  python move specific files from source to destination including duplicates mg24 3 1,163 Jan-21-2023, 04:21 AM
Last Post: deanhystad
  Python Snippets Source kucingkembar 0 677 Oct-18-2022, 12:50 AM
Last Post: kucingkembar
  python read iperf log and parse throughput jacklee26 4 2,931 Aug-27-2022, 07:04 AM
Last Post: Yoriz
  How to parse a live feed in Python? Daring_T 2 4,318 Jan-20-2022, 04:17 AM
Last Post: Daring_T
  multi-line CMD in one-line python kucingkembar 5 4,175 Jan-01-2022, 12:45 PM
Last Post: kucingkembar
  Long-term stable source to get news headlines with Python? sandufi 4 2,035 Dec-23-2021, 09:48 AM
Last Post: sandufi
  Navigating cpython source using python quazirfan 3 2,097 Aug-09-2021, 07:52 PM
Last Post: quazirfan
  Using lambdas and map() to parse substrings in a single line Drone4four 5 3,171 Sep-20-2020, 10:38 AM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020