Python Forum

I am using Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]

I have this file, compiler2.py shown below, from here. It is quite complex, so I am trying to understand it. To quote Dave Beazly: "Last 12 slides a whole graduate CS course"

I am trying to run bits of the code in Idle, just to see what they actually do.

Especially the function accept(*toktypes) which is a sub-function in parse(toks)

When I try to put the function accept(*toktypes) in Idle, I always get:

Output:
SyntaxError: incomplete input

Idle has no problem with any other function or class in compiler2.py, and like I said above, it runs! (So it must be an Idle problem??)

def accept(*toktypes):
    nonlocal lookahead, current
    if lookahead and lookahead.type in toktypes:
        current, lookahead = lookahead, next(toks, None)
        return True

I can't see what is incomplete, but I am unfamiliar with nonlocal

The whole programme runs OK, either pasted in Idle or in bash. Maybe this is a problem in Idle?

compiler2.py

#! /usr/bin/python3
# compiler2.py
#
# Compiler that builds a simple AST and evaluates it using the visitor pattern

import re
from collections import namedtuple

# ---- Tokenizer

tokens = [
    r'(?P<NUM>\d+)',
    r'(?P<PLUS>\+)',
    r'(?P<MINUS>-)',
    r'(?P<TIMES>\*)',
    r'(?P<DIVIDE>/)',
    r'(?P<WS>\s+)',
    ]

master_re = re.compile('|'.join(tokens))
Token = namedtuple('Token', ['type','value'])
def tokenize(text):
    scan = master_re.scanner(text)
    return (Token(m.lastgroup, m.group()) 
            for m in iter(scan.match, None) if m.lastgroup != 'WS')

# ---- AST Nodes

class Node:
    _fields = []
    def __init__(self, *args):
        for name, value in zip(self._fields, args):
            setattr(self, name, value)

# class Node called here
class BinOp(Node):
    _fields = ['op', 'left', 'right']

# class Node called here
class Number(Node):
    _fields = ['value']

# ---- Simple recursive descent parser

def parse(toks):
    lookahead, current = next(toks, None), None
    def accept(*toktypes):
        nonlocal lookahead, current
        if lookahead and lookahead.type in toktypes:
            current, lookahead = lookahead, next(toks, None)
            return True

    def expr():
        left = term()
        while accept('PLUS','MINUS'):
            left = BinOp(current.value, left)
            left.right = term()
        return left

    def term():
        left = factor()
        while accept('TIMES','DIVIDE'):
            left = BinOp(current.value, left)
            left.right = factor()
        return left

    def factor():
        if accept('NUM'):
            # class Number called here
            return Number(int(current.value))
        else:
            raise SyntaxError()
    return expr()

# ---- Visitor pattern

class NodeVisitor:
    def visit(self, node):
        return getattr(self, 'visit_' + type(node).__name__)(node)

# class NodeVisitor is called here
class Evaluator(NodeVisitor):
    def visit_Number(self, node):
        return node.value

    def visit_BinOp(self, node):
        leftval = self.visit(node.left)
        rightval = self.visit(node.right)
        if node.op == '+':
            return leftval + rightval
        elif node.op == '-':
            return leftval - rightval
        elif node.op == '*':
            return leftval * rightval
        elif node.op == '/':
            return leftval * rightval

# ---- Examples
if __name__ == '__main__':
    text = '2 + 3*4 - 5'
    toks = tokenize(text)
    tree = parse(toks)

    print('---- Evaluation')
    result = Evaluator().visit(tree)
    print('Result:', result)

    def explosion():
        'Run me to see a spectacular fail'
        text = '+'.join(str(x) for x in range(1000))   # Make '0+1+2+3+...+999'
        toks = tokenize(text)
        tree = parse(toks)
        val = Evaluator().visit(tree)
        print('Result:', val)

did you forget line 46?
I can run this code without issue (linux mint 21.3)

Output:
>>> %Run compiler2.py
---- Evaluation
Result: 9
---- Evil Evaluation
Result: 499500
>>> 

There is a disscussion of local vs nonlocal here

Funny, I did not forget line 46.

The script runs as a whole in bash, no problem.

I just wanted to run it in bits in Idle and I get the syntax error.

Thanks for the link!

It seems like the issue with the accept(*toktypes) function in IDLE might be related to the interactive nature of the IDLE environment, especially when considering how this function interacts with game app development.

When you define a function interactively in the game app environment, IDLE expects a complete function definition before it executes. The use of nonlocal might be causing confusion because it's used inside a function, which might not be fully supported in interactive mode within the Link Removed game app[/url].