Nested Python functions (Dan Bader's book)

[Drone4four]

Check out this script:

def get_speak_func(volume):
   def whisper(text):
       return text.lower() + '...'
   def yell(text):
       return text.upper() + '!'
   if volume > 0.5:
       return yell
   else:
       return whisper

print(get_speak_func(0.3))
print(get_speak_func(0.7))
speak_func = get_speak_func(0.7)
print(speak_func)
print(speak_func('Hello'))

Here is the output:

Output:
<function get_speak_func.<locals>.whisper at 0x7ff195cb9e50>
<function get_speak_func.<locals>.yell at 0x7ff195cb9ee0>
<function get_speak_func.<locals>.yell at 0x7ff195cb9e50>
HELLO!

In the above code snippet, what I do understand is this: the get_speak_func() takes in a parameter on a scale of 0.0 to 1.0 which is a measure of volume. Nested within that function are two lesser functions which take in some text and either return the text as lower case or upper case. If the volume parameter is 0.5 or greater, then under this condition, the yell() function is triggered and the text entered is returned to the parent function as upper case. In all other cases, the whisper() function is triggered and returned. One advantage that this function offers is that it accepts a behaviour as an argument but also returns a behaviour.

What I don’t understand about the above code snippet is how (or why) Python manages to return a value at line 3 for example when no text data is passed in. When get_speak_func() is called at line 11 with 0.3 as a Float value when volume is passed in, Python proceeds to the next line where the whisper() function is defined taking in a text string variable which is not declared anywhere. In this situation, my initial inclination is to expect Python to throw ValueError because no text is passed in. Yet as you can see in the output, the function is proven to exist at a location in memory (as it appears in the output).

Only at line 15 is there a string passed in as the text variable as input. That makes a little bit more sense to me however what is throwing me off now is this: the original function definition handles two variables volume and text but they only work if you call the function two separate times. Python won’t accept them if you pass them in together:

print(speak_func(0.1, 'Good Bye'))
Traceback (most recent call last):
  File "/home/<user>/dev/projects/python/2018-and-2020/rpf.py", line 16, in <module>
    print(get_speak_func(0.1, 'Good Bye'))
TypeError: get_speak_func() takes 1 positional argument but 2 were given

Why does Python require nested functions to be called separately, with a new parameter each time? How does Python do this? Are the inner functions kind of like ‘yielding’ the lower()/upper() transformation methods as Python proceeds to evaluate them with the complete text string on the second call?

For my future reference, I encountered this code snippet while reading Dan Bader’s Real Python book in the section: “3.1 Python’s Functions Are First-Class”.

[Gribouillis]

I think the aim of the chapter is to show that functions can be manipulated like any other values, for example the value of a variable can be a function. See if you understand better the following code

>>> def spamfunc(text):
...     return f'{text.upper()}!'
... 
>>> xxx = spamfunc
>>> xxx('hello')
'HELLO!'
>>> yy = xxx
>>> yy('hello')
'HELLO!'
>>> def foo():
...     return spamfunc
... 
>>> foo()('hello')
'HELLO!'
>>> 

The value of the variables xxx and yy can be the function spamfunc. This function can also be returned by another function foo().

[deanhystad]

get_speak_func() is a function that returns functions. It does not call it's internal functions, and it does not pass arguments to the functions. It just returns function whisper or yell based on the value of the volume arg. Perhaps the how this works and why you might do it would be clearer if I step you through a different example.

This code implements a simple calculator. Imagine the functions add, sub, mul and div are far more complicated than they are and that is why they are broken out into functions.

def add(a, b):
    return a + b

def sub(a, b):
    return a - b

def mul(a, b):
    return a * b

def div(a, b):
    return a / b


a, op, b = input('Enter Equation: ').split()
a = float(a)
b = float(b)
if op == '+':
    print(sum(a, b))
if op == '-':
    print(sub(a, b))
if op == '*':
    print(mul(a, b))
if op == '/':
    print(div(a, b))

This has a lot of duplicate code with all the print statements. I decide to clean it up by making a function that takes an operator character and returns an operator function.

def add(a, b):
    return a + b

def sub(a, b):
    return a - b

def mul(a, b):
    return a * b

def div(a, b):
    return a / b

def operator(op):
    if op == '+':
        return add
    if op == '-':
        return sub
    if op == '*':
        return mul
    if op == '/':
        return div

a, op, b = input('Enter Equation: ').split()
a = float(a)
b = float(b)
print(operator(op)(a, b))

operator() in my calculator does the same thing as get_speak_func() in your example. The only difference is the functions are defined in the module scope instead of the function scope. I could embed add(), sub(), mul() and div() inside operator() and it would work exactly the same. If I call operator('+') it returns the add() function. It does not call the add function. It does not pass arguments to the add function. If I call operator('+')(3, 2) it calls the add function with the arguments 3, 2 and returns the value 5. These are equivalent:

add(3, 2)  # Call directly
func = add  # Reference the function using a variable
func(3, 2)
func = operator('+')  # Get the function by calling operator()
func(3, 2)
operator('+')(3, 2)  # Get the function by calling operator() and evaluate

I notice the operator function is really a dictionary, so I make it a dictionary.

def add(a, b):
    return a + b

def sub(a, b):
    return a - b

def mul(a, b):
    return a * b

def div(a, b):
    return a / b

operator = {'+':add, '-':sub, '*':mul, '/':div}

a, op, b = input('Enter Equation: ').split()
a = float(a)
b = float(b)
print(operator[op](a, b))

There is a Python library of operators as functions, so I use them instead of writing my own.

import operator

operators = {'+':operator.add, '-':operator.sub, '*':operator.mul, '/':operator.truediv}

a, op, b = input('Enter Equation: ').split()
a = float(a)
b = float(b)
print(operators[op](a, b))

[knackwurstbagel]

I like @deanhystad response because it covers returning a function, and assigning it to a variable which can then be used to access or call the appropriate function. Lets assume that you now understand that part, that a function is a first class citizen, can be returned and assigned to a variable and such... its core to whats going on.

Your function get_speak_func simply acts as a dispatcher if I could use that term. It decides as you know based on the volume weather to yell or whisper, however it does not care about what arguments each of whisper(text and yell(text) deal with, or what it does with that text argument at all. It just decides which function signature to give back, and its not until you call the returned function that this matters. Does that make any sence. I am assuming that you might be confused about line 12 and why you get <function get_speak_func.<locals>.yell at 0x7ff195cb9ee0> is this true

hey m.r. get_speak_fun can you please tell me what function to call when my volume is 0.8... hey man I know of this great function that could help you out here is its blueprint.

[ndc85430]

Being able to pass a function to another function is incredibly useful. Let's look at an example: say I have the following program that reads some numbers from a file, finds their total and outputs that:

program.py

import functools
import operator
import sys


def read_file(filename):
    with open(filename) as file:
        return [int(line.strip()) for line in file.readlines()]

def total(data):
    return functools.reduce(operator.add, data, 0)


def run(path_to_file):
    data = read_file(path_to_file)
    print(total(data))


if __name__ == '__main__':
    run(sys.argv[1])

Like a good developer, I want to write tests to check that my program works. I could test the total and read_file functions individually, but I also want a test that checks the program works as a whole - that these parts are integrated correctly. The problem is the call to print - it doesn't return anything, rather it just prints to the standard output stream and so I can't check the value that was printed in test code. Because functions can just be passed around, I can pass a function into run that controls where the output is written - in the main program, I can pass print and in a test, I can pass a function that just writes to a string whose value I can check:

program.py

import functools
import operator
import sys


def read_file(filename):
    with open(filename) as file:
        return [int(line.strip()) for line in file.readlines()]

def total(data):
    return functools.reduce(operator.add, data, 0)


def run(path_to_file, write):
    data = read_file(path_to_file)
    write(total(data))


if __name__ == '__main__':
    run(sys.argv[1], print)

run now takes a second argument, write - a function that will write the output somewhere (line 14). On line 20, we pass print as that second argument.

test_program.py

import tempfile
import unittest

import program


def make_file_containing(contents):
    _, path = tempfile.mkstemp(text=True)
    with open(path, "w") as file:
        file.write(contents)

    return path

class TestProgram(unittest.TestCase):
    def test_it_outputs_the_sum_of_the_numbers_in_the_file(self):
        path_to_file = make_file_containing(
            """1
               2
               3""")

        output = ""
        def write(line):
            nonlocal output
            output += str(line)
            
        program.run(path_to_file, write)

        self.assertEqual(output, "6")

if __name__ == '__main__':
    unittest.main()

In test_program.py, I declare a function called write that just appends what it's given to the variable output (lines 21-24) and then I pass that to run on line 26. Once run has completed, I check the value in output to see that it's what I expect ("6" in this case). This idea is known as dependency injection - the writing function is a dependency of run, so we inject it (i.e. pass it in), so that we can make that function easier to test.

Incidentally, the way I implemented total was to use reduce, which is a function that combines the values in an iterable in some way. The way you specify how to combine values is by passing in a function - here I pass operator.add to add them. The point here is that by allowing you to specify how to combine the values, the reduce function is quite a general purpose abstraction - it can be used in different contexts by passing different combining functions.