Unusual things to do with unittest/HTMLTestRunner

[AndyHolyer]

As mentioned in my previous test, I'm preparing a worksheet for undergraduates learning intorductory Python.

The worksheet will be a Google Colab document, with a code cell for the student to complete, and a bunch of unittest classes which will check if they've got them right.

Source code as at the time of posting at the bottom of this post.

I've RTFM'd and can't see how to do these specific things. Any advice greatly recieved. I'll summarise to the thread if need be.

The things I'd like to do (some of them come from feedback from my boss, so they might well not be possible):

1. I know how to put failure messages into assert methods, which are shown in the appropriate place in the HTML summary.
   It would be nice to be able to put a message on successful test outputs - to provide links to model answers, etc.
   Is that possible?
   
2. So far I've got five functions defined, all basically trivial. Once I've devised the whole sheet, there will be a lot more than that, and a couple of pages of red lines will be intimidating for a student who lacks confidence.
   It would be nice to flag a test class so that if it fails, it aborts all tests beyond that - or even better, it will display two failed tests and then abort. That way as students wor through the sheet, the list of succesful tests will get longer and longer.
   Is this possible (to annotate a test to say "if you've got here, then abort all future tests")?
   
3. I'd like to monitor semi-automatically how far different students have got with the worksheet.
   What I'm thinking of is to put a call into the testing code to make an HTTP posting every time the tests are run. I'll then write a little script on a web host to log all of these to a database, and afterwards I can process the outputs to show me who's got however far (there are projected to be a maximum of 50 students taking the module, so even if they run the tests 100 times each, that's only 5,000 lines of data, so it's not worth being more clever than that).
   So, I'd like to pull the results of a test run out, ideally in JSON. Is this easy/possible? I can't just use json.dump because the HTMLResponse object has circular references.
   At the moment I'm thinking that I'll post the whole HTML response, and then scrape that using BeautifulSoup etc. Which I admit, is a bit silly.
   
4. Continuing on from that, I'd like to be able to see what each student has put in the cell containing their work (especially I'd like to be able to find their registration number - which is why I have that as the first challenge :-) Is in possible to select a code cell by name and insert that into my posting?
   

Many thanks in advance for any help.
===================== Source Code below =====================

# Programming Fundamentals Introduction to Python
This worksheet will allow you to get a start at programming in Python without having to install anything on your computer (yet).
It consists of text blocks (like this one), and code blocks (like the one below).

Code blocks contain Python code: you can run the Python code by clicking the Run button to the left hand side of the python code. You can change the Python code and run it again. When you have completed all the code sections in this document you will have completed the first set of exercises for this module.

Try it now: 

print("Hello World")

You should see `Hello World` printed out.

Now edit the red text in quotes to say something else, (like "Hello Andy"). 

Press the run button again.


You can use Python to do calculations:

2+2

This expression is a lot more complicated, but Python is still happy to work it out:

$\frac{(7.8165^5 + 8.179)}{(9.54 - 1.73)}$

(7.8165**5 + 8.79)/(9.54 - 1.73)

Try changing the values in the two sums above, and then hitting the run button again.


---



One of the best things about Python is that you always have the compiler available. When you have Python installed on your own computer (we'll do this in a few weeks) you can run an interactive session from a terminal widow.

Here in Google Colab, you can try out things in a code cell. When you're writing Python, the correct thing to do is to try thnig sout to check if you're doing things right (so if you're not quite sure if you've remembered the correct syntax for a loop, say, then write a little bit of Pyton and try it out to check that it does what you want it to do.

The code block below this is for you to use to try things out. Think of it as your Rough notebook:

# Type in any little fragments of python in this code block and try them out as you go along


The cell below this is where you put the python code to complete this worksheet.

I've filled in the first three functions - all you need to do is to put the right values for you in them, and you're done.

Then I've given you the basic definition of the next function - you will need to write the python code to get it to do the right thing.

As you go along you will need to add other functions in this cell.

Run the cell, then look further down the sheet to find a unit test cell to see how you're doing.

# This is where you should put the function definitions for your work

#=========================== Fill in your personal details in these functions ==========================
def student_id():
  """Edit this so that it returns your UPnnnnn number"""
  return("UP65432")

def givenname():
  """Return your Given name (first name etc.)"""
  return("Replace this with your given name")

def surname():
  """Return your Surname"""
  return("Replace this with your Surname")

#=========================== The next few functions are to get you used to defining functions ==========
#=========================== You will need to add the definitions after the first one ==================
def add( x, y):
  """Create an add function which accepts two parameters and returns their sum"""
  


# Unit Testing

The cell below runs *Unit Tests* on the functions defined in the worksheet. Unit testing is a powerful tool for making your code more more reliable and easier to maintain.
We will be looking at Unit Testing in more detail later on in this module.

*I've hidden the code in the cell to make it clearer (though there's no real reason why you can't look at the code if you really want to - but it probably won't help you solve the problems on the sheet, and the code is a lot more complex than you're probably used to - yet*

Click the "run" button and you'll see a button labelled "Run Tests". Click this button, and after a little while you'll see a table giving the results of the tests

The first time you run this, you'll just get a whole list of failed tests - don't panic! Just go back to the cell above this, modify your function (don't forget to run the cell again!) and try again. There's no problem with running the tests repeatedly - as long as it takes you further towards solving your problem.

Once you have a whole page of green lights, you've finished the worksheet.

#@title Test how you are doing with this worksheet

test_result = None

!pip install html-TestRunner --quiet

import inspect
import HtmlTestRunner
import unittest
import sys

class Test1NameMethods(unittest.TestCase):

  def test_1_student_id(self):
    self.assertNotEqual(
        student_id(),
        "UP654321", 
        "Change the function 'student_id' so that it has your UP number, not the one originally in there"
    )

  def test_2_givenname(self):
    self.assertNotEqual(givenname(),"Replace this with your given name", "Edit the function 'givenname' so that it returns your given (christian, first) name")

  def test_3_surname(self):
    self.assertNotEqual(surname(), "Replace this with your Surname", "Edit the function 'surname' so that it returns your surname (last name, family name)")

class Test2DataMethods(unittest.TestCase):

  def test_4_dataAdd(self):
    self.assertIs(
        hasattr(sys.modules['__main__'], "add") and inspect.isfunction(add), 
        True,
        "Create an 'add' function which takes two parameters and returns their sum"
    )
    self.assertEqual(
        add(7,17),
        24,
        "Put some code inside the 'add' function so that it takes two parameters and returns their sum"
    )
    self.assertEqual(
        add(-7,-17),
        -24,
        "Put some code inside the 'add' function so that it takes two parameters and returns their sum"   
    )
  
  def test_5_dataSubtract(self):
    self.assertIs(
        hasattr(sys.modules['__main__'], "subtract") and inspect.isfunction(subtract),
        True,
        "Create a 'subtract' function which takes two arguments and subtracts the second from the first (Hint: use the 'def' keyword and look at what you did in your 'add' function"
    )

    self.assertEqual(
        subtract(10, 5),
        5
    )

    self.assertEqual(
        subtract(5, 10),
        -5
    )

import  ipywidgets as widgets
from IPython.display import display,clear_output
button = widgets.Button(
    description="Run Tests",
#    layout=widgets.Layout('description_width=initial')
    )
widget = widgets.Output()
printout = widgets.Output()

def on_button_clicked(b):
  # Display the message within the output widget.

  # Turn off Standard Error Output
  import os
  import sys

  #f = open(os.devnull, 'w')
  #old_stderr = sys.stderr
  #sys.stderr = f
  
  # Also, turn of Standard output

  #g = open(os.devnull, "w")
  #old_stdout = sys.stdout
  #sys.stdout = g
  global test_result

  !rm ./reports/* &>/dev/null

  # Run the Unit Tests
  
  with printout: 
    test_result = unittest.main(
      argv=['ignored'],  
      verbosity=2,
      exit=False,
      testRunner=HtmlTestRunner.HTMLTestRunner(
        output='./reports/', 
        add_timestamp=False, 
        combine_reports=True, 
        open_in_browser=True
      )
    )
  # Restore Stndard Error and Stsndard Output
  #sys.stderr=old_stderr
  #sys.stdout=old_stdout
  #f.close()
  #g.close()

  #import json
  #!pip install -U jsonpickle
  #import jsonpickle

  #json = jsonpickle.encode(test_result.result)
  
  #with open("./result-file", "w") as result_file:
    #result_file.write(json)
    #json.dump(test_result.result, result_file, default=lambda x: x.__dict__)
  
  # Clear the code cell output (in case we're running this more than once)
  
  printout.clear_output()

  # HTMLTestRunner puts the HTML output into a file in ./roports/
  # Find that file using the "glob" library
  import glob
  report_files = glob.glob('./reports/TestResults*')
  for report_file in report_files:
    html = ''
    with open( report_file, "r") as html_file:
      html = html_file.read()

    from IPython.core.display import HTML
    
    display(HTML(html))


button.on_click(on_button_clicked)
display(button)