Testing, Unit Tests and Test-Driven Development

Levels and kinds of testing

Testing is not one activity. The specification names several, and the exam asks you to say which one a scenario describes, and why.

The levels of testing

Two named stages of testing a finished product with real users:

• Alpha testing is carried out inside the development organisation, by staff who are not the programmers. It finds the problems before outsiders see them.
• Beta testing is carried out by a selected group of real users, in their own environment, before the general release. It finds the problems that the developers' own machines and habits could not.

Black box and white box

A white box test aims for coverage: every statement, every branch, and every path executed by at least one test. 100 per cent coverage proves the code was run. It does not prove that it is right.

Other words the exam uses

Unit tests with unittest

A unit test is a small program that calls one piece of your code with chosen inputs and checks the result automatically. Python's unittest module provides the framework. A test case is a class that inherits from unittest.TestCase, and each method whose name starts with test is one test.

import unittest

def grade_for(mark):
    if mark < 0 or mark > 100:
        raise ValueError("mark must be between 0 and 100")
    if mark >= 70:
        return "Distinction"
    if mark >= 50:
        return "Merit"
    if mark >= 40:
        return "Pass"
    return "Fail"

class TestGradeFor(unittest.TestCase):
    def test_normal(self):
        self.assertEqual(grade_for(85), "Distinction")
        self.assertEqual(grade_for(55), "Merit")

    def test_boundaries(self):
        self.assertEqual(grade_for(70), "Distinction")
        self.assertEqual(grade_for(69), "Merit")
        self.assertEqual(grade_for(49), "Pass")
        self.assertEqual(grade_for(39), "Fail")
        self.assertEqual(grade_for(0), "Fail")
        self.assertEqual(grade_for(100), "Distinction")

    def test_invalid(self):
        with self.assertRaises(ValueError):
            grade_for(101)
        with self.assertRaises(ValueError):
            grade_for(-1)

def run(case):
    suite = unittest.defaultTestLoader.loadTestsFromTestCase(case)
    result = unittest.TestResult()
    suite.run(result)
    print(result.testsRun, "tests,", len(result.failures), "failed,", len(result.errors), "errors")
    for test, trace in result.failures:
        print("  FAILED", test.id().split(".")[-1], "-", [line for line in trace.split("\n") if line.startswith("AssertionError")][0])

run(TestGradeFor)

Run it: three tests, none failed. The run function is only there to give a short, readable summary. In a normal program you would end the file with unittest.main() and let the framework print its own report.

A test that finds a bug

Change one character in the function, the classic boundary mistake of >= written as >, and run the same tests again.

import unittest

def grade_for(mark):
    if mark >= 70:
        return "Distinction"
    if mark > 50:
        return "Merit"
    return "Pass"

class TestGradeFor(unittest.TestCase):
    def test_normal(self):
        self.assertEqual(grade_for(85), "Distinction")
        self.assertEqual(grade_for(55), "Merit")

    def test_boundaries(self):
        self.assertEqual(grade_for(70), "Distinction")
        self.assertEqual(grade_for(50), "Merit")

def run(case):
    suite = unittest.defaultTestLoader.loadTestsFromTestCase(case)
    result = unittest.TestResult()
    suite.run(result)
    print(result.testsRun, "tests,", len(result.failures), "failed")
    for test, trace in result.failures:
        print("  FAILED", test.id().split(".")[-1], "-", [line for line in trace.split("\n") if line.startswith("AssertionError")][0])

run(TestGradeFor)

The normal test passes. The boundary test fails, and the report says exactly which assertion and what it received: 'Pass' != 'Merit'. A mark of 50 earned a Pass instead of a Merit. It is the boundary test that catches it, every time.

The assertions

A fresh start for every test

Tests must be independent: one must not depend on the leftovers of another, or the order they run in changes the result. setUp runs before each test and builds a fresh object.

import unittest

class Stack:
    def __init__(self):
        self.items = []
    def push(self, item):
        self.items.append(item)
    def pop(self):
        if not self.items:
            raise IndexError("stack is empty")
        return self.items.pop()

class TestStack(unittest.TestCase):
    def setUp(self):
        self.stack = Stack()

    def test_last_in_first_out(self):
        self.stack.push(1)
        self.stack.push(2)
        self.assertEqual(self.stack.pop(), 2)
        self.assertEqual(self.stack.pop(), 1)

    def test_pop_from_empty_raises(self):
        with self.assertRaises(IndexError):
            self.stack.pop()

    def test_a_new_stack_is_empty(self):
        self.assertEqual(self.stack.items, [])

suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestStack)
result = unittest.TestResult()
suite.run(result)
print(result.testsRun, "tests,", len(result.failures) + len(result.errors), "problems")

Test-driven development

Test-driven development (TDD) turns the order of work round. You write the test first, watch it fail, and only then write the code. It is a short cycle repeated many times:

1. Red. Write one small test for a behaviour that does not exist yet. Run it. It fails, which proves that the test can fail.
2. Green. Write the simplest code that makes the test pass. No more.
3. Refactor. Tidy the code, with every test still passing, so that the tests protect the clean up.

A worked cycle: leap years

The rule: a year is a leap year if it divides by 4, except centuries, which must also divide by 400. So 2024 is, 1900 is not, 2000 is.

Red. The function does not do anything yet, so the very first test fails.

import unittest

def is_leap_year(year):
    return False

class TestLeapYear(unittest.TestCase):
    def test_ordinary_leap_year(self):
        self.assertTrue(is_leap_year(2024))

suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestLeapYear)
result = unittest.TestResult()
suite.run(result)
print(result.testsRun, "tests,", len(result.failures), "failed")
for test, trace in result.failures:
    print("  FAILED", test.id().split(".")[-1], "-", [line for line in trace.split("\n") if line.startswith("AssertionError")][0])

Green. The simplest code that passes is to divide by 4. Add the next tests, which fail for 1900 and then pass for 2000, and each makes the function a little more right.

import unittest

def is_leap_year(year):
    return year % 4 == 0 and (year % 100 != 0 or year % 400 == 0)

class TestLeapYear(unittest.TestCase):
    def test_ordinary_leap_year(self):
        self.assertTrue(is_leap_year(2024))

    def test_ordinary_year(self):
        self.assertFalse(is_leap_year(2023))

    def test_century_is_not_a_leap_year(self):
        self.assertFalse(is_leap_year(1900))

    def test_every_fourth_century_is(self):
        self.assertTrue(is_leap_year(2000))

suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestLeapYear)
result = unittest.TestResult()
suite.run(result)
print(result.testsRun, "tests,", len(result.failures), "failed")

Refactor. With four tests watching, the condition could be rewritten as clearer code, and the tests would say at once if the rewrite broke it.

Why do it this way round

The cost is time at the start, and the discipline to write a test when it is not wanted. TDD does not remove the need for the other levels of testing: unit tests do not show that the units fit together.

Exceptions, and failing loudly

A function that cannot do its job should say so, not return something plausible. Raise an exception with a message that explains what was wrong, and let the caller decide what to do. A custom exception gives the problem a name.

class InsufficientFunds(Exception):
    pass

class Account:
    def __init__(self, balance):
        self.balance = balance

    def withdraw(self, amount):
        if amount <= 0:
            raise ValueError("amount must be positive")
        if amount > self.balance:
            raise InsufficientFunds("balance is " + str(self.balance) + ", asked for " + str(amount))
        self.balance -= amount
        return self.balance

account = Account(100)
print(account.withdraw(30))

for amount in [500, -5, 20]:
    try:
        print("left:", account.withdraw(amount))
    except InsufficientFunds as error:
        print("refused:", error)
    except ValueError as error:
        print("invalid:", error)
    finally:
        print("   checked", amount)

Catching the specific exception matters. A bare except: hides every bug, including ones you did not know about, and is a common reason for a program that fails silently.

Key terms you must be able to define

Unit testing

Testing a single subprogram or class in isolation, with chosen inputs and expected outputs.

Integration testing

Testing that several units work correctly together.

System testing

Testing the whole program against its requirements.

Acceptance testing

Testing by the client, to decide whether the finished system meets what they asked for.

Alpha testing

Testing of software by people within the development organisation, before release to outsiders.

Beta testing

Testing of software by a selected group of real users, in their own environment, before general release.

Black box testing

Testing based only on the specification, without knowledge of the code.

White box testing

Testing designed using knowledge of the code, so that every path is exercised.

Regression testing

Repeating earlier tests after a change, to check that nothing has been broken.

Test-driven development

A way of working in which a failing test is written first, then the code to pass it, then the code is tidied.

Exception

An event that stops the normal flow of a program, raised when something cannot be done and handled elsewhere.

Flashcards

Say the definition out loud before you flip the card. Retrieving it from memory is what builds recall, reading it again is not.

Reach the top grade on this topic

Testing answers are marked on matching the term to the situation, and on tests that would really find a fault.

• Name the level of testing from the scenario: unit, integration, system, acceptance, alpha or beta
• Say what the tester knows in black box and in white box testing, and what each can and cannot find
• In a test plan, include normal, boundary and erroneous data, and write the expected result first
• Describe test driven development as red, green, refactor, with the test written before the code
• Say that regression testing is repeating earlier tests after a change, and why that is needed

Mistakes that cost marks here

import unittest

def is_palindrome(text):
    cleaned = text.lower()
    return cleaned == cleaned[::-1]

class TestIsPalindrome(unittest.TestCase):
    def test_normal(self):
        self.assertTrue(is_palindrome("racecar"))
        self.assertTrue(is_palindrome("Level"))
        self.assertFalse(is_palindrome("python"))

    def test_boundary(self):
        self.assertTrue(is_palindrome(""))
        self.assertTrue(is_palindrome("a"))
        self.assertFalse(is_palindrome("ab"))

    def test_erroneous(self):
        with self.assertRaises(AttributeError):
            is_palindrome(12321)

suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestIsPalindrome)
result = unittest.TestResult()
suite.run(result)
print(result.testsRun, "tests,", len(result.failures) + len(result.errors), "problems")