Working Like a Professional

Readable code: style, naming and docstrings

Code is read many more times than it is written, usually by someone who has forgotten what it does. That someone is often you, six months later. Readable code is not a luxury. It is how a program stays correct as it grows.

Python's style guide is PEP 8. These are the rules that matter most:

Here is the same function written badly and well. Read the first and try to say what it does:

def calc(d, r):
    x = 0
    for k in d:
        if d[k][1] == 1:
            x = x + d[k][0] * (1 + r)
        else:
            x = x + d[k][0]
    return x

def basket_total(basket, vat_rate):
    '''Return the cost of a basket of shopping.

    basket maps an item name to a pair of (price, is_taxable).
    VAT is added only to the taxable items.
    '''
    total = 0
    for price, is_taxable in basket.values():
        if is_taxable:
            total += price * (1 + vat_rate)
        else:
            total += price
    return total

shopping = {"bread": (1.5, 0), "pen": (2.0, 1), "toy": (10.0, 1)}
print(round(calc(shopping, 0.2), 2), round(basket_total(shopping, 0.2), 2))

The two give the same answer. The second is far easier to trust, because every name says what it holds and the docstring says what the data looks like. The difference is only names, structure and a docstring.

The habits of clear code

• Name things for what they mean, not for how they are used: days_overdue, not d. A long accurate name beats a short vague one.
• Replace magic numbers with named constants. if age >= 18: becomes if age >= VOTING_AGE:.
• One function, one job, short enough to see on a screen. If you describe it with "and", split it.
• Return early instead of nesting: a check that fails can return straight away, keeping the main path unindented.
• Do not repeat yourself (DRY). Code that appears twice will one day be fixed in only one place.
• Comments say why, not what. # the API counts from 1 is useful. # add one to x is noise.

Docstrings

A docstring is a string in the first line of a function, class or module. It becomes the object's built-in documentation, shown by help() and by editors. The first line is a short summary. After it come the arguments and what is returned.

Doctests: documentation that tests itself

A docstring can hold an example written as an interactive session, starting >>>. The doctest module runs these examples and checks the answers, so the documentation can never quietly go out of date:

import doctest

def clamp(value, low, high):
    '''Return value, limited to the range from low to high.

    >>> clamp(5, 0, 10)
    5
    >>> clamp(-3, 0, 10)
    0
    >>> clamp(99, 0, 10)
    10
    '''
    return max(low, min(value, high))

finder = doctest.DocTestFinder()
runner = doctest.DocTestRunner(verbose=False)
for test in finder.find(clamp, "clamp", globs=globals()):
    runner.run(test)
print(runner.summarize(verbose=False))

attempted=3, failed=0 means the three examples in the docstring all gave the answers shown. On a real project you run python -m doctest module.py, or put the checks in a test suite. A failing example is reported like this:

File "clamp.py", line 5, in clamp
Failed example:
    clamp(5, 0, 10)
Expected:
    6
Got:
    5

Tools that do the checking for you

pip install ruff black
black program.py        reformats the file in the standard style
ruff check program.py   finds unused imports, undefined names, and many bugs

Modules, packages and virtual environments

Any Python file is a module. Importing it runs its code once and gives you its names. Several modules in a folder with an __init__.py file make a package. Breaking a program into modules keeps each file small, lets you reuse code, and gives every name a home.

shop/
    pyproject.toml          how the project is built and what it needs
    README.md               what it is and how to use it
    src/shop/
        __init__.py
        basket.py           the Basket class
        pricing.py          VAT and discounts
        cli.py              the command line
    tests/
        test_pricing.py
    .gitignore

Ways to import

Running a file or importing it

Every module has a __name__. When you run a file directly it is "__main__". When another file imports it, it is the module's own name. That lets one file be both a library and a program:

import sys
import types

source = '''
PI = 3.14159

def area(radius):
    return PI * radius ** 2

if __name__ == "__main__":
    print("running as a script")
'''

shapes = types.ModuleType("shapes")            # make a module in memory
exec(source, shapes.__dict__)
sys.modules["shapes"] = shapes                  # register it so that import can find it

import shapes as imported
print(imported.area(2), "| name is", imported.__name__)

print("--- the same code, run directly:")
exec(source, {"__name__": "__main__"})

When imported, the guard if __name__ == "__main__": is false, so the script part does not run. A file you intend to import should never do real work at the top level.

Virtual environments and dependencies

Different projects need different versions of the same library. If they all install into the one Python on your computer, they collide. A virtual environment is a private copy of Python and its libraries for one project. You create it, activate it, and install into it, and nothing else on your computer is touched.

python -m venv .venv
source .venv/bin/activate         (on Windows: .venv\Scripts\activate)
pip install requests
pip freeze > requirements.txt     records the exact versions in use
pip install -r requirements.txt   rebuilds the same environment elsewhere

A modern project describes itself in pyproject.toml:

[project]
name = "shop"
version = "1.2.0"
dependencies = ["requests>=2.31"]

[project.scripts]
shop = "shop.cli:main"

Version numbers follow a convention, semantic versioning: in 1.2.0, the first number changes when something breaks, the second when features are added, the third for fixes. Each library you depend on is also a risk. Keep the list short, and check it for known security problems with a tool such as pip-audit.

Version control with Git

Version control records every change to a project, who made it and why. You can see what the code was last Tuesday, find the change that broke something, and undo it. It also lets several people work on one project at once without overwriting each other. Git is the standard tool.

The commands for a normal day:

git init                       start a repository here
git status                     what has changed?
git add scoring.py             choose what goes in the next snapshot
git commit -m "Reject scores below zero"
git log --oneline              the history, one line per commit
git diff                       exactly what changed
git switch -c high-scores      start a new branch
git merge high-scores          bring it back
git push                       send commits to the server

A good commit message says what and, when it is not obvious, why: Reject scores below zero, not fixed stuff. Commit small, often, and only things that work. A .gitignore file lists what never to commit: virtual environments, compiled files, and above all passwords and keys.

For a coursework project the commit history is evidence. It shows the project was developed step by step, and when each part was written.

How it works inside

Git is built on ideas you already know. Each commit stores a snapshot of the files, named by a hash of their content. A commit also records its parent, so the history is a linked list. Two files with identical content have the same hash and are stored once. Here is a tiny version control system in a few lines:

import difflib
import hashlib
import json

class Repo:
    def __init__(self):
        self.snapshots = {}               # hash of the content -> the files
        self.commits = []

    def commit(self, files, message):
        content = json.dumps(files, sort_keys=True)
        tree = hashlib.sha1(content.encode()).hexdigest()[:7]
        self.snapshots[tree] = dict(files)       # the same content is stored only once
        parent = self.commits[-1]["id"] if self.commits else None
        ident = hashlib.sha1((content + str(parent) + message).encode()).hexdigest()[:7]
        self.commits.append({"id": ident, "parent": parent, "message": message, "tree": tree})
        return ident

    def log(self):
        for commit in reversed(self.commits):
            print(commit["id"], "<-", commit["parent"], commit["message"])

    def checkout(self, ident):
        for commit in self.commits:
            if commit["id"] == ident:
                return dict(self.snapshots[commit["tree"]])

    def diff(self, old_id, new_id):
        old, new = self.checkout(old_id), self.checkout(new_id)
        for name in sorted(set(old) | set(new)):
            before = old.get(name, "").splitlines()
            after = new.get(name, "").splitlines()
            for line in difflib.unified_diff(before, after, "a/" + name, "b/" + name, lineterm="", n=0):
                print(line)

repo = Repo()
first = repo.commit({"main.py": "print('hello')\n"}, "first version")
second = repo.commit({"main.py": "print('hello')\nprint('world')\n", "notes.txt": "todo\n"}, "say world, add notes")
third = repo.commit({"main.py": "print('hello, world')\n", "notes.txt": "todo\n"}, "merge the two prints")

repo.log()
print()
repo.diff(first, third)
print()
print("going back to the first commit:", repo.checkout(first))

repo.commit({"main.py": "print('hello, world')\n", "notes.txt": "todo\n"}, "no change")
print(len(repo.snapshots), "snapshots stored for", len(repo.commits), "commits")

The last line shows the saving. Four commits, but only three snapshots, because the fourth is identical to the third. A branch, in Git, is nothing more than a movable label pointing at one commit, which is why creating one is instant.

Measuring and improving performance

Programmers are famously bad at guessing which part of a program is slow. The rule is measure, then change. The steps are:

1. Make it correct first. A fast wrong answer is no use.
2. Measure with real data, to see whether there is a problem at all.
3. Find the bottleneck, the small part where most of the time goes.
4. Change one thing, and measure again.
5. Stop when it is fast enough.

Timing with timeit

timeit runs a small piece of code many times to average out the noise. This compares looking for a value in a list and in a set:

# varies
import timeit

in_list = timeit.timeit("99999 in data", setup="data = list(range(100000))", number=200)
in_set = timeit.timeit("99999 in data", setup="data = set(range(100000))", number=200)

print("set lookup was faster:", in_set < in_list)
print("by roughly", round(in_list / in_set), "times")

A list has to be searched item by item, O(n). A set finds the item from its hash, O(1). Your computer will give different times, but the comparison will not change.

Algorithms beat tuning

Counting the steps is more reliable than timing, because it does not depend on the computer. Find two numbers in a list that add up to a target. The obvious way tries every pair. A smarter way remembers the numbers it has seen:

numbers = list(range(0, 1200, 2))              # 600 even numbers
target = 1                                     # no two even numbers can add up to 1

def pair_by_brute_force(numbers, target):
    checks = 0
    for i in range(len(numbers)):
        for j in range(i + 1, len(numbers)):
            checks += 1
            if numbers[i] + numbers[j] == target:
                return (numbers[i], numbers[j]), checks
    return None, checks

def pair_with_a_set(numbers, target):
    seen = set()
    checks = 0
    for number in numbers:
        checks += 1
        if target - number in seen:
            return (target - number, number), checks
        seen.add(number)
    return None, checks

print("brute force :", pair_by_brute_force(numbers, target))
print("with a set  :", pair_with_a_set(numbers, target))

Nearly 180 thousand checks against 600. That is the difference between O(n squared) and O(n), and it grows: with 6000 numbers it would be 18 million checks against 6000. No amount of clever micro tuning will find a gap that size.

Profiling: find the bottleneck

A profiler records how long each function takes and how often it was called, so you can see where the time goes instead of guessing:

# varies
import cProfile
import pstats

def parse(line):
    return [int(part) for part in line.split(",")]

def total(lines):
    return sum(sum(parse(line)) for line in lines)

lines = ["1,2,3,4,5,6,7,8"] * 20000

profiler = cProfile.Profile()
profiler.runcall(total, lines)
pstats.Stats(profiler).sort_stats("cumulative").print_stats(5)

In the table, ncalls is how many times a function ran, tottime is the time spent in the function itself, and cumtime includes everything it called. Look for the functions with a large cumtime. On your own computer, python -m cProfile -s cumulative program.py profiles a whole program.

Quick wins worth knowing

Command line tools, packaging and documentation

A program that runs from the command line can be scheduled, chained with others and used by people who never see the code. The shape of a well made one is always the same:

import argparse
import sys
from collections import Counter

def count_words(text, top, lower):
    '''Return the most common words in text.'''
    words = text.lower().split() if lower else text.split()
    return Counter(words).most_common(top)

def main(argv=None):
    parser = argparse.ArgumentParser(prog="counter", description="Count the words in some text")
    parser.add_argument("text", help="the text to count")
    parser.add_argument("-n", "--top", type=int, default=3, help="how many words to show")
    parser.add_argument("--lower", action="store_true", help="ignore the case of the words")
    args = parser.parse_args(argv)

    if args.top < 1:
        print("counter: --top must be at least 1", file=sys.stderr)
        return 2                                  # a non zero exit code means failure

    for word, count in count_words(args.text, args.top, args.lower):
        print(count, word)
    return 0

print("exit code:", main(["The cat saw the dog and the bird", "-n", "2", "--lower"]))
print("exit code:", main(["anything", "-n", "0"]))
try:
    main(["anything", "-n", "lots"])
except SystemExit as stop:
    print("argparse stopped the program with code", stop.code)

In a file of its own, the last two lines would be the entry point. They run only when the file is started directly:

if __name__ == "__main__":
    sys.exit(main())

This pattern has several good features:

• the real work is in small functions that can be tested without a command line
• main takes the arguments as a parameter, so it can be tested too
• it returns an exit code: 0 for success, non zero for a failure, which is how other programs and scripts tell what happened
• errors go to the error stream (stderr), not to the normal output, so they do not mix into data that is being piped to another program
• if __name__ == "__main__" makes the file importable as well as runnable

Packaging

A project with a pyproject.toml can be built into a file that others install with pip. A [project.scripts] entry creates a command, such as shop, that runs your main. Choose a licence, because code without one is by default not free for others to use. Licences such as MIT allow nearly anything, and others, such as the GPL, require changes to be shared too.

Documentation

Documentation has different readers, and each needs something different:

For a coursework project the final report is made of exactly these. A project that keeps its README, its tests and its commit history up to date while it is built finds the write-up is largely done by the end.

Key terms you must be able to define

PEP 8

The style guide for Python code, covering naming, layout and imports.

Docstring

A string at the start of a function, class or module that documents it.

Doctest

An example in a docstring, written as an interactive session, which is run automatically to check it.

Module

A single Python file whose code and names can be imported and used elsewhere.

Package

A folder of modules with an __init__.py file.

Virtual environment

A private copy of Python and its libraries for one project, so that projects do not interfere.

Version control

A system that records every change to files, so that earlier versions can be recovered and teams can work together.

Commit

A saved snapshot of a project at a point in time, with a message and unique identifier.

Branch

A separate line of development that can later be merged back.

Profiler

A tool that measures how often each function is called and how long it takes.

Exit code

A number returned by a program when it finishes, where 0 means success.

Semantic versioning

A version number scheme of major.minor.patch, where the numbers show how big a change is.

Flashcards

Say the definition out loud before you flip the card. Retrieving it from memory is what builds recall, reading it again is not.

Reach the top grade on this topic

Professional practice questions want a practice linked to a specific benefit.

• For version control, give two benefits: a history that can be restored, and several people working without overwriting each other
• For readability, name features: meaningful names, docstrings, small functions, constants instead of magic numbers
• For a virtual environment, say it isolates the project's libraries so that versions cannot clash
• For performance, say to measure with a profiler before changing code, then change the algorithm before the details
• For a third party library, give a benefit and a risk

Mistakes that cost marks here