The science of debugging

PyData Amsterdam · September 20, 2024

About me

ML engineer at
Maintainer of
@SdgJlbl

What does an ML engineer do?

Productionizing research code
Implementing tooling to facilitate data scientist job
Debugging

Everyone is debugging

There is a logical explanation to bugs.

Approaching debugging as

an experimental science

Experimental science = theory + experiments

Don't try to fix your bug

Understand what happens

I don't want to fix my bug by chance, and forever be haunted by the fear of it coming back. I want to understand what happened, so I can build on this knowledge to solve more complex cases. I want to be able to transfer this knowledge to my coworkers, so they can solve their own mazes. I want to know more about Python in general, and my codebase in particular, and being able to share that I've learned. And seeing things that way, debugging is not longer a chore, but it's actually an exciting puzzle to solve. Let's get started to improve your skills for solving mazes efficiently. And we'll start with drawing your map.

Drawing your map

Drawing your map

You have assumptions about how your code is working

Make them explicit

Focus on the most obvious at first

First, you do have some assumptions about your code expected behaviour. Not high-level, but more detailed. Eg: how an algorithm is registered in Substra. Make them explicit, because we want to be able to test them (and maybe invalidate them). Now, you have too many things to keep track of, how much details should you go into? Start from the beginning, with simple explanations / assumptions, close to where the bug is observed. It might come from earlier. Occam's razor: the simplest explanation is the most plausible. So it's okay to assume at first that Python is not buggy, the hardware is not hit by a rogue neutrino, ...

Drawing your map

Divide and conquer

Drawing your map

Use documentation and architecture diagrams

But don't trust them

Using your map

Make some hypotheses about what is happening during the bug

Using your map

Follow the flow

Remember

At least one of your assumptions is wrong

Finding your way

Finding your way

Experiments:

What is actually happening when you run your code

Finding your way

Experiments are used to test your hypotheses

Finding your way

Be systematic

Trust nothing

Check everything

Finding your way

Inspect values at interfaces.

Finding your way

Focus on data, not the code flow.

Remember

It's all about putting your theory to the test.

Keeping track of your progress

How?

Scientists use a research log

Debugging log

Write down experiment settings and results
Write down your assumptions and hypotheses explicitly
Write down ideas you can't follow right now

drawing of an open notebook

a circular diagram showing how assumptions are tested by experiments, which either confirm or infirm the assumptions

Debugging log is great to write incident retrospectives

Let's apply this approach to a toy example.

I iterate on cumulative lists:

[0], [0, 1], [0, 1, 2], ...

I add a special value at the beginning of each list:
-100


                        def prepend(l: list, beginning: list = [-100]):
                            beginning.extend(l)
                            return beginning


                        for i in range(5):
                            zero_to_i = list(range(i))
                            l = prepend(zero_to_i)

                        print(len(l))


                        def prepend(l: list, beginning: list = [-100]):
                            beginning.extend(l)
                            return beginning


                        for i in range(5):
                            zero_to_i = list(range(i))
                            l = prepend(zero_to_i)

                        print(len(l))

i = 4

zero_to_i = [0, 1, 2, 3]

l = [-100, 0, 1, 2, 3]

Expected: 5

Observed: 11

Experiment 1


                            def prepend(l: list, beginning: list = [-100]):
                                beginning.extend(l)
                                return beginning

                            print(prepend([0, 1, 2]))

Hypothesis 1

prepend is adding -100 in front of the list.

[-100, 0, 1, 2]

Experiment 2


                            def prepend(l: list, beginning: list = [-100]):
                                beginning.extend(l)
                                return beginning


                            for i in range(5):
                                zero_to_i = list(range(i))
                                print(zero_to_i)
                                l = prepend(zero_to_i)

Hypothesis 2

Let's check the value of zero_to_i

[]

[0]

[0, 1]

[0, 1, 2]

[0, 1, 2, 3]

Experiment 3


                            def prepend(l: list, beginning: list = [-100]):
                                beginning.extend(l)
                                return beginning


                            for i in range(5):
                                zero_to_i = list(range(i))
                                l = prepend(zero_to_i)
                                print(l)

Hypothesis 3

Let's check the value of l

[-100]

[-100, 0]

[-100, 0, 0, 1]

[-100, 0, 0, 1, 0, 1, 2]

[-100, 0, 0, 1, 0, 1, 2, 0, 1, 2, 3]

Experiment 4


                            def prepend(l: list, beginning: list = [-100]):
                                beginning.extend(l)
                                return beginning

                            print(prepend([0, 1, 2]))

                            print(prepend([0, 1, 2]))

Hypothesis 4

prepend is always behaving in the same way.

1st print: [-100, 0, 1, 2]

2nd print: [-100, 0, 1, 2, 0, 1, 2]

Experiment 5


                            def prepend(l: list, beginning: list = [-100]):
                                print("l: ", l)
                                print("beginning: ", beginning)
                                beginning.extend(l)
                                return beginning

                            prepend([0, 1, 2])

                            prepend([0, 1, 2])

Hypothesis 5

The default argument beginning is always [-100].

l: [0, 1, 2]

beginning: [-100]

l: [0, 1, 2]

beginning: [-100, 0, 1, 2]

What is happening

Default arguments are instantiated once.
If a default argument is changed, the new value will be used as default for subsequent calls.

beginning.extend(l)

DO NOT use mutable default arguments.

Fixed code


                    def prepend(l: list, beginning: list | None = None):
                        if beginning is None:
                            beginning = [-100]
                        beginning.extend(l)
                        return beginning


                    for i in range(5):
                        zero_to_i = list(range(i))
                        l = prepend(zero_to_i)

Other useful tools for your adventure

Other useful tools for your adventure

drawing of a magnifying glass Qualify the bug

drawing of a hourglass Leverage temporality

bad drawing of two dices Identify sources of randomness

drawing of various tools Invest in tooling

drawing of a rubber duck Enroll another adventurer

drawing of a compass, I guess Architect your maze better