Automate Debugging in Git with Unit Tests

A while ago, I published an article on debugging a codebase in Git using two commands blame and bisect. Git blame involved checking the author of each line of a file, whereas bisect involves traversing through the commits (using binary search) to find the one that introduced the bug. In this post, we will see how to automate the process of bisect.

To refresh your memory, git bisect involved a few steps, which are summarized below:

• Start the bisect wizard with git bisect start
• Select “good” and “bad” commits, or known commits where the bug was absent and present, respectively
• Assign commits to be tested as “good” or “bad” until Git finds out the commit which introduced the bug
• Exit the wizard with git bisect reset

To get an idea of the whole process, you could have a look at this screencast, which shows in detail how the debugging process works.

Naturally, the third step was time consuming — Git would show you commits one by one and you had to label them as “good” or “bad” after checking if the bug was present in that commit.

When we write a script to automate the process of debugging, we’ll basically be running the third step. Let’s get started!

Staging the environment

In this post, I will write a small module in Python that contains a function which adds two numbers. This is a very simple task and I’m going to do this for demonstration purposes only. The code is self explanatory, so I won’t go into details.

#add_two_numbers.py
def add_two_numbers(a, b):
    '''
        Function to add two numbers
    '''
    addition = a + b
    return addition

To automate the process of Git Bisect, you need to write tests for your code. In Python, we’ll use the unittest module to write our test cases. Here’s what a basic test looks like.

#tests.py
import unittest
from add_two_numbers import add_two_numbers

class TestsForAddFunction(unittest.TestCase):

    def test_zeros(self):
        result = add_two_numbers(0, 0)
        self.assertEqual(0, result)

if __name__ == '__main__':
    unittest.main()

We could write more of these tests, but this was just to demonstrate how to get on with it. In fact, you should definitely write more test cases as your programs and apps are going to be far more complex than this.

To run the unit tests, execute the tests.py file containing your test cases.

python tests.py

If the tests pass, you should get the following output.

Let’s now introduce an error in our function and commit the code.

def add_two_numbers(a, b):
    '''
        Function to add two numbers
    '''
    addition = a + 0
    return addition

To verify that the tests fail, let us run them again.

Let us add a few more commits so that the commit that introduced the error is not the last.

Start the bisect process

For the git bisect wizard, we will select the latest commit as bad (b60fe2cf35) and the first one as good (98d9df03b6).

git bisect start b60fe2cf35 98d9df03b6

At this point, Git points us to a commit and asks us whether it’s a good or a bad commit. This is when we tell Git to run the tests for us. The command for it is as follows.

git bisect run [command to run tests]

In our case, it will turn out to be the following.

git bisect run python tests.py

When we provide Git the command to run the tests itself, rather than asking us, Git runs these tests for every revision and decides whether the commit should be assigned good or bad.

Once Git is done running tests for every commit, it figures out which commit introduced the error, like magic!

Once you have found your commit, don’t forget to reset the wizard with git bisect reset.

In place of your unit tests, you can also create a custom shell script with custom exit codes. In general an exit code of 0 is considered a success, everything else is a failure.

Final thoughts

As the size of your code base increases, writing unit tests for every little piece of code that you write becomes necessary. Writing tests may seem time-consuming, but as you’ve seen in this case, they help you in debugging and save you time in the long run.

How does your team debug errors in code? Let us know in the comments below.