The How and Why of Property-Based Testing in Ruby

Benjamin Tan Wei Hao
Benjamin Tan Wei Hao
Share

Think about the last time you wrote unit tests (which hopefully is pretty recent). You had to come up with the happy-path, the tragic-path, and those hard-to-find edge cases. Since we are not infallible human beings, we tend to miss things more often than not. We miss an edge case here, forget about handling an error there, and discover our omissions during production.

Property-based testing totally flips the notion of unit testing on its head. Instead of writing specific examples of a test, you would write a property instead. A property, in this context, signifies a condition that would hold for all inputs that you define.

An example here is certainly helpful, if not required. Say you want to test a function that reverses an array. What could be some useful properties? Well, I can think of the following:

  • The first element of the array becomes the last element
  • The length of the array remains the same before and after reversing
  • Reversing the array twice will give back the original array

You could probably come up with a few more. With a property-based testing tool, you could express the above as code and tell it to generate hundreds or even thousands of test cases with randomly generated arrays.

A cool feature of many property-based testing tools is shrinking. This means that the tool, to the best of its ability, will find the smallest input that causes the property to be unsatisfied.

This Sounds Familiar…

If the above description sounds familiar, then you might have heard of a testing tool called QuickCheck. QuickCheck was originally written in Haskell, but has found its way into languages like Erlang. There are other QuickCheck implementations, like ScalaCheck for Scala and FsCheck for F#.

While the above languages are mainly functional languages, this hasn’t stopped the creators of Rantly from coming up with a QuickCheck-like tool for Ruby along with a cute name.

In this article, we will expand our testing repertoire by trying out property-based testing in Ruby.

Setting Up Rantly with RSpec

We will first set up a demo project to use Rantly with RSpec. The instructions are similar for MiniTest and TestUnit. Let’s create a demo project:

% mkdir rantly-demo && cd rantly-demo

Then create a skeleton Gemfile:

% bundle init
Writing new rantly-demo/Gemfile

We only need rantly:

source "https://rubygems.org"

gem "rantly"

Time to install the dependencies:

% bundle
Fetching gem metadata from https://rubygems.org/..
Fetching version metadata from https://rubygems.org/.
Resolving dependencies...
Using rantly 0.3.2
Using bundler 1.10.6

Assuming you already have rspec installed, you can quickly set up your project to use RSpec:

rspec --init

Using Rantly

Here is how you would express a Rantly property:

it "define your property here" do property_of { Rantly { <GENERATOR GOES HERE> } }.check { |a_generated_value| <EXPECTATION GOES HERE> } end

Therefore, an example would be:

it "integer property only returns Integer type" do property_of { integer # the generator }.check { |i| # i is the generated value expect(i).to be_a(Integer) # the expectation } end

A Failing Test and Shrinking

Let’s see what Rantly tells us when a property fails. We will tell Rantly to create arrays of integers, and then check that every generated array has completely even elements. Obviously, that will fail. The interesting thing is how will it fail? Let’s write the shady property in spec/array_spec.rb :

require 'rantly'
require 'rantly/rspec_extensions'
require 'rantly/shrinks'

RSpec.describe "Array" do
  it "even numbers" do
    property_of {
      Rantly { array { integer } }
    }.check { |i|
      expect(i).to all(be_even)
    }
  end
end

When you run the file, this is the output:

[0, 0, 0, 0, -1324248444, -819907805037675589]
found a reduced failure case:
...
[0, 0, 0, 0, -10102, -819907805037675589]
found a reduced failure case:
...
[0, 0, 0, 0, -77, -819907805037675589]
found a reduced failure case:
...
[0, 0, 0, 0, -1, -819907805037675589]
found a reduced failure case:
...
minimal failed data is:
[0, 0, 0, 0, 0, -819907805037675589]
F

Failures:

  1) Array even numbers
     Failure/Error: expect(i).to all(be_even)
       expected [-1384706466568309853, -2143298094606122148, 2181188094126790798, 1908884087348911076, -710950470620772656, -819907805037675589] to all be even

          object at index 0 failed to match:
             expected `-1384706466568309853.even?` to return true, got false

          object at index 5 failed to match:
             expected `-819907805037675589.even?` to return true, got false

Here, we can see that Rantly tries to find the smallest failure case by reducing the number of elements to as small as possible. This process is called shrinking. Rantly is able to perform shrinking on integers, strings, arrays, and hashes.

Unfortunately it didn’t decrease the last element. However, it is relatively obvious (after some squinting) that -819907805037675589 is a negative number.

Coming Up with Properties

By far the hardest task when doing property-based testing is coming up with the properties in the first place. In this section, we will cover some helpful techniques to figure out what kinds of properties to write. This list is not exhaustive, but serves as a good starting point.

1. Inverse Functions

These are usually pretty obvious to spot. Examples of inverse functions include:

  • Encoding and Decoding (e.g. Base64)
  • Serializing and Unserializing (e.g. JSON)
  • Adding and Removing

While writing a property that exploits “inverse-ness” doesn’t really cover a lot, this property is extremely useful as a sanity check. Couple this with one hundred or more generated test cases and you should feel pretty confident.

Here’s an example of how to test Base 64 encoding and decoding. Create base64_spec.rb in spec:

require 'rantly'
require 'rantly/rspec_extensions'
require 'rantly/shrinks'

require 'base64'

RSpec.describe "Base64" do
  it "encoding and decoding are inverses of each other" do
    property_of {
      Rantly { sized(30) { string } }
    }.check(1000) { |s|
      puts s
      expect(Base64.decode64(Base64.encode64(s))).to eq(s)
    }
  end
end

This basically creates random 30-character strings and generates 1000 tests, asserting that encoding and decoding are indeed inverses of each other. Here’s a sampling:

`` ... I.F@5!x}PI({m[8XPw=r1Vep(\*uIi Cz)ZkAcUUE],xoOI/@g*&;I):JVn$ J\Oo”PTR-8[A3);k*5Li0+v;[e8o= R{IL2Vz]$.KcOG<uy<gBpPc}T|+j7n .gsw?:?#"Iy%0O>-V0!]y#K}6>M!Ny xnkLFCLeim)VR9r|qaZuoYrNWd1GOU ?~m|6;;N~w.b)_+L-mIx-X?Lkb55Z7C”_Ps Kb[Or?V’ypYI!kwk>xB>:s-!D?GxS !lb%bw>u0V6xHwh9+D6c$/Q0!UkM\Y

success: 1000 tests . “`

2. Idempotence

Idempotent – A word to impress your friends and annoy your co-workers! This basically means doing it once is the same as doing it multiple times. For example, calling Array#uniq multiple times will result in the same value. Sorting functions also belong to the same category.

Let’s try out Array#uniq (you can put this in spec/uniq.rb):

require 'rantly'
require 'rantly/rspec_extensions'
require 'rantly/shrinks'

RSpec.describe "Array" do
  it "uniq is idempotent" do
    property_of {
      Rantly { array { Rantly { i = integer; guard i >= 0; i } } }
    }.check { |a|
      expect(a.uniq.uniq).to eq(a.uniq)
    }
  end
end

Here, we are generating an array of non-negative integers. We are using generator guard to limit the generated integers to only non-negative ones:

Rantly { i = integer; guard i >= 0; i }

3. Using an existing implementation

Say you have discovered a new sorting algorithm called, QuickerSort, that you know is faster than the existing sorting algorithm implemented in Ruby. Now all you need to do is to make sure that your QuickerSort implementation produces the exact same results as the Ruby implementation.

With QuickCheck, we can easily express a property like so:

RSpec.describe "Array" do
  it "Array#quicker_sort works produces the same result as Array#sort" do
    property_of {
      Rantly { array(range(0, 100)) { integer }}
    }.check { |a|
      expect(a.quicker_sort).to eq(a.sort)
    }
  end
end

Here we are generating an array of random integers that can be an empty array all the way to an array of 100 elements.

Custom Generators: Generating a DNA Sequence

Let’s learn how to create a custom generator. Custom generators are useful when your input data has to fit certain requirements. For example, if a method only operates on binary digits, then using the default integer generator will not be very useful.

In this example, we will create a DNA sequence generator. For our purposes, a DNA sequence is basically a array that contains a combination of A, T, G and C.

An example would be ["C", "G", "A", "G", "A", "T", "G"]. Our first stop is Rantly#choose, which let’s the generator pick a value from the specified choices:

choose("A", "T", "G", "C")

Next, we know that we need an array. The array generator accepts a block, which is called to generate an element of the array. This is exactly what we need:

Rantly { array { choose("A", "T", "G", "C") } }

To add some variation, we can also have the generator produce various length arrays by specifying a range:

Rantly { array(range(0, 20)) { choose("A", "T", "G", "C") } }

Try this out on a console. You would need to do a require "rantly":

> 10.times { p Rantly { array(range(1,20)) { choose("A", "T", "G", "C") } } }
["T", "A", "A", "G", "A", "A", "T", "G", "G", "T", "A", "T", "T", "T"]
["T", "A", "T", "T", "G"]
["T", "T", "C", "G", "T", "T", "C", "A"]
["T", "C", "C"]
["G", "G", "T", "C"]
["A", "A", "A", "A", "C", "G", "T", "G", "G", "T"]
["C", "A", "G"]
["T", "G", "C", "C", "A", "C", "C", "T", "G", "C", "T", "C", "G", "C"]
["G", "C", "T", "T", "T", "A", "C", "A"]
["G", "G", "G", "A", "C", "T", "G", "C"]
 => 10

Pretty cool, eh?

Summary

Property-based testing proposes another way to think about tests. Instead of writing specific examples, why not write generic properties and let the tool generate test cases for you?

However, don’t go throwing away your unit-tests yet! Property-based testing would probably be most useful for testing things like data-structures, stateless functions (that is, functions in the functional programming sense of the word), and algorithms. It probably isn’t a great fit for testing business logic. In other words, Rantly is a new tool in your belt, not the entire belt.

Try out Rantly and let me know how it goes. Happy testing!

Frequently Asked Questions (FAQs) on Property-Based Testing in Ruby

What is the main difference between example-based testing and property-based testing in Ruby?

Example-based testing, also known as traditional unit testing, involves writing test cases for specific scenarios or inputs. In contrast, property-based testing doesn’t focus on specific scenarios. Instead, it verifies the properties or behaviors of a system by generating random inputs. This approach can uncover edge cases that might be overlooked in example-based testing. Property-based testing in Ruby can be implemented using libraries like Rantly or PropCheck.

How does property-based testing improve the quality of my code?

Property-based testing can significantly improve the quality of your code by uncovering edge cases that you might not have considered. It generates random inputs to test the properties of your system, which can lead to the discovery of bugs that would have otherwise gone unnoticed. This can make your code more robust and reliable.

How can I implement property-based testing in my existing Ruby project?

To implement property-based testing in your existing Ruby project, you can use libraries like Rantly or PropCheck. These libraries allow you to define properties of your system and generate random inputs to test these properties. You can integrate these tests into your existing test suite, running them alongside your example-based tests.

What are some common challenges when implementing property-based testing in Ruby, and how can I overcome them?

One common challenge when implementing property-based testing is defining the properties of your system. This requires a deep understanding of your system and its expected behavior. To overcome this, spend time upfront to thoroughly understand your system and its requirements. Another challenge is dealing with the randomness of the inputs. To handle this, make sure your tests are deterministic, meaning they produce the same results given the same inputs.

Can property-based testing replace example-based testing in Ruby?

While property-based testing can uncover edge cases that might be overlooked in example-based testing, it doesn’t necessarily replace example-based testing. Both testing methods have their strengths and can complement each other. Example-based testing is great for testing specific scenarios, while property-based testing is excellent for testing the overall behavior of your system.

How can I generate random inputs for property-based testing in Ruby?

Libraries like Rantly or PropCheck can be used to generate random inputs for property-based testing in Ruby. These libraries provide functions to generate random data of various types, such as integers, strings, arrays, and more.

How can I ensure my property-based tests are effective?

To ensure your property-based tests are effective, make sure you define clear and accurate properties for your system. Also, ensure your tests are deterministic. This means they should produce the same results given the same inputs, despite the randomness of the inputs.

What is the role of shrinking in property-based testing?

Shrinking is a process in property-based testing that simplifies failing test cases to their minimal form. This makes it easier to understand why a test failed and to fix the underlying issue. Libraries like PropCheck support shrinking.

Can property-based testing be used for integration testing in Ruby?

Yes, property-based testing can be used for integration testing in Ruby. It can be particularly useful for testing complex interactions between different parts of your system.

How can I learn more about property-based testing in Ruby?

There are many resources available to learn more about property-based testing in Ruby. You can start with the documentation for libraries like Rantly or PropCheck. There are also many tutorials and articles available online.