Understanding Test Concurrency In Elixir

This post refers to Elixir 1.8 and Ecto 3.0, which are the latest versions at publication time.

I remember the first time I ran a really fast software test suite.

I had created a small library which unlike the applications I’d worked on, didn’t touch a database. I typed in the test command, pressed enter, and before I could blink, the tests were finished.

Developing that library was a joy. Every time I made a change, I ran the whole suite. There was never a pause to interrupt my train of thought, and I always knew everything my change affected. The fast feedback cycle kept me happy and productive.

Years later when I had the same experience on an Elixir web development project, I was floored. With no special effort, the entire Phoenix application’s test suite ran in two seconds.

In Elixir, the fastest applications take full advantage of concurrency, and the same thing can be said about test suites. Because database access is a common test performance bottleneck, Ecto SQL provides tools for concurrent database tests. But to use them well, you need to understand how they work.

Text Concurrency in ExUnit

The timeline of an ExUnit test run looks something like this:

First, test_helper.exs is run, with nothing else running concurrently.

Next, using one process per test, ExUnit runs the tests for all modules marked async: true. The tests in one async module will run concurrently with the tests in another async module. However, tests within the same test module run serially.

Finally, using one process per test, ExUnit runs the tests for each module marked async: false. This time, tests are run for one module at a time. As before, tests within the same test module run serially.

In an umbrella project, the timeline above repeats once per application, serially.

This fact is worth highlighting: tests within the same test module always run one at a time, even if that module is marked async.

We can use this knowledge to control test concurrency. If we have two tests which should not run at the same time, we can ensure that by either marking both of their test modules as async: false or by putting them in the same test module.

On the other hand, if test A and test B should run concurrently, we should put them in separate test modules and mark both modules async: true.

Breaking apart a large test module into smaller ones will automatically increase test concurrency. They could even be submodules within the same file:

defmodule MyApp.MathTest do

  defmodule AdditionTests do
    use ExUnit.Case, async: true

    test "it adds" do
      # ...
    end
  end

  defmodule SubtractionTests do
    use ExUnit.Case, async: true

    test "it subtracts" do
      # ...
    end
  end
end

Now let’s take a look at how database access works in concurrent tests.

Database Connection Pools

A database connection pool is a bit like a local library. If every copy of a printed book has been checked out, you won’t be able to read it until someone checks in a copy.

Like paper books, database connections are limited resources. This is because they each require some memory for things like transaction state and local settings. They also take a moment to set up. Whether your database is configured to allow 10 simultaneous connections or 5,000, Elixir can run a much larger number of processes. Because of this Elixir processes must take turns.

In dev or production we typically configure Ecto to use Poolboy. In tests we use Ecto.Adapters.SQL.Sandbox. It works as a connection pool, but its main purpose is to help tests run cleanly.

SQL Sandbox

Ecto.Adapters.SQL.Sandbox is “a pool for concurrent transactional tests” per Repo in your system.

The basic idea is that whenever a test is running it can pretend it has its own database which no other test uses. Whatever one test does with the database will have no effect on another test. This works by taking advantage of database transactions.

By default, when you checkout/1 a connection, it’s set to sandbox: true. This means that the adapter starts a database transaction for your test, making all changes to the database tentative and therefore invisible to other connections. When the test is complete, the adapter does not commit the transaction but rolls it back, abandoning all changes made within it and leaving other tests happily unaware.

(The fact that changes made in a database transaction are invisible until the transaction is committed is known as “Isolation,” which is the “I” in ACID. It’s possible to set the “isolation level” for a given transaction to be more or less restrictive, but the default level for both PostgreSQL and MySQL prevents “dirty reads” in which “a transaction reads data written by a concurrent uncommitted transaction.” However, Ecto doesn’t support concurrent tests with MySQL, due to problems with deadlocking.)

When a process checks out a connection, only that process may use it. But what if we need to test a GenServer, Task, web server, or any other process which needs access to the database? For instance, maybe we want our test to make a web request, have the server insert a record, and have the test confirm that it was inserted. In production, the web server would commit its transaction to the database, making its changes visible to other processes. But in tests, we don’t commit our transactions. So how can the test see what the server did?

The answer is to let multiple processes share a database connection. Each will see the others’ changes because they’ll all be working with the database during the same open transaction.

Let’s look at how SQL.Sandbox handles sharing.

Sandbox Modes

The sandbox can initially be set to one of two modes:

• :auto - any process that wants a connection gets one automatically
• :manual - each process must explicitly checkout/1 a connection

I haven’t actually seen :auto used, but it seems likely to create confusion. Any additional process spun up during a test would get its own connection, which means that the new process and the test process would be unable to see each others’ changes.

A safer option is to use :manual and then checkout/1 the connection in a setup block, which each test process runs before its actual test. This makes checking out a connection seem automatic, since we don’t need an explicit checkout/1 inside every test body. The test process will own that connection until it either exits or calls checkin/1.

Once a test process has a connection, there are a couple of ways it can share it. The first is to remain in :manual mode and use allow/3, which says “make the connection specified for this repo and owned by process A accessible to process B.” Taking an example straight from the documentation:

test "create two posts, one sync, another async" do
  parent = self()
  task = Task.async(fn ->
    Ecto.Adapters.SQL.Sandbox.allow(Repo, parent, self())
    Repo.insert!(%Post{title: "async"})
  end)
  assert %Post{} = Repo.insert!(%Post{title: "sync"})
  assert %Post{} = Task.await(task)
end

This is very clear, but isn’t convenient for many of our testing needs. We don’t want to litter our code with test-only logic and ownership pids.

So Ecto.Adapters.SQL.Sandbox provides :shared mode, which a process can switch to after checkout/1. Ecto.Adapters.SQL.Sandbox.mode(Repo, {:shared, pid}) means “all processes running concurrently with pid should use the connection which pid owns.”

As the docs for SQL.Sandbox say, using :shared mode means that “tests can no longer run concurrently.” With the understanding we’ve built, this makes sense. If shared mode means “all concurrent processes share this connection,” we only want one one test process to be in that concurrent group. If any other test ran at the same time, the two tests would be working with the database within the same transaction and the changes that one test made would leak to the other test. This would destroy our illusion that each test has a database to itself.

So any test using :shared mode must be in a test module marked async: false. This means no tests from other modules will run concurrently with it. Since tests in the same module run serially, it will be the only test running until it completes and may share its database connection fearlessly with the processes whose behavior it tests.

Conclusion

Before we wrap up, there are two other things I’d like to mention.

First, sharing database connections can lead to connection ownership errors. This happens when the owner of the connection has checked it in (whether explicitly or by terminating) and a process that shared the connection tries to keep using it. Often the owner process was a test which terminated when it finished. The SQL.Sandbox docs explain this issue and related ones that you may encounter.

Second, mixing concurrent tests with a shared mutable state of any kind will cause problems. If that mutable state is in a database, transactions make it non-shared. If it’s in a process (such as a GenServer), you can’t run your tests concurrently unless you build similar isolation. For example, you could have each test spin up its own instance of the GenServer to work with.

I may write more about these things in a future post, but this one is long enough. Until then, may your tests be fast and green.

DockYard is a digital product agency offering exceptional product strategy, design, full stack engineering, web app development, custom software, Ember, Elixir, and Phoenix services, consulting, and training.