In the latest release of phoenix_ecto (version 3.3.0), Chris McCord introduced some exciting changes, bringing the power of concurrent, transactional browser tests to our doorstep. The revised Phoenix.Ecto.SQL.Sandbox plug makes it a breeze to write client-side acceptance tests that hit actual API endpoints, rather than ones that execute against mocks.
While this push was motivated by DockYard’s internal needs to support end-to-end tests for our own Ember and Phoenix stack, the implementation that landed in phoenix_ecto itself is completely agnostic of any client frameworks.
Before we take a closer look at phoenix_ecto and how we can write Ember acceptance tests to take advantage of the new feature, let’s briefly address an inevitable question.
What’s wrong with mocks?
Mocks still have their place in the testing landscape. Developers may find themselves in need to develop features and write associated tests before the API endpoints are ready. Perhaps they are working within an environment that doesn’t give them full control over the backend. Mocks also result in tests that run faster, since server-side logic and database operations are removed from the picture altogether.
On the other hand, mocks do have the potential downside of concealing errors when they start growing out of sync with the actual API, or when invalid assumptions are made about how the server would respond to certain requests.
Just as integration tests complement unit tests, and acceptance tests complement integration tests, end-to-end tests provide developers with additional confidence that their application works as expected when pieced together. Many developers avoid going down this path, arguing that these tests are often too slow and flaky. Perhaps it is time to revisit this strategy.
Phoenix.Ecto.SQL.Sandbox
Previous releases of phoenix_ecto already shipped with a plug named Phoenix.Ecto.SQL.Sandbox, which made it possible for Elixir developers to write concurrent, transactional tests for their Phoenix apps with ease. In the Elixir space, this is typically done in conjunction with testing libraries such as Hound or Wallaby, which simulate user interactions via headless browsers or Selenium WebDriver.
Let’s take some time to understand how the underlying sandbox functionality works.
Ecto’s SQL Sandbox
By default, the test database for a Phoenix app is configured to use a sandbox connection pool called Ecto.Adapters.SQL.Sandbox. Using this, a process is able to check out a connection from the pool and take ownership of it. The sandbox manages that connection by wrapping it in a transaction, which then gets rolled back when the process either dies or when the process checks the connection back in. The sandbox can also allow other processes to participate in the same transaction as the owning process.
# Check out a connection
Ecto.Adapters.SQL.Sandbox.checkout(Repo)
# Allow another process (`pid`) to use the same connection
Ecto.Adapters.SQL.Sandbox.allow(Repo, self(), pid)
# Check in a connection
Ecto.Adapters.SQL.Sandbox.checkin(Repo)
This fulfills the need to not only have isolated tests, but also to return the database to its initial state once a test is complete.
Maintaining a sandbox session
Over the course of a single test, a client may send multiple requests to the server, each resulting in a different process being spawned. For these to share the same sandboxed connection as the owning process that initiated the checkout, there must be a way to identify the active session.
The Phoenix.Ecto.SQL.Sandbox plug provides a function called metadata_for/2 that can be called to get a map that identifies a session by its repository and owner’s PID:
%{repo: repo, owner: pid}
The setup for a test in Phoenix using Hound would look like this:
use Hound.Helpers
setup do
:ok = Ecto.Adapters.SQL.Sandbox.checkout(YourApp.Repo)
metadata = Phoenix.Ecto.SQL.Sandbox.metadata_for(YourApp.Repo, self())
Hound.start_session(metadata: metadata)
end
Here, a connection is checked out from the pool, and the metadata is fed into Hound in order to associate all subsequent operations with that same sandbox session. The plug takes care of inspecting all incoming requests for the presence of a user-agent header that includes this metadata, and grants access to the sandboxed connection accordingly.
Transactional Ember acceptance tests
When we explored using the plug to support end-to-end tests written with Ember, prior to phoenix_ecto 3.3.0, some limitations became apparent.
The first was that the plug only looked for the session metadata in the user-agent request header, and most browsers do not let you overwrite this value. This one was easily solved by making the request header configurable, a fix that made it into the latest release of phoenix_ecto.
The second issue was that there was no built-in functionality to check out and check in connections from an external HTTP client.
Introducing Sandbox API endpoints
The client needs to be able to communicate with the server that a test is starting and that a new connection needs to be checked out from the sandbox pool. It also needs to be able to tell it to check the connection back in once the test is complete, so that we can roll back the transaction, along with any associated side effects.
This can be achieved very simply by introducing the following endpoints on the server, and protecting them so that they are only exposed when running Phoenix in a test environment:
POST http://localhost:4000/api/sandboxDELETE http://localhost:4000/api/sandbox
We will take a look at the implementation details for those endpoints in a moment, but for now, let’s remain focused on the client.
Armed with those endpoints, the setup and teardown for each acceptance test in Ember are straightforward:
// In tests/acceptance/some-test.js
beforeEach() {
return fetch('/api/sandbox', { method: 'POST' })
.then((response) => response.text())
.then((metadata) => {
// Set the required header for all subsequent requests
setHeader(this.application, 'x-user-agent', metadata);
});
},
afterEach() {
return fetch('/api/sandbox', { method: 'DELETE' });
}
The beforeEach hook makes a request to the server to start a new sandbox session, and stores the session metadata in a header for all subsequent requests (the implementation of setHeader will vary depending on whether you use jQuery.ajax() or fetch). Finally, the afterEach hook sends a request to the server at the end of the test to destroy the session. You can add this logic to the module-for-acceptance.js helper and continue to write acceptance tests as you always have.
Of course, the Phoenix server must be up and running during these tests, in an environment configured with the sandbox. In addition, Ember CLI must be configured to proxy API requests to the backend:
// Snippet config from testem.js
"proxies": {
"/api": {
"target": "http://localhost:4000"
}
}
Sandbox API: Behind the scenes
The tricky part isn’t on the Ember side of the implementation. Rather, it has to do with how the sandbox endpoints are implemented. One might be tempted to think we could just set up a controller action to respond to the checkout request this way:
defmodule MyApp.Test.Endpoints.SandboxController do
use MyApp.Web, :controller
def checkout(conn, _) do
:ok = Ecto.Adapters.SQL.Sandbox.checkout(MyApp.Repo)
metadata = Phoenix.Ecto.SQL.Sandbox.metadata_for(MyApp.Repo, self())
send_resp(200, encode_metadata(metadata))
end
def checkin(conn, _) do
# ...
end
defp encode_metadata(metadata) do
# ...
end
end
However, this approach quickly falls apart because the process that owns the connection would have already been terminated by the time the check-in is requested.
A solution for keeping the owner process alive is to create a GenServer to handle the session:
defmodule SandboxSession do
use GenServer
def start_link(repo, opts) do
GenServer.start_link(__MODULE__, [repo, opts])
end
def init([repo, opts]) do
:ok = Ecto.Adapters.SQL.Sandbox.checkout(repo)
{:ok, %{repo: repo}}
end
def handle_call(:checkin, _from, state) do
:ok = Ecto.Adapters.SQL.Sandbox.checkin(state.repo)
{:stop, :shutdown, :ok, state}
end
end
When a POST /api/sandbox request is received, we spawn a new process to own the sandboxed connection, so that it persists even after the server has finished responding to the request. The code snippet above is a simplified example. In reality, we would need to make the code more robust by adding proper timeout handling, in case there is a longer than acceptable delay between checkouts and checkins. We also need to have this sandbox session be a worker that is managed by its own supervisor process.
You can explore the details of the final solution in the PR that brought forth this feature in phoenix_ecto.
Today, the configuration to enable all of this functionality couldn’t be any simpler. After adding the Phoenix.Ecto.SQL.Sandbox plug to your application endpoint, you just need to configure the sandbox route name, the name of the request header, and the repository:
plug Phoenix.Ecto.SQL.Sandbox,
at: "/sandbox",
header: "x-user-agent",
repo: MyApp.Repo
Behind the scenes, phoenix_ecto takes care of all the necessary routing, process spawning, supervising, and timeout handling.
Concurrent Ember acceptance tests
The story wouldn’t be complete without a word on concurrency. While hitting actual API endpoints and interacting with the database has its advantages, it also makes the tests slower than if the server responses were mocked out.
On the backend, concurrency is a no-brainer if you use PostgreSQL, as the database already supports concurrent tests with the SQL Sandbox.
On the frontend, concurrency can be enabled both by setting the parallel option in Testem, as well as installing ember-exam to your Ember app. The test suite can then be split into a configured number of partitions, and invoked with the --parallel flag:
ember exam --split=4 --parallel
The partitions would each execute within a separate browser instance, resulting in a significant decrease in total runtime. In one internal app, running 197 tests in 4 separate partitions, the total test duration dropped from 73 seconds down to 35 seconds. Of course, your mileage may vary greatly as the distribution of the tests across partitions isn’t guaranteed to be balanced. ember-exam only splits by ES module (in other words, by test file), and some modules could contain a much larger number of tests than others. Nevertheless, the clear advantage here is that we are able to run tests that hit the API server concurrently, without having to worry about data from one test affecting another.
Where do we go from here?
This post explained how we went about implementing end-to-end tests in a Phoenix-powered Ember app. With all those details now being hidden away inside phoenix_ecto, there is actually very little configuration required to enable this kind of testing.
As we strive to improve the end-to-end testing landscape, we can start shifting our focus onto another important piece of the picture: how to provision and manage the seed data that the tests will end up consuming. Stay tuned for more on this in a future post!
