Smart compression for your Ember app

Here at DockYard we always try to deliver the best experience possible in our web apps: pages that load fast and stay fast.

As a digital agency, we are always starting new projects and we aim to make the good practices that keep our apps in good shape part of our regular workflow. In the development world that can only mean one thing: Tooling.

Ember.js is a fantastic choice to be able to reuse knowledge, patterns, libraries, and good practices between all the projects we create. One of its core values is about sharing efforts instead of reinventing wheels. The tooling the community has built around Ember CLI is solid and a lot of complex tasks are just one ember install ... away, which is something that still blows my mind from time to time.

Sometimes though, we identify gaps to be filled in the community tools and when we do, we don’t hesitate in rolling up our sleeves and contributing back upstream so all the community can take advantage of the improvements. One good example is the awesome work that Marten Schilstra has done with ember-service-worker to make building PWAs a no-brainer for all Ember.js apps out there.

This is the story of how we decided we could reduce a lot of people’s app for up to 20% with virtually no work for them.

Act 1: The gap

It turns out that with the new year we just started building several apps that, to our joy, do not have to support Internet Explorer anymore.

Thanks Santa! We must have been good this year.

Personally this is the first time I have the luxury of building an app only for evergreen browsers and we got very excited because all the cool new features we will be able to take advantage of. And we did: CSS grid, Service Workers, async/await… It has been nothing short of liberating.

Then the time came where we had to start deploying the app and as usual we tried to take advantage of Ember CLI Deploy using the lightning approach.

We installed the ember-cli-deploy-lightning-pack and we had the app deployed in a couple hours with gzip compression and storing assets in S3 behind a CDN for superfast load times from everywhere.

As I’ve done several times in the last weeks when I started a task, one of the first questions I asked myself was: How can I make this better now that I only have to support evergreen browsers?

I remembered reading articles about new compression algorithms better than the traditional gzip the web has been using for the last 18 years and I investigated zopfli, a backwards-compatible gzip/deflate compression algorithm that can result in files up to 10% smaller than standard gzip), and brotli, a totally new compression algorithm that produces files up to 25% smaller than gzip and 15% better than zopfli.

I checked the browser support in caniuse.com and it was very good, so I went for it!

A quick look in emberobserver.com pointed me to ember-cli-deploy-brotli which seemed to be almost a drop-in replacement for ember-cli-deploy-gzip which comes with the lightning pack.

There is an addon for almost anything today!

To make use of it I had to break apart the lightning-pack into its individual addons and replace the gzip one with the new one:

--- a/package.json
+++ b/package.json
@@ -24,7 +24,13 @@
     "ember-cli-deploy": "^1.0.2",
-    "ember-cli-deploy-lightning-pack": "^1.2.2",
+    "ember-cli-deploy-brotli": "^0.2.1",
+    "ember-cli-deploy-build": "^1.1.1",
+    "ember-cli-deploy-display-revisions": "^1.0.0",
+    "ember-cli-deploy-manifest": "^1.1.0",
+    "ember-cli-deploy-redis": "^1.0.2",
+    "ember-cli-deploy-revision-data": "^1.0.0",
+    "ember-cli-deploy-s3": "^1.2.0",
     "ember-cli-eslint": "^4.2.1",
     "ember-cli-htmlbars": "^2.0.1",
@@ -40,6 +46,7 @@
     "eslint-plugin-ember": "^5.0.0",
+    "iltorb": "^2.0.3",
     "loader.js": "^4.2.3"
   },

I found a couple of rough edges that I fixed upstream but today that is all that you have to do today to take advantage of brotli compression if you’re using the lightning approach.

The results were great immediately.

Before:

After:

The JS files got ~18% smaller on average and CSS files even a bit more. Fantastic, right?

Not quite.

Act 2: The human factor

We are humans and we make mistakes. We’re inherently flawed. That’s why having code styleguides is not enough and we invented linters to enforce them. Best intentions and pinky-swearing to do things right never ends well.

What was my mistake then?

This is the support matrix for brotli:

This is the /config/targets.js for my application.

module.exports = {
  browsers: [
    'last 2 Chrome versions',
    'last 2 Firefox versions',
    'last 2 Edge versions',
    'last 2 ios versions',
    'last 2 ChromeAndroid versions'
  ]
};

If the problem doesn’t seem evident to you that makes me feel better, because I didn’t see it either. Let’s expand the iOS column a bit:

It turns out that the app broke miserably when tested in iOS 10.3.

I misread the support table and wrongly assumed that brotli was supported across the board when it wasn’t. Apple’s iOS 11.3 should be around the corner and we’ll be able to leverage it soon but not yet.

It happens that it’s hard to know exactly what features are supported by your target browsers, especially when your policy is not "last 2 XXX version" but something less clear like supporting all browsers with more than a 5% market share in Canada:

module.exports = {
  browsers: ['>5% in CA']
};

In those situations, it is almost impossible to know what features you can use, and even worse, it’s changing every day as browser usage changes.

Act 3: Declare your browsers, let the software decide the best compression.

The bottom line is that, as humans, we shouldn’t be making these kinds of decisions that are easy to get wrong and which answer changes over time. Instead, it should be our tooling the one that gives us the best compression possible for our supported browsers automatically, without us having to worry about it.

For a while now, Ember.js has enjoyed a universal spot to declare what browsers you intend to support: the /config/targets.js. Every piece of tooling knows exactly where to lookup that information to automate flows for you.

That’s why I created ember-cli-deploy-compress, which is sort of the love child of ember-cli-deploy-gzip and ember-cli-deploy-brotli.

It automatically reads your project’s supported browsers and uses the evergreen information from caniuse.com to decide wether or not you can leverage brotli or not.

If you start using it today maybe you have to continue to use gzip for a little while like I do, but as soon as iOS 11.3 is released your app will become almost 20% smaller without you having to worry about the details.

Go try it. Perhaps you find you save 20% on your app with a 5-minute change. And even if you can’t right now, you’re set to do it as soon as you remove support for the browser preventing you from doing it.

If you have to support old browsers, there is still hope

The addon has decided to take the smart approach of automatically switching from gzip to brotli when all your browsers support it. When in combination with ember-cli-deploy-s3 assets will be uploaded with the configuration to be served with the right Content-Encoding: br header.

This is as far as the addon can take you out of the box, but if you are serving your assets yourself with some sort of web server like nginx or you use a CDN that supports custom rules, you can configure the plugin to generate both gzip and brotli versions every asset (with separated .gz and .br file extensions) with the following configuration in your config/deploy.js:

module.exports = function(deployTarget) {
  let PROJECT_NAME = 'scripts-ui';

  let ENV = {
    compress: {
      compression: ['gzip', 'brotli']
    },
    ...
  };
}

Then, browsers that support brotli compression will send requests with Accept-Encoding: gzip, deflate, br, which your server can use to conditionally serve the best compression when supported.

But this is an exercise for the reader 😜