Testing server errors with Mirage

Testing network requests that fail in an Ember app is currently a somewhat inconsistent experience.

To help us sort through it all, let's categorize an Ember app's network requests into two buckets: routing requests and non-routing requests.

Testing routing requests

Routing requests are network requests that Ember's router is aware of, which include any promises returned by the model, beforeModel and afterModel hooks:

// post/route.js
export default Route.extend({
  model() {
    return this.store.findRecord('post', 1);
  }
})

These requests are special, because Ember has a built-in mechanism for dealing with routing requests that fail: the error substate. By defining an error template for a particular route (or for the application route which serves as a catch-all), Ember's router will smoothly transition an app that makes a routing request and gets a 400/500 in response from the server to the corresponding error state.

Let's say we have a post route that displays a blog post. We can define an error template for it in case this route's model hook fails:

{{! post/error/template.hbs }}
<h1>Whoops!</h1>
<p>
  Sorry, we couldn't load that post.
</p>

{{#each model.errors as |error|}}
  <p>{{error}}</p>
{{/each}}

Now if our server responds with an error, this template will render and show our user some feedback about what's happened.

Let's write a test for this behavior:

test('if the index route errors, I see a message', async function(assert) {
  server.create('post');
  server.get('/posts/:id', { errors: ['The site is down'] }, 500); // force Mirage to error

  await visit('/posts/1');

  assert.ok(find(':contains(The site is down)').length > 1);
});

Our new test works great! – with one exception. Our console is noisy:

This is because Ember Data throws an exception when the server errors, and Ember's test adapter logs the exception.

This exception from Ember Data is helpful in development because we get notified when one of our endpoints isn't functioning as we'd expect. But in this test, we're deliberately putting our server in a state where it will respond with a 4xx/5xx payload, and our app is behaving exactly as we expect it to. Thus, the console message is noisy and distracting.

Testing non-routing requests

All other network requests in an Ember app fall outside of Ember's router, and thus the error handling via route substates doesn't apply. So, what happens when we try to write a test for one of these requests?

Let's say we can edit blog posts in our app. When we click “Save”, we invoke an Ember Concurrency task:

export default Component.extend({
  save: task(function*() {
    yield this.get('model').save();
  })
})

In our template, we use the task's derived state to show our user feedback if the request fails:

<button onclick={{perform save}}>
  Save
</button>

{{#if save.last.isError}}
  <p>Whoops - your changes weren't saved! Try again.</p>
{{/if}}

Let's write a test for this.

test('if saving a post errors, I see a message', async function(assert) {
  server.create('post');
  server.patch('/posts/:id', { errors: ['The site is down'] }, 500); // force Mirage to error

  await visit('/posts/1');
  await click('button:contains(Edit post)');
  await fillIn('textarea', “New post body');
  await click(':contains(Save changes)');

  assert.ok(find(':contains(Woops - your changes weren't saved!)').length > 1, 'I see the message');
});

If we try to run this test, it will fail:

Notice that our assertion actually passed - we can see the green bar next to “I see the message”, so our template and logic is working as we expect. But, there's a failing assertion that caused the suite to go red, and that's because the exception from Ember Data is no longer being managed by Ember's router. QUnit sees this as an unexpected error, and as a result fails the suite.

We could use a try...catch block on our Ember Concurrency task so our code never throws, but then the save task wouldn't be in an isError state, and we wouldn't be able to use the nice declarative template we already have here.

The truth is that our code is working fine, and the app is behaving exactly as we expect. The real problem lies in how we've written our test and how QUnit is responding to our app's behavior.

A single approach for all server errors

Ok, so we’ve covered out-of-the-box behavior for both routing requests and non-routing requests, and they both left something to be desired: routing requests pass our test suite, but they dirty up our console; and non-routing requests add a failing assertion because of Ember Data’s uncaught exception.

What we really want is a single way to write a test that says,

• GIVEN that my server will error in response to XYZ network request
• WHEN my app makes a request to XYZ
• THEN I expect my app to behave in this way

If our app behaves in exactly that way, we want our test suite to pass without any extra console noise or failing assertions. So, let's work through these issues and do just that.

We'll start with our first test, the routing request for the index route that passes but leaves us with a noisy console. The console message comes from the function Ember.Logger.error. If we replace it with a noop

test('If the index route errors, I see a message', async function(assert) {
  Ember.Logger.error = () => {};
  // rest of test

the console message goes away!

But, Ember.Logger.error is there for good reason, and if something happens that we don't expect, we'd want to know. So let's not throw the baby out with the bathwater.

test('If the index route errors, I see a message', async function(assert) {
  // Intercept error logging
  let originalLoggerError = Ember.Logger.error;
  Ember.Logger.error = () => {};

  server.create('post');
  server.get('/posts/:id', { errors: ['The site is down'] }, 500); // force Mirage to error

  await visit('/posts/1');

  assert.ok(find(':contains(The site is down)').length > 1);

  // Restore error logging
  Ember.Logger.error = originalLoggerError;
});

Awesome! Let's pull the intercept/restore bits out into separate helpers.

let originalLoggerError;

function intercept() {
  originalLoggerError = Ember.Logger.error;
  Ember.Logger.error = () => {};
}

function restore() {
  Ember.Logger.error = originalLoggerError;
}

test('If the index route errors, I see a message', async function(assert) {
  intercept();
  server.create('post');
  server.get('/posts/:id', { errors: ['The site is down'] }, 500); // force Mirage to error

  await visit('/posts/1');

  assert.ok(find(':contains(The site is down)').length > 1);

  restore();
});

Now intercept/restore are reusable! But we have one problem. What if a message we don't expect is logged while all error messages are being intercepted? We'd want to know about such things.

So let's improve our intercept function to take a string that must match the error the developer expects the actual Logger to display. And if they don't match, we'll invoke the original function.

function intercept(expectedText) {
  originalLoggerError = Ember.Logger.error;
  Ember.Logger.error = (...args) => {
    let loggedText = args.join(' ');
    let actualMatched = expectedText && loggedText.match(expectedText);

    if (!actualMatched) {
      originalLoggerError(...args);
    }
  };
}

Now in order to clean the console, the developer must pass a string into intercept that will match the text that's being logged out:

test('If the index route errors, I see a message', async function(assert) {
  server.create('post');
  server.get('/posts/:id', { errors: ['The site is down'] }, 500); // force Mirage to error
  intercept('GET /posts/1 returned a 500');

  await visit('/posts/1');

  assert.ok(find(':contains(The site is down)').length > 1);

  restore();
});

Now we have a clean console, and we'll also see any errors thrown that we didn't expect to see. Pretty neat!

Improving the interceptor for non-routing requests

Now that routing requests are taken care of, let's try our interceptor on non-routing requests. If we look at the image above, we see that during our editing test the console error said “PATCH /posts/1 returned a 500”. So we'll use that in our intercept function.

test('if saving a post errors, I see a message', async function(assert) {
  server.create('post', { text: 'Old post body' });
  server.patch('/posts/:id', { errors: [ 'A bad thing happened' ] }, 500);
  intercept('PATCH /posts/1 returned a 500');

  await visit('/posts/1/edit');
  await fillIn('textarea', 'New post body');
  await click('button:contains(Save)');

  assert.ok(find(':contains(Woops - something happened!)').length > 0, 'I see the message');
  restore();
});

This takes care of the console message, but unfortunately we still have the failing assertion (and thus red suite) we saw above. The problem is there's one more piece of the puzzle. We need to also overwrite Ember.Test.adapter.exception. This is another function that's designed to notify us of unhandled exceptions in our apps. But the default implementation bubbles rejected XHR requests as errors up to QUnit's runner, which is why we have the additional failing assertion.

So, let's force this to be a noop in our intercept method, and restore it alongside Ember.Logger.error in our restore method.

let originalLoggerError;
let originalTestAdapterException;

function intercept(expectedText) {
  originalLoggerError = Ember.Logger.error;
  originalTestAdapterException = Ember.Test.adapter.exception;

  Ember.Logger.error = (...args) => {
    let loggedText = args.join(' ');
    let actualMatched = expectedText && loggedText.match(expectedText);

    if (!actualMatched) {
      originalLoggerError(...args);
    }
  };
  Ember.Test.adapter.exception = () => {};
}

function restore() {
  Ember.Logger.error = originalLoggerError;
  Ember.Test.adapter.exception = originalTestAdapterException;
}

Boom - our test passes, and our console is clean!

We now have a clean, consistent way to test errors for any type of network request in our Ember applications.

An asynchronous assertion helper

After using this pattern a few times, what I really wanted was something that mirrored QUnit's throws API:

// This lets us assert that we expect some code to error
assert.throws(() => {
  throw 'something happened';
}, 'something happened');

throws lets you run some arbitrary code that you expect to raise an exception, and assert against the text of that exception.

Let's look at our first test again:

test('If the index route errors, I see a message', async function(assert) {
  intercept('GET /posts/1 returned a 500');
  server.create('post');
  server.get('/posts/:id', { errors: ['The site is down'] }, 500); // force Mirage to error

  await visit('/posts/1');

  assert.ok(find(':contains(The site is down)').length > 1);

  restore();
});

We know that it's the call to visit that triggers the network request that ultimately fails.

What if we could wrap that visit line in something like a throws block, assert that the correct error was thrown, and then go on to assert that the app rendered the UI in the correct state and presented the error message to the user? We'd also still want to know about any other exceptions that were thrown that we didn't explicitly whitelist.

I use Ember CLI Custom Assertions on most of my projects, so I fleshed out this idea as a custom async assertion. Here's what I ended up with. It uses the pushResult API from QUnit to explicitly add assertions to our tests - a passing one if an exception is raised with the matching text, and a failing one otherwise. This way, we will have an explicit test failure if there's an exception, instead of just a console warning. Much better!

With this, we can now write our two server-error tests like this:

test('If the index route errors, I see a message', async function(assert) {
  server.create('post');
  server.get('/posts/:id', { errors: ['The site is down'] }, 500); // force Mirage to error

  await assert.asyncThrows(() => {
    return visit('/posts/1');
  }, 'GET /posts/1 returned a 500');

  assert.ok(find(':contains(The site is down)').length > 1);
});

test('if saving a post errors, I see a message', async function(assert) {
  server.create('post', { text: 'Old post body' });
  server.patch('/posts/:id', { errors: [ 'A bad thing happened' ] }, 500);

  await visit('/posts/1/edit');
  await fillIn('textarea', 'New post body');

  await assert.asyncThrows(() => {
    return click('button:contains(Save)');
  }, 'PATCH /posts/1 returned a 500');

  assert.ok(find(':contains(Woops - something happened!)').length > 0, 'I see the message');
});

asyncThrows gives us a block which we can use to return a promise that we expect to raise an exception. This saves us from having to restore() the original loggers, like we did above. Also, note that we now await on these assertions, since they're async.

We now have a clean API for testing server errors, our UI and developer exceptions are fully tested, and our CI console is clean and happy again! Our work here is done. 🕵️‍♂️

A big thank you to Wesley Workman and Toran Billups for sharing the code in this post on this issue, and Robert Jackson and Ryan Toronto for review.