Levels of abstraction in testing

When writing tests, always consider the level of abstraction of the system under test.

The System Under Test (SUT) is the thing whose behavior is being verified by the test. It could be a function, component or entire application. For acceptance tests, the SUT is the entire application.

SUTs operate at different levels of abstraction. Applications are consumed by users, but components are consumed by other developers. It is important to be aware of the level of abstraction of the SUT, and to write your testing code at the same level. Otherwise, you will write brittle tests that depend on things that change more often than the behavior you're testing.

When writing tests, act as if the user of the SUT was sitting next to you, and you were telling them in words how the behavior you're testing actually works. Then, write your test code using similar words.

Something that had a big impact on how I write tests was learning about levels of abstraction in testing.

So the idea here is that, all the code within our test suite is at a certain abstraction level, and it's up to us to ensure that any code within a single test is all at the same consistent level pf abstraction all the way through.

So what does that mean? Well let's take a look at a test.

test('visiting /login', function() {
  let session = this.application.__container__.lookup('service:session');

  visit('/login');
  fillIn('input.username', 'bobwiley');
  fillIn('input.password', '12345');
  find('input').trigger('click');

  andThen(() => {
    assert.equal(session.get('userIsLoggedIn'), true);
  });
});

Here we're looking at an example of an acceptance test that's verifying some login behavior. At the top it looks up this session service in the global container; the test then fills out a form; and then in the andThen block we assert that the session's isAuthenticated property is true.

So, what do I mean when I say levels of abstraction? Well, this test has at least two different levels of abstraction in it. First we have these helpers in the middle that are being called: visit, fillIn and so on. And second we have this session object that we get at the beginning of the test, and we also verify the property down here.

Now, the code here in the middle that uses the test helpers lives at a very high level of abstraction. This API is designed to mimic what a user would do when interacting with our application - and so fillIn lets us the user fill in a text box, without worrying about how that text actually gets into that input field.

A lower-level equivalent of fillIn might look something like this:

let input = $('input.username');
$('input').trigger("keyup", {which: 50});

This is pseudocode but you get the idea - this is lower-level because it's using these lower-level APIs to interact with our web app, in the same way that a programmer would in code. But fillIn is an abstraction around this low-level code that's designed from the perspective of a user actually using our app - and because of this we would say that fillIn is a more expressive way to represent what it is we're trying to test.

Now, "what it is we're trying to test" is of critical importance when it comes to identifying these abstraction layers. There's even a term for this: it's called System Under Test, which is sometimes abbreviated as SUT. The idea behind System Under Test is that every test we write should be focused on verifying the behavior of some piece of our software, whether it's a function, a component, or something else.

So, for acceptance tests, the System Under Test is actually our application as a whole, and this is how we can know whether or not the code in our test is at the right level of abstraction. The abstraction level of code in the test should match the abstraction level of the System Under Test.

And this is why the test helpers like visit and fillIn exist - again, these are more expressive of the behavior of the end-user, which is exactly what this test is trying to capture.

So, now we know this test should really be from the perspective of the end user - but we have this code that's dealing with the session service. The question we need to ask is, do our users care about a session service object - or any other Ember object for that matter? Well, no - they interact with our app by using a browser, a mouse and a keyboard, and seeing what's actually rendered to the screen.

When you go to Twitter.com and log in, how do you know that you were able to successfully log in? Do you open the JavaScript console and start inspecting variables? Well no - you know you were able to log in because the home page rendered, you're able to see your tweets and so on. And that's the kind of behavior we want to capture in these acceptance tests.

So, we want to fix this test so that the assertion is at the same level of abstraction as our helper code. Instead of checking the value of a service, we'll assert against the rendered output of our application - let's say that in this application, when you successfully log in, you see a title rendered to the screen somewhere and it says "Your posts".

assert.equal(find('.title').text(), 'Your posts'))

Now, it's important to note that this session object's isAuthenticated property will still be true - we haven't changed that behavior - but this property on this service object is just an implementation detail the app as it currently stands, and this is not actually demonstrating that the app is functioning correctly from the perspective of the user

Implementation details in tests are typically at the wrong levels of abstraction, and they make our tests brittle - so, even though session.get('isAuthenticated') is true, we want to remove this code from our test. And we'll remove the container lookup here as well.

Now before we leave, there's two more things we want to do.

First, we're using the visit and fillIn helpers here, but then on line 5 we use find('button') and then we call trigger('click'). Well there's actually a test helper called click that we can use instead, like this

click('button');

Now, this might seem trivial, but again its important to have all the code of your test at the same level of abstraction all the way through - and find('button').trigger('click') is just not how you would tell a user to use your application.

One thing I've found helpful is to imagine yourself talking with the user of the System Under Test. Now sometimes that user could be another developer, but in this case, the user is an end user. And you might say, How would I talk to this user in real life, describing the behavior of this application? If I were telling them how to log in, what would I say? I would say things like visit the homepage, fill in your username and password, click the login button, and then we see the homepage being rendered.

So you could imagine even changing this part of the test to something like

click('button:contains(Log in)');

Now, there's many techniques for how to actually write these selectors - that's not what I want to focus on in this video. The important point here is that, these test helpers are a more expressive way of describing what it is we're trying to capture in this test, which is why they're an improvement over those lower-level helpers we looked at earlier.

So we'll delete this, keep our click helper, and we have one more improvement we want to make - and that has to do with the name of the test.

If we were talking about this user behavior to someone on our team, how might we describe it? We probably wouldn't say "visiting /login" - we'd say something like "a user can login to the app". Sometimes you'll see a style where this would be written as "I can log in to the app", but again the general point here is that all the code in this test should exist at the same level of abstraction.

Ok, and now our test is looking much better.

---

To sum up, when writing tests it's good to be aware of the abstraction level of the system under test, so the code that you're writing can match it.

When tests are written at a consistent level of abstraction, they are less brittle - because they don't depend on implementation details that can change more frequently than the actual behavior that you're testing; and they're easier to read, write and understand - because they're written from a uniform perspective.

In acceptance testing, we're testing our entire application - so our code should be written from the perspective of an end user using our product with a mouse and keyboard. When writing acceptance tests, imagine yourself sitting next to somebody using your app, and walking them through the behavior that you're testing.

If we were writing tests for a component, well now the user of the System Under Test is other developers, and they interact with our component in code. And so that's the level of abstraction of the test, and our testing code should reflect that. So in that case, our assertions would be on things like the properties of the component, whether certain actions were fired and so on.

But in general, the idea is to match the level of abstraction of your testing code to the actual System Under Test.

Now one last thought before we go: if I look at this test, I see a problem — why is Bob able to log in with this username and password? Does he even have an account? It's not really clear from looking at this test.

That'll be the subject of another video.