Under-the-hood of test runners (e.g. Mocha)

Test runners are a very important part to the modern JavaScript application. Without which we could not be able to run any tests at all. They are fairly straight forward to understand in terms of what they do, they run your tests and print output. However something which is not as straight forward is how they work and the mechanisms they utilise to enable the command-line interface that we are all so familiar with.

A video for this post can be found here. This is part of my “under-the-hood of” series:

I have been a member of the MochaJS core team for a little over 2 years so I feel I have a good understanding how the Mocha test runner works. This post aims to shine a light on how the Mocha test runner works and the codebase design, by walking through a slimmed down version of the tool. The real one is many thousands of lines of code so I have created a slimmed down version which can be found here. All of it has been taken from the real MochaJS codebase, however only the core functionality has been included. Hopefully this will help us focus on and learn about the individual aspects which are critical to its core functionality. It will be a mix of code snippets, links to mochas repository and links to my repository.

The article today will be broken down into 3 parts:

  1. Introduction to our test runner
  2. Building our runner: Parsing phase
  3. Building our runner: Execution phase (the bulk of the article)

“Execution phase” is broken down into:

  1. Loading step
  2. Running step
  3. Mocha.run
  4. Reporters
  5. Runner.run
  6. In-depth look at the code for runner.run
  7. “Running step” summary

Before we move on here is some of the basic terminology for the core functionality of any test runner:

  • Test - Functions with an assertion/s inside
  • Suite - Collection of tests
  • Hook - Functions run at specific times in a test runners lifecycle
  • Reporter - Determines the presentation of the output
  • Interface - The methods that will be use inside each test e.g. describe and it (also known as UI)
  • Runner - A given run of a suite, tests, hooks, reporter etc (uses an instance of a Runnable)

1. Our test runner

Given that test runners can include a huge variation of functionality, not all of which is necessary for the basic job of the test runner, today we will be building our own test runner which includes:

This means we will be ignoring all other features, for example option validation, timeout management, slow test flagging, pending states etc.

Our runner output will look like this.

  • One test fails
  • Two tests in different suites pass
  • Second suite logs all hooks

Modules in MochaJS

It is necessary to understand the types of Modules that can be used in and are found in the MochaJS codebase, as we will be sticking to the same.

  • MochaJS uses CJS module inside the core, so it does not support ESM natively yet.
  • MochaJS accepts ESM for test files.

It is also worth noting that Mocha does not have a transpilation step (yet, plans are at work) so most of the code is ES5.

Our runners basic breakdown

There are 3 parts to our runner

1. The entities required for a test runner

This includes everything from above i.e Tests, Suites, Hooks, Reporters, Interfaces, Runner and Runnable.

2. The Parsing phase

This is the first of 2 steps for the runner. The goal with this phase is to build a coherent CLI for the user.

MochaJS makes use of yargs. You can see the starting point in lib/cli/cli.js. As it is ES5 it uses prototypes so it can use ES5 class instances.

3. Execution phase

The final part of the runner relies on a Mocha instance to be handed to it from the parsing phase. This part creates an instance of a Runnable and executes everything from the suites and tests to the hooks and reporters.

2. Parsing phase

Here we create a yars instance, it includes the basics required to have an intuitive command-line interface (e.g. asking for version information or help). We attach a commands object (see details below).

I have included in comments the location of the code in the real Mocha codebase.

Below details the commands object using a child yargs instance. It will set options and run validation checks (see methods options() and check())

Finally if it passes all these it will build a new Mocha instance (see new Mocha(argv)) and hand it to next phase ( runMocha(mocha, argv))

My code for the parser can be found here.

Building the Mocha instance

Let us look at what creating a new Mocha instance does by examining its constructor plus the ui and reporter instance methods.

So above:

  • Constructor: Creates instance of our Root Suite and attaches to this.suite (see this.suite = new Suite). Suite inherits pub/sub from EventEmitter.
  • ui method: Based on the interface we bind events, which attach interface methods, onto our suite context this.suite. See details on bindInterface below.
  • reporter method: Creates instance of a Reporter and attaches to this._reporter (see this._reporter = _reporter). An example of a reporter is the Spec function (see Mocha.reporters.spec)

How does the Interface work with our Mocha instance?

For our BDD interface it is set below, called from the ui() method ( here) and initially the constructor.

Let us walk-through whats is going on.

Each time the event is fired (inside the execution phase coming later) the listener callback runs. The callback is given the params context, file and our mocha instance..

So based on an event (i.e. on("EVENT_FILE_PRE_REQUIRE") ) it attaches the interface methods, see context.after = ..., context.describe = ..., context.it = ..... Many of which come from a utility object called common.

Here the describe() method creates and returns a new Suite via suite.create(). The it() method creates and returns a new Test via new Test().

So thats it for the parsing phase, we now have our Mocha instance which has the reporter and the suite with the interface getting attached each time.

3. Execution phase

Now we have everything that we need to execute our test runner. We will be loading files and then running it.

1. Loading

Our parser calls runMocha inside the handler() (in parsing phase above).

We start by building an array of all the files to run. We pass them from the spec array.

We have a simplified version of this, but in the real mocha codebase there are many checks done against the spec items to check for globs, files and directories (see lib/cli/collect-files.js and lookupFiles).

Next we call loadFilesAsync. This loads the ESM and CJS files asynchronously, emits the "EVENT_FILE_PRE_REQUIRE" event and then runs the root suite.

The "EVENT_FILE_PRE_REQUIRE" event had the listener set in our bddInterface function above ( here). So it is now that the interface methods are added to the suite context.

It is under esm-utils.js in the real mocha codebase.

Finally it runs the mocha.run() which moves us onto the running step.

2. Running

This phase is split into the following sections:

  1. Mocha.run
  2. Reporters
  3. Runner.run
  4. In-depth look at the code for runner.run
  5. “Running step” summary

mocha.run

Lets walk-through what it does

Create instance of a Runner (file here) and add stats collecting.

Similarly to the Suite, the Runner inherits from EventEmitter. This is for the publish and subscribe events functionality that Mocha relies on.

Here is our stats collector. On pass/fail/end events increment pass/fail/end on the runner.stats object. In the real mocha codebase found lib/stats-collector.js.

Back to our Mocha.prototype.run. Next we create an instance of a Reporter using the Runner (see new this._reporter(runner, options))

Reporters

Inside the reporter we add spec-specific listeners onto the runner events. For example on EVENT_RUN_END even will run reporter.epilogue().

See below for a snippet from the spec reporter which includes events for

  • Run begin
  • Test pass
  • Test fail
  • Run end

So essentially as the runner is running our tests, the Spec reporter is printing output.

Below is the epilogue from the Base reporter. It processes the details on this.stats and once the tests are finished, outputs a final statement detailing the summary of the run.

See the spec and base files for more.

Finally our mocha.run triggers return runner.run(done).

runner.run

So lets walk-through runner.run

  • Creates a listener for EVENT_RUN_END events. It will run the fn callback with failure details
  • Emits an EVENT_RUN_BEGIN event
  • Executes each suite on Root suite, via runSuite. This is very important; for each suite it will run all tests and hooks. Once that is finished it emits anEVENT_RUN_END event

So not that crazy but we need to dig deeper into runSuite.

This is where all the suite, tests and hook are executed. To summarise how it works, starting with the Root Suite it:

  1. Executes all before hooks for the suite
  2. Then runs all tests for the suite
  3. Then runs all after hooks for the suite
  4. The moves onto the next suite.

If you are interested in more details, the next section looks at the code necessary to run the above. Alternatively skip to the summary section after this to see an overview.

In-depth look at the code for runner.run.

The total size of all the code required is quite large, so in an attempt to make it easier to follow the mechanics below I have included most of the required code for them.

The Runner methods we will cover are:

  • runSuite
  • hook
  • runTests
  • runTest
  • hooks

I will explain each one as we go along by doing both code comments and text outside the function. Use whichever you prefer.

Runner.runSuite

  • emit EVENT_SUITE_BEGIN
  • trigger the beforeAll hooks with a callback.

hook callback (after beforeAll)

  • trigger runTests handing suite and next callback.

next function

  • grab the next suite in the suites array
  • if no suite is left trigger done
  • else call itself via runSuite with the next suite.

done function

  • run all afterAll hooks
  • emit EVENT_SUITE_END event

Runner.hook

  1. Grab all hooks with the name given
  2. trigger next callback with hook index

next function

  • using index grab the current hook
  • if no hook call fn
  • else set test context
  • lastly execute the hook (hook.run) with a callback

hook.run method

This comes from the Runnable.run method below as the hooks inherit Runnable.

Call the function with the current context.

Runnable.run(fn) {
var ctx = this.ctx;
fn.call(ctx);
}

run callback

  • increment the hook index and call next again (triggering the next hook under the given name)

Runner.runTests

  • trigger next callback

next function

  • grab the next test
  • if no tests are left trigger callback
  • else run beforeEach hooks

hooks callback

  • trigger runTest

runTest callback

  • if there is a test failure emit EVENT_TEST_FAIL event
  • else emit EVENT_TEST_PASS event
  • for both branches emit the EVENT_TEST_END event and run afterEach hook

Runner.runTest

  • early return if no test available
  • run the test via test.run
  • handle test errors

Runner.hooks

  • call the next function

next function

  • if there are no suites left to run trigger the callback
  • trigger the hook for the given name
  • after that run hooks for the next suite

“Running step” summary

So essentially the core flow involved in the runner mentioned above is:

1. The Runner

  • Attaches event listeners
  • Updates stats property
  • Triggers the Root suite

2. The Suite

  • Executes the beforeAll hooks
  • Begins execution of the tests
  • Executes the afterAll hooks
  • Executes the next suite

3. The Tests

  • Executes the beforeEach hooks,
  • Executes the tests
  • Emits the test pass, fail and end events
  • Handle any unexpected failures
  • Executes the afterEach hooks

4. The Hooks

  • Executes all the hooks under the given name

5. The Reporter

  • Attaches event listeners
  • Processes the stats
  • Logs output to terminal/browser etc.

Thats it

So there you have it, the Mocha test runner broken down from the parsing to the execution.

Thanks so much for reading or watching. I hope you have found this article useful or at least interesting in some ways. You can find the repository for all this code here.

Thanks, Craig 😃

JS "under-the-hood of" series https://bit.ly/36GLhlo. dev @ fiit.tv. Formerly BBC. https://craigtaub.dev/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store