DDG search drop in powered by ddg.patdryburgh.com
tl;dr I extend my UI component unit/integration testing methodology to do E2E tests. Look at the code, I basically add lots of nice context-setter-upper functions (that’s the official term) that makes it work and make it relatively clean.
As hinted-to at the end (PS) of my previous post, recently after attempting (and succeeding, mostly) to set up full E2E testing in the backend of a haskell app I’m writing, I decided to switch to writing the tests in JS land (instead of Haskell land) due to some lack of library support for PhantomJS/Selenium.
This prompted a second look/improvement of some code I wrote quite a while ago that does my testing, with tape. Feel free to read the blog post where I switch from mocha to tape as it contains background on the scheme.
This post contains a bunch of odds and ends from revisiting that testing code, to extend it to support full E2E tests (using chromedriver and webdriverio).
While I was revisiting/porting some tests to this new project, I had to go back through a bunch of UI components on the frontend to make sure that all references to window were guarded with a fairly nonsensical line like var window = window || window, so that SystemJS would be happy, and not give me an undefined variable error. This problem was supposedly fixed, but I guess not, because I’m still dealing with it.
During the port I also found out (the hard way) that JSDOM changed their API/use patterns (since the last time I used it). Luckily for me, the code was written in such a way that this only required changing a few function calls on a few lines.
A bunch of UI components that I just copy-pasted from the old project to the new one underwent some breaking changes (as I found bugs). The right decision would probably have been to put these shared components in a proper repository and make changes there, but… I don’t want to spend too much time managing such a structure just yet, with only 2 projects using the components as-is.
wd driver (before I found out about webdriver.ioI spent a bunch of time trying to get wd to work like I expected it to (I thought I was being reasonble), and even after reading the docs and adjusting how I thought it should work to how it DID (or should have) worked, I still got nowhere. Turns out wd library is just kind of hard to use – ((it would launch the chrome driver through selenium**, but not do anything after that, which was weird to me, almost like it lost access to he session or something and all the commands just went into the ether.
This lead me to switch to webdriverio, which was much easier to use, and enabled me to use chromedriver directly. Here’s a github issue that really helped me getting webdriver.io set up to run with chromedriver.
test("login-page component should render properly on the app", {timeout: 10000}, t => {
TestUtil.withRunningTestAppInstance(opts => {
return TestUtil.withNewChromeDriverInstance(t, browser => {
return browser
.url(opts.info.appUrl)
.getTitle()
.then(title => t.assert(title.should.be.eql("App Title") , "Title is what we think it is"))
.then(() => t.end())
.then(() => browser.end());
});
});
});
TestUtil.withRunningTestAppInstance is a utility function that starts an actual instance of the application (like, it runs an executable), and executes a function you give it. The function you pass to withRunningTestAppInstance is expected to return a promise, which is how it knows when the test is done (and when to clean up the app instance it made.
TestUtil.withNewChromeDriverInstance is a utility function that returns a promise (needs to, for withRunningTestAppInstance’s sake), that requires you to give it a tape instance (the t argument), along with a function to run, that will return a promise. It does whatever setup is necessary to make a new chrome driver instance, and then runs your function, and does necessary cleanup afterwards.
I won’t show the implementations for those functions here because they ended up being reworked quite a bit, but take my word for it (and be glad you only have to see the working implementation later).
This setup gave me new app instances and chromedrivers on a per-test basis. Later I would realize that I didn’t need so many chromedrivers (because they do session management).
One of the big reasons I preferred the chromedriver + webdriver.io approach is that I didn’t have to subscribe to NightWatchJS’s way of writing tests. While I’m a fan of how easy the project is to use and the extensive documentation they provide, I’m not a fan of how hard it seems to use just the bits you want without subscribing to their way of writing tests.
After initial exploration, and lots of mistakes here’s what the code look like:
Test utility code
Here’s the NodeJS testing utility code that helps my stuff do the thing
var process = require("process");
var tape = require("tape");
// These things help to kill all the processes started by test code
process.on("SIGINT", () => PROCESSES_TO_KILL.forEach(p => p.kill()));
process.on("exit", () => PROCESSES_TO_KILL.forEach(p => p.kill()));
tape.onFinish(() => {
setTimeout(() => { // nexttick is not enough
PROCESSES_TO_KILL.forEach(p => p.kill());
}, WAIT_BEFORE_KILL_AND_EXIT_MS);
});
A bunch of things set up to try and make sure that the process closes out the started webdriver/app processes when it’s stopped.
var childProcess = require("child_process");
var process = require("process");
var getPort = require("get-port");
var chromedriver = require("chromedriver");
var wdio = require("webdriverio");
/**
* Run some code in the context of a running app instance
*
* @param {Function} tape - `test` function provided by Tape
* @param {string} name - The name of the child process that is going to be created
* @param {Object} options - Options to use with the running app instance
* @param {String|Function} options.cmd - Command to spawn in a child process
* @param {Object} [options.process] - Object containing config to use for the spawned child process
* @param {Object|Function} [options.process.env] - ENV variables that will be passed to the started process
* @param {Function} fn - The function to run with the app instance. This function is a passed an object with shape {info: {...}, process: ChildProcess }, with a possibly present error. If the function returns a promise, the child process will be alive as long as the promise is alive.
* @returns a Promise that resolves to an object containing information about the running child process
*/
var startChildProcess = (tape, name, options) => {
return new Promise(resolve => {
(options.port ? Promise.resolve(options.port) : getPort())
.then(function(port) {
options.port = port;
// Generate appropriate ENV to be passed to child process
if (options.process &&
options.process.env &&
typeof options.process.env === "function") {
options.process.env = options.process.env(options);
}
// Generate the data that will be passed back to the
if (typeof options.info === "function") { options.info = options.info(options); }
// Generate the command, if a function was provided
if (typeof options.cmd === "function") { options.cmd = options.cmd(options); }
// Spawn child process, keep a pointer to it to clean it up later
var generatedProcess = childProcess.spawn(
options.cmd.command,
options.cmd.args,
{env: options.process.env}
);
PROCESSES_TO_KILL.push(generatedProcess);
// Log output of childprocess if specified
if (options.logStdout) { generatedProcess.stdout.on("data", d => tape.comment(`${name}.stdout: ${d}`)); }
if (options.logStderr) { generatedProcess.stderr.on("data", d => tape.comment(`${name}.stderr: ${d}`)); }
tape.comment(`Process ${name} started on port [${options.port}]`);
setTimeout(() => resolve({info: options.info, process: generatedProcess, cleanup: () => generatedProcess.kill}), options.afterStartDelayMs || 0);
});
});
};
/**
* Run the provided function with an app instance running in a child process.
*
* @param {Function} fn - The function to run (likely containing tests/assertions). This function should expect an objects with shape {info: {...}, process: ChildProcess}, along with a possibly present error.
*/
exports.startTestAppInstance = tape => startChildProcess(tape, "App", {
afterStartDelayMs: 500,
process: {
env: (options) => {
if (!process.env.DB_MIGRATION_FOLDER) { throw new Error("DB_MIGRATION_FOLDER ENV variable missing!"); }
if (!process.env.FRONTEND_FOLDER) { throw new Error("FRONTEND_FOLDER ENV variable missing!"); }
return {
ENVIRONMENT: "Test",
DB_MIGRATION_FOLDER: process.env.DB_MIGRATION_FOLDER,
FRONTEND_FOLDER: process.env.FRONTEND_FOLDER,
PORT: options.port
};
}
},
info: options => {
if (!process.env.TEST_APP_BIN_PATH) { throw new Error("TEST_APP_BIN_PATH ENV variable missing!"); }
return Object.assign({
binPath: process.env.TEST_APP_BIN_PATH,
appUrl: `http://localhost:${options.port}`
}, options);
},
cmd: options => ({command: options.info.binPath, args: ["RunServer"]})
});
/**
* Executes a function in a context in which a chromedriver browser session has been made
*
* @param {Function} tape - The tape (`test`) function
* @param {Promise} chromeDriverInfoPromise - A promise that evaluates to information about the chromedriver
* @param {Function} fn - The function to execute
* @returns A promise that resolves to the result of the function execution with a browser passed into it
*/
const startNewChromeDriverInstance = tape => {
return new Promise(resolve => {
return getPort()
.then(port => {
var process = chromedriver.start([
"--url-base=wd/hub",
`--port=${port}`
]);
PROCESSES_TO_KILL.push(process);
process.stdout.on("data", d => {
if (`${d}`.match(/local connections are allowed/)) {
tape.comment(`Chromedriver expected to be running at localhost:${port}`);
// Setup browser
var browser = wdio.remote({port, desiredCapabilities: {browserName: "chrome"}});
browser.on("error", tape.fail);
resolve({browser, cleanup: () => browser.end()});
}
});
});
});
};
exports.startNewChromeDriverInstance = startNewChromeDriverInstance;
There’s a lot in there to unpack, and a fair amount of details that are specific to my application (like ENV variables), but I hope that gives an idea of what I had to do.
I spent an entirely unreasonable amount of time and effort struggling with how to use webdriver.io. For me, my intuition headbutted with reality in that when using webdriver.io, I had to wait on the browser object (what the promise returned) to make sure it had a session, but call methods on the INITIAL promise itself that is created by init(). webdriver.io’s init just produces a promise-like thing, that also takes what to do when it’s actually been set up, and that was infuriatingly difficult for me to figure out.
I am proud I didn’t file a ticket since I seem to be the only one having this problem (does everyone else understand this already?) and the package maintainers definitely don’t owe me anything. As people mostly consider filing issues as contributing to a project, I guess I should also be ashamed of myself for struggling in silence…
One of the cleaner tests
test("search page component renders on homepage", T.defaultTestConfig, t => {
Promise
.all([T.startTestAppInstance(t), CHROMEDRIVER_PROMISE])
.then(([app, driver]) => driver.browser.then(() => { // Wait for browser to load
driver.browser
.init()
.url(app.info.appUrl)
.isExisting("section.search-page-component")
.then(T.assertAndReturnBrowser(t, driver.browser, it => it.should.be.true("component should be present")))
.then(() => app.cleanup())
.then(() => driver.cleanup())
.then(() => t.end(), t.end);
}))
.catch(t.end);
});
// A slightly longer test with more complex repetitive interactions hidden away
test.only("login works and redirects to app page", {timeout: T.defaultTimeout}, t => {
Promise
.all([T.startTestAppInstance(t), T.startNewChromeDriverInstance(t)])
.then(([app, driver]) => {
T.makeAdminUser(t, app.info.appUrl)
.then(T.doLoginForUser(t, driver.browser, app.info.appUrl, USER_FIXTURES.ADMIN))
.then(() => driver.browser.isExisting("section#app"))
.then(T.assertAndReturnBrowser(t, driver.browser, it => it.should.be.true("app page should render")))
.then(() => app.cleanup())
.then(() => driver.cleanup())
.then(() => t.end(), t.end);
});
});
As you can see, I went full promises with the interface, which makes the code a lot cleaner. If you’re new to Promises, or functions, or the difference between when a function runs and when you just pass a function to BE run later, this code may be pretty confusing, and for that I apologize (also if you’re new to ES6 () => functions, you’re probably real confused right now, but hopefully not for long).
Chromedriver not closing properly
One issue I ran into while developing this stuff was the difficulty of actually closing chromedriver processes (lots of not-good memories trying to close webdriver-related processes…). Weirdly enough, if I tried to KILL the chrome driver process, either by using chromedriver.stop or <child process that chromedriver.start creates>.kill() or even childProcess.spawn('kill -9 <process>') and process.kill(-childproc.pid), chromedriver will actually not shut down, it actually turns into like a full-powered chrome process with ~8 chrome processes.
Super frustrating/rage-inducing, but I’m sure if I had just taken a deeper look into chrome-driver/linux process internals I would have found why. Rather than doing that deep dive however, I just didn’t attempt to manually close the chromedriver process. I used browser.end and just waited for ~500ms to kill the process.`. A lot of trial and error went into making that work, and I probably wasted more time than if I had just done the deep-dive but I’ve got a working solution now and just don’t care.
All in all I’m pretty happy with what the tests look like now:
T.makeAdminUser(t, app.info.appUrl). Yay for promises and async-capable abstractions with unified interfaces.Every time I run the full test suite and see tape make the app, make the browser windows, run the tests (I watch it as it’s doing the stuff), I get the warm fuzzies.