Scripted and Exploratory Testing
Scripted testing is a necessary evil. Repeated tests like smoke tests are fastest and most efficient when they are written down (similar to our feature test plans that are closer to a “requirements sheet” that has a few edge cases, than a “traditional” test plan). Scripted tests are also better measurable compared to exploratory tests. We use a rotation system for repeated tests, so not every release is testing every important feature — instead, we rotate important cases to make sure we can cover more features during a cycle of three to four releases. Thanks to this approach, our scripted tests are fast and lean but keep a high level of coverage.
On the other hand, we consider exploratory testing — an incredibly underrated approach — the best way to find bugs. Every feature that hits QA is tested first without guidelines (apart from a basic design document), trusting the tester and their knowledge of the game. Historically, most issues are found during an exploratory test and it’s way quicker than writing extensive test plans and following test cases that are more detailed than most game design documents.
Our testing approach here follows a clear “keep it as close as possible to live” mentality, and thanks to our extensive testing setup in the game on the development branches (over 100 Cheats!) and our ability to load any player’s save game in this setup, it’s very easy for us to simulate any player’s progress.
The same mentality applies to our test session at the end of the sprint. Based on a few testing guidelines, Devs, Artists, Game Designer, and Product Management all get their hands on the new features in a 20 to 30-minute meeting and find the last issues or bring up last-minute changes. This is also a great opportunity to make sure that every team member plays the game and gets a feel for the new features.
We keep an eye out on all crashes and exceptions that occur while the version is live. We barely have any crashes thanks to a stable development process (let’s forget low-memory devices for a moment), but exceptions are a great indication for bugs and usually give enough hints which parts of the game are having issues. We track recurring exceptions and investigate new exceptions with every release.
Most of the time a new exception is a minor issue, easy to reproduce and fix, or an exception without user-facing problems coming from an SDK. For issues that require deeper digging, we can load player save games in our testing environment with more than 150 test devices and take a look at the logs from there, which is usually enough to find the issue.
Our process is built in a way that avoids long tests (like performance) getting in the way of the sprint. Thanks to a clear integration of the QA Team, our communication with Developers, Design, and Product is super smooth, even when things get hectic.
We test as early as possible, use as much exploratory testing as possible, integrate stakeholders in the whole process, take ownership of the issues on live, and make scripted testing extremely lean. We don’t do hotfixes, because we focus on the critical area first, bring edge cases to the Dev Team before they start implementing the features, implement the rest of the Game Team in the QA process, while observing every exception, ready to fix it at any time before it escalates.
Product Management trusts us — we intentionally don’t do inefficient tests (like resolution testing), we wave a lot of our issues within QA, we take calculated risks knowing players might run into bugs, and only in a few rare cases does our testing slow down the sprint. Our first step to testing truly lean was to forget a lot of things that people taught us about QA.
One More Thing…
While I would like to highlight the success of our testing process, QA can’t claim all the credit alone: everything I write here is only possible because of a great company culture, a dedicated and efficient Development Team, hardworking Artists, super awesome Designers and a great set of Product Managers.
To a hotfix-free 2020!