Automated test generation? Yes, please.
This post shows how to write business logic for your software/app with Markdown tables and auto-generate tests for Xcode with a command line tool called AcceptanceMark.
But first, some context. Your code may have some test coverage in the form of:
- Unit Tests
- Integration Tests
- Acceptance Tests
- UI Tests
Read here for a good overview of the differences between them.
Additionally, you may have a dedicated Q/A team, and you regularly run your app manually to verify its correctness.
While writing good tests is essential for building quality software, it’s also something that takes valuable time. I’ll show you how to get your code under test faster, so that you can spend your time on more important things!
Testing, the Xcode way
In Xcode you can write Unit Tests or UI Tests by subclassing
Let’s suppose that your system under test (SUT) is a date formatter that needs to work correctly with a variety of possible languages / input dates. The test class may look like this:
In practice, you can get full test coverage in three steps:
- Generate all input combinations
- Pass each input to the SUT and retrieve the output
- Compare the output with the expected value
XCTestCase methods do not take parameters, you need write a separate test method for each input combination. Example:
- This quickly becomes unpractical as the number of possible inputs increases.
- All the inputs and expected values are scattered across test methods. This could be improved by encoding inputs and expected values in an array of structs, and have each test process one item in the array.
- All your business requirements are confined to your test classes, which are typically not accessible or easy to understand by your Q/A team or product managers.
Making things better
What if you could write all your inputs and expectations in a table like this:
|2016||9||16||french||16 sept. 2016|
|2016||9||16||english||Sep 16, 2016|
And have this automatically translated into an Xcode test class?
Well, I have written a tool called AcceptanceMark that does exactly that.
Just run this on the terminal:
amtool -i DateFormattingSimple.md
And the tool will generate a Swift test file like this:
NOTE: What this tool really does is generate the code to execute a test runner with all the inputs and compare the outputs with the expected values, for each row of the table. It is up to you to write the test runner code needed to pass the input to your SUT and return the output.
Advantages over writing tests manually 💯
- This is a huge time saver because you no longer have to write all the tests manually. You only write one test runner to exercise your SUT with a given input and return an output.
- Your business logic is written in Markdown, which can be easily edited by non-developers in your team, and it can also be exported to HTML or PDF.
- You can add type information to your test Markdown tables to easily support basic value types such as String, Bool, Int, Float, so that you don’t have to do basic string-to-number conversions in your test code.
Note that the test runner is the place to encode application-specific domain knowledge about how to configure and run your SUT.
Also, the test runner takes one strongly-typed input struct, returns one strongly-typed output struct and is completely stateless. This is by design, and it ensures that tests will have no side effects.
🚀 Sounds great! Show me how to use this! 🚀
Now that I’ve covered the core principles behind AcceptanceMark, I can introduce a more detailed example.
Let’s suppose that the SUT is a
This date formatter takes the following inputs as parameters:
- Date Components (year, month, day, hour, minute, seconds)
- Language, derived from a locale identifier
- Date Formatter Style
I’ll now show you how to add test coverage to this class with a step-by-step guide. Ready? Let’s go!
Step 1: Create a new
DateFormatting.md file in Xcode and add it to your tests folder. This will contain all your test data:
- The year, month, day, hour, minute, second columns are annotated as Int. AcceptanceMark does not support dates as a built-in type yet, but it’s ok to use separate components here so that the date input is format agnostic.
- The language column is a locale identifier string.
- The dateStyle column is a string.
- There is an empty column between dateStyle and localizedDate. This is called a separator column. AcceptanceMark treats all columns before it as inputs and all columns after it as outputs. A separator column is always required by AcceptanceMark. Also note that the second row of the table has three dashes (
---) for the empty column. This is because valid Markdown tables always have at least three dashes for each column.
- The localizedDate column represents the only output of the table, which will be treated as an expected value in the tests.
- All column names use the camel case convention, in line with Swift naming conventions for variables.
Step 2: Generate a Swift test class from the Markdown file.
Prerequisite: for this to work,
amtool needs to be installed on your machine. Please follow the instructions the AcceptanceMark GitHub page for how to compile and install
Open the terminal,
cd to the test folder and run the following command:
amtool -i DateFormatting.md -l swift3
If everything goes well, the console output will confirm that a Swift file has been exported:
Exported swift3 code: /Users/andrea/AcceptanceMark/AcceptanceMarkDemoTests/DateFormatting_DateFormattingTests.swift
Open the folder and see that the file of interest is in there:
Step 3: Drag & drop this file to Xcode and add it to your test target:
Step 4: Try to compile and run the unit tests. A Swift compiler error is presented:
Use of unresolved identifier: 'DateFormatting_DateFormattingRunner' 🤔
This is because the test class is now in the project but a test runner is missing. As I said before, this is required to pass the table inputs to the SUT (the date formatter), and retrieve the outputs (the formatted string) to be compared with the expected values.
Step 5: Create a test runner class.
Notice that AcceptanceMark kindly provides a template for the test runner as part of the generated code in the test class:
- Add a new Swift file named
DateFormatting_DateFormattingRunner.swiftto the test target.
- Paste and uncomment the code from the template in the test class.
Step 6: Implement the test runner.
The inputs of the date formatter are a
DateComponents value, a
Locale and a
- Create a
DateComponentsinstance and configure it with the values from the input struct:
- Calculate the locale directly by using the input language string as an identifier:
- Calculate the
DateFormatter.Style. This can be done by creating an extension with a custom initialiser that takes a style string and maps it to the desired value:
With this, the date style can be calculated directly from the dateStyle value in the input struct:
- Finally, create a date formatter with the required parameters and use it to calculate the formatted date and return it:
The whole implementation of the date formatter test runner class looks like this:
Step 7: Compile and run the tests again. This time, all the tests are passing.
Getting a system under test with AcceptanceMark is simple:
- Write a Markdown table with all the inputs and expected outputs.
- Generate the corresponding Swift test class with
- Write the test runner (map the inputs / outputs if needed + create the SUT and run it).
This can save a lot of time compared to traditional unit tests, especially if the number of inputs / outputs to test is high. Additional benefits:
- Writing tests becomes partially automated and data-driven.
- Tests are written in a more uniform way across the codebase, and are stateless by design.
- All the business rules are in one place and outside the code, rather than scattered across tests.
As for the test runners, when the inputs or outputs of the SUT are not a basic value type such as String, Float, Int, Bool, some mapping is required to transform them into objects that the SUT can understand.
This can be as simple as creating a
Locale object from an input string identifier, or slightly more complex and require to map a set of strings to an enumeration, class or struct by providing an extension with a custom initialiser.
Congratulations for making it to the end! This post has presented a tool to partially automate the process of writing tests for your classes.
Visit the AcceptanceMark GitHub page to find all the source code for this tutorial and further information on how to compile and use
I hope that you can use AcceptanceMark to save time on your projects as I’m already doing on mine! This is something that I have created just recently and I hope that feedback from the community will help it evolve into an great tool in every developer’s testing toolkit.
If you have any feedback on this tutorial or on how I can improve AcceptanceMark, please let me know in the comments!