Documentation: Add proposal for firmware unit testing

Signed-off-by: Jan Dabros <jsd@semihalf.com>
Change-Id: I552d6c3373219978b8e5fd4304f993d920425431
Reviewed-on: https://review.coreboot.org/c/coreboot/+/39893
Tested-by: build bot (Jenkins) <no-reply@coreboot.org>
Reviewed-by: Paul Fagerburg <pfagerburg@chromium.org>
Reviewed-by: Julius Werner <jwerner@chromium.org>

1 files changed, 319 insertions, 0 deletions

diff --git a/Documentation/technotes/2020-03-unit-testing-coreboot.md b/Documentation/technotes/2020-03-unit-testing-coreboot.md
new file mode 100644
index 0000000000..0d1d8ece49
--- /dev/null
+++ b/Documentation/technotes/2020-03-unit-testing-coreboot.md
@@ -0,0 +1,319 @@
+# Unit testing coreboot
+
+## Preface
+First part of this document, Introduction, comprises disambiguation for what
+unit testing is and what is not. This definition will be a basis for the whole
+paper.
+
+Next, Rationale, explains why to use unit testing and how coreboot specifically
+may benefit from it.
+
+This is followed by evaluation of different available free C unit test
+frameworks. Firstly, collection of requirements is provided. Secondly, there is
+a description of a few selected candidates. Finally, requirements are applied to
+candidates to see if they might be a good fit.
+
+Fourth part is a summary of evaluation, with proposal of unit test framework
+for coreboot to be used.
+
+Finally, Implementation proposal paragraph touches how build system and coreboot
+codebase in general should be organized, in order to support unit testing. This
+comprises couple of design considerations which need to be addressed.
+
+## Introduction
+A unit test is supposed to test a single unit of code in isolation. In C
+language (in contrary to OOP) unit usually means a function. One may also
+consider unit under test to be a single compilation unit which exposes some
+API (set of functions). A function, talking to some external component can be
+tested if this component can be mocked out.
+
+In other words (looking from C compilation angle), there should be no extra
+dependencies (executables) required beside unit under test and test harness in
+order to compile unit test binary. Test harness, beside code examining a
+routines, may comprise test framework implementation.
+
+It is hard to apply this strict definition of unit test to firmware code in
+practice, mostly due to constraints on speed of execution and size of final
+executable. coreboot codebase often cannot be adjusted to be testable. Because
+of this, coreboot unit testing subsystem should allow to include some additional
+source object files beside unit under test. That being said, the default and
+goal wherever possible, should be to isolate unit under test from other parts.
+
+Unit testing is not an integration testing and it doesn't replace it. First of
+all, integration tests cover larger set of components and interactions between
+them. Positive integration test result gives more confidence than a positive
+unit test does. Furthermore, unit tests are running on the build machine, while
+integration tests usually are executed on the target (or simulator).
+
+## Rationale
+Considering above, what is the benefit of unit testing, especially keeping in
+mind that coreboot is low-level firmware? Unit tests should be quick, thus may
+be executed frequently during development process. It is much easier to build
+and run a unit test on a build machine, than any integration test. This in turn
+may be used by dev to gather extra confidence early during code development
+process. Actually developer may even write unit tests earlier than the code -
+see [TDD](https://en.wikipedia.org/wiki/Test-driven_development) concept.
+
+That being said, unit testing embedded C code is a difficult task, due to
+significant amount of dependencies on underlying hardware. Mocking can handle
+some hardware dependencies. However, complex mocks make the unit test
+susceptible to failing and can require significant development effort.
+
+Writing unit tests for a code (both new and currently existing) may be favorable
+for the code quality. It is not only about finding bugs, but in general - easily
+testable code is a good code.
+
+coreboot benefits the most from testing common libraries (lib/, commonlib/,
+payloads/libpayload) and coreboot infrastructure (console/, device/, security/).
+
+## Evaluation of unit testing frameworks
+
+### Requirements
+Requirements for unit testing frameworks:
+
+* Easy to use
+* Few dependencies
+
+    Standard C library is all we should need
+
+* Isolation between tests
+* Support for mocking
+* Support for some machine parsable output
+* Compiler similarity
+
+   Compiler for the host _must_ support the same language standards as the target
+   compiler. Ideally the same toolchain should be used for building firmware
+   executables and test binaries, however the host complier will be used to build
+   unit tests, whereas the coreboot toolchain will be used for building the
+   firmware executables. For some targets, the host compiler and the target
+   compiler could be the same, but this is not a requirement.
+
+* Same language for tests and code
+
+   Unit tests will be written in C, because coreboot code is also written in C
+
+### Desirables
+
+* Easy to integrate with build system/build tools
+
+   Ideally JUnit-like XML output format for Jenkins
+
+* Popularity is a plus
+
+   We want a larger community for a couple of reasons. Firstly, easier access to
+   people with knowledge and tutorials. Secondly, bug fixes for the top of tree
+   are more frequent and known issues are usually shorter in the pending state.
+   Last but not least, larger reviewer pool means better and easier upstream
+   improvements that we would like to submit.
+
+* Extra features may be a plus
+* Compatible license
+
+   This should not be a blocker, since test binaries are not distributed.
+   However ideally compatible with GPL.
+
+* IDE integration
+
+### Candidates
+There is a lot of frameworks which allow unit testing C code
+([list](https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C) from
+Wikipedia). While not all of them were evaluated, because that would take an
+excessive amount of time, couple of them were selected based on the good
+opinions among C devs, popularity and fitting above criteria.
+
+* [SputUnit](https://www.use-strict.de/sput-unit-testing/)
+* [GoogleTest](https://github.com/google/googletest)
+* [Cmocka](https://cmocka.org/)
+* [Unity](http://www.throwtheswitch.org/unity) (CMock, Ceedling)
+
+We looked at several other test frameworks, but decided not to do a full evaluation
+for various reasons such as functionality, size of the developer community, or
+compatibility.
+
+### Evaluation
+* [SputUnit](https://www.use-strict.de/sput-unit-testing/)
+  * Pros
+    * No dependencies, one header file to include - that’s all
+    * Pure C
+    * Very easy to use
+    * BSD license
+  * Cons
+    * Main repo doesn’t have support for generating JUnit XML reports for
+      Jenkins to consume - this feature is available only on the fork from
+      SputUnit called “Sput_report”. It makes it niche in a niche, so there are
+      some reservations whether support for this will be satisfactory
+    * No support for mocks
+    * Not too popular
+    * No automatic test registration
+* [GoogleTest](https://github.com/google/googletest)
+  * Pros
+    * Automatic test registration
+    * Support for different output formats (including XML for Jenkins)
+    * Good support, widely used, the biggest and the most active community out
+      of all frameworks that were investigated
+    * Available as a package in the most common distributions
+    * Test fixtures easily available
+    * Well documented
+    * Easy to integrate with an IDE
+    * BSD license
+  * Cons
+    * Requires C++11 compiler
+    * To make most out of it (use GMock) C++ knowledge is required
+* [Cmocka](https://cmocka.org/)
+  * Pros
+    * Self-contained, autonomous framework
+    * Pure C
+    * API is well documented
+    * Multiple output formats (including XML for Jenkins)
+    * Available as a package in the most common distributions
+    * Used in some popular open source projects (libssh, OpenVPN, Samba)
+    * Test fixtures available
+    * Support for exception handling
+  * Cons
+    * No automatic test registration
+    * It will require some effort to make it work from within an IDE
+    * Apache 2.0 license (not compatible with GPLv2)
+* [Unity](http://www.throwtheswitch.org/unity) (CMock, Ceedling)
+  * Pros
+    * Pure C (Unity testing framework itself, not test runner)
+    * Support for different output formats (including XML for Jenkins)
+    * There are some (rather easy) hints how to use this from an IDE (e.g. Eclipse)
+    * MIT license
+  * Cons
+    * Test runner (Ceedling) is not written in C - uses Ruby
+    * Mocking/Exception handling functionalities are actually separate tools
+    * No automatic test registration
+    * Not too popular
+
+### Summary & framework proposal
+After research, we propose using the Cmocka unit test framework. Cmocka fulfills
+all stated evaluation criteria. It is rather easy to use, doesn’t have extra
+dependencies, written fully in C, allows for tests fixtures and some popular
+open source projects already are using it. Cmocka also includes support for
+mocks.
+
+Cmocka's limitations, such as the lack of automatic test registration, are
+considered minor issues that will require only minimal additional work from a
+developer. At the same time, it may be worth to propose improvement to Cmocka
+community or simply apply some extra wrapper with demanded functionality.
+
+## Implementation
+
+### Framework as a submodule or external package
+Unit test frameworks may be either compiled from source (from a git submodule
+under 3rdparty/) or pre-compiled as a package. The second option seems to be
+easier to maintain, while at the same time may bring some unwanted consequences
+(different version across distributions, frequent changes in API). It makes sense
+to initially experiment with packages and check how it works. If this will
+cause any issues, then it is always possible to switch to submodule approach.
+
+### Integration with build system
+To get the most out of unit testing framework, it should be integrated with
+Jenkins automation server. Verification of all unit tests for new changes may
+improve code reliability to some extent.
+
+### Build configuration (Kconfig)
+While building unit under test object file, it is necessary to apply some
+configuration (config) just like when building usual firmware. For simplicity,
+there will be one default tests .config `qemu_x86_i440fx` for all unit tests. At
+the same time, some tests may require running with different values of particular
+config. This should be handled by adding extra header, included after config.h.
+This header will comprise #undef of old CONFIG values and #define of the
+required value. When unit testing will be integrated with Jenkins, it may be
+preferred to use every available config for periodic builds.
+
+### Directory structure
+Tests should be kept separate from the code, while at the same time it must be
+easy to match code with test harness.
+
+We create new directory for test files ($(toplevel)/tests/) and mimic the
+structure of src/ directory.
+
+Test object files (test harness, unit under tests and any additional executables
+are stored under build/tests/<test_name> directory.
+
+Below example shows how directory structure is organized for the two test cases:
+tests/lib/string-test and tests/device/i2c-test:
+
+```bash
+├── src
+│   ├── lib
+│   │   ├── string.c <- unit under test
+│   │
+│   ├── device
+│       ├── i2c.c
+│
+├── tests
+│   ├── include
+│   │   ├── mocks <- mock headers, which replace original headers
+│   │
+│   ├── Makefile.inc <- top Makefile for unit tests subsystem
+│   ├── lib
+│   │   ├── Makefile.inc
+│   │   ├── string-test.c <- test code for src/lib/string.c
+│   │   │
+│   ├── device
+│   │   ├── Makefile.inc
+│       ├── i2c-test.c
+│
+├── build
+│   ├── tests <-all test-related executables
+        ├── config.h <- default config used for tests builds
+        ├── lib
+        │   ├── string-test <- all string-test executables
+        │   │   ├── run     <- final test binary
+        │   │   ├── tests   <- all test harness executables
+        │   │       ├── lib
+        │   │           ├── string-test.o  <-test harness executable
+        │   │   ├── src    <- unit under test and other src executables
+        │   │       ├── lib
+        │   │           ├── string.o       <- unit under test executable
+        ├── device
+            ├── i2c-test
+                ├── run
+                ├── tests
+                │   ├── device
+                │       ├── i2c-test.o
+                ├── src
+                    ├── device
+                        ├── i2c.o
+```
+
+### Adding new tests
+For purpose of this description, let's assume that we want to add a new unit test
+for src/device/i2c.c module. Since this module is rather simple, it will be enough
+to have only one test module.
+
+Firstly (assuming there is no tests/device/Makefile.inc file) we need to create
+Makefile.inc in main unit test module directory. Inside this Makefile.inc, one
+need to register new test and can specify multiple different attributes for it.
+
+```bash
+# Register new test, by adding its name to tests variable
+tests-y += i2c-test
+
+# All attributes are defined by <test_name>-<attribute> variables
+# <test_name>-srcs is used to register all input files (test harness, unit under
+# test and others) for this particular test. Remember to add relative paths.
+i2c-test-srcs += tests/device/i2c-test.c
+i2c-test-srcs += src/device/i2c.c
+
+# We can define extra cflags for this particular test
+i2c-test-cflags += -DSOME_DEFINE=1
+
+# For mocking out external dependencies (functions which cannot be resolved by
+# linker), it is possible to register a mock function. To register new mock, it
+# is enough to add function-to-be-mocked name to <test_name>-mocks variable.
+i2c-test-mocks += platform_i2c_transfer
+
+# Similar to coreboot concept, unit tests also runs in the context of stages.
+# By default all unit tests are compiled to be ramstage executables. If one want
+# to overwrite this setting, there is <test_name>-stage variable available.
+i2c-test-stage:= bootblock
+```
+
+### Writing new tests
+Full description of how to write unit tests and Cmocka API description is out of
+the scope of this document. There are other documents related to this
+[Cmocka API](https://api.cmocka.org/) and
+[Mocks](https://lwn.net/Articles/558106/).