diff options
Diffstat (limited to 'Documentation/tutorial/part3.md')
| -rw-r--r-- | Documentation/tutorial/part3.md | 284 |
1 files changed, 149 insertions, 135 deletions
diff --git a/Documentation/tutorial/part3.md b/Documentation/tutorial/part3.md index dc487182c1..83b34088c3 100644 --- a/Documentation/tutorial/part3.md +++ b/Documentation/tutorial/part3.md @@ -1,31 +1,32 @@ # Writing unit tests for coreboot ## Introduction -General thoughts about unit testing coreboot can be found in -[Unit testing coreboot](../technotes/2020-03-unit-testing-coreboot.md). -Additionally, [code coverage](../technotes/2021-05-code-coverage.md) support -is available for unit tests. - -This document aims to guide developers through the process of adding and writing -unit tests for coreboot modules. - -As an example of unit under test, `src/device/i2c.c` (referred hereafter as UUT -"Unit Under Test") will be used. This is simple module, thus it should be easy -for the reader to focus solely on the testing logic, without the need to spend -too much time on digging deeply into the source code details and flow of -operations. That being said, a good understanding of what the unit under test is -doing is crucial for writing unit tests. +General thoughts about unit testing coreboot can be found in [Unit +testing coreboot](../technotes/2020-03-unit-testing-coreboot.md). +Additionally, [code coverage](../technotes/2021-05-code-coverage.md) +support is available for unit tests. + +This document aims to guide developers through the process of adding and +writing unit tests for coreboot modules. + +As an example of unit under test, `src/device/i2c.c` (referred hereafter +as UUT "Unit Under Test") will be used. This is simple module, thus it +should be easy for the reader to focus solely on the testing logic, +without the need to spend too much time on digging deeply into the +source code details and flow of operations. That being said, a good +understanding of what the unit under test is doing is crucial for +writing unit tests. This tutorial should also be helpful for developers who want to follow -[TDD](https://en.wikipedia.org/wiki/Test-driven_development). Even though TDD -has a different work flow of building tests first, followed by the code that -satisfies them, the process of writing tests and adding them to the tree is the -same. +[TDD](https://en.wikipedia.org/wiki/Test-driven_development). Even +though TDD has a different work flow of building tests first, followed +by the code that satisfies them, the process of writing tests and adding +them to the tree is the same. -## Analysis of unit under test -First of all, it is necessary to precisely establish what we want to test in a -particular module. Usually this will be an externally exposed API, which can be -used by other modules. +## Analysis of unit under test First of all, it is necessary to +precisely establish what we want to test in a particular module. Usually +this will be an externally exposed API, which can be used by other +modules. ```eval_rst .. admonition:: i2c-test example @@ -34,66 +35,70 @@ used by other modules. .. code-block:: c - int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg, uint8_t *data, - uint8_t mask, uint8_t shift) - int i2c_write_field(unsigned int bus, uint8_t chip, uint8_t reg, uint8_t data, - uint8_t mask, uint8_t shift) + int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg, + uint8_t *data, uint8_t mask, uint8_t shift) + int i2c_write_field(unsigned int bus, uint8_t chip, uint8_t reg, + uint8_t data, uint8_t mask, uint8_t shift) - For sake of simplicity, let's focus on `i2c_read_field` in this document. + For sake of simplicity, let's focus on `i2c_read_field` in this + document. ``` -Once the API is defined, the next question is __what__ this API is doing (or -what it will be doing in case of TDD). In other words, what outputs we are -expecting from particular functions, when providing particular input parameters. +Once the API is defined, the next question is __what__ this API is doing +(or what it will be doing in case of TDD). In other words, what outputs +we are expecting from particular functions, when providing particular +input parameters. ```eval_rst .. admonition:: i2c-test example .. code-block:: c - int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg, uint8_t *data, - uint8_t mask, uint8_t shift) + int i2c_read_field(unsigned int bus, uint8_t chip, uint8_t reg, + uint8_t *data, uint8_t mask, uint8_t shift) - This is a method which means to read content of register `reg` from i2c device - on i2c `bus` and slave address `chip`, applying bit `mask` and offset `shift` - to it. Returned data should be placed in `data`. + This is a method which means to read content of register `reg` from + i2c device on i2c `bus` and slave address `chip`, applying bit `mask` + and offset `shift` to it. Returned data should be placed in `data`. ``` -The next step is to determine all external dependencies of UUT in order to mock -them out. Usually we want to isolate the UUT as much as possible, so that the -test result depends __only__ on the behavior of UUT and not on the other -modules. While some software dependencies may be hard to be mock (for example -due to complicated dependencies) and thus should be simply linked into the test -binaries, all hardware dependencies need to be mocked out, since in the -user-space host environment, targets hardware is not available. +The next step is to determine all external dependencies of UUT in order +to mock them out. Usually we want to isolate the UUT as much as +possible, so that the test result depends __only__ on the behavior of +UUT and not on the other modules. While some software dependencies may +be hard to be mock (for example due to complicated dependencies) and +thus should be simply linked into the test binaries, all hardware +dependencies need to be mocked out, since in the user-space host +environment, targets hardware is not available. ```eval_rst .. admonition:: i2c-test example `i2c_read_field` is calling `i2c_readb`, which eventually invokes - `i2c_transfer`. This method simply calls `platform_i2c_transfer`. The last - function in the chain is a hardware-touching one, and defined separately for - different SOCs. It is responsible for issuing transactions on the i2c bus. - For the purpose of writing unit test, we should mock this function. + `i2c_transfer`. This method simply calls `platform_i2c_transfer`. The + last function in the chain is a hardware-touching one, and defined + separately for different SOCs. It is responsible for issuing + transactions on the i2c bus. For the purpose of writing unit test, + we should mock this function. ``` ## Adding new tests -In order to keep the tree clean, the `tests/` directory should mimic the `src/` -directory, so that test harness code is placed in a location corresponding to -UUT. Furthermore, the naming convention is to add the suffix `-test` to the UUT -name when creating a new test harness file. +In order to keep the tree clean, the `tests/` directory should mimic the +`src/` directory, so that test harness code is placed in a location +corresponding to UUT. Furthermore, the naming convention is to add the +suffix `-test` to the UUT name when creating a new test harness file. ```eval_rst .. admonition:: i2c-test example Considering that UUT is `src/device/i2c.c`, test file should be named - `tests/device/i2c-test.c`. When adding a new test file, it needs to be - registered with the coreboot unit testing infrastructure. + `tests/device/i2c-test.c`. When adding a new test file, it needs to + be registered with the coreboot unit testing infrastructure. ``` -Every directory under `tests/` should contain a Makefile.inc, similar to what -can be seen under the `src/`. Register a new test in Makefile.inc, by -__appending__ test name to the `tests-y` variable. +Every directory under `tests/` should contain a Makefile.inc, similar to +what can be seen under the `src/`. Register a new test in Makefile.inc, +by __appending__ test name to the `tests-y` variable. ```eval_rst .. admonition:: i2c-test example @@ -103,10 +108,11 @@ __appending__ test name to the `tests-y` variable. tests-y += i2c-test ``` -Next step is to list all source files, which should be linked together in order -to create test binary. Usually a tests requires only two files - UUT and test -harness code, but sometimes more is needed to provide the test environment. -Source files are registered in `<test_name>-srcs` variable. +Next step is to list all source files, which should be linked together +in order to create test binary. Usually a tests requires only two files +- UUT and test harness code, but sometimes more is needed to provide the +test environment. Source files are registered in `<test_name>-srcs` +variable. ```eval_rst .. admonition:: i2c-test example @@ -117,9 +123,10 @@ Source files are registered in `<test_name>-srcs` variable. i2c-test-srcs += src/device/i2c.c ``` -Above minimal configuration is a basis for further work. One can try to build -and run test binary either by invoking `make tests/<test_dir>/<test_name>` or by -running all unit tests (whole suite) for coreboot `make unit-tests`. +Above minimal configuration is a basis for further work. One can try to +build and run test binary either by invoking `make +tests/<test_dir>/<test_name>` or by running all unit tests (whole suite) +for coreboot `make unit-tests`. ```eval_rst .. admonition:: i2c-test example @@ -135,31 +142,34 @@ running all unit tests (whole suite) for coreboot `make unit-tests`. make unit-tests ``` -When trying to build test binary, one can often see linker complains about -`undefined reference` to couple of symbols. This is one of solutions to -determine all external dependencies of UUT - iteratively build test and resolve -errors one by one. At this step, developer should decide either it's better to -add an extra module to provide necessary definitions or rather mock such -dependency. Quick guide through adding mocks is provided later in this doc. +When trying to build test binary, one can often see linker complains +about `undefined reference` to couple of symbols. This is one of +solutions to determine all external dependencies of UUT - iteratively +build test and resolve errors one by one. At this step, developer should +decide either it's better to add an extra module to provide necessary +definitions or rather mock such dependency. Quick guide through adding +mocks is provided later in this doc. ## Writing new tests -In coreboot, [Cmocka](https://cmocka.org/) is used as unit test framework. The -project has exhaustive [API documentation](https://api.cmocka.org/). Let's see -how we may incorporate it when writing tests. +In coreboot, [Cmocka](https://cmocka.org/) is used as unit test +framework. The project has exhaustive [API +documentation](https://api.cmocka.org/). Let's see how we may +incorporate it when writing tests. ### Assertions -Testing the UUT consists of calling the functions in the UUT and comparing the -returned values to the expected values. Cmocka implements -[a set of assert macros](https://api.cmocka.org/group__cmocka__asserts.html) to -compare a value with an expected value. If the two values do not match, the test +Testing the UUT consists of calling the functions in the UUT and +comparing the returned values to the expected values. Cmocka implements +[a set of assert +macros](https://api.cmocka.org/group__cmocka__asserts.html) to compare a +value with an expected value. If the two values do not match, the test fails with an error message. ```eval_rst .. admonition:: i2c-test example - In our example, the simplest test is to call UUT for reading our fake devices - registers and do all calculation in the test harness itself. At the end, let's - compare integers with `assert_int_equal`. + In our example, the simplest test is to call UUT for reading our fake + devices registers and do all calculation in the test harness itself. + At the end, let's compare integers with `assert_int_equal`. .. code-block:: c @@ -191,24 +201,25 @@ fails with an error message. ### Mocks #### Overview -Many coreboot modules are low level software that touch hardware directly. -Because of this, one of the most important and challenging part of -writing tests is to design and implement mocks. A mock is a software component -which implements the API of another component so that the test can verify that -certain functions are called (or not called), verify the parameters passed to -those functions, and specify the return values from those functions. Mocks are -especially useful when the API to be implemented is one that accesses hardware -components. - -When writing a mock, the developer implements the same API as the module being -mocked. Such a mock may, for example, register a set of driver methods. Behind -this API, there is usually a simulation of real hardware. +Many coreboot modules are low level software that touch hardware +directly. Because of this, one of the most important and challenging +part of writing tests is to design and implement mocks. A mock is a +software component which implements the API of another component so that +the test can verify that certain functions are called (or not called), +verify the parameters passed to those functions, and specify the return +values from those functions. Mocks are especially useful when the API to +be implemented is one that accesses hardware components. + +When writing a mock, the developer implements the same API as the module +being mocked. Such a mock may, for example, register a set of driver +methods. Behind this API, there is usually a simulation of real +hardware. ```eval_rst .. admonition:: i2c-test example - For purpose of our i2c test, we may introduce two i2c devices with set of - registers, which simply are structs in memory. + For purpose of our i2c test, we may introduce two i2c devices with + set of registers, which simply are structs in memory. .. code-block:: c @@ -266,16 +277,17 @@ this API, there is usually a simulation of real hardware. }; ``` -Cmocka uses a feature that gcc provides for breaking dependencies at the link -time. It is possible to override implementation of some function, with the -method from test harness. This allows test harness to take control of execution -from binary (during the execution of test), and stimulate UUT as required -without changing the source code. +Cmocka uses a feature that gcc provides for breaking dependencies at the +link time. It is possible to override implementation of some function, +with the method from test harness. This allows test harness to take +control of execution from binary (during the execution of test), and +stimulate UUT as required without changing the source code. -coreboot unit test infrastructure supports overriding of functions at link time. -This is as simple as adding a `name_of_function` to be mocked into -<test_name>-mocks variable in Makefile.inc. The result is that the test's -implementation of that function is called instead of coreboot's. +coreboot unit test infrastructure supports overriding of functions at +link time. This is as simple as adding a `name_of_function` to be +mocked into <test_name>-mocks variable in Makefile.inc. The result is +that the test's implementation of that function is called instead of +coreboot's. ```eval_rst .. admonition:: i2c-test example @@ -284,44 +296,45 @@ implementation of that function is called instead of coreboot's. i2c-test-mocks += platform_i2c_transfer - Now, dev can write own implementation of `platform_i2c_transfer`. This - implementation instead of accessing real i2c bus, will write/read from - fake structs. + Now, dev can write own implementation of `platform_i2c_transfer`. + This implementation instead of accessing real i2c bus, will + write/read from fake structs. .. code-block:: c - int platform_i2c_transfer(unsigned int bus, struct i2c_msg *segments, - int count) + int platform_i2c_transfer(unsigned int bus, struct i2c_msg + *segments, int count) { } ``` #### Checking mock's arguments -A test can verify the parameters provided by the UUT to the mock function. The -developer may also verify that number of calls to mock is correct and the order -of calls to particular mocks is as expected (See -[this](https://api.cmocka.org/group__cmocka__call__order.html)). The Cmocka -macros for checking parameters are described -[here](https://api.cmocka.org/group__cmocka__param.html). In general, in mock -function, one makes a call to `check_expected(<param_name>)` and in the -corresponding test function, `expect*()` macro, with description which parameter -in which mock should have particular value, or be inside a described range. +A test can verify the parameters provided by the UUT to the mock +function. The developer may also verify that number of calls to mock is +correct and the order of calls to particular mocks is as expected (See +[this](https://api.cmocka.org/group__cmocka__call__order.html)). The +Cmocka macros for checking parameters are described +[here](https://api.cmocka.org/group__cmocka__param.html). In general, in +mock function, one makes a call to `check_expected(<param_name>)` and in +the corresponding test function, `expect*()` macro, with description +which parameter in which mock should have particular value, or be inside +a described range. ```eval_rst .. admonition:: i2c-test example - In our example, we may want to check that `platform_i2c_transfer` is fed with - number of segments bigger than 0, each segment has flags which are in - supported range and each segment has buf which is non-NULL. We are expecting - such values for _every_ call, thus the last parameter in `expect*` macros is - -1. + In our example, we may want to check that `platform_i2c_transfer` is + fed with number of segments bigger than 0, each segment has flags + which are in supported range and each segment has buf which is + non-NULL. We are expecting such values for _every_ call, thus the + last parameter in `expect*` macros is -1. .. code-block:: c static void mock_expect_params_platform_i2c_transfer(void) { - unsigned long int expected_flags[] = {0, I2C_M_RD, I2C_M_TEN, - I2C_M_RECV_LEN, I2C_M_NOSTART}; + unsigned long int expected_flags[] = {0, I2C_M_RD, + I2C_M_TEN, I2C_M_RECV_LEN, I2C_M_NOSTART}; /* Flags should always be only within supported range */ expect_in_set_count(platform_i2c_transfer, segments->flags, @@ -330,8 +343,8 @@ in which mock should have particular value, or be inside a described range. expect_not_value_count(platform_i2c_transfer, segments->buf, NULL, -1); - expect_in_range_count(platform_i2c_transfer, count, 1, INT_MAX, - -1); + expect_in_range_count(platform_i2c_transfer, count, 1, + INT_MAX, -1); } And the checks below should be added to our mock @@ -347,11 +360,11 @@ in which mock should have particular value, or be inside a described range. ``` #### Instrument mocks -It is possible for the test function to instrument what the mock will return to -the UUT. This can be done by using the `will_return*()` and `mock()` macros. -These are described in -[the Mock Object section](https://api.cmocka.org/group__cmocka__mock.html) of -the Cmocka API documentation. +It is possible for the test function to instrument what the mock will +return to the UUT. This can be done by using the `will_return*()` and +`mock()` macros. These are described in [the Mock Object +section](https://api.cmocka.org/group__cmocka__mock.html) of the Cmocka +API documentation. ```eval_rst .. admonition:: Example @@ -361,17 +374,18 @@ the Cmocka API documentation. ``` ### Test runner -Finally, the developer needs to implement the test `main()` function. All tests -should be registered there and cmocka test runner invoked. All methods for -invoking Cmocka test are described +Finally, the developer needs to implement the test `main()` function. +All tests should be registered there and cmocka test runner invoked. All +methods for invoking Cmocka test are described [here](https://api.cmocka.org/group__cmocka__exec.html). ```eval_rst .. admonition:: i2c-test example - We don't need any extra setup and teardown functions for i2c-test, so let's - simply register test for `i2c_read_field` and return from main value which is - output of Cmocka's runner (it returns number of tests that failed). + We don't need any extra setup and teardown functions for i2c-test, so + let's simply register test for `i2c_read_field` and return from main + value which is output of Cmocka's runner (it returns number of tests + that failed). .. code-block:: c |