I’d like to describe how Linaro is currently executing Zephyr tests, describe how I think we should do it and to discuss broadly what set of test cases we want to include in our test suite. So let’s get into it:
Linaro Zephyr testing
What we do in Linaro as part of the LITE (IoT and embedded) group is currently not the most optimal solution. We build the Zephyr image with test cases being already embedded in the image. We flash the device with this image and the tests are run as soon as the image boots. LAVA then monitors for test case results and pick them up. Because most of the devices under test have really small amount of flash memory, the images only include small number of test cases, sometimes only one. The amount of LAVA jobs needed to run in order to cover the plethora of test cases in Zephyr for each device goes up to almost 1000 as can be seen here.
I’ve talked to a engineer which currently maintains Zephyr tests in Linaro and we have come up with new design which can benefit everyone:
ASOS Zephyr testing
The idea is to run twister in pretty much the same fashion that LAVA currently runs adb when doing android testing - inside the LAVA worker using docker container. An example job definition for android can be seen here. So build the tests and run them on the connected DUT, then get the results back to the LAVA server. Sounds simple but in theory it will require some development on the LAVA side. I will soon know more and have a rough estimate once I do the initial skimming through the code.
Twister uses board configuration files found in zephyr repo to describe how and which tests to build and to execute the on an arbitrary DUT. I guess this covers test suites that we want to include in our testing. I’m not sure what’s the test coverage like but this seems ok to start with.
Edit: Thanks for the comments all. Amit raised a good point about how twister handles board limitation (specifically flash memory). The point is that twister will re-flash the board for each test scenario with the new image containing the necessary tests. Twister will control our test execution flow and we skim the results of the top, all in a single job.