Zephyr testing using LAVA

I’d like to describe how Linaro is currently executing Zephyr tests, describe how I think we should do it and to discuss broadly what set of test cases we want to include in our test suite. So let’s get into it:

Linaro Zephyr testing

What we do in Linaro as part of the LITE (IoT and embedded) group is currently not the most optimal solution. We build the Zephyr image with test cases being already embedded in the image. We flash the device with this image and the tests are run as soon as the image boots. LAVA then monitors for test case results and pick them up. Because most of the devices under test have really small amount of flash memory, the images only include small number of test cases, sometimes only one. The amount of LAVA jobs needed to run in order to cover the plethora of test cases in Zephyr for each device goes up to almost 1000 as can be seen here.

I’ve talked to a engineer which currently maintains Zephyr tests in Linaro and we have come up with new design which can benefit everyone:

ASOS Zephyr testing

The idea is to run twister in pretty much the same fashion that LAVA currently runs adb when doing android testing - inside the LAVA worker using docker container. An example job definition for android can be seen here. So build the tests and run them on the connected DUT, then get the results back to the LAVA server. Sounds simple but in theory it will require some development on the LAVA side. I will soon know more and have a rough estimate once I do the initial skimming through the code.

Twister uses board configuration files found in zephyr repo to describe how and which tests to build and to execute the on an arbitrary DUT. I guess this covers test suites that we want to include in our testing. I’m not sure what’s the test coverage like but this seems ok to start with.

Edit: Thanks for the comments all. Amit raised a good point about how twister handles board limitation (specifically flash memory). The point is that twister will re-flash the board for each test scenario with the new image containing the necessary tests. Twister will control our test execution flow and we skim the results of the top, all in a single job.

Comments welcome.

1 Like

Do you have an estimate of the rough timeline of when we could see some initial tests running using the new method?

Being unfamiliar with twister but familiar with adb, can you help me understand what twister does in the zephyr world and how we would use it from LAVA?

Depends. Seems to me at this point that update to LAVA codebase will be minimal or non-existing but I will know more tomorrow.

Twister has many use cases but the one most interesting to us is it’s ability to use serial to connect to the board and execute tests on it using zephyr SDK. We will do it all from docker since we don’t want anyone running arbitrary code on our LAVA worker. So essentially we will have a pre-built docker image with everything Zephyr (SDK, dependencies, twister) and use that to run twister inside the LAVA worker. The output can be easily parsed in LAVA test definition with lava-test-case.

1 Like

@stevanr How do you solve the above problem with the new proposal?

With Zephyr, the image and applications (including tests) are simply one artifact. You can in theory have a single meta-application that calls other applications ala. busybox. But you still suffer from the RAM and storage constraints.

Twister will re-flash the board for each test scenario with the new image containing the necessary tests. The point is that twister will control our test execution flow and we skim the results of the top, all in a single job.

1 Like

@stevanr OK, makes sense now. I think the quoted bit above should be in the original proposal to make it clear.

1 Like