How to Profile Oniro boot

According to Wikipedia, Profiling is dynamic analysis of a system, this mean that a system can be observed at runtime with some extra tooling.

Just like Schrödinger’s cat changing the system may have some side effect on the behavior of the system itself, but it’s still useful to find bottlenecks or hints to where to optimize.

Each boot steps will need to be profiled since hardware initialization, BIOS, bootloader etc. Today we’ll focus on OS launching (once the linux kernel is loaded).

bootchart

Bootchart is a well know tool in Linux world, this project has been more or less maintained over years, but there is one of its offspring now as part of systemd utiliites.

For the demonstration the vending machine blueprint will be profiled, it’s composed of a touch panel application and a lightweight controller app that we will ignore for now.

While raspberry-pi 4 is convenient for prototyping, measure will be done on Seco b68 an industrial grade product supported by this oniro blueprint.

Bootchart is able to produce this kind of graph.

Usage

First download sources and setup oniro env as explained in online.

systemd-bootchart is already packaged in yocto so it can be used in Oniro very easily by tuning “local.conf” file:

# file: conf/local.conf
CORE_IMAGE_EXTRA_INSTALL += "systemd-bootchart"
APPEND_append += "init=/lib/systemd/systemd-bootchart"

Before (re)building an Oniro image

bitbake oniro-linux-blueprint-vending-machine

Then on boot init change can be double checked by looking at boot args:

cat /proc/cmdline
BOOT_IMAGE=/vmlinuz (...)  init=/lib/systemd/systemd-bootchart  (...) 

If lucky you’ll find a svg file generated by a systemd service:

root@seco-intel-b68:~# systemctl status systemd-bootchart
● systemd-bootchart.service - Boot Process Profiler
(...)
Nov 18 20:47:13 seco-intel-b68 systemd-bootchart[148]: Bootchart created: /run/log/bootchart-20211118-2047.svg
(...)

If it failed to work you can double check that Kernel scheduler has been property configured as explained in next chapter.

Then svg file can be copied elsewhere, by the way to enable ssh you can to add few lines to local.conf:

IMAGE_FEATURES += " ssh-server-dropbear "
CORE_IMAGE_EXTRA_INSTALL += "avahi-daemon"
ROOT_FSTYPE = "ext4"

Then file can be extracted using:

scp root@seco-intel-b68.local:/run/log/bootchart-*.svg .

Kernel configuration

For development purposes we can also rely on qemux86-64 virtual machine which uses oniro’s linux-kernel.

Using bitbake kernel can be configured manually (In “Kernel hacking” submenu):

bitbake -c menuconfig linux-oniro

CONFIG_SCHEDSTATS variable will be changed, it can be also set in fragment file like: meta-oniro-core/recipes-kernel/linux/linux/misc.cfg.

After rebuild use “runqemu” to boot system to check that feature is enabled:

bitbake blueprint-vending-machine-image
zcat /proc/config.gz | grep CONFIG_SCHEDSTATS
CONFIG_SCHEDSTATS=y

Optimization

Note that VM absolute time to boot is less relevant because it depends on the host speed
however it can be interesting to focus on relative time, and where optimization should be tried first:

What can we observe from this chart is that there is a process that can be removed is the keyboard process for weston compositor, it will not be used so let’s remove this, it will save 1 second for cheap on every platforms.

Back to b68, let’s rebuild and deploy on faster SDcard (first screenshot was on USB), according to chart the vending-machine UI (2d process) should show up in 5 seconds (much less than time between the board’s power and to BIOS).

Then we can also optimize at the kernel level, specifically for this blueprint, by removing unwanted modules or features (eg: debug, drivers, filesystem, i18n, crypto, pw) since the hardware is well known and static, it should be faster to uncompress before booting.

Work in progress are staging in this development branch:

shouldn’t this content be part of the documentation @Nawab @rzr ? what do you think? I still feel the forum is used to provide content that instead should be published as blog post on a resource center on the webportal or on the documentation repo

I think it’s fine to have documentation in raw form here as well. Adding things to the documentation repository means it’s a supported and maintained thing. We don’t always have resources or process to do that.

Yes that’s something that worth to be shared inside documentation, if we’re committed to support such tools (be a req), meanwhile to forum can be also added if anyone search for it.

FYI I needed to share this post quickly that’s the reason I use this forum as publishing tool (with video inlined too).

It can be a list of HOWTOs and “Tip and Tricks” for Oniro debugging somewhere on the wiki. Do we have usable wiki :slight_smile: ?

1 Like

Yes the need for wiki is something I felt since day 1,
but on the other hand discourse is flexible enough to be used as wiki
so we could create a such page and then link to this one.

BTW, please use https://forum.ostc-eu.org/t/improving-forum-structure/186 for comments not related to this page content.

@Chiara_Del_Fabbro, I agree with you, this would have been best suited as a blog post, or a separate page in docs. I don’t feel this is correct place in perspective of attracting users on UX front.

@rzr if it is finalized, can you raise a MR in Docs, we will put this up in ToC at an appropriate location.

1 Like