Boot Time Optimization¶
This is a guide for optimizing boot times for automotive needs. Certain components may need to be booted more quickly than on a traditional Linux distribution. In the automotive industry, cold-boots are common and frequent, so there is an expectation to boot quickly, to have a drivable system. Some examples of components that may need to be prioritized could be camera, ethernet, CAN bus, etc.
Bootloader¶
Many bootloaders contain only the minimal implementation required to boot a system, but you can tune the following settings to optimize your system:
- Disable serial logging
- Maximize CPU clock for performance
- Use multiple cores and parallelization
- Use accelerators for decompression
Kernel¶
Some kernel arguments can be used to optimize boot time by disabling features. You can also remove some kernel parameters. For more information about kernel parameters, see https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/kernel-parameters.txt. Two such examples include:
quietdisables most log messageslibahci.ignore_sss=1ignore staggered spin-up and spin-up all drives at once
Initrd/Initramfs¶
Both the kernel and initrd require decompression of data, reading of the data and writing it into memory, along with other initialization steps, like populating a filesystem in the case of an initrd.
You may achieve faster build times by decreasing the size of your initrd
removing unused files, to list the files in initrd you can use the lsinitrd
command.
Building an initrd is done via a tool called dracut which pulls various files
from the main root filesystem into a compressed initrd archive.
The instructions for how to build the initrd are contained in the
/usr/lib/dracut/modules.d/ directory and primarily in the module-setup.sh files.
When altered these will change how your initrd is composed on next rebuild of
initrd.
Kernel modules are pulled from /usr/lib/modules/, removing a module and
rebuilding an initrd can be sufficient to decrease the size of an initrd.
Userspace (and systemd)¶
systemd is designed to run as many units as possible in parallel that satisfies
the dependencies of the defined unit. Directives such as Wants=, Requires=,
Before=, After=, determine when your unit will start. For units that have to
start particularly early in boot DefaultDependencies=no can disable some
dependencies that are enabled by default. Here is an example of an application
that uses these features of systemd to start displaying a camera within 2
seconds.
[Unit]
Description=Show Twincam Boot Screen
DefaultDependencies=no
ConditionVirtualization=!container
IgnoreOnIsolate=true
[Service]
ExecStart=/usr/bin/twincam -d -s -u
Type=simple
RemainAfterExit=yes
KillMode=mixed
SendSIGKILL=no
Tools¶
Here is a list of tools that can be used to help analyze and optimize performance.
Consolidated boot time data¶
You can record and augment important boot time data through multiple subsystems. The https://gitlab.com/CentOS/automotive/src/boot-time-analysis-tools/ repository contains tools that you can use to group, normalize, and visualize the data and KPIs from various sources.
Boot timings script¶
The boot_timings script collects boot time log information from dbus, dmesg, and
systemd, and creates outputs to satisfy a diverse set of needs.
The script source is maintained in GitLab:
Running the boot_timings script with no parameters uses a default set of KPIs for
timestamps and outputs all generated files into the local directory. Some parameters
are available to change script behaviors:
$ boot_timings --help
usage: boot_timings [-h] [-o OUTPUT_DIRECTORY] [-d DESCRIPTION] [--kpi-re-pattern KPI_RE_PATTERN]
[--max-time MAX_TIME] [--offset OFFSET] [--post-process POST_PROCESS]
Collects logs and timings relevant to boot time performance.
options:
-h, --help show this help message and exit
-o, --output-directory OUTPUT_DIRECTORY
Directory where output files will be stored (default: .)
-d, --description DESCRIPTION
Free form description to be added to the metadata (default: None)
--kpi-re-pattern KPI_RE_PATTERN
Regular expression to match log messages on for KPI highlighting (default:
multi-user.target|sysinit.target|Load Kernel Modules|mounted)
--max-time MAX_TIME Time in ms to limit the collection. Default to completion of switchroot. 0 to
not limit collection. (default: )
--offset OFFSET Time in ms to offset SystemInit to measure wall clock time. Default to 0.
(default: 0)
--post-process POST_PROCESS
Post-process an existing '-boot_time_logs.json' from the boot_timings script.
Other options except '-o' will be ignored and all information will be read from
the provided file. The summary, log, and chart files will be created, and the
report will be displayed. (default: None)
The script produces various formats of data to stdout as well as several artifact
files. The direct output sections include an ASCII graphical timeline, durations of boot
stages, a hierarchical summary of stages, and specific KPIs and their timestamps.
The logs are normalized and output as a _logs.json file. The normalized logs are also
fed into a charting module, which creates an HTML/JavaScript interactive chart output as
a _chart.html file. Important boot time summary data that may be futher processed by
automation and change detection systems is output as a _summary.json file.
Example console output:
NOTES:
======
- CNTVCT offset of 1860.662 ms has been applied to SystemInit.
- Offset(s) applied. Times will be reported with an offset of 1860.662 ms.
- Default to logging until completion of switch root.
- Logs limit: 16729.288 ms
# System boot time analysis
## Timeline of Boot Stage Durations
[=======/===/==/=========================================================/=>
^ SystemInit: 1861 ms
^ Kernel: 914 ms
^ InitRD: 785 ms
^ Switchroot: 13170 ms
## Durations (ms)
Stage Components Start End
SystemInit 0.0 1860.662
KernelTimerOffset 0.0 0.0
SystemInitCNTVCT 0.0 1860.662
Kernel 1860.662 2774.326
InitRD 2774.326 3559.303
InitRDSystemdInit 2774.326 2884.709
InitRDSecurity 2778.72 2779.231
InitRDGenerators 2864.125 2864.477
InitRDUnitsLoad 2864.487 2884.709
Switchroot 3559.303 16729.288
SystemdInit 3559.303 4051.715
Security 3572.869 3583.445
Generators 3865.117 3965.909
UnitsLoad 3965.941 4051.715
## Summary
SystemInit: 1860.662 ms
KernelTimerOffset: 0.0 ms
SystemInitCNTVCT: 1860.662 ms
Kernel: 913.664 ms
InitRD: 784.977 ms
InitRDSystemdInit: 110.383 ms
InitRDSecurity: 0.511 ms
InitRDGenerators: 0.352 ms
InitRDUnitsLoad: 20.222 ms
Initrd systemd units load finished at 2.884709 ms: 0.0 ms
Switchroot: 13169.985 ms
SystemdInit: 492.412 ms
Security: 10.576 ms
Generators: 100.792 ms
UnitsLoad: 85.774 ms
Root systemd units load finished at: 4051.715 ms
Overall: 16729.288 ms
NInstalledJobs: 172
NFailedJobs: 4
## KPI Timestamps (ms)
KPI Timestamp
Initrd systemd units load finished at 2.88470… 2884.709
Root systemd units load finished at 4051.715
systemd[1]: Load Kernel Modules was skipped b… 2928.859
kernel: EXT4-fs (mmcblk0p6): mounted filesyst… 3053.253
systemd[1]: Starting Load Kernel Modules... 4929.261
systemd[1]: Finished Load Kernel Modules. 5054.084
kernel: EXT4-fs (mmcblk0p5): mounted filesyst… 9730.999
sysinit.target 11109.965
multi-user.target 13937.198
## Artifacts
Created log JSON file at ./localhost_06_10_2025_17_04_12-boot_time_logs.json
Created log chart at ./localhost_06_10_2025_17_04_12-boot_time_chart.html
Created summary JSON file at ./localhost_06_10_2025_17_04_12-boot_time_summary.json
Created output log file at ./localhost_06_10_2025_17_04_12-boot_time_output.log
Example interactive chart:
Wall clock time and offsets versus kernel time¶
One major challenge of testing boot time is reporting the data in wall clock time, that is, the true elapsed time from power on to the log events as measured by an external clock. The Linux kernel starts its internal timer only after hardware initialization and firmware phases, and after an early kernel initialization phase for memory and other subsystem, so those log times are necessarily missing the time of that early initialization.
There are a couple of mechanisms to address this limitation. The first is a tool called
cntvct-log, which is maintained alongside the boot_timings script:
When installed and enabled as a systemd service against a target such as
@basic.service, this adds a secondary timestamp that is based on the CPU clock cycle.
The boot_timings tool looks for this timestamp, and if found will automatically
calculate the system init offset and use the new value as the basis for all timestamps,
thus reporting all logs more closely relative to true wall clock time.
Sometimes there is also an additional time offset from hardware initialization that may
be outside the scope of the primary system CPU, such as co-processor initialization. In
these cases, even the cntvct offset may not be entirely accurate. Typically these
additional initialization phases are very consistent for a particular hardware platform,
so the boot timing tool also allows the addition of a static user-provided offset.
For even more explicit observability, cntvct is also provided as a kernel patch that
allows the hardware timestamp to be entered into the logs as early as possible with
kernel activities. This allows for separately identifying the time taken during this
early kernel initialization phase (labeled here as pre-timer). Because this requires a
kernel patch, it is an optional feature that you can implement manually.
The cntvct kernel patch is available in a GitLab branch here:
https://gitlab.com/echanude/centos-stream-9/-/commit/d5e04706047b9bfb75f8f3297219df5d33f98f27
systemd-analyze¶
systemd-analyze is a tool that may be used to determine system boot-up performance statistics and retrieve other state and tracing information from the system and service manager. Some examples of it’s usage are below.
Basic command just to get high level boot statistics:
If a systemd unit’s dependencies are satisfied it proceeds, critical-chain can be used to see the various points in a chain of units.
Plot an svg graphic to file, viewable in an svg viewer (most modern web browsers are capable svg viewers):
systemd-bootchart¶
systemd-bootchart is a tool that can be run at boot-up to collect CPU, memory,
I/O, and per-process timing information. systemd-bootchart can be run at
boot-up by adding init=/usr/lib/systemd/systemd-bootchart to the kernel
command line.
systemd-bootchart collects data for 20 seconds and then outputs an SVG file
to /run/log, which can be viewed in a browser.
initcall_debug can also be added to the kernel command line in order to
include timing information for kernel init threads.
Options can be set in the file /etc/systemd/bootchart.conf.