One of our customers builds a connected electrical panel for homes. Its gateway runs Linux on an i.MX 8M Mini system-on-module, in two hardware generations from two module vendors, with Yocto, a 5.4 kernel and over-the-air updates. The units sit in homes with no serial console attached, and when the kernel has a problem, the network link is often the first thing to go. When a unit rebooted on its own, we needed its last kernel messages, kept on the unit itself.
pstore provides exactly that. When the kernel panics, pstore writes the end of the kernel log into a reserved area of RAM; that content survives a warm reset, and after the reboot the record shows up as a file. We enabled it on both hardware generations, with persistent function tracing available for units under investigation. The reserved area relies on the same /reserved-memory mechanism that keeps Linux away from the firmware’s memory, described in our article on the arm64 boot flow.
Contents
pstore and Its RAM Back End
pstore is the generic part. It brings a file system mounted on /sys/fs/pstore, front ends that produce records (kernel log dumps on oops and panic, the console, ftrace, and pmsg for user space), and back ends that store them. In 5.4, the back ends include ramoops for RAM, efi-pstore for UEFI variables and erst for ACPI platforms. On an Arm SoC without UEFI variables or ACPI, ramoops is the one that applies. pstore/blk, which keeps records on a block device, came with 5.8 (see its documentation).
The whole scheme depends on DRAM keeping its content across the reset. A warm reset, after a panic or from the watchdog of the SoC, leaves DRAM powered, as long as the watchdog resets only the SoC and does not drive the PMIC through its WDOG_B output, which TF-A can enable and which boards often wire to it. A power cut, or a reset line that also cycles the power rails, loses the records. ramoops can protect each zone with an error correcting code, through the ecc-size property, against the few bits that may flip in between.
Kernel Configuration
Each module vendor shipped its own kernel recipe and defconfig. Rather than patch both, we set the options from a class that every kernel recipe of the product inherits:
# classes/product-linux-common-config.bbclass
kernel_do_configure_prepend() {
# pstore
# kernel_conf_variable PSTORE_PMSG y
# kernel_conf_variable PSTORE_CONSOLE y
kernel_conf_variable PSTORE_RAM y
kernel_conf_variable FTRACE y
kernel_conf_variable FUNCTION_TRACER y
kernel_conf_variable FUNCTION_GRAPH_TRACER y
kernel_conf_variable PSTORE_FTRACE y
kernel_conf_variable FTRACE_SYSCALLS y
}
kernel_conf_variable, from kernel.bbclass, writes an option into .config before the configuration is completed. The machine configuration adds the class to the kernel recipes through KERNEL_CLASSES:
# conf/machine/include/product-imx8mm-common.inc (Dunfell syntax) KERNEL_CLASSES_append = " product-linux-common-config"
From Honister on, the override syntax becomes KERNEL_CLASSES:append and kernel_do_configure:prepend. The options themselves:
CONFIG_PSTORE_RAM: the ramoops back end. It depends onCONFIG_PSTORE, which has to be enabled too, and pstore records oops and panic dumps by default.CONFIG_PSTORE_FTRACE, withCONFIG_FTRACEandCONFIG_FUNCTION_TRACER: lets the function tracer write into ramoops, as described in the last section.CONFIG_FUNCTION_GRAPH_TRACERandCONFIG_FTRACE_SYSCALLSare not needed by pstore. We added them for tracing sessions on units at the bench.CONFIG_PSTORE_CONSOLEandCONFIG_PSTORE_PMSGstayed disabled on the products. We enabled them on the evaluation kit used to validate the setup, for the tests of the pmsg section.
The kernel must also reboot after a panic. kernel.panic = 5 in sysctl, or panic=5 on the command line, resets the board five seconds later.
Reserving the RAM in the Device Tree
ramoops takes its region from a child of /reserved-memory with the compatible string “ramoops”. Linux leaves the region out of its page allocator, and ramoops maps it at boot. The properties that size it:
reg: base address and size of the region.record-size: size of each oops or panic record. What remains of the region after the other zones is split into as many records as fit.console-size,ftrace-size,pmsg-size: one zone each. A size of zero disables the front end.
ramoops rounds each of these four sizes down to a power of two. We added the node to the existing reserved-memory node of the board device tree:
reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;
ramoops@b0000000 {
compatible = "ramoops";
reg = <0x0 0xb0000000 0x0 0x100000>; /* 1 MB */
record-size = <0x8000>; /* 32 kB per dump */
console-size = <0x8000>; /* 32 kB */
ftrace-size = <0x8000>; /* 32 kB */
pmsg-size = <0x1000>; /* 4 kB */
};
};
DRAM starts at 0x40000000 on the i.MX 8M Mini. On a 2 GB module, the region above sits well inside DRAM and below the top area that OP-TEE takes when it is enabled. ramoops lays out the zones in the order of figure 2: the dump records first, then console, ftrace and pmsg. With 1 MB and these sizes, 956 kB remain for dumps, that is 29 records of 32 kB.
Without a device tree node, the same settings go on the kernel command line, as ramoops.mem_address, ramoops.mem_size, ramoops.record_size and so on. They are read at boot and cannot be changed at run time; /sys/module/ramoops/parameters/ shows the values in use.
Testing with a Forced Panic
We validated the setup in the lab on an i.MX 8M Mini evaluation kit running the same 5.4 kernel, before enabling it on the products. With systemd, /sys/fs/pstore is mounted at boot. On an image without systemd, mount it by hand:
root@imx8mmevk:~# mount -t pstore pstore /sys/fs/pstore
The sysrq c command forces a panic. With kernel.panic at 5, the board resets five seconds later:
root@imx8mmevk:~# echo c > /proc/sysrq-trigger
After the reset, the record is waiting:
root@imx8mmevk:~# cat /sys/fs/pstore/dmesg-ramoops-0 Panic#1 Part1 <6>[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034] <5>[ 0.000000] Linux version 5.4.154 (oe-user@oe-host) (gcc version 9.3.0 (GCC)) #1 SMP PREEMPT Fri Nov 26 15:33:03 UTC 2021 [...] <6>[ 238.779510] sysrq: Trigger a crash <0>[ 238.783033] Kernel panic - not syncing: sysrq triggered crash <4>[ 238.788803] CPU: 0 PID: 1602 Comm: sh Not tainted 5.4.154 #1 <4>[ 238.796994] Hardware name: NXP i.MX8MMini LPDDR4 EVK board <4>[ 238.805095] Call trace: <4>[ 238.807556] dump_backtrace+0x0/0x148 <4>[ 238.811228] show_stack+0x24/0x30 <4>[ 238.814555] dump_stack+0xb4/0x110 <4>[ 238.817969] panic+0x160/0x368 <4>[ 238.821037] sysrq_handle_reboot+0x0/0x28 <4>[ 238.825054] __handle_sysrq+0x90/0x188 <4>[ 238.828809] write_sysrq_trigger+0x98/0xb8 <4>[ 238.832912] proc_reg_write+0x8c/0xe0 <4>[ 238.836582] __vfs_write+0x48/0x90 <4>[ 238.839990] vfs_write+0xe4/0x1c8 <4>[ 238.843310] ksys_write+0x78/0x100 <4>[ 238.846720] __arm64_sys_write+0x24/0x30 <4>[ 238.850657] el0_svc_common.constprop.0+0x74/0x180 <4>[ 238.855460] el0_svc_handler+0x34/0xa0 <4>[ 238.859223] el0_svc+0x8/0x208 <2>[ 238.862294] SMP: stopping secondary CPUs <0>[ 238.866231] Kernel Offset: disabled <0>[ 238.869727] CPU features: 0x0002,2000200c <0>[ 238.873742] Memory Limit: none
The header gives the reason, Panic, the dump counter and the part number; pstore can split a long log into parts, but ramoops keeps only the first one. A dump holds the end of the kernel log, compressed by default in 5.4 so that more than record-size bytes of text fit, and the call trace of the panic is always there; on this board, a 32 kB record also covered the boot messages. The trace shows sysrq_handle_reboot+0x0 where sysrq_handle_crash called panic(): since panic() never returns, the call is the last instruction of sysrq_handle_crash, and the return address points to the start of the next function.
Within one boot, each new dump takes the next record, and when all are used, the oldest is overwritten. The index starts again at record 0 at every boot, so the first dump after a reboot overwrites dmesg-ramoops-0 unless it was removed. Deleting a file erases its record:
root@imx8mmevk:~# rm /sys/fs/pstore/dmesg-ramoops-0
systemd 243 and later ships systemd-pstore.service for this job. At boot, it copies the records to /var/lib/systemd/pstore and removes them from pstore, so the zones are free for the next crash and the logs survive later power cuts.
Records from User Space: pmsg
With CONFIG_PSTORE_PMSG, the kernel creates /dev/pmsg0. What a process writes there goes into the pmsg zone and comes back as pmsg-ramoops-0 after a reboot. We tried it on the evaluation kit:
root@imx8mmevk:~# echo "Hello world" > /dev/pmsg0 root@imx8mmevk:~# reboot
After the reboot:
root@imx8mmevk:~# cat /sys/fs/pstore/pmsg-ramoops-0 Hello world
The zone is small and the device only appends, so it suits short markers, such as the version an update just installed or the last state of a service, read next to the kernel record of a crash.
Persistent Function Tracing
A panic record shows where the kernel died. Some failures leave none. A hard lock-up ends with a watchdog reset, and the kernel log stops before anything useful. pstore ftrace writes every kernel function call into the ftrace zone, so the last calls before the reset survive it. Recording is off by default and is switched on at run time:
root@imx8mmevk:~# mount -t debugfs debugfs /sys/kernel/debug root@imx8mmevk:~# echo 1 > /sys/kernel/debug/pstore/record_ftrace
The setting does not survive a reboot, so a boot script has to switch it on again on a unit under investigation. Each function call costs a write into the zone, which slows the whole kernel down while recording is on. After the next reset, the directory holds one more file:
root@imx8mmevk:~# ls /sys/fs/pstore/ dmesg-ramoops-0 dmesg-ramoops-1 ftrace-ramoops-0 pmsg-ramoops-0
In 5.4, a pstore file name is the record type, the back end and an index. The ftrace zone is a single record, ftrace-ramoops-0, unless the per-CPU flag of ramoops splits it into one zone per CPU. Each line of the file comes from this format, in fs/pstore/inode.c:
seq_printf(s, "CPU:%d ts:%llu %08lx %08lx %ps <- %pSn",
pstore_ftrace_decode_cpu(rec),
pstore_ftrace_read_timestamp(rec),
rec->ip, rec->parent_ip, (void *)rec->ip,
(void *)rec->parent_ip);
A line gives the CPU, a timestamp, the addresses of the function and of its caller, then both as symbols. The zone wraps around, so with 32 kB only the most recent calls remain. That is enough to see which driver and which path the kernel was in, though not a full history.
What We Got from It
Every kernel panic on a gateway now leaves a record on the unit itself, with its call trace, as long as the reset keeps DRAM powered, for 1 MB of RAM and no extra hardware. On units where recording is switched on, the function trace covers the cases where the kernel stops without a panic. With a 5.8 or later kernel and a block device to spare, pstore/blk also keeps records across power cuts; on a 5.4 kernel, and for warm resets, ramoops does the job.
How do you collect kernel logs from units in the field? Tell us in the comments.


