Inside the system: resources, kernel, processes / Stage 4 of 12 · step 9 of 10

A kernel setting disappears

~20 min

Not running Starting usually ~20 s

Situation

The lab-sensors service — the sensor-hub application — receives the warehouse sensor readings over UDP. Last week it stopped starting. A colleague raised a kernel parameter for it with a command, and the service worked. The server was rebooted overnight, and lab-sensors fails to start again. The colleague insists the setting was saved to a file.

There is no need to reboot the machine: at boot the kernel parameters come from the same files, in the same order, as with sudo sysctl --system.

Task

The lab-sensors service is running, enabled at boot and answers curl http://127.0.0.1:8126/health. The application code stays the same.
The value survives a reboot: sudo sysctl --system sets the parameter the service needs, and takes it from a file in /etc.
The other parameters from the settings files stay as they were — both in the files and in the running kernel.

There is no internet on the machine; everything you need is already installed. Run administrative commands with sudo.

Lab goals

0 of 3

The lab-sensors service is running, enabled at boot and answering; the application code is unchanged — not checked yet
sudo sysctl --system sets net.core.rmem_max to at least 8388608 (8 MiB) — from a file in /etc — not checked yet
The other parameters from the settings files are unchanged — in the files and in the kernel — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Boot · picked for this lab

sysctl — kernel parameters at runtime

Cheat sheet

$_ — insert into the terminal

sysctl vm.swappiness read a parameter
sysctl -n vm.swappiness the value only
sudo sysctl -w vm.swappiness=10 until reboot
echo 'vm.swappiness = 10' \
| sudo tee /etc/sysctl.d/90-swappiness.conf permanently
sudo sysctl --system apply all files
sudo sysctl -p /etc/sysctl.d/90-swappiness.conf one file
sysctl -a 2>/dev/null | grep forward find a parameter

Common parameters:

Parameter	What for
`vm.swappiness`	eagerness to swap (60 by default)
`net.ipv4.ip_forward`	packet forwarding: router, NAT, containers
`fs.nr_open`	ceiling of the per-process open files limit
`fs.file-max`	open files for the whole system
`fs.inotify.max_user_watches`	inotify watches (editors, `tail -f`)
`kernel.pid_max`	the largest PID (4194304 on Ubuntu)
`kernel.dmesg_restrict`	1 — `dmesg` for root only
`kernel.hung_task_timeout_secs`	after how many seconds in `D` the kernel reports a hung task
`net.core.somaxconn`	connection queue of a listening socket

The command's own options are in the table below.

Find a parameter

bash

sysctl -a 2>/dev/null | grep swappiness
sudo sysctl -a | grep -i forward

2>/dev/null hides the permission denied on key lines; with sudo all parameters are visible.

Check that a value survives a reboot

bash

grep -r swappiness /etc/sysctl.d/ /etc/sysctl.conf /usr/lib/sysctl.d/
sudo sysctl --system | grep -E 'Applying|swappiness'

If sysctl --system gives the right value, boot will give the same: the systemd-sysctl service reads the same files in the same order.

Who set the value

sysctl --system first prints all the * Applying … lines in order, then all the values. systemd-analyze shows more clearly which file set a parameter last: the files in the order they apply, each after a # /path line:

bash

systemd-analyze cat-config sysctl.d | grep -B3 'swappiness'

text

kernel.pid_max = 4194304

# /etc/sysctl.d/95-test.conf
vm.swappiness = 33

A value from a file lower in this list overrides the ones above it.

Return the default value

Remove your file from /etc/sysctl.d/ and set the value by hand (sudo sysctl -w vm.swappiness=60) or reboot — sysctl --system does not restore old values, it only applies files.

Changed it, reset after a reboot

sysctl -w and writes to /proc/sys last until reboot. You need a *.conf file in /etc/sysctl.d/ (section "Permanently" above).

permission denied on key, yet the value is printed

text

student@ubuntu:~$ sysctl -w vm.swappiness=20; echo rc=$?
vm.swappiness = 20
sysctl: permission denied on key "vm.swappiness", ignoring
rc=0
student@ubuntu:~$ sysctl vm.swappiness
vm.swappiness = 60

Without sudo, sysctl prints the line with the new value and exits with 0, but does not change the value. The error is on standard error. Repeat with sudo and check by reading.

A file in sysctl.d is not applied

The name does not end in .conf — the file is skipped silently.
Another file later in alphabetical order sets its own value — find it: grep -r <parameter> /etc/sysctl.d/ /usr/lib/sysctl.d/ /etc/sysctl.conf.
A typo in the parameter name: sysctl --system skips it without a message and with exit code 0. sudo sysctl -p <file> shows cannot stat /proc/sys/…: No such file or directory and returns 1.

cannot stat /proc/sys/…: No such file or directory

bash

sudo sysctl -w net.ipv4.no_such=1   # exit code 1

The answer is sysctl: cannot stat /proc/sys/net/ipv4/no_such: No such file or directory.

There is no such parameter: a typo, or the parameter appears only with a kernel module (for example, net.netfilter.* after nf_conntrack is loaded, see kernel-modules).

invalid syntax, continuing

text

student@ubuntu:~$ sudo sysctl -w net.ipv4.ip_forward = 1
sysctl: command line(0): invalid syntax, continuing...

On the command line name=value has no spaces. In sysctl.d files spaces around = are fine.

Forwarding is on, but packets do not pass

Besides net.ipv4.ip_forward = 1 you need routes (ip) and allowing firewall rules, plus a masquerade rule for NAT.

sysctl --system --dry-run writes to the kernel

--dry-run should only show what would be applied. In procps 4.0.4 the order of keys matters: sudo sysctl --dry-run --system changes nothing, while sudo sysctl --system --dry-run applies the files for real. Checked on the training machine: after the second command vm.swappiness went from 60 to 33. To try a file, use sudo sysctl --dry-run -p <file>.

A kernel setting disappears

Situation

Task

Signing in is needed to start the machine

Find a parameter

Check that a value survives a reboot

Who set the value

Return the default value

Changed it, reset after a reboot

permission denied on key, yet the value is printed

A file in sysctl.d is not applied

cannot stat /proc/sys/…: No such file or directory

invalid syntax, continuing

Forwarding is on, but packets do not pass

sysctl --system --dry-run writes to the kernel