Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

cgroups and namespaces — how containers work

6 min Linux · cgroups v2

Isolation (namespaces) and resource limits (cgroups) — the foundation of containers and systemd.

Cheat sheet

6 commands Printable version
  • lsns namespaces (isolation)
  • systemd-cgls cgroup tree
  • systemd-cgtop per-cgroup resource usage
  • sudo nsenter -t PID -a bash enter all namespaces of a process
  • sudo systemctl set-property nginx.service MemoryMax=512M CPUQuota=50% TasksMax=100
  • systemctl show nginx.service -p MemoryCurrent -p MemoryMax

Overview

Containers (Docker) are ordinary Linux processes that the kernel handles with two mechanisms:

  • namespaces — isolation: a process sees its own set of PIDs, network, mount points, users, hostname (as if it were a separate system);
  • cgroups — resource limits and accounting: CPU, memory, I/O for a group of processes.

systemd uses the same mechanisms (each service gets its own cgroup).

Recipes

3

namespaces

bash 
lsns                              # all namespaces and which processes are in them
lsns -t net                       # only network namespaces
ls -l /proc/$$/ns/                # namespaces of the current shell
sudo nsenter -t PID -n ss -tlnp   # run inside the NETWORK namespace of process PID
sudo nsenter -t PID -a bash       # enter ALL namespaces of a process ("inside the container")
unshare --net --pid --fork bash   # start a shell in new namespaces (isolation demo)
ip netns list                     # named network namespaces (see ip)

Types: pid (processes), net (network/interfaces), mnt (mounts), user (UID), uts (hostname), ipc, cgroup.

cgroups (v2)

bash 
systemd-cgls                      # cgroup tree (who's where)
systemd-cgtop                     # resource usage per cgroup (like top)
cat /sys/fs/cgroup/cgroup.controllers   # available controllers
systemctl status nginx            # near the bottom — CGroup and current usage

# limits for a service — through systemd (the correct way):
sudo systemctl set-property nginx.service MemoryMax=512M CPUQuota=50% IOWeight=50
systemctl show nginx -p MemoryMax -p CPUQuota -p MemoryCurrent
# or in the unit's [Service]: MemoryMax=512M / CPUQuota=50% / TasksMax=100

CPUQuota=50% is half of one core; MemoryMax= is a hard ceiling: going over it causes an OOM inside that service's group (see oom-killer); TasksMax= limits the number of processes.

Slices and CPU sharing

systemd places processes in a tree of groups. Services live in the system.slice slice, user sessions in user.slice; a group of services can get a slice of its own with Slice=shop.slice in the unit. Whose process it is shows in the CGroup line of systemctl status and in ps -o unit:

text 
student@ubuntu:~$ ps -o pid,unit,comm -p 1211,$$
    PID UNIT                            COMMAND
   1021 session-c1.scope                bash
   1211 ssh.service                     sshd

In the training image system.slice and user.slice have the cpu controller on (cat /sys/fs/cgroup/system.slice/cgroup.subtree_control). When several groups need the CPU, the kernel first shares it between the groups by CPUWeight (100 by default), and only inside a group between processes by their nice value. An experiment on one CPU with two stress-ng --cpu 1:

Where the processes are CPU shares
two services, CPUWeight=100 and CPUWeight=900 9.7 % and 88.7 %
two services, one with nice -n 19 49.1 % and 49.1 %
one service, one process with nice -n 19 96.6 % and 1.4 %

nice works only between neighbours in a group. To make one service yield to another, lower its weight or set a ceiling:

bash 
systemctl show app -p CPUWeight -p Slice
sudo systemctl set-property app CPUWeight=50     # now and for good
sudo systemctl set-property app CPUQuota=20%     # at most 20 % CPU
cat /sys/fs/cgroup/system.slice/app.service/cpu.weight

Troubleshooting

4

A process was "killed", logs mention memory cgroup / oom-kill

The service hit its cgroup's MemoryMax — the kernel killed it (a cgroup-OOM, not a global one). systemctl show svc -p MemoryMax -p MemoryCurrent; raise the limit or fix the leak (see oom-killer).

A service is "eating" all the CPU/disk

Limit it through systemd: CPUQuota=, IOWeight=/IOReadBandwidthMax=. This is cgroups, not ulimit (that limits a process/user, different things).

A container can't see other processes/network — that's normal

It's by design (namespaces isolate). To "look inside", use nsenter -t PID -a into the process's namespaces.

Where to see actual usage by service

systemd-cgtop (like top, but by cgroup/service) — shows right away which service is loading the system.

See also

  • veth — network namespaces and the veth pair in practice.
  • docker — Basic container work — running, logs, exec, images, compose, cleanup.
  • systemd — The init system and service manager (PID 1): units, targets, dependencies, journals.
  • ulimit — Limits on processes — open files, number of processes; limits.conf and systemd.

Found an inaccuracy? Report an error in the article

Practice on this topic

Lessons on this topic

Topics
Loading…