Containers: Docker and compose / Stage 10 of 11 · step 8 of 9

A container restarts under load

~25 min

Not running Starting usually ~20 s

Situation

The ingest service processes the nightly sales exports: a CSV file appears in /srv/ingest/inbox, and a summary of it appears in /srv/ingest/done. Until last week the service ran as the ingest system user, and the /srv/ingest directory belongs to that user. Then it was moved into a Docker container: the container is created by the /opt/ingest/run.sh script, and the import code sits next to it in /opt/ingest.

Since the move, not a single export has been processed: today's one, 40 MiB, is still sitting in inbox, and docker ps shows ingest up for a couple of seconds one moment and restarting the next.

Task

ingest processes exports and does not restart: the check will drop its own 40 MiB export into inbox and wait for the summary. Do not change the import code in /opt/ingest.
The memory limit of the ingest container is 128 MiB: that is how much the service is given on this machine.
The process in the container runs as the ingest user and group, as it did before the move.
ingest comes up by itself after the machine reboots.

Run docker commands with sudo: the student user has no direct access to Docker. The machine has no internet access; the alpine image is already loaded.

Lab goals

0 of 4

ingest processes a 40 MiB export without restarting, and the import code is unchanged — not checked yet
The memory limit of the ingest container is 128 MiB — not checked yet
The process in the container runs as the ingest user and group — not checked yet
ingest comes up by itself after the machine reboots — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Containers · picked for this lab

Docker — containers: running, ports, volumes, networks

Cheat sheet

$_ — insert into the terminal

sudo systemctl enable --now docker enable the service
sudo docker run -d --name web -p 8080:80 nginx:alpine background
sudo docker ps -a all containers and status
sudo docker logs --tail 20 web main process output
sudo docker exec -it web sh a shell inside
sudo docker stop web; sudo docker start web
sudo docker rm -f web remove, even if running
sudo docker port web published ports
sudo docker inspect -f '{{.State.Status}}' web one field
sudo docker volume ls volumes
sudo docker network ls networks
sudo docker stats --no-stream container CPU and memory

The service is configured in /etc/docker/daemon.json — JSON, no comments:

json

{
  "log-driver": "json-file",
  "log-opts": {"max-size": "10m", "max-file": "3"},
  "data-root": "/srv/docker"
}

Apply with sudo systemctl restart docker. A JSON error keeps the service from starting: check journalctl -u docker. What took effect:

bash

sudo docker info -f '{{.DockerRootDir}} {{.LoggingDriver}}'

On the lab machine that is /var/lib/docker json-file. A service restart stops every container. The restart policy decides which ones come back (see the section above). Change data-root only to an empty directory: the service does not move old images and volumes by itself.

Why the container stopped

bash

sudo docker ps -a --filter name=web
sudo docker logs --tail 30 web
sudo docker inspect -f '{{.State.ExitCode}} {{.State.OOMKilled}}' web

Code 0 means the command simply finished: for a service it means it went to the background. 137 means SIGKILL: out of memory (OOMKilled true) or a docker stop the process did not answer. 1, 2 and others are errors of the program itself, with the text in the logs.

Re-create a container with new options

bash

sudo docker inspect -f '{{json .Config.Env}} {{json .Mounts}}' web
sudo docker rm -f web
sudo docker run -d --name web --restart unless-stopped \
  -p 8080:80 -v data:/data -e MODE=prod app:1.1

Data in the volume survives re-creation; data in the container layer does not.

Find a container among host processes

text

student@ubuntu:~$ sudo docker inspect -f '{{.State.Pid}}' web1
2055
student@ubuntu:~$ ps -o pid,user,cmd -p 2055
    PID USER     CMD
   2055 root     sleep 600

A command in a container from a script

docker exec -it needs a terminal. In a script or a pipeline -t fails with the input device is not a TTY — drop it: docker exec web cat /etc/hosts.

permission denied … docker.sock

The user is not in the docker group. Run with sudo. Group membership gives root on the machine — see the Danger zone.

Cannot connect to the Docker daemon

text

Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Is the docker daemon running?

The service is not running: sudo systemctl enable --now docker. If you stopped only docker.service, systemd warns its triggering units are still active: docker.socket — the next docker command starts the service again through the socket.

port is already allocated

Bind for 0.0.0.0:8081 failed: port is already allocated — another container or service holds the host port: sudo ss -tlnp | grep :8081 (ss). The container has already been created — remove it with docker rm and start again with another port on the left.

The container name … is already in use

A container with that name exists, maybe a stopped one (docker ps -a). Remove the old one with docker rm or pick another name.

The container exits at once with code 0

The main process finished without an error. A service that went to the background does this: httpd without -f, nginx without daemon off. The main process has to run in the foreground.

docker stop takes 10 seconds and gives 137

docker stop sends SIGTERM and waits 10 seconds. A PID 1 process without its own SIGTERM handler never gets it — sleep and busybox httpd behave this way — so Docker finishes it with SIGKILL: Exited (137). For such containers use docker stop -t 1 or docker rm -f.

container … is not running

docker exec works only in a running container. Start a stopped one with docker start; if it dies at once, read docker logs.

docker logs | grep finds nothing

docker logs returns the container output on two streams: what the process wrote to stderr goes to stderr. Program errors usually live there, and a pipe does not see them. Merge the streams: sudo docker logs web 2>&1 | grep error.

Data gone after re-creating

The data was in the container layer. Mount a volume: -v data:/path. An old volume is still there if the container had one: docker volume ls.

Containers cannot see each other by name

Both are on the default bridge network: ping: bad address. Create your own network and connect both (docker network connect).

volume is in use / has active endpoints

A container needs the volume or network, maybe a stopped one. List them with docker ps -a --filter volume=data1 or --filter network=appnet; remove the containers, then the volume or network.

The docker group is root

Whoever can write to docker.sock can start a container with -v /:/host and get the whole host disk as root. Adding a user to the docker group is the same as giving them passwordless sudo.

docker rm -f and data in the container layer

rm -f stops and removes a container at once, together with its layer. Anything the container wrote outside a volume or bind mount is gone.

docker volume prune -a and docker system prune

docker volume prune removes anonymous volumes no container needs; with -a, named ones too. A volume whose container you are re-creating right now is "needed by nobody" at that moment — its data is deleted for good. Check docker volume ls before cleaning up.

A container restarts under load

Situation

Task

Signing in is needed to start the machine

Why the container stopped

Re-create a container with new options

Find a container among host processes

A command in a container from a script

permission denied … docker.sock

Cannot connect to the Docker daemon

port is already allocated

The container name … is already in use

The container exits at once with code 0

docker stop takes 10 seconds and gives 137

container … is not running

docker logs | grep finds nothing

Data gone after re-creating

Containers cannot see each other by name

volume is in use / has active endpoints

The docker group is root

docker rm -f and data in the container layer

docker volume prune -a and docker system prune