Containers: Docker and compose / Stage 5 of 11 · step 6 of 7

Data after re-creation

~20 min

Not running Starting usually ~20 s

Situation

The on-call shift keeps its notes in the notes application. It runs in a Docker container of the same name and lists the notes since the start of the month on port 8080. Last night a colleague updated the application to version 1.1: they removed the container and created a new one from the notes:1.1 image with the same port, NOTES_TITLE variable and restart policy. The container is running, but this morning curl http://localhost:8080/ shows notes: 0.

Run docker commands with sudo: the student user has no direct access to Docker. The machine has no internet access; the notes:1.0 and notes:1.1 images are already built.

Task

http://localhost:8080/ answers with version 1.1 of the application and shows all the notes that existed before the update.
The notes container keeps the notes where they were before the update and has no extra volumes.
The earlier notes are unchanged.
notes runs with the same port, NOTES_TITLE and restart policy as before: it will come up by itself after the machine reboots.

Lab goals

0 of 4

http://localhost:8080/ answers with notes 1.1 and shows all the notes that existed before the update — not checked yet
The notes container keeps the notes where they were before the update and has no extra volumes — not checked yet
The earlier notes are unchanged — their checksums match — not checked yet
notes runs with the same port, NOTES_TITLE and restart policy as before — it will come up by itself after a reboot — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Containers · picked for this lab

Docker — containers: running, ports, volumes, networks

Cheat sheet

$_ — insert into the terminal

sudo systemctl enable --now docker enable the service
sudo docker run -d --name web -p 8080:80 nginx:alpine background
sudo docker ps -a all containers and status
sudo docker logs --tail 20 web main process output
sudo docker exec -it web sh a shell inside
sudo docker stop web; sudo docker start web
sudo docker rm -f web remove, even if running
sudo docker port web published ports
sudo docker inspect -f '{{.State.Status}}' web one field
sudo docker volume ls volumes
sudo docker network ls networks
sudo docker stats --no-stream container CPU and memory

The service is configured in /etc/docker/daemon.json — JSON, no comments:

json

{
  "log-driver": "json-file",
  "log-opts": {"max-size": "10m", "max-file": "3"},
  "data-root": "/srv/docker"
}

Apply with sudo systemctl restart docker. A JSON error keeps the service from starting: check journalctl -u docker. What took effect:

bash

sudo docker info -f '{{.DockerRootDir}} {{.LoggingDriver}}'

On the lab machine that is /var/lib/docker json-file. A service restart stops every container. The restart policy decides which ones come back (see the section above). Change data-root only to an empty directory: the service does not move old images and volumes by itself.

Why the container stopped

bash

sudo docker ps -a --filter name=web
sudo docker logs --tail 30 web
sudo docker inspect -f '{{.State.ExitCode}} {{.State.OOMKilled}}' web

Code 0 means the command simply finished: for a service it means it went to the background. 137 means SIGKILL: out of memory (OOMKilled true) or a docker stop the process did not answer. 1, 2 and others are errors of the program itself, with the text in the logs.

Re-create a container with new options

bash

sudo docker inspect -f '{{json .Config.Env}} {{json .Mounts}}' web
sudo docker rm -f web
sudo docker run -d --name web --restart unless-stopped \
  -p 8080:80 -v data:/data -e MODE=prod app:1.1

Data in the volume survives re-creation; data in the container layer does not.

Find a container among host processes

text

student@ubuntu:~$ sudo docker inspect -f '{{.State.Pid}}' web1
2055
student@ubuntu:~$ ps -o pid,user,cmd -p 2055
    PID USER     CMD
   2055 root     sleep 600

A command in a container from a script

docker exec -it needs a terminal. In a script or a pipeline -t fails with the input device is not a TTY — drop it: docker exec web cat /etc/hosts.

permission denied … docker.sock

The user is not in the docker group. Run with sudo. Group membership gives root on the machine — see the Danger zone.

Cannot connect to the Docker daemon

text

Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Is the docker daemon running?

The service is not running: sudo systemctl enable --now docker. If you stopped only docker.service, systemd warns its triggering units are still active: docker.socket — the next docker command starts the service again through the socket.

port is already allocated

Bind for 0.0.0.0:8081 failed: port is already allocated — another container or service holds the host port: sudo ss -tlnp | grep :8081 (ss). The container has already been created — remove it with docker rm and start again with another port on the left.

The container name … is already in use

A container with that name exists, maybe a stopped one (docker ps -a). Remove the old one with docker rm or pick another name.

The container exits at once with code 0

The main process finished without an error. A service that went to the background does this: httpd without -f, nginx without daemon off. The main process has to run in the foreground.

docker stop takes 10 seconds and gives 137

docker stop sends SIGTERM and waits 10 seconds. A PID 1 process without its own SIGTERM handler never gets it — sleep and busybox httpd behave this way — so Docker finishes it with SIGKILL: Exited (137). For such containers use docker stop -t 1 or docker rm -f.

container … is not running

docker exec works only in a running container. Start a stopped one with docker start; if it dies at once, read docker logs.

docker logs | grep finds nothing

docker logs returns the container output on two streams: what the process wrote to stderr goes to stderr. Program errors usually live there, and a pipe does not see them. Merge the streams: sudo docker logs web 2>&1 | grep error.

Data gone after re-creating

The data was in the container layer. Mount a volume: -v data:/path. An old volume is still there if the container had one: docker volume ls.

Containers cannot see each other by name

Both are on the default bridge network: ping: bad address. Create your own network and connect both (docker network connect).

volume is in use / has active endpoints

A container needs the volume or network, maybe a stopped one. List them with docker ps -a --filter volume=data1 or --filter network=appnet; remove the containers, then the volume or network.

The docker group is root

Whoever can write to docker.sock can start a container with -v /:/host and get the whole host disk as root. Adding a user to the docker group is the same as giving them passwordless sudo.

docker rm -f and data in the container layer

rm -f stops and removes a container at once, together with its layer. Anything the container wrote outside a volume or bind mount is gone.

docker volume prune -a and docker system prune

docker volume prune removes anonymous volumes no container needs; with -a, named ones too. A volume whose container you are re-creating right now is "needed by nobody" at that moment — its data is deleted for good. Check docker volume ls before cleaning up.

Data after re-creation

Situation

Task

Signing in is needed to start the machine

Why the container stopped

Re-create a container with new options

Find a container among host processes

A command in a container from a script

permission denied … docker.sock

Cannot connect to the Docker daemon

port is already allocated

The container name … is already in use

The container exits at once with code 0

docker stop takes 10 seconds and gives 137

container … is not running

docker logs | grep finds nothing

Data gone after re-creating

Containers cannot see each other by name

volume is in use / has active endpoints

The docker group is root

docker rm -f and data in the container layer

docker volume prune -a and docker system prune