Containers: Docker and compose / Stage 2 of 11 · step 7 of 8

A service from a container

~20 min

Not running Starting usually ~20 s

Situation

The team's status page on this machine is served by the status-legacy service on port 8080, version 0.9. The developers sent the new version as the Docker image status-page:1.0: it holds httpd from busybox and the page itself. The image is already loaded; now the page has to move into a container.

Run docker commands with sudo: the student user has no direct access to Docker. There is no internet on the machine.

Task

Container status from the status-page:1.0 image is running and not restarting.
curl http://localhost:8080/ gets the new page, status-page 1.0, from container status.
After a reboot, port 8080 belongs to container status again: Docker starts at boot, and the old service does not.
The restart policy of status is unless-stopped: if the on-call admin stops the container by hand, it does not come back by itself. The page still answers after sudo systemctl restart docker — the check restarts docker itself.

Lab goals

0 of 4

Container status from the status-page:1.0 image is running and not restarting — not checked yet
http://localhost:8080/ serves the status-page 1.0 page from container status — not checked yet
After a reboot, port 8080 goes to the container — docker starts at boot, status-legacy does not — not checked yet
The restart policy of status is unless-stopped, and the page answers after a docker restart too — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Containers · picked for this lab

Docker — containers: running, ports, volumes, networks

Cheat sheet

$_ — insert into the terminal

sudo systemctl enable --now docker enable the service
sudo docker run -d --name web -p 8080:80 nginx:alpine background
sudo docker ps -a all containers and status
sudo docker logs --tail 20 web main process output
sudo docker exec -it web sh a shell inside
sudo docker stop web; sudo docker start web
sudo docker rm -f web remove, even if running
sudo docker port web published ports
sudo docker inspect -f '{{.State.Status}}' web one field
sudo docker volume ls volumes
sudo docker network ls networks
sudo docker stats --no-stream container CPU and memory

The service is configured in /etc/docker/daemon.json — JSON, no comments:

json

{
  "log-driver": "json-file",
  "log-opts": {"max-size": "10m", "max-file": "3"},
  "data-root": "/srv/docker"
}

Apply with sudo systemctl restart docker. A JSON error keeps the service from starting: check journalctl -u docker. What took effect:

bash

sudo docker info -f '{{.DockerRootDir}} {{.LoggingDriver}}'

On the lab machine that is /var/lib/docker json-file. A service restart stops every container. The restart policy decides which ones come back (see the section above). Change data-root only to an empty directory: the service does not move old images and volumes by itself.

Why the container stopped

bash

sudo docker ps -a --filter name=web
sudo docker logs --tail 30 web
sudo docker inspect -f '{{.State.ExitCode}} {{.State.OOMKilled}}' web

Code 0 means the command simply finished: for a service it means it went to the background. 137 means SIGKILL: out of memory (OOMKilled true) or a docker stop the process did not answer. 1, 2 and others are errors of the program itself, with the text in the logs.

Re-create a container with new options

bash

sudo docker inspect -f '{{json .Config.Env}} {{json .Mounts}}' web
sudo docker rm -f web
sudo docker run -d --name web --restart unless-stopped \
  -p 8080:80 -v data:/data -e MODE=prod app:1.1

Data in the volume survives re-creation; data in the container layer does not.

Find a container among host processes

text

student@ubuntu:~$ sudo docker inspect -f '{{.State.Pid}}' web1
2055
student@ubuntu:~$ ps -o pid,user,cmd -p 2055
    PID USER     CMD
   2055 root     sleep 600

A command in a container from a script

docker exec -it needs a terminal. In a script or a pipeline -t fails with the input device is not a TTY — drop it: docker exec web cat /etc/hosts.

permission denied … docker.sock

The user is not in the docker group. Run with sudo. Group membership gives root on the machine — see the Danger zone.

Cannot connect to the Docker daemon

text

Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Is the docker daemon running?

The service is not running: sudo systemctl enable --now docker. If you stopped only docker.service, systemd warns its triggering units are still active: docker.socket — the next docker command starts the service again through the socket.

port is already allocated

Bind for 0.0.0.0:8081 failed: port is already allocated — another container or service holds the host port: sudo ss -tlnp | grep :8081 (ss). The container has already been created — remove it with docker rm and start again with another port on the left.

The container name … is already in use

A container with that name exists, maybe a stopped one (docker ps -a). Remove the old one with docker rm or pick another name.

The container exits at once with code 0

The main process finished without an error. A service that went to the background does this: httpd without -f, nginx without daemon off. The main process has to run in the foreground.

docker stop takes 10 seconds and gives 137

docker stop sends SIGTERM and waits 10 seconds. A PID 1 process without its own SIGTERM handler never gets it — sleep and busybox httpd behave this way — so Docker finishes it with SIGKILL: Exited (137). For such containers use docker stop -t 1 or docker rm -f.

container … is not running

docker exec works only in a running container. Start a stopped one with docker start; if it dies at once, read docker logs.

docker logs | grep finds nothing

docker logs returns the container output on two streams: what the process wrote to stderr goes to stderr. Program errors usually live there, and a pipe does not see them. Merge the streams: sudo docker logs web 2>&1 | grep error.

Data gone after re-creating

The data was in the container layer. Mount a volume: -v data:/path. An old volume is still there if the container had one: docker volume ls.

Containers cannot see each other by name

Both are on the default bridge network: ping: bad address. Create your own network and connect both (docker network connect).

volume is in use / has active endpoints

A container needs the volume or network, maybe a stopped one. List them with docker ps -a --filter volume=data1 or --filter network=appnet; remove the containers, then the volume or network.

The docker group is root

Whoever can write to docker.sock can start a container with -v /:/host and get the whole host disk as root. Adding a user to the docker group is the same as giving them passwordless sudo.

docker rm -f and data in the container layer

rm -f stops and removes a container at once, together with its layer. Anything the container wrote outside a volume or bind mount is gone.

docker volume prune -a and docker system prune

docker volume prune removes anonymous volumes no container needs; with -a, named ones too. A volume whose container you are re-creating right now is "needed by nobody" at that moment — its data is deleted for good. Check docker volume ls before cleaning up.

A service from a container

Situation

Task

Signing in is needed to start the machine

Why the container stopped

Re-create a container with new options

Find a container among host processes

A command in a container from a script

permission denied … docker.sock

Cannot connect to the Docker daemon

port is already allocated

The container name … is already in use

The container exits at once with code 0

docker stop takes 10 seconds and gives 137

container … is not running

docker logs | grep finds nothing

Data gone after re-creating

Containers cannot see each other by name

volume is in use / has active endpoints

The docker group is root

docker rm -f and data in the container layer

docker volume prune -a and docker system prune