Inside the system: resources, kernel, processes / Stage 7 of 12 · step 7 of 8

An application exits silently

~25 min

Not running Starting usually ~20 s

Situation

The warehouse moved to a new server. The inventory service lab-inventory — the inventory-api application — does not start in its new home: systemctl status lab-inventory shows status=1/FAILURE, and the journal has not a single line from the application itself. The application settings were moved over. The backup of the old server is in /var/backups/inventory. The application records every request through the event service lab-relay — that one is running.

There is no source code of the application on the machine, only the built program. Its author left long ago.

Task

The lab-inventory service runs as the inventory user and answers curl http://127.0.0.1:8085/health; the warehouses in the answer are the same as on the old server. The application code stays the same.
Do not widen permissions "for everyone": no file, directory or socket needs write access for any user. lab-relay runs as relay, as before.
The application still answers after both services restart: sudo systemctl restart lab-relay lab-inventory.

There is no internet on the machine; everything you need is already installed. Run administrative commands with sudo.

Lab goals

0 of 3

The lab-inventory service runs as inventory and answers /health with the old server's warehouses; the application code is unchanged — not checked yet
Write access is not opened to everyone anywhere, and lab-relay runs as relay, as before — not checked yet
The application answers after both services restart — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Processes · picked for this lab

strace — which system calls a process makes

Cheat sheet

$_ — insert into the terminal

strace cat /etc/hostname all calls of a command
strace -f sh -c '…' and of its children
strace -e trace=openat cmd openat only
strace -f -e trace=file cmd everything with file paths
strace -e trace=network cmd sockets and connections
strace -o /tmp/t.log -f cmd to a file, apart from the output
strace -c ls /etc > /dev/null summary: calls of each kind
sudo timeout 10 strace -p 1234 attach to a process

The main options are in the table below. The full list is man strace.

A program exits silently with code 1

bash

strace -f -o /tmp/t.log ./app; echo $?
grep -E '= -1 E' /tmp/t.log | tail

-o writes the trace to a file, and the program's output stays on screen. The last failures before exited with 1 are the cause: a missing file, no permission, a port that did not open.

Run as the service user

root can read any file, so sudo strace shows no permission failures. A service runs as its own user — trace as that user:

bash

sudo strace -e trace=openat cat /etc/app.conf
sudo -u appsvc strace -e trace=openat cat /etc/app.conf

text

openat(AT_FDCWD, "/etc/app.conf", O_RDONLY) = 3
…
openat(AT_FDCWD, "/etc/app.conf", O_RDONLY) = -1 EACCES (Permission …
cat: /etc/app.conf: Permission denied

systemctl show -p User <service> shows the service's user.

Which files a program reads

bash

strace -f -e trace=openat -o /tmp/t.log cmd
grep -v ENOENT /tmp/t.log | grep -o '"[^"]*"' | sort -u

Where it connects

text

student@ubuntu:~$ strace -e trace=network curl -s -m 2 127.0.0.1:9
…
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 5
connect(5, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr(…

The address and port are in the connect arguments.

What a process is stuck on

bash

sudo timeout 5 strace -p 1112

A single unfinished line means the process waits in that call: flock waits for a file lock (who holds it — lsof and /proc/locks, see proc-filesystem), read for data, connect for the network. A process in state D waits for the kernel inside a call, and strace may print nothing until the wait ends (see load-average).

Summary: which calls and how many

text

student@ubuntu:~$ strace -c ls /etc > /dev/null
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 22.41    0.000221         221         1           execve
 18.86    0.000186           5        34        12 openat
…
100.00    0.000986           6       150        18 total

calls is how many times, errors how many failed, % time the share of time in the kernel.

attach: ptrace(PTRACE_SEIZE, …): Operation not permitted

ptrace_scope = 1 forbids attaching to a process that is not your child. Run sudo strace -p <PID>. PID 1 and other users' processes need sudo too.

Program output is mixed with strace

Both write to the terminal. -o file sends the trace to a file.

Strings are cut: "hello-wo"...

strace shows 32 characters of a string, the rest is .... -s 200 shows 200.

No calls from children

A child process does the work, and strace without -f does not see it.

An application exits silently

Situation

Task

Signing in is needed to start the machine

A program exits silently with code 1

Run as the service user

Which files a program reads

Where it connects

What a process is stuck on

Summary: which calls and how many

attach: ptrace(PTRACE_SEIZE, …): Operation not permitted

Program output is mixed with strace

Strings are cut: "hello-wo"...

No calls from children