Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

strace — which system calls a process makes

8 min strace 6.8 · Ubuntu 24.04

strace shows the system calls of a program and their results: a file that did not open, a failed connect, a wait.

Cheat sheet

8 commands Printable version
  • strace cat /etc/hostname all calls of a command
  • strace -f sh -c '…' and of its children
  • strace -e trace=openat cmd openat only
  • strace -f -e trace=file cmd everything with file paths
  • strace -e trace=network cmd sockets and connections
  • strace -o /tmp/t.log -f cmd to a file, apart from the output
  • strace -c ls /etc > /dev/null summary: calls of each kind
  • sudo timeout 10 strace -p 1234 attach to a process

Overview

A program does not open or send anything by itself: it asks the kernel. Each such request is a system call: openat opens a file, read and write read and write, connect connects, exit_group ends the process. strace runs the program and prints every call with its arguments and result — to standard error, so as not to mix with the program's output.

text 
student@ubuntu:~$ strace -e trace=openat,read,write cat /etc/hostname
…
openat(AT_FDCWD, "/etc/hostname", O_RDONLY) = 3
read(3, "ubuntu\n", 131072)             = 7
write(1, "ubuntu\n", 7ubuntu
)                 = 7
read(3, "", 131072)                     = 0
+++ exited with 0 +++

Reading a line

name(arguments) = result. openat(…, "/etc/hostname", O_RDONLY) = 3 means the file was opened and got descriptor 3; then read(3, …) reads exactly that file. write(1, …) writes to standard output — that is why ubuntu cut into the strace line. The last line, +++ exited with 0 +++, is the exit code.

A failure is the result -1 plus an error name:

text 
openat(AT_FDCWD, "/nope", O_RDONLY)     = -1 ENOENT (No such file or …
openat(AT_FDCWD, "/etc/shadow", O_RDONLY) = -1 EACCES (Permission …
Error Meaning
ENOENT no such file or directory
EACCES permission denied
ECONNREFUSED nobody listens on that port
EMFILE the open files limit ran out (see ulimit)
EAGAIN the resource is busy, try later

Not every ENOENT is a problem. A program looks for files in several places in turn: sh tried /usr/local/sbin/ls, /usr/local/bin/ls, /usr/sbin/ls before it found /usr/bin/ls. What matters is the failure after which the program gave up — usually near the end.

Child processes: -f

Without -f, strace sees only the process it started. A shell, a service, a script do their work in children — with -f their calls are visible, each line with a PID:

text 
student@ubuntu:~$ strace -f -e trace=file sh -c 'cat /etc/shadow'
…
[pid  1060] openat(AT_FDCWD, "/etc/shadow", O_RDONLY) = -1 EACCES (…
cat: /etc/shadow: Permission denied

Picking calls: -e trace=

The full output runs to hundreds of lines. -e trace= keeps what you need:

Filter What stays
-e trace=openat openat only
-e trace=openat,read,write the listed calls
-e trace=file everything with a file path: openat, statx, execve, access
-e trace=network socket, connect, sendto, recvfrom
-e trace=process starting and ending processes

Attach to a process: -p

strace -p <PID> attaches to a running process and prints its calls from that moment on. Ubuntu sets kernel.yama.ptrace_scope = 1: without root you can trace only your own children. Attaching to an already running process — even your own — needs sudo:

text 
student@ubuntu:~$ strace -p 1081
strace: attach: ptrace(PTRACE_SEIZE, 1081): Operation not permitted

A waiting process shows an unfinished call — with no = and no result. Here flock waits for a lock held by another process:

text 
student@ubuntu:~$ sudo timeout 3 strace -p 1112
strace: Process 1112 attached
flock(3, LOCK_EXstrace: Process 1112 detached
 <detached ...>

timeout 3 detaches strace after 3 seconds (exit code 124); the process keeps working as if nothing happened. Ctrl+C does the same.

Options

9

The main options are in the table below. The full list is man strace.

Option What it does Example
-f Follow forked child processes and threads. strace -f cmd
-p PID Attach to an already running process by PID. sudo strace -p 1234
-e trace=... Filter by call group — file, network, openat,read, and more. strace -e trace=network cmd
-c Summary profile — which calls were made and how much time they took. strace -c cmd
-T Show the time spent in each call. strace -T -e trace=openat cmd
-y Show paths and addresses instead of raw file-descriptor numbers. strace -y -p 1234
-s N Length of displayed strings (default is 32 — often too short). strace -s 200 -e trace=write -p 1234
-o FILE Write the output to a file instead of the terminal. strace -o trace.log -f cmd
-tt A microsecond timestamp before each line. strace -tt -o trace.log cmd

Nothing found. Try a shorter query.

Recipes

6

A program exits silently with code 1

bash 
strace -f -o /tmp/t.log ./app; echo $?
grep -E '= -1 E' /tmp/t.log | tail

-o writes the trace to a file, and the program's output stays on screen. The last failures before exited with 1 are the cause: a missing file, no permission, a port that did not open.

Run as the service user

root can read any file, so sudo strace shows no permission failures. A service runs as its own user — trace as that user:

bash 
sudo strace -e trace=openat cat /etc/app.conf
sudo -u appsvc strace -e trace=openat cat /etc/app.conf
text 
openat(AT_FDCWD, "/etc/app.conf", O_RDONLY) = 3
…
openat(AT_FDCWD, "/etc/app.conf", O_RDONLY) = -1 EACCES (Permission …
cat: /etc/app.conf: Permission denied

systemctl show -p User <service> shows the service's user.

Which files a program reads

bash 
strace -f -e trace=openat -o /tmp/t.log cmd
grep -v ENOENT /tmp/t.log | grep -o '"[^"]*"' | sort -u

Where it connects

text 
student@ubuntu:~$ strace -e trace=network curl -s -m 2 127.0.0.1:9
…
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 5
connect(5, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr(…

The address and port are in the connect arguments.

What a process is stuck on

bash 
sudo timeout 5 strace -p 1112

A single unfinished line means the process waits in that call: flock waits for a file lock (who holds it — lsof and /proc/locks, see proc-filesystem), read for data, connect for the network. A process in state D waits for the kernel inside a call, and strace may print nothing until the wait ends (see load-average).

Summary: which calls and how many

text 
student@ubuntu:~$ strace -c ls /etc > /dev/null
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 22.41    0.000221         221         1           execve
 18.86    0.000186           5        34        12 openat
…
100.00    0.000986           6       150        18 total

calls is how many times, errors how many failed, % time the share of time in the kernel.

Troubleshooting

4

attach: ptrace(PTRACE_SEIZE, …): Operation not permitted

ptrace_scope = 1 forbids attaching to a process that is not your child. Run sudo strace -p <PID>. PID 1 and other users' processes need sudo too.

Program output is mixed with strace

Both write to the terminal. -o file sends the trace to a file.

Strings are cut: "hello-wo"...

strace shows 32 characters of a string, the rest is .... -s 200 shows 200.

No calls from children

A child process does the work, and strace without -f does not see it.

Terms

1
system call
A program's request to the kernel: open a file (openat), read, write, connect, exit (exit_group). strace prints every call with its result; a failure is -1 plus the error name. Also appears in: Load average: how many tasks wait for the CPU and the disk, The kernel, /proc: processes and the kernel as files, Service hardening: User=, ProtectSystem, NoNewPrivileges and 1 more

See also

Found an inaccuracy? Report an error in the article

Practice on this topic

Lessons on this topic

Topics
Loading…