Observability: metrics, alerts, logs / Stage 7 of 9

Logs in one place

Not running Starting usually ~20 s

Step 6 of 7 · lab · 30 min

The second machine's logs are not on the collector

The team collects the logs of all machines on node1, each machine in its own directory. node2 was set up to forward its journal there, yet the collector has not a single line from it. Get the forwarding working end to end so that it survives a reboot of both machines.

Situation

The team collects the logs of all servers in one place — on node1. The collector is rsyslog: it sorts the lines of other machines into directories, one per machine — /var/log/remote/<machine name>/syslog.log. Yesterday node2 was connected to it: there rsyslog forwards to node1 everything that lands in its journal. Last night something happened on node2, and the collector has not a single line from it — not even the /var/log/remote/node2 directory.

node1 is your terminal. student logs in to node2 with a key: ssh node2 'command' or ssh node2, back — exit. sudo works without a password on both machines. The collector directory is readable with sudo.

Task

A line written by logger on node2 shows up on node1 in /var/log/remote/node2/ within a few seconds.
On node1 the logs are taken by rsyslog on the standard syslog port — 514: other machines will be connected to it too.
The node1 firewall stays on: open in it only what the log intake needs.
The setup survives a reboot of both machines.

Constraints

The check itself writes a logger line with its own tag on node2 and looks for it on node1. It also sends its own lines from node2 to node1:514 directly and opens an extra port on node1 for a moment — to make sure the firewall keeps it closed.
You do not need to reboot the machines for the check: it makes sure the running rsyslog was started with the config from its files and the nftables rules in the kernel are the ones /etc/nftables.conf will load.

There is no Internet in the lab — only the network between node1 and node2. Everything you need is installed on both machines.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 4

A line written by logger on node2 shows up on node1 in /var/log/remote/node2/ within a few seconds — not done yet
On node1 port 514 is held by rsyslog, on an address node2 can connect to — not done yet
The node1 firewall stays on — it lets traffic reach the log intake, while other ports stay closed to node2 — not done yet
The setup survives a reboot — rsyslog on both machines is enabled and runs the config from its files, the nftables rules in the kernel come from /etc/nftables.conf — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 4

A line written by logger on node2 shows up on node1 in /var/log/remote/node2/ within a few seconds — not done yet
On node1 port 514 is held by rsyslog, on an address node2 can connect to — not done yet
The node1 firewall stays on — it lets traffic reach the log intake, while other ports stay closed to node2 — not done yet
The setup survives a reboot — rsyslog on both machines is enabled and runs the config from its files, the nftables rules in the kernel come from /etc/nftables.conf — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 4

A line written by logger on node2 shows up on node1 in /var/log/remote/node2/ within a few seconds — not done yet
On node1 port 514 is held by rsyslog, on an address node2 can connect to — not done yet
The node1 firewall stays on — it lets traffic reach the log intake, while other ports stay closed to node2 — not done yet
The setup survives a reboot — rsyslog on both machines is enabled and runs the config from its files, the nftables rules in the kernel come from /etc/nftables.conf — not done yet

Logs and audit · picked for this lab

rsyslog: text logs and forwarding to a collector

Cheat sheet

$_ — insert into the terminal

tail -f /var/log/syslog the general log in real time
sudo rsyslogd -N1 check the whole config
sudo systemctl restart rsyslog apply the config
logger -t myapp 'check' to the journal and syslog
logger -p user.warning -t myapp 'warn' a line with warning severity
sudo ss -tlnp | grep ':514' is TCP input listening
sudo ls -R /var/log/remote what the collector received
journalctl -u rsyslog -n 20 errors of rsyslog itself

Network input: imudp and imtcp

A collector loads an input module and opens a port, usually 514:

text

module(load="imtcp")
input(type="imtcp" port="514")
module(load="imudp")
input(type="imudp" port="514")

Module	Protocol	In `ss`
`imtcp`	TCP: a connection, a line is not lost silently	`sudo ss -tlnp`: `LISTEN … 0.0.0.0:514`
`imudp`	UDP: no connection, simpler, lines can get lost	`sudo ss -ulnp`: `UNCONN … 0.0.0.0:514`

Without address the input listens on every address of the machine. address="127.0.0.1" means the machine itself only, and other machines cannot reach it:

text

$ sudo ss -tlnp | grep ':514'
LISTEN 0  25  127.0.0.1:514  0.0.0.0:*  users:(("rsyslogd",pid=1829,…))

One command shows what the inputs listen on: sudo ss -lntup | grep 514, with tcp and udp lines. Open the input port in the firewall for the senders' addresses. A UDP line sent to a port with no input is lost without an error (rsyslog).

Sorting by host: a template

Without rules of their own, received lines follow the same rules as local ones and land in the shared /var/log/syslog, mixed with the machine's own. A path template sorts them into per-machine directories, and a separate rule set (ruleset) keeps other machines' lines apart from local ones:

text

# /etc/rsyslog.d/10-collector.conf
module(load="imtcp")
template(name="perhost" type="string"
         string="/var/log/remote/%HOSTNAME%/syslog.log")
ruleset(name="remote") {
  action(type="omfile" dynaFile="perhost")
}
input(type="imtcp" port="514" ruleset="remote")

Property	What it inserts
`%HOSTNAME%`	the machine name from the line itself: `node2`
`%FROMHOST-IP%`	the address the connection came from
`%PROGRAMNAME%`	the program: `sshd`, `myapp`

rsyslog creates the directories itself, with mode 0700 and owner syslog: student cannot read them without sudo. Other permissions come from dirCreateMode and fileCreateMode in the action: dirCreateMode="0755" fileCreateMode="0644".

Forwarding: @, @@ and omfwd

The sender gets a file in /etc/rsyslog.d/ with a rule whose action is another machine. The short form:

text

*.* @@node1:514      # TCP
*.* @node1:514       # UDP

The full form is the omfwd action, where everything is spelled out:

text

*.* action(type="omfwd" target="node1" port="514" protocol="tcp")

The rule lives in its own file in /etc/rsyslog.d/ and takes effect after restart: only new lines go out, and old ones from /var/log are not forwarded again. protocol defaults to udp. The selector works as in any rule: *.warn @@node1:514 forwards only warning and above. The sender's protocol and port must match the collector's input: lines sent over UDP to a collector that loads only imtcp are lost without a single error (rsyslog).

Apply the config

bash

sudo rsyslogd -N1
sudo systemctl restart rsyslog

rsyslogd -N1 parses the whole configuration without touching the running service; without errors it prints End of config validation run. Bye.. The config of a second rsyslog instance, with its own file, is checked with rsyslogd -N1 -f <file>. systemctl reload rsyslog does not work on the training machine, only restart does (rsyslog).

Collector: receive and sort

On the collector machine, put the file from rsyslog in place, then:

bash

sudo rsyslogd -N1 && sudo systemctl restart rsyslog
sudo ss -tlnp | grep ':514'

A LISTEN line on 0.0.0.0:514 means the input listens on every address. Open port 514 in the firewall for the senders' addresses.

Sender: everything to the collector

bash

echo '*.* @@node1:514' | sudo tee /etc/rsyslog.d/60-forward.conf
sudo rsyslogd -N1 && sudo systemctl restart rsyslog

Check the chain with logger

On the sender:

bash

logger -t lab-check "check $(date +%s)"

On the collector a couple of seconds later:

text

$ sudo grep lab-check /var/log/remote/*/syslog.log
/var/log/remote/node2/syslog.log:2026-10-22T05:18:46+00:00 node2
  lab-check: check 1792646326

The output is one line, wrapped here. logger sends the line through rsyslog, so it tests the whole chain: the selector, the forwarding, the network, the input and the template. rsyslog helps tell the input from the forwarding.

logger straight to the collector

bash

logger -n node1 -P 514 -T -t lab-check 'over TCP'
logger -n node1 -P 514 -d -t lab-check 'over UDP'

-n is the collector address, -P the port, -T TCP, -d UDP. The sender's rsyslog takes no part here: if the line arrives, the collector's input is fine and the forwarding is at fault. Always give the port: with -T and no -P, logger looks up the syslog-conn service port and fails:

text

logger: failed to resolve name 127.0.0.1 port syslog-conn: Servname
  not supported for ai_socktype

Over UDP logger exits with code 0 even when the port is closed: UDP waits for no answer (rsyslog).

Lines do not reach the collector

Walk the chain from the sender to the collector:

Sender: is there a forwarding rule, and does the line pass its selector: grep -r '@\|omfwd' /etc/rsyslog.d/.
Sender: forwarding errors, journalctl -u rsyslog -n 20 (rsyslog).
Network: is the port open, nc -zv node1 514 for TCP (nc).
Collector: does the input listen on the right address and protocol, sudo ss -tlnp and sudo ss -ulnp.
Collector: does the firewall let port 514 in from the sender.
Collector: where the template writes, sudo ls -R /var/log/remote.

cannot connect … Connection refused

text

$ journalctl -u rsyslog -o cat | tail -2
cannot connect to 127.0.0.1:10514: Connection refused
  [v8.2312.0 try https://www.rsyslog.com/e/2027 ]
action 'action-10-builtin:omfwd' suspended (module 'builtin:omfwd'),
  next retry is Thu Oct 22 05:19:22 2026, retry nbr 0. …

The sender knocks over TCP on a port nobody listens on: the wrong port, no input loaded, or an input on 127.0.0.1 only. rsyslog suspends the forwarding and tries again about every 30 seconds; when the collector answers, the journal says resumed. Lines that arrive meanwhile are lost by default: on the training machine two lines sent while the port was silent never reached the collector. To make them wait, give the action a queue: queue.type="LinkedList" and action.resumeRetryCount="-1".

UDP loses lines without errors

The sender uses UDP (@ or protocol="udp") and the collector accepts only TCP (imtcp). Neither side shows an error: UDP does not wait for an answer. Compare the sender's protocol (@ or @@) with the input module and with ss -ulnp / ss -tlnp on the collector.

The input listens on 127.0.0.1 only

ss -tlnp shows 127.0.0.1:514: the input has address="127.0.0.1". Lines from the machine itself arrive, lines from others do not. Remove address or set the network interface address.

Only warnings get through

The collector has warning lines, while logger without -p does not arrive: the forwarding selector is *.warn or stricter. logger writes user.notice. For all lines, use *.*.

Job type reload is not applicable

text

$ sudo systemctl reload rsyslog
Failed to reload rsyslog.service: Job type reload is not applicable
  for unit rsyslog.service.

The service has no reload command. sudo systemctl restart rsyslog applies a new config.

rsyslogd -N1 blames another file

text

rsyslogd: error during parsing file /etc/rsyslog.d/20-ufw.conf, on or
  before line 2: invalid character ':' in object definition …
rsyslogd: could not interpret master config file '/etc/rsyslog.conf'.

The rsyslog.d files are read one after another as a single text. An unclosed input(… in 11-broken.conf swallowed the start of the next file by name, and rsyslog saw the error in 20-ufw.conf. Look at the file before the one named.

No access to /var/log/remote

ls: cannot open directory '/var/log/remote': Permission denied: the input directories are created with mode 0700 for syslog. Read them with sudo or set dirCreateMode (rsyslog).

Logs grow and fill the disk

Rotation is the job of logrotate; rsyslog does not clean files itself. Received lines in /var/log/remote are not in the packaged logrotate rules, so they need a file of their own in /etc/logrotate.d/.

Lines are duplicated

The same line matches several rules and goes into several files. Review the selectors: none excludes a facility, and a separate ruleset keeps received lines apart from local ones.

journald or rsyslog: where to look

Searching by service, field or time is journalctl (-u, --since, -o json). Text files, the usual grep and forwarding to a collector are rsyslog.

Input 514 open to everyone

Whoever reaches the port writes anything into the collector's logs: forged lines in someone else's name, or a flood that fills the disk. Open 514 in the firewall only to the addresses of your own machines.

Logs in one place

Situation

Task

Constraints

Signing in is needed to start the machine

Collector: receive and sort

Sender: everything to the collector

Check the chain with logger

logger straight to the collector

Lines do not reach the collector

cannot connect … Connection refused

UDP loses lines without errors

The input listens on 127.0.0.1 only

Only warnings get through

Job type reload is not applicable

rsyslogd -N1 blames another file

No access to /var/log/remote

Logs grow and fill the disk

Lines are duplicated

journald or rsyslog: where to look

Input 514 open to everyone