Where a line ends up is decided by rsyslog rules — they also show why nothing went to the collector.
Services and programs write to the systemd journal, and rsyslog gets a copy of every line from it and sorts the lines into files under /var/log. Besides its text, a line has a tag — the program name, lab-orders — and a priority made of two parts: the facility — kern for the kernel, auth for logins, mail, user for ordinary programs, local0–local7 for your own — and the level: err, warning, info and others.
An rsyslog rule is a line of two parts: which lines to select and what to do with them. Ubuntu's rules live in /etc/rsyslog.d/50-default.conf:
On the left is the facility.level selector: * means "any", none excludes a facility, and a level selects itself and everything more severe. On the right is the action, here a file path; a minus before the path lets rsyslog skip flushing every line to disk at once. A line goes through all the rules in turn and lands in every file whose selector matched, until a rule with stop halts it. logger sends a line of your own with a tag and a priority (in the Wiki):
A mail program sent a line: logger -p mail.err -t shop "mail disk is full". Which files will the 50-default.conf rules above write it to?
-
The
*.*;auth,authpriv.noneselector excludes only logins — mail goes to /var/log/syslog as well. -
The first matching rule does not stop the line:
mail.*and*.*select it too. Check withsudo grep shop: /var/log/mail.log /var/log/syslog. -
They do have one: the
mail.errrule writes to /var/log/mail.err, andmail.*also to /var/log/mail.log.
Three rules select the line: *.*;auth,authpriv.none, mail.* and mail.err — and rsyslog writes it to all three files. Only a rule with stop can halt a line halfway.
Key points
- A syslog line has a program tag and a "facility.level" priority; rsyslog picks the files by them.
-
A line goes through all the rules in a row and is written to every matching file until
stophalts it.