Auditing listening services

Not running Starting usually ~20 s

Step 6 of 7 · lab · 35 min

Too much is visible from the network

The warehouse server runs a site, a database and an admin panel, and only the site and SSH should be visible from the network. Find what else is open to the outside and close it so that everything still works on the server itself, after a reboot too.

Situation

node1 runs the Sever warehouse. nginx serves the customer site on port 80. The stock is kept in the PostgreSQL database stock and edited in the admin panel, the lab-admin service. The admin panel is used only from the server itself, through an SSH tunnel, and only the admin panel needs the database. A security scanner has sent a report: more than the site and SSH is visible on node1 from the network.

node2 is a neighbouring machine on the same network. From there node1 looks the way it does to any outside address: ssh node2 'command'. sudo works without a password on both machines.

Task

From the network, node1 answers only on SSH (port 22) and the site (port 80). Everything else is closed to node2 and to other addresses on the network, over IPv4 and over IPv6.
SSH and the site http://node1/ are open to node2 and to other addresses on the network.
On node1 itself the database and the admin panel work as before: curl http://127.0.0.1:8080/ shows the stock from the stock database.
It stays so after node1 reboots.

The lab has no internet access, only the network between node1 and node2. Everything you need is installed on both machines. You reach node2 from node1 over SSH, and so does the check: don't close that path.

Lab · on your own

No commands are given in this lab: type them in the terminal yourself. Goals are checked on the machine after every command.

Goals

0 of 4

Everything on node1 except SSH (22) and the site (80) is closed to the network — to node2 and other addresses, over IPv4 and IPv6 — not done yet
SSH (22) and the site http://node1/ (80) are open to node2 and other addresses on the network — not done yet
The database and the admin panel work on node1 — curl http://127.0.0.1:8080/ shows the stock from the stock database — not done yet
It stays so after node1 reboots — the settings and rules are saved in files and the services are enabled — not done yet

Ctrl+Enter

Starts the machine node1 · Ubuntu 24.04

The machine is starting

Hints

0 of 3

Report a problem

Lab goals

0 of 4

Everything on node1 except SSH (22) and the site (80) is closed to the network — to node2 and other addresses, over IPv4 and IPv6 — not done yet
SSH (22) and the site http://node1/ (80) are open to node2 and other addresses on the network — not done yet
The database and the admin panel work on node1 — curl http://127.0.0.1:8080/ shows the stock from the stock database — not done yet
It stays so after node1 reboots — the settings and rules are saved in files and the services are enabled — not done yet

node1 Ubuntu 24.04

Lab goals

0 of 4

Everything on node1 except SSH (22) and the site (80) is closed to the network — to node2 and other addresses, over IPv4 and IPv6 — not done yet
SSH (22) and the site http://node1/ (80) are open to node2 and other addresses on the network — not done yet
The database and the admin panel work on node1 — curl http://127.0.0.1:8080/ shows the stock from the stock database — not done yet
It stays so after node1 reboots — the settings and rules are saved in files and the services are enabled — not done yet

Network · picked for this lab

ss — sockets and ports

Cheat sheet

$_ — insert into the terminal

ss -tlnp listening ports + who
ss -tnp active connections
ss -tulnp TCP+UDP listeners
ss -tlnp 'sport = :80' who's on port 80
ss -tan state close-wait a socket leak?
ss -s statistics

Mnemonic for the common combo -tlnp: tcp listening numeric process. -n matters on slow networks: without it ss tries to resolve every address into a name, and the output noticeably slows down. -p needs root to show other users' processes — without sudo you'll only see your own.

Commands

bash

ss -tlnp                          # listening TCP ports + processes
ss -ulnp                          # listening UDP ports
ss -tnp                           # established TCP connections
ss -tanp                          # all TCP: both listening and established
ss -s                             # summary: total sockets by type
ss -tnp state established         # established only
ss -tnp state time-wait          # "stuck" in TIME-WAIT
ss -tlnp '( sport = :80 or sport = :443 )'   # who's listening on 80/443
ss -tnp dst 10.0.0.5             # connections to a specific host
ss -tnp 'dport = :5432'          # connections to the postgres port
ss -tnp dst 10.0.0.0/24          # to a whole subnet
ss -tonp                          # with keepalive/retransmit timers
ss -tnp | wc -l                   # roughly how many active TCP connections

States and filters

Filter by TCP state — state <name> (or state connected/listening):

bash

ss -tan state established         # working connections
ss -tan state syn-sent            # client waiting for a reply (a stuck connect?)
ss -tan state close-wait          # app isn't closing sockets (fd leak!)
ss -tan state time-wait | wc -l   # lots of TIME-WAIT — frequent short connections

Address filters combine with and/or: src/dst (address), sport/dport (port).

bash

ss -tnp 'dst 10.0.0.5 and dport = :443'
ss -tnp 'sport >= :1024'         # ephemeral outgoing ports

"Address already in use" when starting a service

The port is already taken. Find who and stop/change the port:

bash

ss -tlnp 'sport = :8000'         # users:(("python",pid=1234,fd=3))

The service is running but unreachable from outside

Is it listening at all: ss -tlnp | grep :PORT.
Bound to 127.0.0.1 instead of 0.0.0.0/*? Then it's local only — fix the bind address in the config.
Is the port open in the firewall? (nft list ruleset / ufw status — see firewall).

The number of CLOSE-WAIT sockets keeps growing

The app isn't closing connections (a descriptor leak) — sooner or later you'll hit ulimit. ss -tanp state close-wait shows the culprit process.

Lots of TIME-WAIT

Normal with many short-lived client connections; usually not a problem. Fixed by keep-alive/connection pooling on the application side, not by randomly tuning the kernel.

Not sure which process holds a connection

ss -tnp shows users:(("nginx",pid=1234,...)). For more depth, lsof.

Auditing listening services

Situation

Task

Signing in is needed to start the machine

Commands

States and filters

"Address already in use" when starting a service

The service is running but unreachable from outside

The number of CLOSE-WAIT sockets keeps growing

Lots of TIME-WAIT

Not sure which process holds a connection