Web stack / Stage 4 of 9 · step 7 of 8

The portal fails over HTTPS

~25 min

Not running Starting usually ~20 s

Situation

Since this morning users have been reporting that their browsers won't open the company portal at https://portal.lab.test, also known as https://www.portal.lab.test: "Your connection is not private". The portal is served by nginx on this machine.

Certificates for the company's sites are issued by an internal training certificate authority (CA). Its certificate and key are in /srv/lab-ca, and every company machine trusts the CA certificate, this one included. The portal's key and certificate are in /etc/ssl/portal, along with portal.cnf, the portal's certificate request settings for openssl req -config.

Don't change the portal's key: the company's mobile app knows the public part of this key and will not connect to the portal with a different one.

Task

curl without -k opens https://portal.lab.test/ and https://www.portal.lab.test/ and gets the portal page.
nginx presents a new certificate issued by the training CA.
The new certificate is issued for the portal's existing key.
The new certificate stays valid for at least 30 more days.

The machine has no internet access; everything you need is already installed.

Lab goals

0 of 4

curl without -k opens https://portal.lab.test/ and https://www.portal.lab.test/ and gets the portal page — not checked yet
nginx presents a new certificate issued by the training CA — not checked yet
The new certificate is issued for the portal's existing key — not checked yet
The new certificate stays valid for at least 30 more days — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Security · picked for this lab

TLS and certificates — openssl, the chain, verification

Cheat sheet

$_ — insert into the terminal

openssl x509 -in site.pem -noout -subject -issuer -dates
openssl x509 -in site.pem -noout -ext subjectAltName names
openssl x509 -in site.pem -noout -checkend 2592000 30 days?
openssl s_client -connect site.lab.test:443 -servername site.lab.test \
-showcerts </dev/null 2>/dev/null | grep -E '^ *[0-9]+ s:|^ +i:'
openssl verify -CAfile ca.pem -untrusted int.pem site.pem
curl --cacert ca.pem https://site.lab.test/ your own CA
sudo cp ca.pem /usr/local/share/ca-certificates/lab.crt
sudo update-ca-certificates trust the CA

What the server sends

bash

openssl s_client -connect secure.lab.test:443 \
  -servername secure.lab.test -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -enddate

The first certificate in the output is the site certificate. For the whole chain, look at the s: (subject) and i: (issuer) lines:

text

 0 s:CN = secure.lab.test
   i:CN = Lab Intermediate CA
 1 s:CN = Lab Intermediate CA
   i:CN = Lab Root CA

Only line 0 — the server does not send the intermediate. -servername passes the site name at the start of TLS (SNI): nginx with several sites on 443 picks the certificate by it. Without a name, for example when connecting to an IP, you get the default server's certificate.

Check a chain locally

bash

openssl verify -CAfile ca.pem site.pem
openssl verify -CAfile ca.pem -untrusted int.pem site.pem

The first command, without the intermediate, says error 20 at 0 depth lookup: unable to get local issuer certificate. The second says site.pem: OK: the chain goes from the site through the intermediate to the root.

Build a full chain file

bash

cat site.pem int.pem > fullchain.pem
grep -c BEGIN fullchain.pem      # 2

The order is the site certificate first, the intermediate after it. The root does not go in the file: the client has its own. Using the file in nginx — nginx.

Do the key and certificate match?

bash

openssl x509 -in site.pem -noout -pubkey | sha256sum
openssl pkey -in site.key -pubout | sha256sum

Equal sums mean a pair. If they differ, nginx rejects the config with key values mismatch.

Issue a certificate with the old key

bash

openssl req -new -key site.key -out site.csr -subj '/CN=shop.lab.test' \
  -addext 'subjectAltName=DNS:shop.lab.test,DNS:www.shop.lab.test'
openssl x509 -req -in site.csr -CA ca.pem -CAkey ca.key \
  -CAcreateserial -days 90 -copy_extensions copy -out site.pem
openssl x509 -in site.pem -noout -ext subjectAltName -enddate

-copy_extensions copy carries SAN from the request into the certificate. Without it the certificate has no names: -ext subjectAltName prints No extensions in certificate, and curl rejects it by name. Put the new certificate in the file the server uses, then nginx -t and reload.

How long until it expires

bash

openssl x509 -in site.pem -noout -enddate
openssl x509 -in site.pem -noout -checkend 2592000; echo $?

-checkend N asks whether the certificate expires within N seconds (2592000 is 30 days): Certificate will expire and code 1, or Certificate will not expire and code 0. Handy in checks and monitoring.

Trust your own CA system-wide

bash

sudo cp ca.pem /usr/local/share/ca-certificates/lab-root.crt
sudo update-ca-certificates

The file must be named *.crt. The command prints 1 added, 0 removed; done., and after that curl and other OpenSSL programs trust this CA's certificates without --cacert.

Let's Encrypt (certbot)

bash

sudo certbot --nginx -d example.com -d www.example.com
sudo certbot renew --dry-run
systemctl list-timers certbot.timer

A public CA for sites on the internet: certificates last 90 days, and a timer renews them in advance. It needs a DNS name pointing at this server and port 80 reachable from outside. The lab VMs have no outside network, so they use a lab CA.

certificate has expired

curl: (60) SSL certificate problem: certificate has expired; openssl s_client says Verify return code: 10. Check notAfter of the certificate the server sends (recipe above) and the machine's clock — timedatectl. Issue a new one, replace the file and reload: nginx keeps the old certificate in memory.

unable to get local issuer certificate

The client did not reach a trusted root. Two causes:

the client does not know the CA — --cacert or update-ca-certificates;
the server does not send the intermediate — s_client shows a single 0 s: line and Verify return code: 21 (unable to verify the first certificate). You need a full chain file.

no alternative certificate subject name matches

curl: (60) SSL: no alternative certificate subject name matches target host name 'other.lab.test' — the name from the URL is not in SAN. curl answers the same when you connect by IP. Look at the names: openssl x509 -noout -ext subjectAltName; if SAN is empty, the certificate was signed without -copy_extensions.

self-signed certificate

curl: (60) SSL certificate problem: self-signed certificate; s_client says code 18. The server sends a certificate it signed itself, and the trust store does not have it. For your own service, issue a certificate from your own CA and put the root in the trust store.

wrong version number

curl: (35) … SSL routines::wrong version number — https:// went to a port that speaks plain HTTP. Check the port and the server's listen … ssl.

The server sends the old certificate

The file is replaced but s_client shows the old notAfter: nginx has not re-read its config. sudo nginx -t && sudo systemctl reload nginx, then check a second later — nginx.

curl -k in scripts and checks

-k (--insecure) silently accepts a spoofed server, an expired certificate and a wrong name. A "site answers over HTTPS" check with -k passes even when users see an error. Use --cacert with the right CA instead.

The private key

The key lets anyone pose as the site. Mode 600, owner root, and it never goes into a repository, an email or a chat. If it leaks, issue a certificate on a new key.

The portal fails over HTTPS

Situation

Task

Signing in is needed to start the machine

What the server sends

Check a chain locally

Build a full chain file

Do the key and certificate match?

Issue a certificate with the old key

How long until it expires

Trust your own CA system-wide

Let's Encrypt (certbot)

certificate has expired

unable to get local issuer certificate

no alternative certificate subject name matches

self-signed certificate

wrong version number

The server sends the old certificate

curl -k in scripts and checks

The private key