Web stack / Stage 5 of 9 · step 6 of 7

The shop after a certificate renewal

~25 min

Not running Starting usually ~20 s

Situation

The company's online shop store.lab.test runs over HTTPS on this machine's nginx. This month the training certificate authority (CA) renewed the shop's certificate, and the new file replaced the old one. Since then two people have written in.

A developer: "The shop opens in my browser, but curl and our payment service get a certificate error". A marketer: "Links from an old newsletter, like http://store.lab.test/catalog?page=2, open the home page instead of the catalog".

Every company machine trusts the training CA's root certificate, this one included. The certificates the CA signs sites with are published in /srv/lab-ca. The shop's files are in /etc/ssl/store.

Task

curl without -k opens https://store.lab.test/ and gets the shop page.
The shop certificate passes verification on a client that knows only the training CA's root certificate (/srv/lab-ca/root.crt), with no changes on the client side.
http://store.lab.test/ answers with code 301 and redirects to https://store.lab.test/.
The redirect to HTTPS keeps the path and parameters: http://store.lab.test/catalog?page=2 redirects to https://store.lab.test/catalog?page=2.

The machine has no internet access; everything you need is already installed.

Lab goals

0 of 4

curl without -k opens https://store.lab.test/ and gets the shop page — not checked yet
The shop certificate passes verification on a client that knows only the training CA's root certificate — not checked yet
http://store.lab.test/ answers with code 301 and redirects to https://store.lab.test/ — not checked yet
http://store.lab.test/catalog?page=2 redirects to https://store.lab.test/catalog?page=2 — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Security · picked for this lab

TLS and certificates — openssl, the chain, verification

Cheat sheet

$_ — insert into the terminal

openssl x509 -in site.pem -noout -subject -issuer -dates
openssl x509 -in site.pem -noout -ext subjectAltName names
openssl x509 -in site.pem -noout -checkend 2592000 30 days?
openssl s_client -connect site.lab.test:443 -servername site.lab.test \
-showcerts </dev/null 2>/dev/null | grep -E '^ *[0-9]+ s:|^ +i:'
openssl verify -CAfile ca.pem -untrusted int.pem site.pem
curl --cacert ca.pem https://site.lab.test/ your own CA
sudo cp ca.pem /usr/local/share/ca-certificates/lab.crt
sudo update-ca-certificates trust the CA

What the server sends

bash

openssl s_client -connect secure.lab.test:443 \
  -servername secure.lab.test -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -enddate

The first certificate in the output is the site certificate. For the whole chain, look at the s: (subject) and i: (issuer) lines:

text

 0 s:CN = secure.lab.test
   i:CN = Lab Intermediate CA
 1 s:CN = Lab Intermediate CA
   i:CN = Lab Root CA

Only line 0 — the server does not send the intermediate. -servername passes the site name at the start of TLS (SNI): nginx with several sites on 443 picks the certificate by it. Without a name, for example when connecting to an IP, you get the default server's certificate.

Check a chain locally

bash

openssl verify -CAfile ca.pem site.pem
openssl verify -CAfile ca.pem -untrusted int.pem site.pem

The first command, without the intermediate, says error 20 at 0 depth lookup: unable to get local issuer certificate. The second says site.pem: OK: the chain goes from the site through the intermediate to the root.

Build a full chain file

bash

cat site.pem int.pem > fullchain.pem
grep -c BEGIN fullchain.pem      # 2

The order is the site certificate first, the intermediate after it. The root does not go in the file: the client has its own. Using the file in nginx — nginx.

Do the key and certificate match?

bash

openssl x509 -in site.pem -noout -pubkey | sha256sum
openssl pkey -in site.key -pubout | sha256sum

Equal sums mean a pair. If they differ, nginx rejects the config with key values mismatch.

Issue a certificate with the old key

bash

openssl req -new -key site.key -out site.csr -subj '/CN=shop.lab.test' \
  -addext 'subjectAltName=DNS:shop.lab.test,DNS:www.shop.lab.test'
openssl x509 -req -in site.csr -CA ca.pem -CAkey ca.key \
  -CAcreateserial -days 90 -copy_extensions copy -out site.pem
openssl x509 -in site.pem -noout -ext subjectAltName -enddate

-copy_extensions copy carries SAN from the request into the certificate. Without it the certificate has no names: -ext subjectAltName prints No extensions in certificate, and curl rejects it by name. Put the new certificate in the file the server uses, then nginx -t and reload.

How long until it expires

bash

openssl x509 -in site.pem -noout -enddate
openssl x509 -in site.pem -noout -checkend 2592000; echo $?

-checkend N asks whether the certificate expires within N seconds (2592000 is 30 days): Certificate will expire and code 1, or Certificate will not expire and code 0. Handy in checks and monitoring.

Trust your own CA system-wide

bash

sudo cp ca.pem /usr/local/share/ca-certificates/lab-root.crt
sudo update-ca-certificates

The file must be named *.crt. The command prints 1 added, 0 removed; done., and after that curl and other OpenSSL programs trust this CA's certificates without --cacert.

Let's Encrypt (certbot)

bash

sudo certbot --nginx -d example.com -d www.example.com
sudo certbot renew --dry-run
systemctl list-timers certbot.timer

A public CA for sites on the internet: certificates last 90 days, and a timer renews them in advance. It needs a DNS name pointing at this server and port 80 reachable from outside. The lab VMs have no outside network, so they use a lab CA.

certificate has expired

curl: (60) SSL certificate problem: certificate has expired; openssl s_client says Verify return code: 10. Check notAfter of the certificate the server sends (recipe above) and the machine's clock — timedatectl. Issue a new one, replace the file and reload: nginx keeps the old certificate in memory.

unable to get local issuer certificate

The client did not reach a trusted root. Two causes:

the client does not know the CA — --cacert or update-ca-certificates;
the server does not send the intermediate — s_client shows a single 0 s: line and Verify return code: 21 (unable to verify the first certificate). You need a full chain file.

no alternative certificate subject name matches

curl: (60) SSL: no alternative certificate subject name matches target host name 'other.lab.test' — the name from the URL is not in SAN. curl answers the same when you connect by IP. Look at the names: openssl x509 -noout -ext subjectAltName; if SAN is empty, the certificate was signed without -copy_extensions.

self-signed certificate

curl: (60) SSL certificate problem: self-signed certificate; s_client says code 18. The server sends a certificate it signed itself, and the trust store does not have it. For your own service, issue a certificate from your own CA and put the root in the trust store.

wrong version number

curl: (35) … SSL routines::wrong version number — https:// went to a port that speaks plain HTTP. Check the port and the server's listen … ssl.

The server sends the old certificate

The file is replaced but s_client shows the old notAfter: nginx has not re-read its config. sudo nginx -t && sudo systemctl reload nginx, then check a second later — nginx.

curl -k in scripts and checks

-k (--insecure) silently accepts a spoofed server, an expired certificate and a wrong name. A "site answers over HTTPS" check with -k passes even when users see an error. Use --cacert with the right CA instead.

The private key

The key lets anyone pose as the site. Mode 600, owner root, and it never goes into a repository, an email or a chat. If it leaks, issue a certificate on a new key.

The shop after a certificate renewal

Situation

Task

Signing in is needed to start the machine

What the server sends

Check a chain locally

Build a full chain file

Do the key and certificate match?

Issue a certificate with the old key

How long until it expires

Trust your own CA system-wide

Let's Encrypt (certbot)

certificate has expired

unable to get local issuer certificate

no alternative certificate subject name matches

self-signed certificate

wrong version number

The server sends the old certificate

curl -k in scripts and checks

The private key