Linux Labs
Where to start First steps track · 15 stages
Linux Labs
EN

HTTP — requests, responses, status codes and headers

11 min HTTP/1.1 · curl 8.5 · nginx 1.24

HTTP is the web protocol: a client sends a method, path and headers; the server replies with a code, headers, body.

Cheat sheet

8 commands Printable version
  • curl -i http://portal.lab.test/ code, headers and body
  • curl -I http://portal.lab.test/ headers only (HEAD)
  • curl -v -o /dev/null http://portal.lab.test/ the whole exchange
  • curl -s -o /dev/null -w '%{http_code}\n' URL just the code
  • curl -H 'Host: docs.lab.test' http://127.0.0.1/ a site by name
  • curl -X POST -d 'a=1' http://portal.lab.test/f POST with a body
  • curl -L http://portal.lab.test/old follow 301 and 302
  • sudo tail /var/log/nginx/access.log which codes nginx returned

Overview

HTTP is the protocol a browser, curl or an application uses to talk to a web server. The client opens a TCP connection — port 80, or 443 for HTTPS — and sends a request. The server sends back a response. Both are text: a first line, headers, an empty line, a body.

curl -v shows the whole exchange: lines with > are the request, < the response, * what curl itself did (address, connection, TLS).

text 
*   Trying 127.0.0.1:80...
* Connected to portal.lab.test (127.0.0.1) port 80
> GET / HTTP/1.1
> Host: portal.lab.test
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.24.0 (Ubuntu)
< Content-Type: text/html
< Content-Length: 12

Request: method, path, Host

The first line of a request is the method, the path and the protocol version: GET / HTTP/1.1.

Method What the client asks for
GET return the resource
HEAD the same without a body (curl -I)
POST accept data: a form, JSON, a file
PUT, DELETE replace or delete a resource (APIs)

nginx serves files from disk for GET and HEAD. A POST or DELETE to such a file gets 405 Not Allowed.

HTTP/1.1 requires exactly one request header — Host, the site name from the URL. The server uses it to pick a site when several share one address and port (virtual hosts). nginx rejects a request without Host with 400 Bad Request.

curl takes Host from the URL, but you can set it by hand: curl -H 'Host: docs.lab.test' http://127.0.0.1/ connects to 127.0.0.1 and asks for the site docs.lab.test. If the server has no such name, the default server answers.

Response: code, headers, body

The first line of a response is the version, the status code and a phrase: HTTP/1.1 200 OK. The first digit is the class:

Class Meaning Common codes
1xx interim response 101
2xx success 200, 204
3xx resource is elsewhere 301, 302, 304
4xx server refused the request 400, 403, 404, 405, 413
5xx failure on the server side 500, 502, 503, 504

Whoever answers sets the code. Behind a reverse proxy that is either nginx or the application behind it. 502 and 504 come from the proxy itself: it could not reach the application or gave up waiting. The logs tell who really answered: nginx access.log records the code of every request, the application log only the requests that reached it.

The response headers you need most: Content-Type (what the body is), Content-Length (body size in bytes), Location (where to go on 3xx) and Server (which server answered).

Redirects: 301 and 302

A 3xx response with a Location header sends the client to another address — a redirect. 301 Moved Permanently is for good: the browser remembers it and goes straight to the new address next time, search engines move the page there. 302 Found is temporary: the old address stays the main one.

text 
$ curl -sI 'http://secure.lab.test/x?y=1'
HTTP/1.1 301 Moved Permanently
Location: https://secure.lab.test/x?y=1

Location is a full address including the path and the parameters after ?. If it loses them, the user lands on the home page instead of the page they wanted. curl -L follows redirects by itself.

HTTPS

HTTPS is the same HTTP inside an encrypted TLS connection. Before the first byte of HTTP the client checks the server certificate; after that methods, codes and headers are the same. The default port is 443. Details are in openssl-tls.

Recipes

4

Just the status code

bash 
curl -s -o /dev/null -w '%{http_code}\n' http://portal.lab.test/

-o /dev/null throws the body away, -w prints the code. Code 000 means there was no HTTP response at all: the connection never opened or broke. The same request with -sS instead of -s shows why.

A site by name when DNS does not know it

bash 
curl -H 'Host: docs.lab.test' http://127.0.0.1/
curl --resolve docs.lab.test:80:127.0.0.1 http://docs.lab.test/

The first command changes only the Host header. It does not work for HTTPS: the name is also needed to check the certificate. --resolve supplies the address for the name, and curl behaves as if DNS had returned it.

A request by hand

bash 
printf '%s\r\n' 'GET / HTTP/1.1' 'Host: portal.lab.test' \
  'Connection: close' '' | nc 127.0.0.1 80

This shows HTTP is plain text: every line ends with \r\n, headers end with an empty line (the last argument ''). Connection: close asks the server to close the connection after the response; otherwise nc keeps waiting.

Who answered: nginx or the application

bash 
curl -sI http://app.lab.test/ | head -1     # the code
sudo tail -n 3 /var/log/nginx/access.log    # the code as nginx saw it
sudo tail -n 3 /var/log/nginx/error.log     # the reason

An access.log line holds the client address, time, request line, code and response size: "POST / HTTP/1.1" 405 166. If the request is in access.log but not in the application log, nginx produced the response — look for the reason in error.log.

Troubleshooting

9

400 Bad Request

The request breaks HTTP rules. Common causes: no Host header (a hand-made request) or plain HTTP sent to an HTTPS port — then the body says The plain HTTP request was sent to HTTPS port. Check the scheme in the URL: https:// for port 443.

403 Forbidden

The server understood the request and refused. nginx has two common causes, both visible in error.log. (13: Permission denied) — the www-data user cannot read the file or enter a directory: namei -l shows permissions along the whole path. directory index of "…" is forbidden — a directory was requested and it has no index file.

404 Not Found

There is no file at the path the server built. nginx writes that path to error.log: open() "/var/www/portal/nope.html" failed (2: No such file or directory). Compare it with where the file actually is.

405 Not Allowed

The method does not fit the resource: POST or DELETE to a static file. Check the method in curl -v and make sure the request goes to the application, not to a file on disk.

413 Request Entity Too Large

The request body is larger than allowed. nginx's default limit is 1 MB; error.log says client intended to send too large body. The client_max_body_size directive raises it.

502 Bad Gateway, 504 Gateway Timeout

A proxy response: it could not reach the application (502) or gave up waiting for an answer (504). How to read the error.log lines is in nginx.

301 when you expect 200

The server sends you to the address in the Location header. nginx does this for a directory without a trailing / (/empty → /empty/) and when moving from HTTP to HTTPS. curl -L follows redirects, curl -I shows where each one leads.

curl: (35) … wrong version number

The reverse of 400 … sent to HTTPS port: https:// went to a port where the server speaks plain HTTP (https://host:80/). Fix the scheme or the port.

curl: (7) Failed to connect … Couldn't connect to server

There is no HTTP response: nothing listens on that address and port, or a firewall drops the connection. This is below HTTP — sudo ss -tlnp on the server shows who listens on the port. curl: (6) Could not resolve host fails even earlier: the name did not turn into an address.

Danger zone

1

A password over http:// is visible on the network

Without TLS the request travels as plain text: a form password, cookies and the Authorization header can be read by anyone on the packet's path — tcpdump shows them in full. Logins and anything carrying secrets go over https:// only.

Terms

5
HTTP status
A three-digit code in an HTTP response describing the server handling result.
HTTP method
The first word of an HTTP request — what the client asks to do with the resource: GET returns it, HEAD returns only its headers, POST accepts data.
HTTP header
A «Name: value» line after the first line of a request or response — Host, Content-Type, Location and others; headers end with an empty line.
Host header
The request header carrying the site name from the URL. The server uses it to choose among several sites on one address and port; HTTP/1.1 requires it. Also appears in: nginx, Unix sockets
redirect (301)
A 3xx response with a Location header — the server sends the client to another address; 301 means permanently, 302 temporarily. curl -L follows them. Also appears in: nginx, Output redirection and pipelines, wget, at and 5 more

See also

  • curl — HTTP requests from the console: -v, -I, -w, -H.
  • nginx — the web server: choosing a site by Host, proxying, 502 and 504.
  • openssl-tls — HTTPS: certificate, chain, verification errors.
  • unix-sockets — HTTP without a network: an application behind nginx on a socket.
  • nc — a raw TCP connection to send a request by hand.
  • ss — who listens on port 80 or 443.
  • dns-resolv — how a site name becomes an address.

Found an inaccuracy? Report an error in the article

Practice on this topic

Lessons on this topic

Tools on this topic

Topics
Loading…