In an hour you will not remember what you did at 13:26 and which of it helped.
A timeline is the list of incident events by time: what was seen, what was done, what changed. It is started in the first minute and gets a line for every action, and after the outage the review is built from it. The time comes first in a line:
Server clocks usually run in UTC — that way the time in the journal, in nginx's access.log and in the timeline matches. Your own action is handy to record at once with a single command: date -u +%T prints the current UTC time, and >> appends the line to the end of the file without wiping the earlier ones.
The time of what happened before you comes from the logs: the second of FIRING from lab-sitecheck is when the outage began for users. A line holds facts — what is seen and what is done, without guesses. “nginx error.log: connect() failed to 127.0.0.1:8000” is a timeline line; “looks like nginx is to blame” is a guess and stays in the on-call's head.
The machine clock runs in UTC, and you work on Moscow time (UTC+3). Which line is worth adding to the timeline?
-
16:27 is Moscow time, and the journal and access.log write UTC: the line lands three hours after FIRING.
-
This is a guess and a judgement of a person. The timeline gets what is seen and what is done: which file and what changed in it.
-
Without a time the line cannot be placed on the scale next to FIRING and the rollback. The time goes first:
date -u +%T.
The time by the machine clock (UTC) first and a fact without judgement — such a line can be matched with the journal and later moved into the review. Moscow time would have to be converted, and a guess checked all over again.
Key points
- A timeline is a “time event” line for every observation and action, from the first minute of the incident.
-
The time follows the machine clock in UTC:
date -u +%Tfor your own actions, the journal for what came before you. - A line holds facts without guesses: the review is later built from them.