A load balancer and health checks

This step shows who moves the address and how fast.

The address 192.0.2.100 is not tied to a machine: whichever machine is the master right now holds it. Such an address is called floating, or a VIP (virtual IP). Clients only know this address, so a change of machine needs no changes on their side.

The machines agree on who holds the address over the VRRP protocol, run by the keepalived service. The master (MASTER) sends an announcement every advert_int seconds: the address is mine, my priority is 150. The standby (BACKUP) listens. When announcements stop, after three misses — about 3 seconds — the standby becomes the master and brings the address up on itself. On systemctl stop, keepalived sends a last announcement with priority 0, and the standby takes the address at once — that is what you saw.

vrrp_instance site {
    state MASTER            # BACKUP on node2
    interface lab-n1
    virtual_router_id 51    # group number: the same on both machines
    priority 150            # 100 on node2: the higher one is the master
    advert_int 1
    virtual_ipaddress {
        192.0.2.100/24
    }
}

When the machine with the higher priority comes back, it takes the address back: on this setup about 3.5 s after systemctl start keepalived.

Check yourself 1 question

keepalived on node1 was killed with kill -9: the service is failed, and the address 192.0.2.100 stayed on lab-n1. Where will this address be 5 seconds later?

Without the master's announcements the standby becomes the master and brings the address up — it does not remove it.
kill -9 lets the process do nothing on its way out: the address stays on lab-n1, and node2 brings it up a second time.
node2 hears no announcements and brings the address up on itself after ~3 s. The address on node1 stays too: nobody is left to remove it.

node2 stopped getting announcements and became the master after ~3 s, while nobody is left to remove the address from node1 — it is on both machines. This split brain shows in ip -br addr on both machines.

Key points

The master machine holds the floating address; clients go to it and need not know the machines' own addresses.
keepalived picks the master over VRRP: an announcement every advert_int, ~3 s of silence and the standby brings the address up.
virtual_router_id is the same on the machines of one group, and priority decides which one is the master.

A load balancer and health checks

Signing in is needed to start the machine