Automation: Git and Ansible / Stage 7 of 8 · step 5 of 6

The role does not apply

~30 min

Not running Starting usually ~20 s

Situation

Every shop in the chain now needs the status page, and Timur did not want to copy the playbook. He moved the tasks of the «Юг» (South) shop playbook into the status_board role and left each shop a short site.yml: its own title and port in vars: and a roles: line. This machine is the «Север» (North) shop, the project is in ~/status-board. ansible-playbook site.yml fails here.

Task

Fix the roles/status_board role so that site.yml brings this machine (localhost from inventory.ini) to the needed state by itself:

ansible-playbook site.yml runs without errors.
The /etc/status/status.ini settings are built from the status.ini.j2 role template with the title and port from site.yml: owner root, group status, mode 0640.
The lab-status service is running, enabled at boot and answers: curl http://127.0.0.1:8096/ gives the title «Магазин «Север»: статус витрины».
A run that has changed the settings restarts the service, and a run without changes leaves it alone.
Running the playbook again changes nothing: the result is changed=0.

"Check" runs the playbook itself. First it removes what the role does: the lab-status service with its unit, /etc/status, /var/lib/status and the status user. Then it runs ansible-playbook site.yml from ~/status-board as you twice, edits the title line in /etc/status/status.ini by hand and runs it a third time — the service must restart. The last run is --check --diff on a copy of the project where site.yml has a different title and port: the settings must get them. So the role must do everything — whatever you set up by hand, the check removes.

Do not change the values in site.yml, the application or the role's lab-status.service unit file.

Run: cd ~/status-board && ansible-playbook site.yml. What the role holds: find roles -type f. There is no internet on the machine.

Lab goals

0 of 5

ansible-playbook site.yml runs without errors on a machine without the status page — not checked yet
/etc/status/status.ini is built from the role template with the values from site.yml (root:status, 0640) — not checked yet
The lab-status service is running, enabled at boot and answers http://127.0.0.1:8096/ with the title from site.yml — not checked yet
The service restarts when a run has changed the settings, and only then — not checked yet
A repeated run changes nothing — changed=0 — not checked yet

Hints

0 of 3 open · they open one at a time

Hint 1

Show the solution after 3 failed checks

Automation · picked for this lab

Ansible: inventory, one-off commands, roles and vault

Cheat sheet

$_ — insert into the terminal

ansible -i hosts.ini all -m ping reach all machines
ansible -i hosts.ini all --list-hosts who a command will touch
ansible-inventory -i hosts.ini --graph groups and machines
ansible -i hosts.ini web -m command -a 'uptime' run a command
ansible -i hosts.ini web -m copy -a 'src=a.conf dest=/tmp/a.conf'
ansible -i hosts.ini web -b -m service -a 'name=nginx state=started'
ansible -i hosts.ini all -m ping -o one-line answers
ansible-playbook -i hosts.ini site.yml a playbook
ansible-galaxy init roles/web a role skeleton
ansible-vault encrypt group_vars/web/vault.yml encrypt
ansible-playbook -i hosts.ini site.yml --ask-vault-pass

Inventory

ini

[web]
node2

[db]
db1 ansible_host=10.0.0.21 ansible_user=deploy

[local]
localhost ansible_connection=local

[web:vars]
ansible_user=student

What	Where	Means
machine name	`node2`	how the machine is named in the output; Ansible also connects by it if there is no `ansible_host`
`ansible_host`	the machine's line	the address or name for SSH
`ansible_user`	the machine's line or `[group:vars]`	who to log in as; without it — your own name
`ansible_connection=local`	the machine's line	work on this machine without SSH
`[web]`	a header	a group; `all` is every machine in the inventory

text

student@node1:~/inv$ ansible-inventory -i hosts.ini --graph
@all:
  |--@ungrouped:
  |--@web:
  |  |--node2
  |--@local:
  |  |--localhost
student@node1:~/inv$ ansible -i hosts.ini all --list-hosts
  hosts (2):
    node2
    localhost

Without -i Ansible looks for /etc/ansible/hosts; the lab machine has none, and the command sees only localhost. Inventory errors in a playbook — ansible-playbook.

One-off commands

ansible <machines> -m <module> -a '<arguments>' runs one module on all machines of a group. The answer comes per machine:

text

student@node1:~/inv$ ansible -i hosts.ini web -m copy \
> -a 'content="hello\n" dest=/tmp/hello.txt'
node2 | CHANGED => {
    "changed": true,
    "dest": "/tmp/hello.txt",
    ...
}

The same command again gives node2 | SUCCESS and "changed": false: the file is already like that. The command module (also the default without -m) does not know what the command does and always reports CHANGED:

text

student@node1:~/inv$ ansible -i hosts.ini web -m command -a 'id -un'
node2 | CHANGED | rc=0 >>
student

In the output	Means
`SUCCESS`	the module ran, there was nothing to change
`CHANGED`	the module changed the machine (`command` always does)
`FAILED!`	the module ran with an error — the cause is in `msg`
`UNREACHABLE!`	cannot connect to the machine — ansible

-o prints each machine's answer on one line. service returns hundreds of unit fields — -o or a playbook is easier to read.

become: through sudo

Modules run as the user Ansible logged in as. Writing to /etc without the rights:

text

node2 | FAILED! => {
    ...
    "msg": "Destination /etc not writable"
}

-b (--become) runs the module through sudo:

text

student@node1:~/inv$ ansible -i hosts.ini web -b -m command -a 'id -un'
node2 | CHANGED | rc=0 >>
root

On the lab machines student's sudo needs no password. Where sudo asks for one, add -K (--ask-become-pass). In a playbook it is the same — become: true.

Playbook

yaml

# site.yml
- hosts: web
  become: true
  vars:
    pkg: nginx
  tasks:
    - name: Install {{ pkg }}
      apt: { name: "{{ pkg }}", state: present }
    - name: Config from a template
      template: { src: nginx.conf.j2, dest: /etc/nginx/nginx.conf }
      notify: reload nginx        # the handler — only on change
  handlers:
    - name: reload nginx
      service: { name: nginx, state: reloaded }

bash

ansible-playbook -i inventory.ini site.yml
ansible-playbook -i inventory.ini site.yml --check --diff
ansible-playbook -i inventory.ini site.yml --limit web1 --tags config

The lab machine has no network to the outside: packages from the network will not install. How to read a run, changed=0 on the second run, handlers, --check --diff and playbook errors — ansible-playbook.

Role

When the same change is needed in three playbooks, it is time to move the tasks into a role: a directory of tasks, handlers, templates and variables that a play pulls in with one line. ansible-galaxy init creates the skeleton (no network needed):

text

$ ansible-galaxy init roles/web
- Role roles/web was created successfully

Inside the role	What is there
`tasks/main.yml`	tasks — run when a play calls the role
`handlers/main.yml`	handlers for `notify:`
`templates/`	`.j2` templates — `template:` looks here first
`files/`	files as they are — for `copy:`
`defaults/main.yml`	default values, weaker than everything else
`vars/main.yml`	role variables, stronger than the inventory and play `vars:`
`meta/main.yml`	author, description, dependencies on other roles

ansible-galaxy also adds README.md and tests/. The directory names and main.yml are fixed: the role will not read handler/ instead of handlers/ or tasks/tasks.yml, and there is no error at that point — there are just no tasks, and notify: answers The requested handler … was not found (ansible-playbook).

A play calls a role from its roles: list:

yaml

- hosts: web
  gather_facts: false
  roles:
    - web

Roles run after pre_tasks: and before the play's tasks:. In the output the role's tasks carry its name: TASK [web : Show vars]. One role serves several playbooks; ansible-playbook a.yml b.yml runs them in turn and prints a PLAY RECAP after each with a running total. Ansible looks for the role in roles/ next to the playbook, then in ~/.ansible/roles, /usr/share/ansible/roles, /etc/ansible/roles; if it is not found — ansible.

template: src: site.conf.j2 in a role task looks for the file in roles/web/templates/ first, then in the role itself, next to the tasks and in the playbook directory. Ansible prints the full list of paths in the error — ansible.

Variable precedence

Ansible reads group_vars/<group>.yml and host_vars/<machine>.yml next to the inventory or the playbook by itself; there is nothing to include. When a variable is set in several places, the value comes from the strongest one. Checked on the lab machine: every next place overrides the previous one.

From weakest to strongest	Example
the role's `defaults/main.yml`	`web_port: 8080`
`group_vars/all.yml`	for every machine
`group_vars/<group>.yml` or the `group_vars/<group>/` directory	for the `web` group
a machine variable: an inventory line or `host_vars/<machine>.yml`	`node1 web_port=8083`
the play's `vars:`	next to `hosts:`
the role's `vars/main.yml`	`web_port: 7000`
`vars:` on the role in the `roles:` list	`- role: web` with `vars:` under it
`-e` (`--extra-vars`)	`-e web_port=9000`

What the role's user will change (a port, a site name) goes into defaults/. A value from vars/main.yml cannot be overridden by the inventory or the play's vars:: it holds what must not change. -e acts for one run and is not stored anywhere. What a machine got from the inventory and group_vars/ (without roles and plays) is shown by ansible-inventory -i inventory.ini --host node1. If a value from the inventory "has no effect", check the role's vars/ and the play's vars: first — ansible. The full order (it has more levels: facts, set_fact, task variables) is in the Variable precedence section of the Ansible docs.

For one group use either the file group_vars/web.yml or the directory group_vars/web/ with files inside, not both: with the directory present the web.yml file next to it is not read, and its variables vanish without a warning (checked on ansible-core 2.16.3).

Secrets: ansible-vault

A plain-text password in group_vars/ is visible to anyone who sees the directory or the repository. ansible-vault encrypts a whole file (AES256), and Ansible decrypts it in memory during a run:

text

$ ansible-vault encrypt group_vars/web/vault.yml \
    --vault-password-file ~/.vault_pass
$ head -1 group_vars/web/vault.yml
$ANSIBLE_VAULT;1.1;AES256
$ ls -l group_vars/web/
-rw-rw-r-- 1 student student  15 Oct 27 14:54 vars.yml
-rw------- 1 student student 419 Oct 27 14:54 vault.yml

The encrypted file got mode 0600. Ansible takes the password in one of these ways:

--ask-vault-pass — asks Vault password: in the terminal;
--vault-password-file ~/.vault_pass — a file with the password on one line, mode 600, outside the project directory;
the environment variable ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass — then no option is needed on the command line;
the line vault_password_file = ~/.vault_pass in the [defaults] section of the project's ansible.cfg — both ansible-playbook and ansible-vault itself read it. If the password file is missing, the run stops before the first play: ansible.

Command	What it does
`ansible-vault encrypt <file>`	encrypt a file
`ansible-vault view <file>`	show it decrypted, the file is unchanged
`ansible-vault edit <file>`	open it in the `$EDITOR` editor and encrypt again
`ansible-vault decrypt <file>`	decrypt it on disk
`ansible-vault encrypt_string '<value>' --name <name>`	one encrypted variable to paste into YAML

A handy layout: group_vars/web/vars.yml says openly db_password: "{{ vault_db_password }}", and the encrypted group_vars/web/vault.yml holds vault_db_password: …. A search by name finds the variable, and the value stays encrypted.

text

$ ansible-vault encrypt_string --vault-password-file ~/.vault_pass \
    'hunter2' --name db_password
db_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          36663965323635323632646430666666316563326133…

Ansible shows the decrypted value to any output that prints it — ansible.

UNREACHABLE!

Ansible could not connect over SSH; the module never ran, the exit code is 4. The cause is in msg, which is the error text of ssh itself (lines wrapped):

text

node3 | UNREACHABLE! => {
    "msg": "Failed to connect to the host via ssh: ssh: Could not
      resolve hostname node3: Temporary failure in name resolution",
node2 | UNREACHABLE! => {
    "msg": "Failed to connect to the host via ssh: admin@node2:
      Permission denied (publickey).",
web1 | UNREACHABLE! => {
    "msg": "Failed to connect to the host via ssh: Host key
      verification failed.",

Text	Cause	What to do
Could not resolve hostname	the name is neither in DNS nor in `/etc/hosts`	fix the name or set `ansible_host`
Permission denied (publickey)	this user does not have your key	the right `ansible_user`, or the key in their `authorized_keys`
Host key verification failed	the host key is not in `known_hosts`, or it changed	`ssh` to that address by hand and accept the key; a changed one — `ssh-keygen -R <address>`

On the lab machines host key checking is off for the names node1 and node2 (/etc/ssh/ssh_config.d/99-lab.conf), but not for an IP address in ansible_host: the same node2 by address gives Host key verification failed. Check by hand as the same user: ssh -o BatchMode=yes node2 true; details — ansible … -vvv. Keys — ssh-keys.

Destination /etc not writable

The module runs without root rights: use -b or become: true — ansible.

sudo: a password is required

You need --become and working sudo. With a password — --ask-become-pass (-K) or NOPASSWD, see sudo.

A task is "changed" every time

You're using command/shell instead of an idempotent module. Use modules (apt, copy, template, lineinfile) — they only change things when needed. For command, use creates=/changed_when:, more in ansible-playbook.

the role '…' was not found

text

ERROR! the role 'nope' was not found in /home/student/r/roles:
  /home/student/.ansible/roles:/usr/share/ansible/roles:
  /etc/ansible/roles:/home/student/r

A typo in the role name, or the role is not in roles/ next to the playbook. Paths count from the playbook directory, not from where the command runs.

Could not find or access '….j2'

text

fatal: [node1]: FAILED! => {"changed": false, "msg": "Could not find
  or access 'nope.conf.j2'\nSearched in:\n\t/home/student/r/roles/web/
  templates/nope.conf.j2\n\t…

After Searched in: the msg lists every place Ansible looked for the template, in order:

text

roles/web/templates/nope.conf.j2
roles/web/nope.conf.j2
roles/web/tasks/templates/nope.conf.j2
roles/web/tasks/nope.conf.j2
templates/nope.conf.j2
nope.conf.j2

The file is in none of them: a typo in src:, the template sits in a directory with another name (template/) or in another role. Put it in roles/<role>/templates/.

Attempting to decrypt but no vault secrets found

ERROR! Attempting to decrypt but no vault secrets found right after the PLAY [web] line — the run has an encrypted file but no vault password. Add --ask-vault-pass or --vault-password-file — ansible.

The vault password file … was not found

ERROR! The vault password file … was not found before the first play — the file from --vault-password-file or from vault_password_file in ansible.cfg is not at that path. Check the path: ~ in ansible.cfg is the home directory of whoever runs the command.

Decryption failed

text

ERROR! Decryption failed (no vault secrets were found that could
  decrypt) on /home/student/r/group_vars/web/vault.yml

The password is not the one the file was encrypted with. Check the password file: one line, no extra spaces.

input is already encrypted

ERROR! input is already encrypted — the file is already encrypted. To change it use ansible-vault edit, to look at it — ansible-vault view.

A value from the inventory has no effect

A stronger place overrides the variable: the play's vars:, the role's vars/main.yml or -e — ansible. To find every place: grep -rn web_port . in the project directory. One more cause: the file group_vars/web.yml sits next to the directory group_vars/web/, and then the file is not read.

A secret in the output

Only the file is encrypted. The decrypted value is printed by ansible -m debug -a var=vault_db_password and by --diff:

text

$ ansible-playbook -i inventory.ini site.yml --check --diff …
--- before: /tmp/db.conf
+++ after: /tmp/db.conf
-password=hunter2
+password=hunter3

Put no_log: true on tasks that handle a secret: the output keeps only changed: [node1], with no value and no diff.

The vault password next to the project

A password file inside the project directory goes into the repository along with the encrypted files, and the encryption hides nothing. Keep it outside the project (~/.vault_pass, mode 600).

1 · Foundations

2 · The system

3 · Admin craft

4 · Server in production

5 · Operations

The role does not apply

Situation

Task

Lab goals

Hints

Ansible: inventory, one-off commands, roles and vault

Cheat sheet

Inventory

One-off commands

become: through sudo

Playbook

Role

Variable precedence

Secrets: ansible-vault

UNREACHABLE!

Destination /etc not writable

sudo: a password is required

A task is "changed" every time

the role '…' was not found

Could not find or access '….j2'

Attempting to decrypt but no vault secrets found

The vault password file … was not found

Decryption failed

input is already encrypted

A value from the inventory has no effect

A secret in the output

The vault password next to the project

Leave the lab?

The role does not apply

Situation

Task

Signing in is needed to start the machine

Inventory

One-off commands

become: through sudo

Playbook

Role

Variable precedence

Secrets: ansible-vault

UNREACHABLE!

Destination /etc not writable

sudo: a password is required

A task is "changed" every time

the role '…' was not found

Could not find or access '….j2'

Attempting to decrypt but no vault secrets found

The vault password file … was not found

Decryption failed

input is already encrypted

A value from the inventory has no effect

A secret in the output

The vault password next to the project