Skip to content

Provisioning

The hardware in the cluster is fully provisioned and managed using Ansible. Specifically the provisioning and updating procedures of all cluster nodes are automated using two separate playbooks.

Inventory

Inside this file all hosts for the cluster and their connection details are defined.

# cluster
[masters]
coruscant.iske.cloud ansible_user=pi
kashyyyk.iske.cloud ansible_user=pi
alderaan.iske.cloud ansible_user=pi

[workers]
dathomir.iske.cloud ansible_user=pi
mustafar.iske.cloud ansible_user=pi
jakku.iske.cloud ansible_user=pi

[cluster:children]
masters
workers

# backup
[backup]
ryloth.iske.cloud ansible_user=pascaliske

Playbooks

provision.yml

TL;DR — ansible/playbooks/provision.yml 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
# masters
- name: Provision Masters
  hosts: masters
  roles:
    - role: common
      tags:
        - masters
        - common
    - role: log2ram
      tags:
        - masters
        - log2ram
    - role: journal
      tags:
        - masters
        - journal
    - role: logrotate
      tags:
        - masters
        - logrotate
    - role: tailscale
      tags:
        - masters
        - tailscale
    - role: keepalived
      tags:
        - masters
        - keepalived
    - role: k3s
      tags:
        - masters
        - k3s

# workers
- name: Provision Workers
  hosts: workers
  roles:
    - role: common
      tags:
        - workers
        - common
    - role: log2ram
      tags:
        - workers
        - log2ram
    - role: journal
      tags:
        - workers
        - journal
    - role: logrotate
      tags:
        - workers
        - logrotate
    - role: tailscale
      tags:
        - workers
        - tailscale
    - role: k3s
      tags:
        - workers
        - k3s

# backup
- name: Provision Backup
  hosts: backup
  become: true
  roles:
    - role: minio
      tags:
        - minio

For a initial and complete provisioning of all nodes the following command can be used:

$ task cluster:provision

To only run specific parts of the playbook the --tags flag can be appended to the command:

$ task cluster:provision -- --tags <tag1>[,<tag2>]

The following tags are available for usage with --tags:

  • masters
  • workers
  • common
  • journal
  • log2ram
  • logrotate
  • tailscale
  • k3s
  • minio

update.yml

TL;DR — ansible/playbooks/update.yml 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# masters
- name: Update Masters
  hosts: masters
  become: true
  tasks:
    - name: Update apt packages
      apt:
        upgrade: safe
        update_cache: true
        autoremove: true
      tags:
        - masters

# workers
- name: Update Workers
  hosts: workers
  become: true
  tasks:
    - name: Update apt packages
      apt:
        upgrade: safe
        update_cache: true
        autoremove: true
      tags:
        - workers

The update playbook allows me to simply update / patch all nodes:

$ task cluster:update

To only run specific parts of the playbook the --tags flag can be appended to the command:

$ task cluster:update -- --tags <tag1>[,<tag2>]

The following tags are available for usage with --tags:

  • masters
  • workers

cleanup.yml

TL;DR — ansible/playbooks/cleanup.yml 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# masters
- name: Clean-up Masters
  hosts: masters
  roles:
    - role: logs
      tags:
        - masters
        - logs

# workers
- name: Clean-up Workers
  hosts: workers
  roles:
    - role: logs
      tags:
        - workers
        - logs

Sometimes, logrotate and log2ram can't keep up with the log files. For this rare cases I have an cleanup playbook which allows me to cleanup the /var/log folders of all cluster nodes to prevent an overflow of the available disk space:

$ task cluster:cleanup

To only run specific parts of the playbook the --tags flag can be appended to the command:

$ task cluster:cleanup -- --tags <tag1>[,<tag2>]

The following tags are available for usage with --tags:

  • masters
  • workers
  • logs

Limit

All playbooks can be executed on a limited set of hosts using the --limit flag:

$ task cluster:<provision|update|cleanup> -- --limit <host1>[,<host2>]

Any hosts from the inventory can be used with this flag.

Vault

Some values needed for the above playbooks are stored as an encrypted secrets file using Ansible Vault.

To encrypt or decrypt I use the following commands:

$ task vault:encrypt

$ task vault:decrypt