Cluster maintenance
Node replacement
In case one of the nodes suffers a difficult to repair situation (lost motherboard for example, or lost disk with no RAID), it might become necessary to replace the server with a blank one.
From the cluster point of view, we will need to remove the old node and add the new one, for both corosync/pacemaker and ceph.
The
ansible/playbooks/replace_machine_remove_machine_from_cluster.yamlplaybook can remove a node from the cluster. For this, themachine_to_removeshould be set to the hostname to remove.
The below command should be launched in the ansible project.cqfd run ansible-playbook -i /path/to/inventory.yaml -e machine_to_remove=HOSTNAME playbooks/replace_machine_remove_machine_from_cluster.yamlA new host should be installed with the ISO installer and the same hostname, ip address, etc... than the old node.
Make the "cluster network" connections between hosts.
Restart the
seapath_setup_main.yamlplaybook to configure the new host in the cluster..
Node disable
If a maintenance is needed on a specific node of the cluster, the deployed resources needs to be migrated to avoid any downtime.
To do so, the concerned node can be temporary disable from the cluster using:
crm node standby <MACHINE_HOSTNAME>Once the node is disabled, all the deployed resources are migrated to another cluster node, if applicable.
Node update
If a node of the cluster needs to be updated, you can use the following Ansible playbooks:
On SEAPATH Yocto:
cqfd run ansible-playbook -i <YOUR_INVENTORY> playbooks/seapath_update_yocto_cluster.yamlNote: You will need to generate first a SWU update file, and provide it in the swu_images directory of your Ansible working directory.
On SEAPATH Debian:
cqfd run ansible-playbook -i <YOUR_INVENTORY> playbooks/seapath_update_debian.yamlMake sure to update one node at a time, and to wait for it to be back in the cluster before updating another one.