# HVM Stretched Cluster Deployment

For high-availability and failover, HVM Clusters can be installed across two or more sites. This section goes over installation considerations and process for deploying such a cluster.

Pre-requisites and Information

- At least six HVM Hosts and one witness. For example, a minimal stretched cluster could have three hosts per site and a witness located in a third site
- Stretched clusters have two layers of quorum: Intra-site quorum and Inter-site quorum\
- The witness will be a distributed worker. See [distributed worker documentation](https://support.hpe.com/hpesc/public/api/document/sd00008014en_us/render?page=GUID-A306B71B-A1BD-449B-AD1C-6EE4E8462ECF.html) for more details on its installation. A host or a VM will be needed to run the worker
- Network connectivity is required between the HVM Hosts and the witness. The HVM Hosts and witness must be able to communicate with Morpheus. The HVM Hosts and witnesses must also be able to communicate with each other
- Install or upgrade HPE Morpheus Software version 9.0.0 or higher

Deploy the Distributed Worker (Witness)

Install the Distributed Worker. This can be done according to the [distributed worker documentation](https://support.hpe.com/hpesc/public/api/document/sd00008014en_us/render?page=GUID-A306B71B-A1BD-449B-AD1C-6EE4E8462ECF.html) and involves installing a package on your designated host or VM, creating a Distributed Worker entry in HPE Morpheus Software UI, and inputting a few values into a config file. The Distributed Worker documentation includes information about other functionality of the Distributed Worker (such as proxying capabilities and use with supported private cloud types) but for the purposes of installing a stretched cluster, you simply need to install it successfully. When you can verify successful connection by checking the logs (`morpheus-worker-ctl tail worker` and `tail -f /var/log/morpheus-worker/worker/current`), you have finished the installation.

Deploy the HVM Cluster

Add a new HVM Cluster. For general information on deploying the HVM Cluster, see [general HVM documentation](https://support.hpe.com/hpesc/public/api/document/sd00008014en_us/render?page=GUID-E5D8BA05-A312-4091-93B4-A1AA9C15E1E9.html) or the [deployment guide](https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007332en_us&docLocale=en_US). At the point where the new HVM Cluster is added in HPE Morpheus Software UI, enter all hosts across all sites. In the case of the example below, six hosts (across two sites) are entered:

![Adding an HVM Cluster to Morpheus UI as part of deploying a stretched cluster](https://support.hpe.com/hpesc/public/api/document/sd00008014en_us/GUID-DA59F0F0-BB92-4A31-B9BD-1F1D8307C2BB-high.png)

IMPORTANT

Do not choose to add the witness now. Although you could choose the witness here, this example will configure the witness in the next section, after the HVM Cluster is added. This allows for time to confirm the cluster is functioning as expected and also shows how a witness would be added to an existing cluster.

Configure the Witness

With the HVM Cluster deployed, a witness must be added before site quorum can be used. Add a witness by editing an existing HVM Cluster (Infrastructure > Clusters, then choose a selected cluster and click Edit). Within the WITNESS WORKER configuration, select the Distributed Worker that was created in a previous section of this guide. If there are no WITNESS WORKERS available to select, return to the section above on deploying the Distributed Worker and ensure the worker was installed according to the documentation. Then, click Save Changes to update the HVM Cluster. Distributed Worker logs can be checked to confirm success (remotely access the worker and use `morpheus-worker-ctl tail worker` and `tail -f /var/log/morpheus-worker/worker/current`).

Add an HPE Clustered Datastore (Shared LUN)

Adding an HPE Clustered Datastore (Shared LUN), which were formerly referred to as GFS2 datastores, will activate the quorum service (as quorum is not needed until a shared LUN is present). If needed, see HVM Cluster storage documentation for complete details on selling up a HPE Clustered Datastore (aka GFS2 datastore). HVM Host and Worker logs can again be used to confirm success (Host: `morpheus-node-ctl tail` and `tail -f /var/log/morpheus-node/morphd/current.log`; Worker: `morpheus-worker-ctl tail worker` and `tail -f /var/log/morpheus-worker/worker/current`).

Configure Site Groups

With quorum established between the HVM Hosts and the witness, the hosts and witness can be assigned to sites within HPE Morpheus Software UI. Sites are used to determine the layers or quorum that was mentioned at the beginning of this section.

1. Navigate to Infrastructure > Clusters, choose the Resources tab, then choose the Host/VM Groups subtab
2. Click Add and then create a Site Group for your first site. You'll enter a **NAME** for the site and choose the **TYPE** of "Site Group" and enter the HVM Host servers which make up the first site.
3. Click Save Changes
4. When the first site is added, the witness is automatically assigned to a site named "siteWitness" and does not need to be manually created. Do not name any of the HVM Host sites "siteWitness" as this name is refered for witness sites
5. Click Add once again and follow the same process for adding the second site. This time, choose a second name for the second site and enter the appropriate HVM Host servers which exist in the second site
6. Click Save Changes
7. If there are any additional sites, add them now
8. Once again, we can monitor quorum status messages via the HVM Hosts and witnesses as needed (Host: `morpheus-node-ctl tail` and `tail -f /var/log/morpheus-node/morphd/current.log`; Worker: `morpheus-worker-ctl tail worker` and `tail -f /var/log/morpheus-worker/worker/current`)

Helpful Commands and Information

1. Additional information can be provided by DLM and Corosync using the following commands:
   - `dlm_tool status`
   - `dlm_tool ls`
   - `corosync-quorumtool -s`

   NOTE

   The witness will not appear in these quorum tools, as HPE Morpheus Software utilizes the witness for quorum but not the underlying services. Corosync is primarily used for a list of hosts, not fencing or quorum
2. Information from host/host files:

- Obtain quorum information via JSON directly from a host IP: `https://192.168.130.44:7443/quorum`
- Obtain similar quorum information from files on the host: `/opt/morpheus/.quorum-nodes`
- Obtain mount information from the files on the host (assuming a GFS2 mount has been added): `/opt/morpheus/.mounts`
