# HVM stretched Cluster deployment

A stretch cluster extends an HVM 1.3 cluster across two physical sites, with a witness node in a third location for tie-breaking arbitration. Stretch clusters provides site-level fault tolerance while maintaining a single cluster management domain.

## Requirements

Table 1. Stretch cluster requirements

    | Requirement | Details |
| --- | --- |
| Hosts | Minimum 6 hosts (3 per site) |
| Witness | One Distributed Worker deployed at a third site |
| Network | Network is correctly configured to allow each hosts to communicate with the witness, Morpheus Manager and all other hosts |
| Quorum | Two layers of quorum: Intra-site quorum and Inter-site quorum |
| HPE Morpheus Software version | 9.0.0 or higher |

## Deploy the distributed worker (Witness)

The witness node is a Morpheus Distributed Worker deployed at a third site, independent from both cluster sites. For details, see [Distributed worker](https://support.hpe.com/hpesc/public/docDisplay?docId=sd00008014en_us&page=GUID-A306B71B-A1BD-449B-AD1C-6EE4E8462ECF.html).

## Deploy the HVM Cluster

Add a new HVM Cluster. For general information on deploying the HVM Cluster, see [Creating an HVM Cluster](https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007332en_us&page=GUID-99397996-8315-49C1-9E2F-2EED51CE03F3.html) or [Deployment Guide](https://support.hpe.com/hpesc/public/docDisplay?docId=sd00007332en_us&page=GUID-4767CCB8-A276-45C4-BD8F-4E42865E28C7.html) for general details. At the point where the new HVM Cluster is added in HPE Morpheus Software UI, enter all hosts across all sites. In the case of the example below, six hosts (across two sites) are entered:  

Figure 1. Stretched Cluster

  
![Stretched Cluster](https://support.hpe.com/hpesc/public/api/document/sd00008014en_us/GUID-DA59F0F0-BB92-4A31-B9BD-1F1D8307C2BB-high.png)

IMPORTANT

Do not choose to add the witness now. Although you could choose the witness here, this example will configure the witness in the next section, after the HVM Cluster is added. This allows for time to confirm the cluster is functioning as expected and also shows how a witness would be added to an existing cluster.

## Configure the Witness

With the HVM Cluster deployed, a witness must be added before site quorum can be used. Add a witness by editing an existing HVM Cluster (Infrastructure > Clusters, then choose a selected cluster and click Edit). Within the WITNESS WORKER configuration, select the Distributed Worker that was created in a previous section of this guide. If there are no WITNESS WORKERS available to select, return to the section above on deploying the Distributed Worker and ensure the worker was installed according to the documentation. Then, click Save Changes to update the HVM Cluster. Distributed Worker logs can be checked to confirm success (remotely access the worker and use `morpheus-worker-ctl tail worker` and `tail -f /var/log/morpheus-worker/worker/current`).

## Creating the worker configuration

1. Navigate to Administration > Intergations > Distributed Workers.
2. Create a new worker configuration.
3. Save the API key provided.

## Installing the worker

1. Download HPE Morpheus Worker on the witness host or VM.
2. Install HPE Morpheus Worker with `dpkg`:

   ```
   ```
   dpkg -i morpheus-worker_<version>.deb
   ```
   ```
3. Modify the HPE Morpheus Worker configuration file /etc/morpheus/morpheus-worker.rb:

   ```
   ```
   worker_url = '<URL from worker config>'  
   worker['appliance_url'] = '<Morpheus appliance URL>'  
   worker['apikey'] = '<any value>'  
   worker['worker_key'] = '<API key from worker config>'
   ```
   ```
4. Reconfigure the worker:

   ```
   ```
   morpheus-worker-ctl reconfigure
   ```
   ```
5. Verify that the worker is running:

   ```
   ```
   morpheus-worker-ctl tail worker
   ```
   ```

## Adding an HPE clustered datastore (Shared LUN)

Adding an HPE Clustered Datastore (Shared LUN), which were formerly referred to as GFS2 datastores, will activate the quorum service (as quorum is not needed until a shared LUN is present). If needed, see HVM Cluster storage documentation for complete details on selling up a HPE Clustered Datastore (aka GFS2 datastore). HVM Host and Worker logs can again be used to confirm success (Host: `morpheus-node-ctl tail` and `tail -f /var/log/morpheus-node/morphd/current.log`; Worker: `morpheus-worker-ctl tail worker` and `tail -f /var/log/morpheus-worker/worker/current`).

## Configuring site groups

Site groups define which hosts belong to each physical site for arbitration decisions.

1. Navigate to Infrastructure > Clusters > [Cluster] > Resources > Host / VM Groups.
2. Click Add.
3. Create a Site Group at each location where:
   - Site Name is the name given to the site, and
   - Servers is the host at the site location.
4. Click Save Changes.

WARNING

`siteWitness` is a reserved site group name for automatic witness assignment and cannot be defined for Site Group. The witness is automatically assigned to the `siteWitness` group when the first group is created.

## Arbitration behavior

A site is considered failed when ALL non-witness nodes at that site become unreachable (60-second timeout). When a site failure occurs, the following arbitration logic is executed:

- If the surviving site is unable to reach a witness, the site self-fences and cannot confirm it is the correct winner.
- If the surviving site can reach the witness, the site can proceed with arbitration.
- The first site listed alphabetically will be selected as the winning site (deterministic tie-breaking).

The winning site adjust Corosync votes to maintain quorum with the remaining nodes and issues `fence_ack` for nodes at the failed sites. The losing site self-fences and stops DLM and Corosync.

After the failed site is restored:

- 3 consecutive healthy ping cycles (~3 minutes) must pass,
- Corosync votes are restored to normal values and
- Fenced nodes are rebooted and rejoin the cluster.

## Helpful commands and information

1. DLM and Corosync can provide additional information using the following commands:
   - `dlm_tool status`
   - `dlm_tool ls`
   - `corosync-quorumtool -s`

   NOTE

   The witness will not appear in these quorum tools, as HPE Morpheus Software utilizes the witness for quorum but not the underlying services. Corosync is primarily used for a list of hosts, not fencing, or quorum
2. Information from host/host files:

   - Obtain quorum information through JSON directly from a host IP: `https://192.168.130.44:7443/quorum`
   - Obtain similar quorum information from files on the host: `/opt/morpheus/.quorum-nodes`
   - Obtain mount information from the files on the host (assuming a GFS2 mount has been added): `/opt/morpheus/.mounts`
