# Full HA Upgrade

Full HA configurations represent multiple app nodes with external (non-system) MySQL, RabbitMQ and Opensearch Clusters or Services.

## HPE Morpheus Enterprise Packages

HPE Morpheus Enterprise Release packages can be downloaded from My HPE Software Center. Contact your account manager with any questions regarding access.

## Overview

When upgrading to v9.0.0 only services local to the Morpheus App node(s) will be upgraded. For fully distributed configurations (Full HA) where MySQL, RabbitMQ and Opensearch are external, the upgrade process will not upgrade these services.

Refer to [breaking changes](https://support.hpe.com/hpesc/public/api/document/sd00008014en_us/render?page=GUID-C131BD09-5DB9-4C31-8106-AA5A7A735F45.html#GUID-0289251C-DBFE-418E-B794-3B9FCFB184CF) for externalized MySQL, Elasticsearch and/or RabbitMQ version requirements.

## Upgrade Instructions

### Full HA Debian / Ubuntu Upgrade

The following covers upgrading the HPE Morpheus Enterprise App nodes in Full HA Architecture configurations to v9.0.0.

IMPORTANT

The following is only for Full HA Architecture configurations, where MySQL, Elasticsearch and RabbitMQ services are external to the App nodes. The following steps assume 3 app nodes, adjust accordingly.

#### HPE Morpheus Enterprise Packages

HPE Morpheus Enterprise Release packages can be downloaded from My HPE Software Center. Contact your account manager with any questions regarding access.

#### Full HA Non-Rolling Upgrade Process (deb)

##### About this task

WARNING

Depending on your current version, a non-rolling upgrade may be required to upgrade to version v9.0.0. Refer to the [upgrade compatibility table](https://support.hpe.com/hpesc/public/api/document/sd00008014en_us/render?page=GUID-C131BD09-5DB9-4C31-8106-AA5A7A735F45.html#GUID-6F0A533B-BEE6-4407-AB67-681A4CC40303) to confirm whether a rolling upgrade is possible. If a rolling upgrade is not possible, refer to the appropriate set of non-rolling upgrade instructions to proceed.

IMPORTANT

It is important to stop the morpheus-ui service on all app nodes prior to upgrade. Failure to do so will result in a flood of log errors due to previous message serialization conflict. The messages will eventually expire and the logs will clear.

WARNING

HPE Morpheus Enterprise v9.0.0 contains new node and VM node packages that require 3.5GB of storage. It is safe to run `sudo rm -Rf /var/opt/morpheus/package-repos/*` after v9.0.0 package installation and before reconfigure to clean old node and vm node packages from the package-repo when room is needed.

##### Procedure

1. Starting with App Node 3, on All App Nodes, stop the morpheus-ui services via `morpheus-ctl stop morpheus-ui`. If you receive a timeout, run `morpheus-ctl graceful-kill morpheus-ui`. 

   ```
   ```
   [root@app-server-3 ~]# morpheus-ctl stop morpheus-ui
   ```
   ```

   ```
   ```
   [root@app-server-2 ~]# morpheus-ctl stop morpheus-ui
   ```
   ```

   ```
   ```
   [root@app-server-1 ~]# morpheus-ctl stop morpheus-ui
   ```
   ```
2. Download the upgrade package from My HPE Software Center and transfer it to each application node. Then, upgrade the deb package on App Node 1, then run a Reconfigure on App Node 1 

   ```
   ```
   [root@app-server-1 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1_amd64.deb  
   [root@app-server-1 ~]# sudo morpheus-ctl stop morpheus-ui  
   [root@app-server-1 ~]# sudo morpheus-ctl reconfigure  
   [root@app-server-1 ~]# sudo morpheus-ctl start morpheus-ui
   ```
   ```

   After the reconfigure has succeeded, tail the UI service to watch the UI startup logs with `morpheus-ctl tail morpheus-ui`.
3. Once App Node 1 upgrade has completed and the UI is available, upgrade the deb package on App Node 2, then run a Reconfigure on App Node 2. 

   ```
   ```
   [root@app-server-2 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1_amd64.deb  
   [root@app-server-1 ~]# sudo morpheus-ctl stop morpheus-ui  
   [root@app-server-1 ~]# sudo morpheus-ctl reconfigure  
   [root@app-server-1 ~]# sudo morpheus-ctl start morpheus-ui  
   After the reconfigure has succeeded, tail the ui service to watch ui startup logs with ``morpheus-ctl tail morpheus-ui``.
   ```
   ```
4. Once App Node 2 upgrade has completed and the UI is available, upgrade the deb package on App Node 3, and run a Reconfigure on App Node 3 

   ```
   ```
   [root@app-server-3 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1_amd64.deb  
   [root@app-server-3 ~]# sudo morpheus-ctl reconfigure
   ```
   ```
5. The upgrade is complete and the HPE Morpheus Enterprise-ui services should be running on the three allocated nodes.

##### Results

IMPORTANT

If reconfigure after a rpm package upgrade stalls or hangs on starting a local service it is possible the `morpheus-runsvdir` service did not start or a process it was managing was manually shutdown or killed. To resolve, run `systemctl stop morpheus-runsvdir` then `systemctl start morpheus-runsvdir`, then run reconfigure again, `morpheus-ctl reconfigure`.

#### Full HA Rolling Upgrade Process (deb)

##### About this task

WARNING

Depending on your current version, a non-rolling upgrade may be required to upgrade to version v9.0.0. Refer to the [upgrade compatibility table](https://support.hpe.com/hpesc/public/api/document/sd00008014en_us/render?page=GUID-C131BD09-5DB9-4C31-8106-AA5A7A735F45.html#GUID-6F0A533B-BEE6-4407-AB67-681A4CC40303) to confirm whether a rolling upgrade is possible. If a rolling upgrade is not possible, refer to the appropriate set of non-rolling upgrade instructions to proceed.

WARNING

HPE Morpheus Enterprise v9.0.0 contains new node and VM node packages that require 3.5GB of storage. It is safe to run `sudo rm -Rf /var/opt/morpheus/package-repos/*` after v9.0.0 package installation and before reconfigure to clean old node and vm node packages from the package-repo when room is needed.

##### Procedure

1. Starting with App Node 3, on All App Nodes, stop the morpheus-ui services via `morpheus-ctl stop morpheus-ui`. If you receive a timeout, run `morpheus-ctl graceful-kill morpheus-ui`.
2. Download the upgrade package from My HPE Software Center and transfer to each application node. Then, upgrade the deb package on App Node 1, then run a Reconfigure on App Node 1 

   ```
   ```
   [root@app-server-1 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1_amd64.deb  
   [root@app-server-1 ~]# sudo morpheus-ctl stop morpheus-ui  
   [root@app-server-1 ~]# sudo morpheus-ctl reconfigure  
   [root@app-server-1 ~]# sudo morpheus-ctl start morpheus-ui
   ```
   ```

   After the reconfigure has succeeded, tail the UI service to watch UI startup logs with `morpheus-ctl tail morpheus-ui`. Once morpheus-ui is started, proceed to the next node.
3. Once App Node 1 upgrade has completed and the UI is available, upgrade the deb package on App Node 2, then run a Reconfigure on App Node 2. 

   ```
   ```
   [root@app-server-2 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1_amd64.deb  
   [root@app-server-2 ~]# sudo morpheus-ctl stop morpheus-ui  
   [root@app-server-2 ~]# sudo morpheus-ctl reconfigure  
   [root@app-server-2 ~]# sudo morpheus-ctl start morpheus-ui
   ```
   ```

   After the reconfigure has succeeded, tail the ui service to watch UI startup logs with `morpheus-ctl tail morpheus-ui`. Once morpheus-ui is started, proceed to the next node.
4. Once App Node 2 upgrade has completed and the ui is available, upgrade the deb package on App Node 3, and run a Reconfigure on App Node 3 

   ```
   ```
   [root@app-server-3 ~]# sudo dpkg -i morpheus-appliance_x.x.x-1_amd64.deb  
   [root@app-server-3 ~]# sudo morpheus-ctl stop morpheus-ui  
   [root@app-server-3 ~]# sudo morpheus-ctl reconfigure  
   [root@app-server-3 ~]# sudo morpheus-ctl start morpheus-ui
   ```
   ```

   After the reconfigure has succeeded, tail the UI service to watch UI startup logs with `morpheus-ctl tail morpheus-ui`. Once morpheus-ui is started, proceed to the next node.
5. The upgrade is complete and the HPE Morpheus Enterprise UI services should be running on the three allocated nodes.

##### Results

IMPORTANT

If reconfigure after a rpm package upgrade stalls or hangs on starting a local service it is possible the `morpheus-runsvdir` service did not start or a process it was managing was manually shutdown or killed. To resolve, run `systemctl stop morpheus-runsvdir` then `systemctl start morpheus-runsvdir`, then run reconfigure again, `morpheus-ctl reconfigure`.

### Full HA RPM Upgrade

The following covers upgrading the HPE Morpheus Enterprise App nodes in Full HA Architecture configurations to v9.0.0.

IMPORTANT

The following is only for Full HA Architecture configurations, where MySQL, Opensearch and RabbitMQ services are external to the App nodes.

#### Full HA Non-Rolling Upgrade Process (rpm)

##### About this task

WARNING

Depending on your current version, a non-rolling upgrade may be required to upgrade to version v9.0.0. Refer to the [upgrade compatibility table](https://support.hpe.com/hpesc/public/api/document/sd00008014en_us/render?page=GUID-C131BD09-5DB9-4C31-8106-AA5A7A735F45.html#GUID-6F0A533B-BEE6-4407-AB67-681A4CC40303) to confirm whether a rolling upgrade is possible. If a rolling upgrade is not possible, refer to the appropriate set of non-rolling upgrade instructions to proceed.

IMPORTANT

It is important to stop the morpheus-ui service on all app nodes prior to upgrade. Failure to do so will result in a flood of log errors due to previous message serialization conflict. The messages will eventually expire and the logs will clear.

WARNING

HPE Morpheus Enterprise v9.0.0 contains new node and VM node packages that require 3.5GB of storage. It is safe to run `sudo rm -Rf /var/opt/morpheus/package-repos/*` after v9.0.0 package installation and before reconfigure to clean old node and vm node packages from the package-repo when room is needed.

##### Procedure

1. Starting with App Node 3, on All App Nodes, stop the morpheus-ui services via `morpheus-ctl stop morpheus-ui`. If you receive a timeout, run `morpheus-ctl graceful-kill morpheus-ui`. 

   ```
   ```
   [root@app-server-3 ~]# morpheus-ctl stop morpheus-ui
   ```
   ```

   ```
   ```
   [root@app-server-2 ~]# morpheus-ctl stop morpheus-ui
   ```
   ```

   ```
   ```
   [root@app-server-1 ~]# morpheus-ctl stop morpheus-ui
   ```
   ```
2. Download the upgrade package from My HPE Software Center and transfer it to each application node. Then, upgrade the deb package on App Node 1, then run a Reconfigure on App Node 1 

   ```
   ```
   [root@app-server-1 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm  
   [root@app-server-1 ~]# sudo morpheus-ctl reconfigure
   ```
   ```

   NOTE

   All services will automatically be stopped and started during the reconfigure process. After the reconfigure has succeeded, tail the ui service to watch ui startup logs with `morpheus-ctl tail morpheus-ui`.
3. Once App Node 1 upgrade has completed and the UI is available, upgrade the RPM package on App Node 2, then run a Reconfigure on App Node 2. 

   ```
   ```
   [root@app-server-2 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm  
   [root@app-server-2 ~]# sudo morpheus-ctl reconfigure
   ```
   ```
4. Then upgrade the RPM package on App Node 3, then run a Reconfigure on App Node 3 

   ```
   ```
   [root@app-server-3 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm  
   [root@app-server-3 ~]# sudo morpheus-ctl reconfigure
   ```
   ```
5. The upgrade is complete and the HPE Morpheus Enterprise-ui services should be running with clustered Elasticsearch and RabbitMQ services across the 3 nodes.

##### Results

IMPORTANT

If reconfigure after a rpm package upgrade stalls or hangs on starting a local service it is possible the `morpheus-runsvdir` service did not start or a process it was managing was manually shutdown or killed. To resolve, run `systemctl stop morpheus-runsvdir` then `systemctl start morpheus-runsvdir`, then run reconfigure again, `morpheus-ctl reconfigure`.

#### Full HA Rolling Upgrade Process (rpm)

##### About this task

WARNING

Depending on your current version, a non-rolling upgrade may be required to upgrade to version v9.0.0. Refer to the [upgrade compatibility table](https://support.hpe.com/hpesc/public/api/document/sd00008014en_us/render?page=GUID-C131BD09-5DB9-4C31-8106-AA5A7A735F45.html#GUID-6F0A533B-BEE6-4407-AB67-681A4CC40303) to confirm whether a rolling upgrade is possible. If a rolling upgrade is not possible, refer to the appropriate set of non-rolling upgrade instructions to proceed.

WARNING

HPE Morpheus Enterprise v9.0.0 contains new node and VM node packages that require 3.5GB of storage. It is safe to run `sudo rm -Rf /var/opt/morpheus/package-repos/*` after v9.0.0 package installation and before reconfigure to clean old node and vm node packages from the package-repo when room is needed.

##### Procedure

1. Download the upgrade package from My HPE Software Center and transfer it to each application node. Then, upgrade the deb package on App Node 1, then run a Reconfigure on App Node 1 

   ```
   ```
   [root@app-server-1 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm  
   [root@app-server-1 ~]# sudo morpheus-ctl stop morpheus-ui  
   [root@app-server-1 ~]# sudo morpheus-ctl reconfigure  
   [root@app-server-1 ~]# sudo morpheus-ctl start morpheus-ui
   ```
   ```

   After the reconfigure has succeeded, tail the UI service to watch UI startup logs with `morpheus-ctl tail morpheus-ui`.
2. Once App Node 1 upgrade has completed and the UI is available, upgrade the RPM package on App Node 2, then run a Reconfigure on App Node 2. 

   ```
   ```
   [root@app-server-2 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm  
   [root@app-server-2 ~]# sudo morpheus-ctl stop morpheus-ui  
   [root@app-server-2 ~]# sudo morpheus-ctl reconfigure  
   [root@app-server-2 ~]# sudo morpheus-ctl start morpheus-ui
   ```
   ```

   After the reconfigure has succeeded, tail the UI service to watch UI startup logs with `morpheus-ctl tail morpheus-ui`.
3. Once App Node 2 upgrade has completed and the UI is available, upgrade the RPM package on App Node 3, then run a Reconfigure on App Node 3 

   ```
   ```
   [root@app-server-3 ~]# sudo rpm -Uhv morpheus-appliance-x.x.x-x.x86_64.rpm  
   [root@app-server-3 ~]# sudo morpheus-ctl stop morpheus-ui  
   [root@app-server-3 ~]# sudo morpheus-ctl reconfigure  
   [root@app-server-3 ~]# sudo morpheus-ctl start morpheus-ui
   ```
   ```

   After the reconfigure has succeeded, tail the ui service to watch ui startup logs with `morpheus-ctl tail morpheus-ui`.
4. The upgrade is complete and the HPE Morpheus Enterprise-ui services should be running with clustered Elasticsearch and RabbitMQ services across the 3 nodes.

##### Results

IMPORTANT

If reconfigure after a rpm package upgrade stalls or hangs on starting a local service it is possible the `morpheus-runsvdir` service did not start or a process it was managing was manually shutdown or killed. To resolve, run `systemctl stop morpheus-runsvdir` then `systemctl start morpheus-runsvdir`, then run reconfigure again, `morpheus-ctl reconfigure`.
