# Kubernetes Clusters

## Requirements

- Agent installation is required for Master and Worker Nodes. Refer to [Morpheus Agent](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/render?page=GUID-91FC2F9F-A3F2-49DA-8C85-D3F634F36209.html) section for additional information.
- Access to Cloud Front, Image copy access and permissions for System and Uploaded Images used in Cluster Layouts
  :   Image(s) used in Cluster Layouts must either exist in destination cloud/resource or be able to be copied to destination by Morpheus, typically applicable for non-public clouds. For the initial provision, Morpheus System Images are streamed from Cloud Front through Morpheus to target destination. Subsequent provisions clone the local Image.
- System Kubernetes Layouts require Master and Worker nodes to access to the following over 443 during K8s install and configuration:

  - Morpheus Appliance url (443)
  - <https://packages.cloud.google.com>
  - <https://storage.googleapis.com>
  - <https://docs.projectcalico.org>
  - <https://openebs.github.io>
  - <https://cloud.weave.works>
- Morpheus Role permission Infrastructure: Clusters  >  Full required for Viewing, Creating, Editing and Deleting Clusters.
- Morpheus Role permission Infrastructure: Clusters  >  Read required for Viewing Cluster list and detail pages.

## Creating Kubernetes Clusters

### About this task

Provisions a new Kubernetes Cluster in selected target Cloud using selected Layout.

NOTE

When deploying a highly-available Kubernetes cluster, it’s important to note that HPE Morpheus Enterprise does not currently auto-deploy a load balancer. Additionally, when HPE Morpheus Enterprise runs the `kubeadm init` command in the background during cluster provisioning, it also sets the `--control-plane-endpoint` flag to the first control plane node. This is a hard-coded behavior. To accomplish a highly-available cluster, users may wish to update the configured control plane endpoint, such as to a DNS name pointing to a load balancer. We are currently investigating updates to the product that would allow the user to specify such a DNS name prior to kicking off cluster provisioning. Additionally, users can circumvent this issue by configuring and deploying their own custom Cluster Layouts.

HPE Morpheus Enterprise maintains a number of default Kubernetes Cluster Layouts which are updated frequently to offer support for current versions. AKS & GKE Kubernetes versions will dynamically update to the providers supported versions. HPE Morpheus Enterprise also supports creation of custom Kubernetes Cluster Layouts, a process which is described in detail in a later section.

To create a new Kubernetes Cluster:

### Procedure

1. Navigate to Infrastructure  >  Clusters
2. Select + ADD CLUSTER
3. Select `Kubernetes Cluster`
4. Select a Group for the Cluster
5. Select NEXT
6. Populate the following: 

   CLOUD
   :   Select target Cloud

   CLUSTER NAME
   :   Name for the Kubernetes Cluster

   RESOURCE NAME
   :   Name for Kubernetes Cluster resources

   DESCRIPTION
   :   Description of the Cluster

   VISIBILITY
   :   Public
       :   Available to all Tenants

       Private
       :   Available to Master Tenant

   LABELS
   :   Internal label(s)
7. Select NEXT
8. Populate the following: 

   NOTE

   VMware sample fields provided. Actual options depend on Target Cloud

   LAYOUT
   :   Select from available layouts. System provided layouts include Single Master and Cluster Layouts.

   PLAN
   :   Select plan for Kubernetes Master

   VOLUMES
   :   Configure volumes for Kubernetes Master

   NETWORKS
   :   Select the network for Kubernetes Master & Worker VM’s

   CUSTOM CONFIG
   :   Add custom Kubernetes annotations and config hash

   CLUSTER HOSTNAME
   :   Cluster address Hostname (cluster layouts only)

   POD CIDR
   :   POD network range in CIDR format ie 192.168.0.0/24 (cluster layouts only)

   WORKER PLAN
   :   Plan for Worker Nodes (cluster layouts only)

   NUMBER OF WORKERS
   :   Specify the number of workers to provision

   LOAD BALANCER
   :   Select an available Load Balancer (cluster layouts only) }

   User Config
   :   CREATE YOUR USER
       :   Select to create your user on provisioned hosts (requires Linux user config in HPE Morpheus Enterprise User Profile)

       USER GROUP
       :   Select User group to create users for all User Group members on provisioned hosts (requires Linux user config in HPE Morpheus Enterprise User Profile for all members of User Group)

   Advanced Options
   :   DOMAIN
       :   Specify Domain override for DNS records

       HOSTNAME
       :   Set hostname override (defaults to Instance name unless an Active Hostname Policy applies)

   Autoscaler
   :   The Cluster Autoscaler is a Kubernetes component that automatically adjusts the number of worker nodes in your cluster based on the resource requirements of your workloads. When application demand increases and existing nodes are unable to schedule new pods, the autoscaler adds additional nodes to the cluster. Conversely, when demand drops and nodes become underutilized, the autoscaler removes unnecessary nodes, helping to optimize resource usage and control costs.
   :   NOTE

       During scale down, the Cluster Autoscaler will skip nodes that have workloads which cannot be safely evicted, such as pods with persistent or local storage (e.g., Rook Ceph), or pods that are constrained by affinity or anti-affinity rules.
   :   |  |  |
       | --- | --- |
       | ENABLE AUTOSCALER | Select this option to activate the Cluster Autoscaler functionality. When enabled, the autoscaler continuously monitors the cluster and automatically scales the number of worker nodes up or down as needed. |
       | MINIMUM NODES | Specifies the minimum number of worker nodes that the autoscaler maintains in the cluster, even during periods of low workload. The cluster will not scale below this number of nodes. |
       | MAXIMUM NODES | Specifies the maximum number of worker nodes that the autoscaler can add to the cluster. The cluster will not scale above this number, even if workload demand increases. This helps to control resource usage and costs. |
9. Select NEXT
10. Select optional Workflow to execute
11. Select NEXT
12. Review and select COMPLETE 

    - The Master Node(s) will provision first.
    - Upon successful completion of VM provision, Kubernetes scripts will be executed to install and configure Kubernetes on the Masters.
      :   NOTE

          Access to the sites listed in the [Requirements](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/render?page=GUID-63B50C5A-C3AD-4543-8A89-58110DDD5064.html#GUID-FD8026F1-B4DB-4778-9588-728A05838892) section is required from Master and Worker nodes over 443
    - After Master or Masters are successfully provisioned and Kubernetes is successfully installed and configured, the Worker Nodes will provision in parallel.
    - Provision status can be viewed:
      :   - From the Status next to the Cluster in Infrastructure  >  Clusters
          - Status bar with eta and current step available on Cluster detail page, accessible by selecting the Cluster name from Infrastructure  >  Clusters
    - All process status and history is available - From the Cluster detail page History tab, accessible by selecting the Cluster name from Infrastructure  >  Clusters and the History tab - From Operations - Activity - History - Individual process output available by clicking i on target process
13. Once all Master and Worker Nodes are successfully provisioned and Kubernetes is installed and configured, the Cluster status will turn green. 

    IMPORTANT

    Cluster provisioning requires successful creation of VMs, Agent Installation, and execution of Kubernetes workflows. Consult process output from ``Infrastructure  >  Clusters - Details and morpheus-ui current logs at `Administration - Health - Morpheus Logs` for information on failed Clusters.

### Intra-Kubernetes Cluster Port Requirements

The table below includes port requirements for the machines within the cluster (not for the HPE Morpheus Enterprise appliance itself). Check that the following ports are open on Control-plane and Worker nodes:

Table 1. Control‐plane node(s)

       | Protocol | Direction | Port Range | Purpose | Used By |
| --- | --- | --- | --- | --- |
| TCP | Inbound | 6443 | Kubernetes API Server | All |
| TCP | Inbound | 6783 | Weaveworks |  |
| TCP | Inbound | 2379‐2380 | etcd server client API | kube‐apiserver, etcd |
| TCP | Inbound | 10250 | kubelet API | Self, Control plane |
| TCP | Inbound | 10251 | kube‐scheduler | Self |
| TCP | Inbound | 10252 | kube‐controller‐manager | Self |

Table 2. Worker node(s)

       | Protocol | Direction | Port Range | Purpose | Used By |
| --- | --- | --- | --- | --- |
| TCP | Inbound | 10250 | kubelet API | Self, Control plane |
| TCP | Inbound | 30000‐32767 | NodePort Services | All |

## Adding Worker Nodes

### Procedure

1. Navigate to `Infrastructure - Clusters`
2. Select `v MORE` for the target cluster
3. Select `ADD (type) Kubernetes Worker` 

   NAME
   :   Name of the Worker Node. Auto=populated with `${cluster.resourceName}-worker-${seq}`

   DESCRIPTION
   :   Description of the Worker Node, displayed in Worker tab on Cluster Detail pages, and on Worker Host Detail page

   CLOUD
   :   Target Cloud for the Worker Node.
4. Select NEXT
5. Populate the following: 

   NOTE

   VMware sample fields provided. Actual options depend on Target Cloud

   SERVICE PLAN
   :   Service Plan for the new Worker Node

   NETWORK
   :   Configure network options for the Worker node.

   HOST
   :   If Host selection is enabled, optionally specify target host for new Worker node

   FOLDER
   :   Optionally specify target folder for new Worker node
       :   Advanced Options
           :   DOMAIN
               :   Specify Domain override for DNS records

               HOSTNAME
               :   Set hostname override (defaults to Instance name unless an Active Hostname Policy applies)
6. Select NEXT
7. Select optional Workflow to execute
8. Select NEXT
9. Review and select COMPLETE

### Results

NOTE

Ensure there is a default StorageClass available when using a Morpheus Kubernetes cluster with OpenEBS so that Kubernetes specs or HELM templates that use a default StorageClass for Persistent Volume Claims can be utilised.

## Configure Autoscaler

1. Navigate to `Infrastructure - Clusters`
2. Select `v MORE` for the target cluster
3. Select `Configure Autoscaler`
4. |  |  |
   | --- | --- |
   | ENABLE AUTOSCALER | Select this option to activate the Cluster Autoscaler functionality. When enabled, the autoscaler continuously monitors the cluster and automatically scales the number of worker nodes up or down as needed. |
   | MINIMUM NODES | Specifies the minimum number of worker nodes that the autoscaler maintains in the cluster, even during periods of low workload. The cluster will not scale below this number of nodes. |
   | MAXIMUM NODES | Specifies the maximum number of worker nodes that the autoscaler can add to the cluster. The cluster will not scale above this number, even if workload demand increases. This helps to control resource usage and costs. |
5. Select `Apply`

## Kubernetes Cluster Detail Pages

The Kubernetes Cluster Detail page provides a high degree of monitoring and control over Kubernetes Clusters. This includes monitoring of all nodes in the Cluster, `kubectl` command line, account and role control, workload management, and more. The upper section of the page (which is persistent regardless of the currently-selected tab) provides high level costing and monitoring information, including a current aggregate metric for the CPU, memory and storage use.

![../../_images/clusterDetail.png](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/GUID-824D0DA3-5400-4EF0-BE66-A96F64902C89-high.png)

The upper section also includes the ACTIONS menu which includes the following functions:

- **REFRESH:** Forces a routine sync of the cluster status
- **PERMISSIONS:** View and edit the Group, Service Plan, and Tenant access permissions for the cluster
- **VIEW API TOKEN:** Displays the API token for the cluster
- **VIEW KUBE CONFIG:** Displays the cluster configuration
- **RUN WORKLOAD:** Run deployments, stateful sets, daemon sets, or jobs and target them to a specific namespace
- **UPGRADE CLUSTER:** Upgrade the cluster to a higher version of Kubernetes

  IMPORTANT

  Kubelet configuration cleanup is required to upgrade clusters. The `--pod-infra-container-image` flag is deprecated from Kubernetes version 1.35.3. Upgrading clusters to v1.35.3 or later will fail if this deprecated flag is present in the Kubelet configuration. This flag must be manually removed from all nodes prior to initiating the upgrade process. See upgrade process below:
  1. Edit the kubelet environment configuration file with `sudo vim /var/lib/kubelet/kubeadm-flags.env`
  2. Locate the `KUBELET_KUBEADM_ARG`S entry and remove the following entry: `--pod-infra-container-image=registry.k8s.io/pause:3.10`
  3. Save the file and exit the editor
  4. Reload the configuration and restart the service with `sudo systemctl daemon-reload` and `sudo systemctl restart kubelet`
  5. This process is repeated for all cluster nodes
- **ADD KUBERNETES WORKER:** Launches a wizard which allows users to configure a new worker for the cluster

Additional monitoring and control panes are located within tabs, some of which contain subtabs.

- [Summary](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/render?page=GUID-3159C5BA-04D5-4035-9842-86197618F7D2.html#GUID-8FA4C005-9D2E-49AE-B13C-4AE07DA54275__DIV_D22330E1_D22331E1_D22332E1_D22333E1_D22334E1_D22335E1)
- [Control](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/render?page=GUID-3159C5BA-04D5-4035-9842-86197618F7D2.html#GUID-8FA4C005-9D2E-49AE-B13C-4AE07DA54275__DIV_D22340E1_D22341E1_D22342E1_D22343E1_D22344E1_D22345E1)
- [Access](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/render?page=GUID-3159C5BA-04D5-4035-9842-86197618F7D2.html#GUID-8FA4C005-9D2E-49AE-B13C-4AE07DA54275__DIV_D22350E1_D22351E1_D22352E1_D22353E1_D22354E1_D22355E1)
- [Nodes](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/render?page=GUID-3159C5BA-04D5-4035-9842-86197618F7D2.html#GUID-8FA4C005-9D2E-49AE-B13C-4AE07DA54275__DIV_D22360E1_D22361E1_D22362E1_D22363E1_D22364E1_D22365E1)
- [Workloads](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/render?page=GUID-3159C5BA-04D5-4035-9842-86197618F7D2.html#GUID-8FA4C005-9D2E-49AE-B13C-4AE07DA54275__DIV_D22370E1_D22371E1_D22372E1_D22373E1_D22374E1_D22375E1)
- [Network](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/render?page=GUID-3159C5BA-04D5-4035-9842-86197618F7D2.html#GUID-8FA4C005-9D2E-49AE-B13C-4AE07DA54275__DIV_D22380E1_D22381E1_D22382E1_D22383E1_D22384E1_D22385E1)
- [Storage](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/render?page=GUID-3159C5BA-04D5-4035-9842-86197618F7D2.html#GUID-8FA4C005-9D2E-49AE-B13C-4AE07DA54275__DIV_D22390E1_D22391E1_D22392E1_D22393E1_D22394E1_D22395E1)
- [Containers](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/render?page=GUID-3159C5BA-04D5-4035-9842-86197618F7D2.html#GUID-8FA4C005-9D2E-49AE-B13C-4AE07DA54275__DIV_D22400E1_D22401E1_D22402E1_D22403E1_D22404E1_D22405E1)
- [Monitoring](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/render?page=GUID-3159C5BA-04D5-4035-9842-86197618F7D2.html#GUID-8FA4C005-9D2E-49AE-B13C-4AE07DA54275__DIV_D22410E1_D22411E1_D22412E1_D22413E1_D22414E1_D22415E1)
- [History](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/render?page=GUID-3159C5BA-04D5-4035-9842-86197618F7D2.html#GUID-8FA4C005-9D2E-49AE-B13C-4AE07DA54275__DIV_D22420E1_D22421E1_D22422E1_D22423E1_D22424E1_D22425E1)
- [Wiki](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/render?page=GUID-3159C5BA-04D5-4035-9842-86197618F7D2.html#GUID-8FA4C005-9D2E-49AE-B13C-4AE07DA54275__DIV_D22430E1_D22431E1_D22432E1_D22433E1_D22434E1_D22435E1)

### Summary

The summary tab contains high-level details on health and makeup of the cluster. ![../../_images/clusterSummary.png](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/GUID-42D2FAC8-D054-4AEC-8D10-59C6B15EE02A-high.png)

### Control

Contains the `kubectl` command line with ability to target commands to specific namespaces. The Control tab also contains the Packages subtab which displays the list of packages and their versions. ![../../_images/clusterControl.png](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/GUID-88388DAC-D1A7-4218-B17A-D605AB1D9095-high.png)

### Access

The Access Tab contains view and edit tools for Namespaces, accounts, roles, and role bindings. ![../../_images/clusterAccess.png](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/GUID-BC2EC859-76DA-4EC4-883C-DBA8BAD9DD52-high.png)

### Nodes

The nodes tab includes a list of master and worker nodes in the cluster, their statuses, and the current compute, memory, and storage pressure on each node. ![../../_images/clusterNodes.png](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/GUID-88E093F0-4EAB-4E6D-A023-F1C70F06812A-high.png)

### Workloads

View and edit existing Pods, Deployments, Replica Sets, Daemon Sets, Stateful Sets, and Jobs. Add new Deployments, Stateful Sets, Daemon Sets, and Jobs through the ACTIONS menu near the top of the Cluster Detail Page. ![../../_images/clusterWorkloads.png](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/GUID-5AA1E4AA-4AD5-4A7F-A274-2D7C01357494-high.png)

### Network

View, add, and edit Services, Endpoints, Ingress and Network Policies ![../../_images/clusterNetwork.png](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/GUID-A94FC46E-8856-411E-8E6A-F2B6DBEAB45F-high.png)

### Storage

View, add, and edit Storage classes, Volume claims, Volumes, Config maps, and Secrets ![../../_images/clusterStorage.png](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/GUID-2A352500-C7FF-43CD-B91B-AD299208090B-high.png)

### Containers

View a list of containers running on the cluster and restart or delete them if needed. This list can be filtered by Namespace or a specific Worker if desired. ![../../_images/clusterContainers.png](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/GUID-8D0ABC87-4D4D-4A4B-91A3-214698B516DD-high.png)

### Monitoring

View logs and events with filtering tools and search functionality available. ![../../_images/clusterMonitoring.png](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/GUID-02EEA1D6-F4BE-4060-AEBB-B6A2BD715EF6-high.png)

### History

View the Cluster lifecycle history. This includes lists of automation packages (Tasks and Workflows) run against the cluster or its nodes, the success of these scripts and the output. ![../../_images/clusterHistory.png](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/GUID-394CC8A6-5257-4B4C-BF41-E10643BEEE8A-high.png)

### Wiki

View the HPE Morpheus Enterprise Wiki entry for this Cluster. This Wiki page may also be viewed in the Wiki section (Operations > Wiki). Edit the Wiki as desired, most standard Markdown syntax will be honored allowing the use of headings, links, embedded images, and more. ![../../_images/clusterWiki.png](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/GUID-BAD395AC-5FAD-495D-B797-EEDA93C17B73-high.png)

## Adding External Kubernetes Clusters

HPE Morpheus Enterprise supports the management and consumption of Kubernetes clusters provisioned outside of HPE Morpheus Enterprise. These are referred to as External Kubernetes Clusters in HPE Morpheus Enterprise UI. This could be used, for example, to onboard and manage OpenShift clusters. In order to fully integrate the Kubernetes cluster with the HPE Morpheus Enterprise feature set, you may need to create a service account for HPE Morpheus Enterprise. Without first taking that step, some features may not work fully, such as listing all namespaces. The process for creating a service account and integrating the Cluster with HPE Morpheus Enterprise is described here.

First, create the Service Account within the Kubernetes cluster:

```
```
kubectl create serviceaccount morpheus
```
```

Next, create the Role Binding:

```
```
kubectl create clusterrolebinding morpheus-admin \  
--clusterrole=cluster-admin --serviceaccount=default:morpheus \  
--namespace=default
```
```

With those items created, we can gather the API URL and the API token which will be used to add the existing cluster to HPE Morpheus Enterprise in the next step:

```
```
kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " "
```
```

```
```
SECRET_NAME=$(kubectl get secrets | grep ^morpheus | cut -f1 -d ' ')  
kubectl describe secret $SECRET_NAME | grep -E '^token' | cut -f2 -d':' | tr -d " "
```
```

After finishing those steps, we can now create the external cluster in HPE Morpheus Enterprise. Navigate to Infrastructure > Clusters. Click + ADD CLUSTER and then select “External Kubernetes Cluster”. Set the following fields, you will have to advance through the pages of the wizard to see all fields indicated:

- **GROUP:** A previously created HPE Morpheus Enterprise Group
- **CLOUD:** A previously-integrated Cloud
- **CLUSTER NAME:** A friendly name for the onboarded cluster in HPE Morpheus Enterprise UI
- **RESOURCE NAME:** The resource name will be pre-pended to Kubernetes hosts associated with this cluster when shown in HPE Morpheus Enterprise UI
- **LAYOUT:** Set an associated Layout
- **API URL:** Enter the API URL gathered in the previous step
- **API TOKEN:** Enter the API Token gathered in the previous step
- **KUBE CONFIG:** Enter Kubeconfig YAML to authenticate the cluster

The above are the required fields, others may be optionally configured depending on the situation. Complete the wizard and HPE Morpheus Enterprise will begin the process of onboarding the existing cluster into management within HPE Morpheus Enterprise UI. Once things are finalized and statuses are green, the cluster can be monitored and consumed as any other cluster provisioned from HPE Morpheus Enterprise.

![../../_images/extKube.png](https://support.hpe.com/hpesc/public/api/document/sd00007732en_us/GUID-E72E1043-FF51-4388-9D80-79CC7660C86A-high.png)
