hvm-docs/docs_mcp/api_lessons.md

# HPE Morpheus VM Essentials — Lessons

Notes and gotchas about running, integrating with, and upgrading
**HPE Morpheus VM Essentials (HVM)** that aren't obvious from the
official docs alone. The official docs (User Manual, Release Notes,
Deployment Guide) describe the product as designed; this file is
what experienced operators actually learn.

> Treat this as living context. Update it when you (or the LLM
> driving this MCP) discover something non-obvious that the docs
> don't say or don't make findable. Each section is an H2 so the
> `hvm_api_lessons(topic=...)` tool can return just the relevant
> piece.

## TL;DR

- **HVM is KVM under the hood.** The "VM Essentials manager" is a
  control-plane VM that orchestrates KVM hosts ("HVM Hosts"). The
  manager runs the same web UI you'd see in HPE Morpheus Enterprise,
  but scoped to the HVM hypervisor capability set. Elevating
  unlocks the full Morpheus Enterprise feature set.
- **Versioning is independent per upgrade hop.** The manager,
  HVM Host agents, and the Plugin API are versioned separately. After
  every manager upgrade, re-check HVM Host agent versions — they
  don't auto-upgrade with the manager.
- **Compatibility matrix lives in the Release Notes.** Each version's
  Release Notes lists the compatible Plugin API + Worker versions
  for that build. Use `weekly_digest` or `get_page` on the Release
  Notes when planning an upgrade.
- **Two consumer surfaces:** the web UI on the manager (port 443),
  and the REST API (same host, `/api/` path). For automation use
  the REST API; for ad-hoc admin use the UI.

## Manager sizing — pick based on cluster count

The Deployment Guide page `VME Manager sizing` is the authoritative
table. Quick mental model:

| Size   | vCPU | RAM   | Max HVM clusters (prod-realistic) |
|--------|------|-------|------------------------------------|
| Small  | 2    | 12GB  | 1 (test/PoC only)                  |
| Medium | 4    | 16GB  | 3                                  |
| Large  | 4    | 32GB  | up to 10                           |

**Pick Medium by default for production.** Small is genuinely
underpowered for anything beyond a lab — UI latency degrades fast
when concurrent provisioning runs.

You CAN'T resize the manager in place by editing the VM. Backup, deploy
a new manager at the new size, restore. See `Backing Up and Restoring
the HPE Morpheus VM Essentials Manager` in the User Manual.

## Upgrade ordering — manager first, then host agents

HVM doesn't auto-upgrade host agents along with the manager. Every
manager upgrade hop is two steps:

1. **Upgrade the manager** (e.g., 8.1.0 → 8.1.1). Standard upgrade
   process per the Deployment Guide.
2. **Upgrade each HVM Host's agent.** Manager → HVM Host detail →
   ACTIONS menu → "Upgrade Agent". Alternative: download the agent
   upgrade script and run it on each host. Scripts are UNIQUE PER
   HOST — don't reuse one script across multiple hosts.

Some upgrade hops won't actually require a host-agent update (the
manager release didn't change the agent). Check the Release Notes for
the target version to see whether a host agent upgrade is included.

> Skipping the agent upgrade step is the #1 cause of cluster issues
> right after a manager upgrade. Symptoms: hosts appear "out of
> sync" or "agent disconnected" in the manager UI, even though the
> hosts themselves are healthy.

## Plugin API + Worker version compatibility

Each HVM version pins a specific compatible Plugin API version and
minimum HVM Host Worker version. The Release Notes for that version
state both. Examples (verify against the live Release Notes):

- HVM 8.1.2 → Plugin API 1.3.4, HVM Host Worker ≥ 5.4.8
- (Earlier versions: check the corresponding `hvm_release_notes_*`
  bundle.)

When writing a plugin or integration:

- Pin the Plugin API version explicitly in your integration's deps.
- Test against the matrix combinations of the HVM versions you're
  promising support for.
- Be aware that minor versions can change the Plugin API surface;
  don't assume drop-in compat across 8.1.x.

## Backups — the "global settings first" trap

`Initial Backups Setup` says to configure Global Backup Settings,
Storage Providers, and Execution Schedules **before** creating any
backup jobs. This is real — if you skip global settings and just
create a backup, the job runs with defaults that may not match what
you want (retention, schedule alignment, synthetic-full cadence).

Order of operations:

1. **Global Backup Settings** (Administration → Settings → Backups)
   — enable scheduled backups globally, set defaults.
2. **Storage Providers** (Infrastructure → Storage) — configure
   where backups land.
3. **Execution Schedules** (Library → Automation → Execute
   Scheduling) — define when jobs run.
4. **Then** create backup jobs (Backups → Backups → ADD).

### Synthetic Full backups

Only meaningful for HVM Instance backups (not VMware/etc.). When
enabled, the manager periodically synthesizes a "new full" from a
chain of incrementals so restore time stays bounded as the chain
grows. Recommended on for production HVM Instance backups.

### VMware-specific quirk

For VMware Cloud-type backups, the manager merges + consolidates
snapshots BEFORE the OVF export. That means a long-held snapshot
chain gets squashed into the backup — fine, but the source VM's
snapshot list changes as a side effect of the backup run. Coordinate
with anyone who relied on those snapshots existing.

## Console access — hypervisor vs guest, and keyboard layouts

HVM VMs expose **two** console paths:

- **Guest console** — what's inside the VM (the OS desktop /
  console). Subject to whatever the guest OS provides.
- **Hypervisor console** — the KVM-level console (pre-OS, BIOS,
  bootloader). Useful for OS install, kernel-panic recovery,
  GRUB editing.

Per-VM keyboard layout is set on the VM, then applied to each
session. Choose based on the keyboard the operator who'll actually
use the console has. Mismatches show up as characters being typed
"wrong" in the console even though the keyboard is fine — usually
operator picks the layout setting once during VM provisioning and
forgets to revisit.

### 8.1.2 added Japanese 101/102 keyboard layout

Only in 8.1.2 and later: Japanese 101/102 hardware keyboards are
supported. The 106/109 layout (the more common JP keyboard form
factor) was announced as a planned future enhancement at 8.1.2.

The 8.1.2 console UI also gained a "keyboard tips" pop-over (the
"?" button) using the HPE Morpheus key extension. Works for VMware,
vCD, and HVM/KVM consoles.

## Elevating to HPE Morpheus Enterprise

HVM ships as the VM-only subset of HPE Morpheus's platform. Elevating
upgrades the manager into the full Morpheus Enterprise SKU, which
unlocks multi-cloud, container, automation, and policy features.

Elevation is an in-place upgrade — you don't redeploy the manager.
Per the User Manual page `Elevating to HPE Morpheus Enterprise`:

- Acquire/install a Morpheus Enterprise license.
- Run the elevation procedure (UI-driven from the manager).
- HVM clusters keep working unchanged after elevation.

> When elevated, the docs you want to read shift to the Morpheus
> Enterprise documentation set (sd0000{7510,7621,7732}en_us). A
> separate `morpheus-docs` MCP exists for that surface; treat HVM
> docs as a strict subset.

## REST API surface

**TODO — fill in as we learn it.** The Deployment Guide includes an
"API reference" entry; the actual API is the Morpheus REST API
(scoped to the HVM-relevant subset on a non-elevated manager). Until
this section gets real content, the practical advice is:

- Hit `https://<manager>/api/` — same shape as Morpheus Enterprise's
  API. Use bearer-token auth (issue tokens from the user profile /
  api-tokens UI).
- The Morpheus public OpenAPI / API docs document the full surface;
  HVM-only managers reject Enterprise-only endpoints with 404.
- Plugin API and REST API are SEPARATE surfaces. Plugins extend the
  manager; the REST API is for external automation.

If you've integrated against HVM REST in anger and have specific
notes (auth quirks, undocumented endpoints, schema gotchas,
pagination behavior, etc.), drop them here.

## Networking & deployment scenarios

The Deployment Guide outlines two recommended network topologies:

- **Scenario 1: Converged networking** (one bonded interface group
  carries management + storage + VM traffic, segmented by VLAN).
  Simpler to deploy; recommended for most environments.
- **Scenario 2: Decoupled networking** (separate physical NICs for
  management vs storage vs VM traffic). More throughput for
  high-density VM hosts but more cabling.

`Network Bonding` covers the bond configuration. Active-Backup is the
safe default; LACP requires switch coordination.

The **Required ports** page is the authoritative list — check it
before firewalling anything between the manager and HVM Hosts. The
list changes as features evolve; don't memorize.

## Common operational gotchas

- **Storage Buckets vs Backup Targets.** Same concept, different
  context. A "Storage Bucket" is a configured storage provider in
  Morpheus terms; a "Backup Target" is what the backup wizard calls
  the same thing during the create-backup flow. They're the same
  underlying object.
- **Disabling 2FA.** Per-user, set from the user's profile page.
  There's no global "disable 2FA for everyone" toggle — for
  environments that need to back out a 2FA rollout, disable
  per-user.
- **VM Essentials Network Pool.** Defined in Infrastructure →
  Network → Pools. The HVM provisioning flow allocates IPs from
  this pool — if pool is exhausted, provisioning fails with a
  confusing error message in the activity log; check pool free
  count first when troubleshooting.

## Adding to this doc

Two ways:

1. Manually edit `docs_mcp/api_lessons.md` in this repo and commit.
   The next image build picks it up.
2. Use `submit_doc_bug` for upstream issues, and append the
   takeaway here once the docs team responds.

The point of this doc is to surface the kind of context an
experienced operator would mention in a hallway conversation but
that doesn't quite fit anywhere in the formal product docs. Keep
sections tight — one H2 = one topic the LLM can return on demand.