webhook-server/docs/recipes/zerto-pre-post-scripts.md

# Recipe: Zerto failover post-script → DNS update + service checks

This is the canonical reason Webhook Server exists.

When Zerto fails a VM over from production to DR, the VM boots fine — but **the things around it** often need attention: DNS records still point at the production IP, dependent services need to be checked, on-call needs a heads-up. Zerto pre/post scripts run on the **Zerto Virtual Manager**, not on a domain controller and not necessarily with admin rights to the things that need fixing. So you want a single webhook URL that the post-script hits, and a Windows host on the DR side that does the actual work with the right identity.

## What we're building

Zerto's post-recovery script (a one-shot PowerShell file pointing at curl) calls `http://webhook.dr.contoso.local:8080/hook/post-failover` with a JSON body identifying the VPG and operation. The Webhook Server, running on a DR-side Windows host as a gMSA with delegated AD/DNS rights, runs PowerShell that:

1. Updates DNS A records to point the failed-over hostnames at their DR IPs
2. Waits for the failed-over VM to come up (ping + WinRM probe)
3. Connects to the VM via PowerShell remoting and starts/checks critical services
4. Sends a Teams notification with the result

The endpoint is **Async** so the Zerto script returns in milliseconds — no risk of timing out Zerto's failover sequence even if the actions take minutes. The script's full output ends up in the webhook log and (optionally) in an outbound callback.

## Why curl and not Invoke-WebRequest?

Zerto's PowerShell runner is intentionally minimal — many environments run an older Windows on the ZVM and don't have full PowerShell modules installed. `curl.exe` ships with Windows 10 1803+ and Server 2019+ and works without any modules. Plus, calling an HTTP endpoint with `curl.exe` doesn't depend on the version of `Invoke-WebRequest` shipped with the host's PowerShell.

## 1. The Zerto post-script (client side)

A ready-to-use script ships in this repo at [`scripts/examples/zerto-post-failover.ps1`](../../scripts/examples/zerto-post-failover.ps1). Copy it to the ZVM, edit `$WebhookUrl` and the bearer-token path at the top, and wire it into the VPG:

> **VPG settings → Recovery → Scripts → Post-Recovery Script**
> Path: `C:\Scripts\zerto-post-failover.ps1`
> Parameters: *(leave empty)*

The script is ~50 lines and only depends on `curl.exe` + a token file readable by the ZVM service account.

The flow:

```
Zerto VPG failover starts
   |
   +-- VM is brought up at DR site
   |
   +-- Zerto post-script fires:
   |     curl POST http://webhook.dr/hook/post-failover  (async, returns 202 in ~50ms)
   |
   +-- Zerto sees success, finishes the failover and reports done
                                                    |
                       (meanwhile, on the webhook server)
                                                    |
                       running PowerShell for several minutes:
                         - update DNS
                         - wait for VM ready
                         - check services on VM
                         - notify Teams
```

## 2. The server-side script (does the actual work)

Save this on the webhook host as `C:\Scripts\post-failover-handler.ps1`:

```powershell
[CmdletBinding()]
param()
$ErrorActionPreference = 'Stop'

$body = $input | ConvertFrom-Json

# ---------- environment specifics; edit for your site ----------
$dnsServer    = 'dc01.contoso.local'
$forwardZone  = 'contoso.local'
$teamsWebhook = 'https://contoso.webhook.office.com/...'
$drIpMap      = @{
    'app01' = '10.42.10.11'
    'app02' = '10.42.10.12'
    'db01'  = '10.42.10.21'
}
$serviceMap   = @{
    'app01' = @('W3SVC','MyAppSvc')
    'app02' = @('W3SVC','MyAppSvc')
    'db01'  = @('MSSQLSERVER','SQLAgent')
}
# ---------------------------------------------------------------

# Default the VM list to "all VMs we know about" if the post-script didn't
# tell us, so the same handler works without having to embed the VM list in
# every Zerto post-script.
$vms = if ($body.vms) { $body.vms } else { $drIpMap.Keys }

$summary = @()

foreach ($vm in $vms) {
    if (-not $drIpMap.ContainsKey($vm)) {
        $summary += "skip  $vm  (no DR IP mapping in handler)"
        continue
    }
    $ip = $drIpMap[$vm]

    # 1. DNS - delete + re-add the A record
    try {
        $existing = Get-DnsServerResourceRecord -ZoneName $forwardZone -Name $vm `
                      -RRType A -ComputerName $dnsServer -ErrorAction SilentlyContinue
        if ($existing) {
            Remove-DnsServerResourceRecord -ZoneName $forwardZone -Name $vm `
                -RRType A -RecordData $existing.RecordData.IPv4Address `
                -ComputerName $dnsServer -Force
        }
        Add-DnsServerResourceRecordA -ZoneName $forwardZone -Name $vm `
            -IPv4Address $ip -ComputerName $dnsServer -TimeToLive 00:05:00
        $summary += "dns   $vm -> $ip"
    } catch {
        $summary += "DNS!  $vm  $($_.Exception.Message)"
        continue
    }

    # 2. Wait for the VM to be reachable (up to 5 minutes)
    $deadline = (Get-Date).AddMinutes(5)
    $reachable = $false
    while ((Get-Date) -lt $deadline) {
        if (Test-Connection -ComputerName $ip -Count 1 -Quiet -ErrorAction SilentlyContinue) {
            try {
                # Quick WinRM probe; succeeds when the VM has finished booting
                Invoke-Command -ComputerName $ip -ScriptBlock { $true } -ErrorAction Stop | Out-Null
                $reachable = $true
                break
            } catch { Start-Sleep -Seconds 10 }
        } else {
            Start-Sleep -Seconds 10
        }
    }
    if (-not $reachable) {
        $summary += "wait! $vm  not reachable after 5 minutes"
        continue
    }

    # 3. Check + start critical services on the VM
    if ($serviceMap.ContainsKey($vm)) {
        $svcReport = Invoke-Command -ComputerName $ip -ArgumentList @(,$serviceMap[$vm]) -ScriptBlock {
            param($services)
            $report = @()
            foreach ($s in $services) {
                $svc = Get-Service -Name $s -ErrorAction SilentlyContinue
                if (-not $svc) { $report += "$s : missing"; continue }
                if ($svc.Status -ne 'Running') {
                    Start-Service $s
                    Start-Sleep -Seconds 2
                    $svc.Refresh()
                }
                $report += "$s : $($svc.Status)"
            }
            return $report
        }
        $summary += "svc   $vm : $($svcReport -join ', ')"
    } else {
        $summary += "svc   $vm  (no services configured)"
    }
}

# 4. Notify Teams
$teamsBody = @{
    text = "Webhook post-failover for VPG **$($body.vpg)**:`n" + ($summary -join "`n")
} | ConvertTo-Json
try {
    Invoke-RestMethod -Uri $teamsWebhook -Method POST -ContentType 'application/json' -Body $teamsBody | Out-Null
} catch {
    $summary += "teams! notification failed: $($_.Exception.Message)"
}

# Return the summary so it shows up in the webhook log + outbound callback
$summary -join "`n"
```

Two things to call out:

- **PowerShell remoting to the VM** uses the gMSA's network identity (or whoever the service runs as). Make sure the gMSA / service account can `Invoke-Command` to the failed-over hosts — usually that means the account is a local admin on the target VMs, or you've configured constrained delegation.
- **WinRM** must be enabled on the failed-over VMs for the remoting calls to work. `Enable-PSRemoting` is the simplest, but most prod environments configure WinRM via Group Policy.

## 3. Configure the endpoint in the GUI

**File → New endpoint:**

| Section | Setting | Value |
|---|---|---|
| Identity | Slug | `post-failover` |
| Identity | Description | "Zerto post-recovery: DNS + service checks" |
| Auth | Mode | **Bearer** |
| Auth | Bearer secret | generate a 32-byte random string; copy it for the Zerto script's token file |
| Allowed clients | (one per line) | `10.0.0.0/8` *(your ZVM's network)* |
| Executor | Type | **Windows PowerShell** |
| Executor | Script path | `C:\Scripts\post-failover-handler.ps1` |
| Data passing | JSON body to stdin | ✓ |
| Run as | Identity | **Service** if the service runs under a gMSA with the right rights, otherwise **SpecificUser** with a delegated account |
| Response | Mode | **Async** ← critical: this is what makes the Zerto script non-blocking |
| Response | Timeout (sec) | `600` *(this is the cap on the long-running handler script, not the Zerto-facing response)* |
| Response | Fail on non-zero exit | unticked *(async hooks have no caller to receive a 502)* |

Save. Right-click the row → **Copy URL** to grab `http://webhook.dr.contoso.local:8080/hook/post-failover` and paste it into `$WebhookUrl` at the top of the Zerto-side script.

> **Why Bearer instead of HMAC?** Both work. Bearer is simpler — drop the token in a file on the ZVM that's readable by the ZVM service account and you're done. HMAC requires the Zerto-side script to compute a signature, which is doable but adds a few lines of code. Pick what fits your environment.

## 4. Wire up the bearer token

Place the bearer token in a file the ZVM service account can read (and nobody else):

```powershell
# on the ZVM, from elevated PowerShell
$token = (New-Guid).ToString('N')   # or paste the value from the GUI
$tokenPath = 'C:\ProgramData\Zerto\webhook-token.txt'
$token | Out-File -LiteralPath $tokenPath -Encoding utf8 -NoNewline
icacls $tokenPath /inheritance:r /grant 'NT SERVICE\Zerto Online Services:R' 'BUILTIN\Administrators:F' /T
```

Adjust the service principal name to whatever Zerto runs as on your version. The script reads from this path automatically; no change needed in the script itself.

## 5. Test before going live

In a maintenance window, fire the webhook by hand:

```powershell
# from any machine that can reach the webhook server
$body = @{
    operation = 'test'
    vpg       = 'SmokeTest'
    timestamp = (Get-Date).ToUniversalTime().ToString('o')
} | ConvertTo-Json -Compress

curl.exe --silent --show-error --max-time 10 -X POST `
    -H "Authorization: Bearer paste-the-token" `
    -H "Content-Type: application/json" `
    -d $body `
    http://webhook.dr.contoso.local:8080/hook/post-failover
```

You'll get back `{"runId":"…","accepted":true}` immediately. Open the Webhook Server GUI and watch the log panel — within 30 seconds or so you'll see lines for the run. Confirm DNS records updated, services on each VM ended in `Running`, and the Teams notification arrived.

## Variations

### Different actions for failover vs. failback

Pass an `operation` field in the body and branch on it. The Zerto-side script already sends `operation = 'failover'`. Add a separate post-failback script (or detect from `$env:ZertoOperationType`) that sends `operation = 'failback'` and have the handler revert DNS to production IPs.

### Per-VPG endpoints

If you want fine-grained access control or different actions per VPG, create one endpoint per VPG (`post-failover-app`, `post-failover-db`, …) and give each its own bearer token. The GUI handles dozens of endpoints fine.

### Audit trail to a SIEM

Each endpoint can have an outbound **Callback** URL. Configure it with your SIEM's HTTP collector + an HMAC secret, and every run produces a JSON record with runId, exit code, duration, stdout, and stderr — perfect for compliance.