Compare commits

...
4 Commits
21 changed files with 1590 additions and 23 deletions
+10
View File
@@ -16,3 +16,13 @@ catalog.json
recovered/
.claude/*
!.claude/gitea-ship.json
# demo harness: live credentials and generated media never go in git
demo/demo_win.json
demo/vo/*.wav
demo/vo/narration.json
demo/vo/durations*.json
demo/*.cast
demo/*.gif
demo/*.mp4
demo/marks.jsonl
+75
View File
@@ -0,0 +1,75 @@
# Demo recording harness
Records the rewind loop running against a live ZVM as a narrated MP4. Terminal
only: no screen capture, no video editor.
```
tmux ──▶ asciinema ──▶ agg ──▶ ffmpeg ──▶ mp4
│ ▲
└── left pane: driver, right pane: journal │
│
xAI /v1/tts ──▶ wav per beat ──────────┘ (mux_vo.py)
```
## Pieces
| file | what it does |
|---|---|
| `windows_driver.py` | Windows demo. Intro over the diagram, then the live loop via WinRM. |
| `driver.py` | The Linux equivalent, over SSH. |
| `diagram.py` | Architecture diagram, revealed in four chunks against intro beats i1..i4. |
| `journal.py` | Right-hand pane. Polls a VPG's checkpoints so the tag appears on camera. |
| `record_win.sh` / `record.sh` | Drive tmux + asciinema, then agg and ffmpeg. |
| `narration.md` | The script. One `[beat]` per block; the source of truth. |
| `vo/build_narration.py` | Parses `narration.md`, synthesises a wav per beat, records durations. |
| `mux_vo.py` | Aligns the wavs to the recorded beat marks and muxes the audio. |
## Running it
```bash
cp demo/demo_win.example.json demo/demo_win.json # then fill in the guest creds
export XAI_KEY_FILE=~/xai-api.key XAI_VOICE_ID=<id from /v1/custom-voices>
python3 demo/vo/build_narration.py 1.0 # synthesise, writes durations.json
demo/record_win.sh win1 # record; writes marks.jsonl
python3 demo/mux_vo.py win1 # -> win1_narrated.mp4
```
`demo_win.json` holds live guest credentials and is gitignored. So are the
generated `.wav`, `.cast`, `.gif` and `.mp4` files.
## How the audio stays in sync
The driver writes `marks.jsonl` as it runs: one line per beat with the real
elapsed time it started. `mux_vo.py` delays each wav to its recorded mark, so
sync survives a slow API call or an FLR mount that takes longer than usual.
Nothing is predicted.
Two things this depends on:
- **`agg --idle-time-limit` must be larger than the longest pause** (the scripts
pass 3600). The default is 5 seconds, which compresses idle time, and that
silently breaks the mapping between wall clock and video time.
- **Each beat holds for its narration length.** `hold()` sleeps out whatever is
left after the work finishes, so a line is never cut off mid-sentence.
Check alignment after a mux: the FLR wait should measure near silence.
```bash
ffmpeg -v error -i out.mp4 -vn -ac 1 /tmp/a.wav
ffmpeg -hide_banner -ss 150 -t 6 -i /tmp/a.wav -af volumedetect -f null /dev/null 2>&1 | grep mean_volume
```
Speech sits around -22 dB; a correctly aligned gap reads about -91 dB.
## Narration gotchas
- **Do not map acronyms to run-together phonetics.** The `replace` map takes
`{"phrase": "pronunciation"}`, and `{"VM": "vee em"}` gets spoken as one word,
"vem". Either leave the acronym alone or expand it: `{"VM": "virtual machine"}`.
- **Write for speech, not for the page.** Short declaratives and fragment stacks
read well and sound robotic out loud. Commas and full stops are what the engine
uses for pacing, so clauses joined with commas breathe; a wall of four-word
sentences marches.
- `volumedetect` reports `n_samples: 0` when pointed at a file whose first stream
is video. Extract the audio first, then measure, or you will think a working
track is silent.
+12
View File
@@ -0,0 +1,12 @@
{
"vpg_id": "1816d7a4-3316-44dc-bafd-6d84f63d4ba9",
"vpg_name": "local",
"vm_query": "ad1(1)",
"vm_identifier": "f7f0835d-a5e4-46fa-b710-4bc32076e820.vm-2033",
"change_id": "win-demo-rewind",
"guest_path": "C:\\demo\\app-config.yaml",
"bad_content": "upstream: http://0.0.0.0:1 # BROKEN-BY-AGENT",
"host": "192.0.2.10",
"user": "EXAMPLE\\administrator",
"password": "change-me"
}
+44
View File
@@ -0,0 +1,44 @@
"""Architecture diagram, revealed in four chunks aligned to intro beats i1..i4."""
C = "\033[36m"; B = "\033[1m"; D = "\033[2m"; G = "\033[32m"; Y = "\033[33m"; R = "\033[0m"
CHUNKS = [
[
f" {B}{C}┌────────────────────────────────┐{R}",
f" {B}{C}│{R} Developer / process owner {B}{C}│{R} {D}\"change this config\"{R}",
f" {B}{C}└───────────────┬────────────────┘{R}",
f" {B}{C} │{R}",
f" {B}{C} ▼{R}",
f" {B}{C}┌────────────────────────────────┐{R}",
f" {B}{C}│{R} Business application on a VM {B}{C}│{R}",
f" {B}{C}└───────────────┬────────────────┘{R}",
],
[
f" {D} │ protected by{R}",
f" {D} ▼{R}",
f" {B}{G}┌────────────────────────────────┐{R} {D}┌───────────────────────────┐{R}",
f" {B}{G}│{R} Zerto continuous protection {B}{G}│{R} vs {D}│ Nightly backup │{R}",
f" {B}{G}│{R} {G}recovery point every few secs{R} {B}{G}│{R} {D}│ recovery point: hours old │{R}",
f" {B}{G}└────────────────────────────────┘{R} {D}└───────────────────────────┘{R}",
],
[
"",
f" {B}{Y}┌────────────────────────────────┐{R}",
f" {B}{Y}│{R} AI agent {B}{Y}│{R} 1. ask Zerto: is this protected?",
f" {B}{Y}│{R} authorised to make the change {B}{Y}│{R} 2. tag a checkpoint in the journal",
f" {B}{Y}└────────────────────────────────┘{R} 3. make the change, and test it",
],
[
"",
f" 4. {B}broken, and the agent cannot fix it{R}",
f" recover the file, or the whole VM,",
f" {G}from the checkpoint taken at step 2{R}",
],
]
def frame(upto: int) -> str:
"""Everything revealed through chunk `upto` (1-based)."""
out = ["\033[H\033[2J", f" {B}{C}ZERTO AI REWIND{R} {D}how it fits together{R}", ""]
for chunk in CHUNKS[:upto]:
out.extend(chunk)
return "\n".join(out)
+254
View File
@@ -0,0 +1,254 @@
"""Left pane: drive zerto_rewind_mcp over real MCP stdio. Full rewind loop."""
from __future__ import annotations
import asyncio, json, os, shlex, subprocess, sys, time
from pathlib import Path
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
SP = Path(__file__).parent
CFG = json.loads((SP / "demo.json").read_text())
REPO = "/home/justin/github/zerto-ai-rewind"
RST, BOLD, DIM = "\033[0m", "\033[1m", "\033[2m"
CYAN, GREEN, RED, YEL, MAG = "\033[36m", "\033[32m", "\033[31m", "\033[33m", "\033[35m"
SPEED = float(os.environ.get("DEMO_SPEED", "1.0"))
def recovered_bytes(rec: dict) -> bytes:
"""zerto_recover_file returns the content now, not a path on the MCP host.
The server may be on another machine, so a path there is of no use here.
"""
import base64
if rec.get("encoding") == "base64":
return base64.b64decode(rec["content"])
return rec["content"].encode()
def w(s=""):
sys.stdout.write(s + "\n"); sys.stdout.flush()
def type_out(s, delay=0.012, color=""):
sys.stdout.write(color)
for ch in s:
sys.stdout.write(ch); sys.stdout.flush()
time.sleep(delay * SPEED)
sys.stdout.write(RST + "\n"); sys.stdout.flush()
def beat(n=1.2):
time.sleep(n * SPEED)
STEP = [0]
def step(title):
STEP[0] += 1
w()
w(f"{BOLD}{CYAN}{'='*66}{RST}")
type_out(f" STEP {STEP[0]} {title}", 0.008, BOLD + CYAN)
w(f"{BOLD}{CYAN}{'='*66}{RST}")
beat(0.5)
def note(s):
type_out(f" {s}", 0.010, DIM)
def call_banner(tool, args):
w(f"{MAG} -> MCP call{RST} {BOLD}{tool}{RST}")
for k, v in args.items():
w(f"{DIM} {k} = {v}{RST}")
def show(payload, keep=None, limit=22):
if keep:
payload = {k: payload[k] for k in keep if k in payload}
text = json.dumps(payload, indent=2, default=str)
lines = text.splitlines()
for line in lines[:limit]:
w(f"{DIM} |{RST} {line}")
if len(lines) > limit:
w(f"{DIM} | ... {len(lines)-limit} more lines{RST}")
def ssh_base():
if CFG.get("password"):
return ["sshpass", "-p", CFG["password"], "ssh", "-o", "StrictHostKeyChecking=no",
"-o", "UserKnownHostsFile=/dev/null", "-o", "LogLevel=ERROR"]
base = ["ssh", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null",
"-o", "LogLevel=ERROR"]
if CFG.get("key"):
base += ["-i", CFG["key"]]
return base
def guest(cmd, show_cmd=True):
target = f"{CFG['user']}@{CFG['host']}"
full = ssh_base() + [target, cmd]
if show_cmd:
w(f"{YEL} $ ssh {target} {shlex.quote(cmd)}{RST}")
r = subprocess.run(full, capture_output=True, text=True, timeout=60)
out = (r.stdout + r.stderr).rstrip()
for line in out.splitlines():
w(f"{DIM} |{RST} {line}")
return r.returncode, out
async def main():
w(f"\n{BOLD}{CYAN} ZERTO AI REWIND{RST} {DIM}live end-to-end{RST}\n")
type_out(" An agent is about to change a file on a protected VM.", 0.014)
type_out(" Zerto already journals that VM. Make the agent use it.", 0.014)
beat(1.5)
params = StdioServerParameters(
command=f"{REPO}/.venv/bin/zerto-rewind-mcp",
env={**os.environ, "ZERTO_REWIND_CONFIG": f"{REPO}/config.json"},
)
devnull = open(os.devnull, 'w')
async with stdio_client(params, errlog=devnull) as (r, s):
async with ClientSession(r, s) as sess:
step("Connect to the rewind MCP")
await sess.initialize()
tools = await sess.list_tools()
note(f"{len(tools.tools)} tools from zerto_rewind_mcp")
for t in tools.tools:
w(f"{DIM} - {t.name}{RST}")
beat(2.0)
async def _ticker(label):
t0 = time.time()
try:
while True:
await asyncio.sleep(5)
el = int(time.time() - t0)
sys.stdout.write(f"\r{DIM} {label} ... {el}s{RST}")
sys.stdout.flush()
except asyncio.CancelledError:
el = int(time.time() - t0)
if el >= 5:
sys.stdout.write(f"\r{DIM} {label} ... {el}s done{RST}\n")
else:
sys.stdout.write("\r" + " " * 50 + "\r")
sys.stdout.flush()
raise
async def call(name, **args):
call_banner(name, args)
tick = asyncio.create_task(_ticker("working"))
try:
res = await sess.call_tool(name, args)
finally:
tick.cancel()
try:
await tick
except asyncio.CancelledError:
pass
return json.loads(res.content[0].text)
# ---- 1. the good state on the guest
step("The file the agent is about to break")
guest(f"cat {CFG['guest_path']}")
beat(1.8)
# ---- 2. find protection
step("zerto_find_protection - is this VM protected?")
note("Reads skip the guard. This is the read.")
found = await call("zerto_find_protection", query=CFG["vm_query"])
show(found)
if not found.get("ok"):
w(f"{RED} find_protection not ok - stopping{RST}"); return 1
vm = found["vm"]
beat(2.5)
# ---- 3. guard
step("zerto_guard_before_mutate - pin the journal FIRST")
note("Capture-before-execute. No tag, no change.")
note("The checkpoint name records which agent, and what it is about to do.")
note("Watch the journal pane on the right.")
guard = await call("zerto_guard_before_mutate",
query=CFG["vm_query"], change_id=CFG["change_id"], agent="claude",
action=f"edit {CFG['guest_path']}")
show(guard, keep=["ok", "tag", "tagged", "skipped", "errors", "message"])
if not guard.get("ok"):
w(f"{RED} GUARD FAILED - refusing the change{RST}"); return 1
tag = guard["tag"]
tagged = guard["tagged"][0]
t_tag = time.strftime("%H:%M:%S")
w(f"{GREEN}{BOLD} tagged checkpoint {tagged['checkpoint_id']} on {tagged['vpg_name']}{RST}")
w(f"{GREEN} inserted at {t_tag}{RST}")
w(f"{GREEN} name: {tag}{RST}")
w(f"{DIM} this checkpoint is the rewind point. remember cp {tagged['checkpoint_id']}.{RST}")
beat(4.0)
# ---- 4. the bad change
step("Now the agent makes the bad change")
guest(f"printf '%s\\n' {shlex.quote(CFG['bad_content'])} > {CFG['guest_path']}")
guest(f"cat {CFG['guest_path']}")
t_bad = time.strftime("%H:%M:%S")
w(f"{RED}{BOLD} the good content is gone from the guest ({t_bad}){RST}")
note("Git never had this file. Last night's backup is hours stale.")
beat(2.5)
# ---- 5. refusal without a human
step("zerto_recover_file without a human yes")
deny = await call("zerto_recover_file",
vpg_identifier=tagged["vpg_identifier"],
vm_identifier=vm["vm_identifier"],
checkpoint_identifier=tagged["checkpoint_id"],
guest_path=CFG["guest_path"], confirmed=False)
show(deny)
w(f"{YEL} refused. FLR mounts a disk; a human says yes.{RST}")
beat(2.5)
# ---- 6. recover
rec_cp = tagged["checkpoint_id"]
step("Human says yes - FLR from the pre-mutation checkpoint")
w(f"{BOLD} provenance{RST}")
w(f" {GREEN}cp {tagged['checkpoint_id']} inserted {t_tag}{RST} {DIM}<- guard, BEFORE the change{RST}")
w(f" {RED}bad write {t_bad}{RST} {DIM}<- the mutation, AFTER{RST}")
assert rec_cp == tagged["checkpoint_id"], "recovery must use the guard checkpoint"
w(f"{DIM} recovering from cp {rec_cp}, not the newest checkpoint.{RST}")
note(f"name: {tag}")
rec = await call("zerto_recover_file",
vpg_identifier=tagged["vpg_identifier"],
vm_identifier=vm["vm_identifier"],
checkpoint_identifier=tagged["checkpoint_id"],
guest_path=CFG["guest_path"], confirmed=True)
show(rec)
if not rec.get("ok"):
w(f"{RED} FLR failed{RST}"); return 1
beat(1.5)
w(f"{YEL} recovered {rec['bytes']} bytes, sha256 {rec['sha256'][:16]}...{RST}")
body = recovered_bytes(rec).decode()
for line in body.splitlines():
w(f"{GREEN}{BOLD} | {line}{RST}")
beat(2.5)
# ---- 7. put it back
step("Copy it back to the guest")
note("This part is scp, not Zerto. Zerto got the bytes back.")
target = f"{CFG['user']}@{CFG['host']}"
local_copy = SP / rec["name"]
local_copy.write_bytes(recovered_bytes(rec))
scp = (["sshpass", "-p", CFG["password"]] if CFG.get("password") else []) + \
["scp", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null",
"-o", "LogLevel=ERROR"] + (["-i", CFG["key"]] if CFG.get("key") else []) + \
[str(local_copy), f"{target}:{CFG['guest_path']}"]
w(f"{YEL} $ scp {rec['name']} {target}:{CFG['guest_path']}{RST}")
subprocess.run(scp, capture_output=True, text=True, timeout=60)
guest(f"cat {CFG['guest_path']}")
beat(1.5)
w()
w(f"{BOLD}{GREEN}{'='*66}{RST}")
type_out(" REWOUND. RPO was the journal, not last night.", 0.016, BOLD + GREEN)
w(f"{BOLD}{GREEN}{'='*66}{RST}")
beat(4.0)
return 0
sys.exit(asyncio.run(main()))
+78
View File
@@ -0,0 +1,78 @@
"""Right pane: live view of a VPG journal, tagged checkpoints highlighted."""
from __future__ import annotations
import asyncio, os, sys, time
os.environ.setdefault("ZERTO_REWIND_CONFIG", "/home/justin/github/zerto-ai-rewind/config.json")
from zerto_rewind_mcp.config import load_config
from zerto_rewind_mcp.client import ZertoClient
from zerto_rewind_mcp.checkpoints import checkpoint_tag, checkpoint_id
from zerto_rewind_mcp.util import pick
VPG_ID = sys.argv[1]
VPG_NAME = sys.argv[2] if len(sys.argv) > 2 else VPG_ID
ROWS = int(os.environ.get("JOURNAL_ROWS", "10"))
DIM, RST, BOLD = "\033[2m", "\033[0m", "\033[1m"
CYAN, GREEN, YEL = "\033[36m", "\033[32m", "\033[33m"
def _wrap_tag(tag, width=52, maxlines=3):
tag = tag.replace("; Used for File Level Restore", " [FLR]")
out = []
while tag and len(out) < maxlines:
out.append(tag[:width]); tag = tag[width:]
return out
def ts_of(row):
return str(pick(row, "TimeStamp", "Timestamp", "timestamp") or "")
def render(rows, spin, tagged):
out = ["\033[H\033[2J"]
out.append(f"{BOLD}{CYAN} ZERTO JOURNAL {RST}{BOLD}{VPG_NAME}{RST}\n")
out.append(f"{DIM} vpg {VPG_ID}{RST}\n")
out.append(f"{DIM} {len(rows)} checkpoints{RST} {YEL}{len(tagged)} ai-tagged{RST} {DIM}{spin}{RST}\n\n")
out.append(f"{BOLD} AI TAGS{RST}\n")
if not tagged:
out.append(f"{DIM} (none yet){RST}\n")
for row in tagged[-4:]:
t = ts_of(row).replace("T", " ").replace(".000Z", "")[11:]
out.append(f"{GREEN}{BOLD} * {t} cp {checkpoint_id(row)}{RST}\n")
for seg in _wrap_tag(checkpoint_tag(row)):
out.append(f"{GREEN} {seg}{RST}\n")
out.append(f"\n{BOLD} NEWEST{RST}\n")
for row in rows[-ROWS:]:
tag = checkpoint_tag(row)
t = ts_of(row).replace("T", " ").replace(".000Z", "")
cid = checkpoint_id(row)
t = t[11:]
if tag.startswith("ai:"):
out.append(f"{GREEN}{BOLD} * {t} cp {cid} <-- new{RST}\n")
elif tag:
out.append(f"{DIM} {t} cp {cid} {tag[:20]}{RST}\n")
else:
out.append(f"{DIM} {t} cp {cid}{RST}\n")
sys.stdout.write("".join(out))
sys.stdout.flush()
async def main():
s = load_config()
c = ZertoClient(base_url=s["zerto_url"], username=s["username"], password=s["password"],
client_id=s.get("client_id", "zerto-client"), verify_tls=bool(s.get("verify_tls")))
spinner = "|/-\\"
i = 0
while True:
try:
rows = await c.list_checkpoints(VPG_ID)
tagged = [r for r in rows if checkpoint_tag(r).startswith("ai:")]
render(rows, spinner[i % 4], tagged)
except Exception as exc:
sys.stdout.write(f"\n journal poll error: {exc}\n")
sys.stdout.flush()
i += 1
await asyncio.sleep(3)
asyncio.run(main())
+39
View File
@@ -0,0 +1,39 @@
"""Build the narration track from recorded beat marks and mux it onto the video."""
import json, subprocess, sys
from pathlib import Path
SP = Path(__file__).parent
VO = SP / "vo"
take = sys.argv[1] if len(sys.argv) > 1 else "win1"
video = SP / f"{take}.mp4"
out = SP / f"{take}_narrated.mp4"
marks = [json.loads(l) for l in (SP / "marks.jsonl").read_text().splitlines() if l.strip()]
dur = json.loads((VO / "durations.json").read_text())
vlen = float(subprocess.run(["ffprobe", "-v", "error", "-show_entries", "format=duration",
"-of", "default=nw=1:nk=1", str(video)],
capture_output=True, text=True).stdout.strip())
inputs, filters, labels = [], [], []
for i, m in enumerate(marks):
wav = VO / f"{m['key']}.wav"
if not wav.exists():
print(f" missing {wav.name}, skipping"); continue
delay_ms = int(m["t"] * 1000)
inputs += ["-i", str(wav)]
filters.append(f"[{i}:a]adelay={delay_ms}|{delay_ms},apad[a{i}]")
labels.append(f"[a{i}]")
end = m["t"] + dur.get(m["key"], 0)
flag = "" if end <= vlen + 0.5 else " <-- OVERRUNS VIDEO"
print(f" {m['key']:9} start {m['t']:7.2f}s len {dur.get(m['key'],0):5.2f}s end {end:7.2f}s{flag}")
mix = "".join(labels) + f"amix=inputs={len(labels)}:normalize=0,atrim=0:{vlen},asetpts=N/SR/TB[a]"
cmd = (["ffmpeg", "-y", "-loglevel", "error"] + inputs + ["-i", str(video),
"-filter_complex", ";".join(filters) + ";" + mix,
"-map", f"{len(labels)}:v", "-map", "[a]",
"-c:v", "copy", "-c:a", "aac", "-b:a", "160k", "-shortest", str(out)])
r = subprocess.run(cmd, capture_output=True, text=True)
if r.returncode != 0:
print("ffmpeg failed:\n", r.stderr[-1500:]); sys.exit(1)
print(f"\n video {vlen:.1f}s | narration ends {max(m['t']+dur.get(m['key'],0) for m in marks):.1f}s")
print(f" -> {out}")
+52
View File
@@ -0,0 +1,52 @@
# Zerto AI Rewind - demo narration (v3, rewritten against the jp-voice profile)
# Edit any line, then tell Claude to re-read this file.
# Blank line separates beats. Lines starting with # are ignored.
## PART 1 - INTRO (architecture diagram on screen, builds as each beat lands)
[i1] Hello, and welcome to the Zerto AI Rewind demo.
[i2] AI agents are acting on production systems faster than ever, and a lot faster
than backup was ever designed to keep up with. Zerto's continuous data
protection is a good fit for that, because the journal is always running.
[i3] In this demo an AI agent makes a change to a production system. But before it
does, it checks whether that VM is protected by Zerto, and if it is, it inserts
a tagged checkpoint into the journal first.
[i4] That gives the agent something to fall back on. If the change breaks the
application, and the agent can't fix it on its own, it can use file level or
full system recovery to get back to the moment before it touched anything.
## PART 2 - THE DEMO (audience: a customer, not an engineer)
[d1] This is the agent connecting to Zerto. Everything from here runs against a live
Zerto environment.
[d2] The machine is a production Windows server, and Zerto is already protecting it.
The agent has been asked to change a configuration file on that server.
[d3] Before it changes anything, it asks Zerto a simple question. Is this machine
protected, and which protection group is it in?
[d4] It is, so Zerto puts a tagged checkpoint into the journal, and the agent waits
until Zerto confirms that checkpoint is really there before it goes any further.
[d5] Now the change goes in, and it breaks the file. The version that was there is
gone from the server. Backup has a copy from last night, so that's hours old
already. Zerto has one from seconds before the change.
[d6] The agent can't just recover on its own. Recovery has to be approved by a
person, so at this point it stops and asks.
[d7] With approval, Zerto goes back to the checkpoint from just before the change,
and hands back the original file exactly as it was.
[d8] Putting that file back onto the server is an ordinary copy. Zerto's job was
keeping the data in the first place.
[d9] This was a single file, so file level recovery was enough. That same checkpoint
would let you bring back the entire machine if the damage were bigger.
[d10] And that's the point. The recovery point is the journal, seconds before the
change, instead of last night's backup window.
Executable
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env bash
# Record the split-pane rewind demo to an MP4.
set -uo pipefail
SP="$(cd "$(dirname "$0")" && pwd)"
REPO=/home/justin/github/zerto-ai-rewind
PY="$REPO/.venv/bin/python"
VPG_ID="$(python3 -c "import json;print(json.load(open('$SP/demo.json'))['vpg_id'])")"
VPG_NAME="$(python3 -c "import json;print(json.load(open('$SP/demo.json'))['vpg_name'])")"
TAKE="${1:-take1}"
CAST="$SP/$TAKE.cast"; GIF="$SP/$TAKE.gif"; MP4="$SP/$TAKE.mp4"
tmux -L demo kill-server 2>/dev/null
tmux -L rec kill-server 2>/dev/null
sleep 1
# demo session: driver left, journal right
tmux -L demo new-session -d -s demo -x 200 -y 50 -c "$REPO" \
"$PY $SP/driver.py; echo; echo ' [take complete]'; sleep 3; tmux -L demo kill-server"
tmux -L demo set-option -g status off
tmux -L demo split-window -h -l 60 -t demo -c "$REPO" \
"JOURNAL_ROWS=10 $PY $SP/journal.py $VPG_ID $VPG_NAME"
tmux -L demo select-pane -t 0
# recorder session gives asciinema a pty
tmux -L rec new-session -d -s rec -x 200 -y 50 \
"$SP/recvenv/bin/asciinema rec '$CAST' --overwrite --quiet -c 'tmux -L demo attach -t demo'"
echo "recording -> $CAST"
for i in $(seq 1 600); do
tmux -L rec has-session -t rec 2>/dev/null || break
sleep 2
done
tmux -L demo kill-server 2>/dev/null
tmux -L rec kill-server 2>/dev/null
sleep 1
[ -s "$CAST" ] || { echo "NO CAST PRODUCED"; exit 1; }
echo "cast: $(du -h "$CAST" | cut -f1) duration: $(python3 -c "
import json,sys
last=0
for l in open('$CAST'):
l=l.strip()
if l.startswith('['):
last=json.loads(l)[0]
print(f'{last:.0f}s')")"
"$SP/bin/agg" --font-size 14 --fps-cap 10 --idle-time-limit 2 --theme asciinema "$CAST" "$GIF"
ffmpeg -y -loglevel error -i "$GIF" \
-movflags +faststart -pix_fmt yuv420p -c:v libx264 -crf 20 \
-vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" "$MP4"
echo "MP4: $MP4 ($(du -h "$MP4" | cut -f1))"
ffprobe -v error -show_entries format=duration:stream=width,height -of default=nw=1 "$MP4"
+42
View File
@@ -0,0 +1,42 @@
#!/usr/bin/env bash
# Record the Windows rewind demo. NO idle compression: the video timeline must
# equal wall clock so the narration can be aligned to the recorded beat marks.
set -uo pipefail
SP="$(cd "$(dirname "$0")" && pwd)"
REPO=/home/justin/github/zerto-ai-rewind
PY="$REPO/.venv/bin/python"
VPG_ID=$(python3 -c "import json;print(json.load(open('$SP/demo_win.json'))['vpg_id'])")
TAKE="${1:-win1}"
CAST="$SP/$TAKE.cast"; GIF="$SP/$TAKE.gif"; MP4="$SP/$TAKE.mp4"
tmux -L demo kill-server 2>/dev/null; tmux -L rec kill-server 2>/dev/null; sleep 1
rm -f "$SP/marks.jsonl"
tmux -L demo new-session -d -s demo -x 200 -y 50 -c "$REPO" \
"$PY $SP/windows_driver.py; echo; echo ' [take complete]'; sleep 2; tmux -L demo kill-server"
tmux -L demo set-option -g status off
tmux -L demo split-window -h -l 60 -t demo -c "$REPO" \
"JOURNAL_ROWS=10 $PY $SP/journal.py $VPG_ID ad1"
tmux -L demo select-pane -t 0
tmux -L rec new-session -d -s rec -x 200 -y 50 \
"$SP/recvenv/bin/asciinema rec '$CAST' --overwrite --quiet -c 'tmux -L demo attach -t demo'"
echo "recording -> $CAST"
for i in $(seq 1 450); do tmux -L rec has-session -t rec 2>/dev/null || break; sleep 2; done
tmux -L demo kill-server 2>/dev/null; tmux -L rec kill-server 2>/dev/null; sleep 1
[ -s "$CAST" ] || { echo "NO CAST"; exit 1; }
echo "cast duration: $(python3 -c "
import json
last=0
for l in open('$CAST'):
l=l.strip()
if l.startswith('['): last=json.loads(l)[0]
print(f'{last:.1f}s')")"
# idle-time-limit huge = no compression, so 1s of cast == 1s of video
"$SP/bin/agg" --font-size 14 --fps-cap 10 --idle-time-limit 3600 --theme asciinema "$CAST" "$GIF"
ffmpeg -y -loglevel error -i "$GIF" -movflags +faststart -pix_fmt yuv420p -c:v libx264 -crf 20 \
-vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" "$MP4"
echo "MP4: $MP4 ($(du -h "$MP4" | cut -f1))"
ffprobe -v error -show_entries format=duration:stream=width,height -of default=nw=1 "$MP4"
+35
View File
@@ -0,0 +1,35 @@
"""Parse the approved markdown script into clips + durations."""
import json, os, re, subprocess, sys
from pathlib import Path
SP = Path(__file__).parent
SRC = Path(os.environ.get("NARRATION_MD", SP.parent / "narration.md"))
KEY = Path(os.environ.get("XAI_KEY_FILE", "~/xai-api.key")).expanduser().read_text().strip()
VOICE = os.environ.get("XAI_VOICE_ID", "") # a custom voice id from /v1/custom-voices
SPEED = float(sys.argv[1]) if len(sys.argv) > 1 else 1.0
beats = re.findall(r"\[([id]\d+)\]\s*(.*?)(?=\n\n|\Z)", SRC.read_text(), re.S)
lines = {k: " ".join(v.split()) for k, v in beats}
(SP / "narration.json").write_text(json.dumps(lines, indent=2) + "\n")
# Do NOT map an acronym to run-together phonetics: {"VM": "vee em"} is spoken
# as one word, "vem". Expand it instead, or leave it alone.
REPLACE = {"VM": "virtual machine"}
out = {}
for key, text in lines.items():
body = {"text": text, "voice_id": VOICE, "language": "en", "speed": SPEED,
"replace": REPLACE, "output_format": {"codec": "wav", "sample_rate": 24000}}
req = SP / "req.json"; req.write_text(json.dumps(body))
dest = SP / f"{key}.wav"
code = subprocess.run(["curl", "-sS", "-o", str(dest), "-w", "%{http_code}", "-X", "POST",
"https://api.x.ai/v1/tts", "-H", f"Authorization: Bearer {KEY}",
"-H", "Content-Type: application/json", "--data-binary", f"@{req}"],
capture_output=True, text=True).stdout.strip()
req.unlink()
if code != "200":
print(f" {key}: HTTP {code} FAILED"); sys.exit(1)
d = float(subprocess.run(["ffprobe", "-v", "error", "-show_entries", "format=duration",
"-of", "default=nw=1:nk=1", str(dest)], capture_output=True, text=True).stdout.strip())
out[key] = round(d, 2)
print(f" {key:5} {d:5.2f}s {len(text.split()):3} words")
(SP / "durations.json").write_text(json.dumps(out, indent=1))
print(f"\n {len(out)} clips, {sum(out.values()):.1f}s of narration")
+204
View File
@@ -0,0 +1,204 @@
"""Windows rewind demo. Beats hold for their narration length and log real start times."""
from __future__ import annotations
import json, os, sys, time
from pathlib import Path
import winrm
import diagram
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
SP = Path(__file__).parent
CFG = json.loads((SP / "demo_win.json").read_text())
DUR = json.loads((SP / "vo" / "durations.json").read_text())
REPO = "/home/justin/github/zerto-ai-rewind"
MARKS = SP / "marks.jsonl"
PAD = 0.8 # breath between beats
RST, BOLD, DIM = "\033[0m", "\033[1m", "\033[2m"
CYAN, GREEN, RED, YEL, MAG = "\033[36m", "\033[32m", "\033[31m", "\033[33m", "\033[35m"
T0 = time.time()
STEP = [0]
def recovered_bytes(rec: dict) -> bytes:
"""zerto_recover_file returns the content now, not a path on the MCP host.
The server may be on another machine, so a path there is of no use here.
"""
import base64
if rec.get("encoding") == "base64":
return base64.b64decode(rec["content"])
return rec["content"].encode()
def w(s=""):
sys.stdout.write(s + "\n"); sys.stdout.flush()
def type_out(s, delay=0.018, color=""):
sys.stdout.write(color)
for ch in s:
sys.stdout.write(ch); sys.stdout.flush(); time.sleep(delay)
sys.stdout.write(RST + "\n"); sys.stdout.flush()
def begin(key, title=None):
"""Mark the beat start so the narration can be aligned to it later."""
t = time.time() - T0
with MARKS.open("a") as fh:
fh.write(json.dumps({"key": key, "t": round(t, 3)}) + "\n")
if title:
STEP[0] += 1
w(); w(f"{BOLD}{CYAN}{'=' * 74}{RST}")
type_out(f" STEP {STEP[0]} {title}", 0.010, BOLD + CYAN)
w(f"{BOLD}{CYAN}{'=' * 74}{RST}")
return t
def hold(start_t, key):
"""Keep this beat on screen until its narration has finished."""
want = DUR.get(key, 6.0) + PAD
spent = (time.time() - T0) - start_t
if spent < want:
time.sleep(want - spent)
def show(payload, keep=None, limit=18):
if keep:
payload = {k: payload[k] for k in keep if k in payload}
for line in json.dumps(payload, indent=2, default=str).splitlines()[:limit]:
w(f"{DIM} |{RST} {line}")
_sess = winrm.Session(f"http://{CFG['host']}:5985/wsman",
auth=(CFG["user"], CFG["password"]), transport="ntlm")
def guest_ps(ps, label):
w(f"{YEL} PS {CFG['host']}> {label}{RST}")
r = _sess.run_ps(ps)
out = r.std_out.decode(errors="replace").strip()
for line in out.splitlines():
w(f"{DIM} |{RST} {line}")
return out
async def main():
import asyncio
MARKS.unlink(missing_ok=True)
for n, key in enumerate(("i1", "i2", "i3", "i4"), start=1):
t = begin(key)
sys.stdout.write(diagram.frame(n)); sys.stdout.write("\n"); sys.stdout.flush()
hold(t, key)
params = StdioServerParameters(
command=f"{REPO}/.venv/bin/zerto-rewind-mcp",
env={**os.environ, "ZERTO_REWIND_CONFIG": f"{REPO}/config.json"})
devnull = open(os.devnull, "w")
async with stdio_client(params, errlog=devnull) as (r, s):
async with ClientSession(r, s) as sess:
t = begin("d1", "Connect to the rewind MCP")
await sess.initialize()
tools = await sess.list_tools()
w(f"{DIM} {len(tools.tools)} tools from zerto_rewind_mcp{RST}")
for x in tools.tools:
w(f"{DIM} - {x.name}{RST}")
hold(t, "d1")
async def call(name, **args):
w(f"{MAG} -> MCP call{RST} {BOLD}{name}{RST}")
for k, v in args.items():
w(f"{DIM} {k} = {v}{RST}")
res = await sess.call_tool(name, args)
return json.loads(res.content[0].text)
t = begin("d2", "The file the agent is about to break")
guest_ps(f"Get-Content '{CFG['guest_path']}'", f"Get-Content {CFG['guest_path']}")
hold(t, "d2")
t = begin("d3", "zerto_find_protection")
found = await call("zerto_find_protection", query=CFG["vm_query"])
show(found, keep=["ok", "outcome", "vm"])
if not found.get("ok"):
w(f"{RED} not ok, stopping{RST}"); return 1
hold(t, "d3")
t = begin("d4", "zerto_guard_before_mutate")
guard = await call("zerto_guard_before_mutate", query=CFG["vm_query"],
change_id=CFG["change_id"], agent="claude",
action=f"edit {CFG['guest_path']}")
show(guard, keep=["ok", "tag", "tagged"])
if not guard.get("ok"):
w(f"{RED} GUARD FAILED, refusing the change{RST}"); return 1
tg = guard["tagged"][0]
t_tag = time.strftime("%H:%M:%S")
w(f"{GREEN}{BOLD} checkpoint {tg['checkpoint_id']} inserted {t_tag} task={tg['task_state']}{RST}")
hold(t, "d4")
t = begin("d5", "Now the agent makes the bad change")
guest_ps(f"Set-Content -Path '{CFG['guest_path']}' -Value '{CFG['bad_content']}' -Encoding ASCII; "
f"Get-Content '{CFG['guest_path']}'", f"Set-Content {CFG['guest_path']} ...")
t_bad = time.strftime("%H:%M:%S")
w(f"{RED}{BOLD} the good config is gone from the guest ({t_bad}){RST}")
hold(t, "d5")
t = begin("d6", "zerto_recover_file without a human yes")
deny = await call("zerto_recover_file", vpg_identifier=CFG["vpg_id"],
vm_identifier=CFG["vm_identifier"],
checkpoint_identifier=tg["checkpoint_id"],
guest_path=CFG["guest_path"], confirmed=False)
show(deny)
w(f"{YEL} refused.{RST}")
hold(t, "d6")
t = begin("d7", "Human says yes: FLR from the pre-mutation checkpoint")
w(f"{BOLD} provenance{RST}")
w(f" {GREEN}cp {tg['checkpoint_id']} inserted {t_tag}{RST} {DIM}<- guard, BEFORE{RST}")
w(f" {RED}bad write {t_bad}{RST} {DIM}<- the mutation, AFTER{RST}")
rec = await call("zerto_recover_file", vpg_identifier=CFG["vpg_id"],
vm_identifier=CFG["vm_identifier"],
checkpoint_identifier=tg["checkpoint_id"],
guest_path=CFG["guest_path"], confirmed=True,
dest_dir=str(SP / "win_demo_out"))
show(rec, keep=["ok", "bytes", "flr_path", "unmount"])
if not rec.get("ok"):
w(f"{RED} FLR failed{RST}"); return 1
body = recovered_bytes(rec).decode()
w(f"{YEL} recovered file:{RST}")
for line in body.splitlines():
w(f"{GREEN}{BOLD} | {line}{RST}")
hold(t, "d7")
t = begin("d8", "Copy it back to the guest")
# Ship the recovered bytes verbatim. Set-Content rewrites line
# endings and appends one, which makes the restored file differ
# from what Zerto handed back. WriteAllBytes does not.
import base64
blob = recovered_bytes(rec)
b64 = base64.b64encode(blob).decode()
guest_ps(
f"[IO.File]::WriteAllBytes('{CFG['guest_path']}',"
f"[Convert]::FromBase64String('{b64}'));"
f"Get-Content '{CFG['guest_path']}';"
f"'md5: ' + (Get-FileHash '{CFG['guest_path']}' -Algorithm MD5).Hash",
f"WriteAllBytes {CFG['guest_path']} <{len(blob)} recovered bytes>")
hold(t, "d8")
t = begin("d9")
w(); w(f"{BOLD}{YEL} the same checkpoint, bigger blast radius{RST}")
type_out(" one file -> file level recovery (what you just saw)", 0.012, DIM)
type_out(" the whole VM -> offsite clone, failover test, or failover", 0.012, DIM)
hold(t, "d9")
t = begin("d10")
w(); w(f"{BOLD}{GREEN}{'=' * 74}{RST}")
type_out(" REWOUND. The recovery point was the journal.", 0.018, BOLD + GREEN)
w(f"{BOLD}{GREEN}{'=' * 74}{RST}")
hold(t, "d10")
return 0
import asyncio
sys.exit(asyncio.run(main()))
+95
View File
@@ -0,0 +1,95 @@
# PreToolUse guard hook
Makes the rewind guard enforceable instead of advisory.
The MCP server cannot see another server's tool calls, so `zerto_check_tool` and
`zerto_guard_before_mutate` only work if the model chooses to call them. A model
that skips the step is not stopped by anything. A `PreToolUse` hook runs in the
host, where the tool call genuinely pauses, so a failed checkpoint stops the
change.
## Decisions
| situation | decision | effect |
|---|---|---|
| tool in `read_only_tools` | none | runs, no checkpoint |
| mutating, checkpoint confirmed | none, plus `additionalContext` | runs, and the model is told which checkpoint to recover from |
| mutating, checkpoint failed | `deny` | **the call never happens** |
| mutating, VM unknown to Zerto | `prompt` | human decides; nothing to rewind to |
| mutating, no VM in the arguments | `prompt` | catalog's `vm_arg` did not match |
| unlisted tool | `prompt` | nobody said it was read-only |
`ZERTO_HOOK_UNKNOWN` switches the unlisted case to `allow` or `deny`.
## Install
```bash
cp hooks/settings.example.json /tmp/x # then merge the hooks block into
# .claude/settings.json
export ZERTO_REWIND_CONFIG=/abs/path/config.json
```
Both paths in the command must be absolute, and the interpreter must be the
venv that has this package installed.
## Timeouts, which matter here
The hook is synchronous: the host waits. That is the point, because the
checkpoint has to exist before the change does.
Budget for the slow path, not the fast one. How long the guard takes is set by
the VPG's checkpoint cadence, which is set in turn by its protected site:
| protected at | cadence | guard takes |
|---|---|---|
| vSphere | 5s | ~7s |
| Azure | 60s | ~40s |
| AWS | 630s | ~111s |
The tag wait is derived from that cadence and capped at 300s, so:
- `timeout` in settings.json: 360 (seconds)
- `ZERTO_HOOK_GUARD_TIMEOUT`: 330 (seconds), kept under it
If the host's timeout fires first it cancels the hook and **discards its
output**, and the tool call carries on through the normal permission flow. A
timeout is therefore a silent failure of the guard, which is why the hook's own
budget is the smaller of the two: it would rather deny than be cancelled.
## Verified behaviour
Against a live ZVM 10.9.10, all six rows of the table above. Two worth naming:
- `jp-ubuntu` (healthy, local VPG) tagged checkpoint 7180 and allowed the call,
passing the tag back through `additionalContext`.
- `win2019-1` (VPG `CMH-AWS-1`, protected site AWS) was **denied**. The deny path
works, but see the known issue below: that particular denial was wrong.
Both decisions held with `permission_mode: bypassPermissions`. A hook still
blocks when the user has turned permissions off, which is when an agent is most
likely to be running unattended.
## Log
`~/.zerto-guard-hook.log`, or `ZERTO_HOOK_LOG`. One line per decision.
## Why the timeouts are derived, not fixed
This hook used to deny every change to a cloud-protected VM.
`wait_for_tag` gave up after a hardcoded 45s. That is generous on a
vSphere-protected VPG, which checkpoints every 5s and surfaces a tag in about
4s, and impossible on an AWS-protected one, where a tag takes ~128s because the
journal only checkpoints every 630s.
So the guard reported "no checkpoint, refusing the change" while Zerto was in the
middle of creating one. The checkpoint landed a minute later, in the journal,
after the agent had already been told there was no rewind point.
That is a worse failure than the one this hook exists to prevent. It is silent,
it looks correct in the log, and it blocks legitimate work on every
cloud-protected VM in the estate.
Both budgets are now derived from the VPG's measured cadence rather than
guessed, which is why the numbers above differ by a factor of fifteen between
platforms.
+17
View File
@@ -0,0 +1,17 @@
{
"_comment": "Copy the hooks block into .claude/settings.json (project) or ~/.claude/settings.json (all projects). Use absolute paths.",
"hooks": {
"PreToolUse": [
{
"matcher": "mcp__.*",
"hooks": [
{
"type": "command",
"command": "/home/you/zerto-ai-rewind/.venv/bin/python /home/you/zerto-ai-rewind/hooks/zerto_guard_hook.py",
"timeout": 360
}
]
}
]
}
}
+251
View File
@@ -0,0 +1,251 @@
#!/usr/bin/env python3
"""Claude Code PreToolUse hook: capture before execute, enforced by the host.
The MCP server cannot see another server's tool calls, so the guard it exposes
is advice the model may skip. This hook sits in the host, where the call really
does pause, so a failed checkpoint stops the change instead of merely
suggesting it should.
Decisions:
read-only tool allow, no checkpoint
mutating, guard ok allow, and tell the model which checkpoint to use
mutating, guard failed DENY. No checkpoint means no rewind, so no change.
mutating, VM unprotected prompt. Zerto cannot rewind it; a human decides.
unknown tool prompt. Nobody said it was read-only.
Exit 0 always, with the decision in stdout JSON. Exit 2 would block
unconditionally and ignore the JSON, which loses the reason text.
Install (project .claude/settings.json). Give it room: the guard inserts a
tagged checkpoint and waits for the Zerto task to reach Completed.
{"hooks": {"PreToolUse": [{
"matcher": "mcp__.*",
"hooks": [{"type": "command",
"command": "/path/to/.venv/bin/python /path/to/hooks/zerto_guard_hook.py",
"timeout": 360}]}]}}
"""
from __future__ import annotations
import asyncio
import json
import os
import re
import sys
import time
LOG = os.environ.get("ZERTO_HOOK_LOG", os.path.expanduser("~/.zerto-guard-hook.log"))
# Seconds the hook will wait for the checkpoint. Must stay under the hook
# timeout configured in settings.json, or the host cancels us and the tool
# call proceeds unguarded through the normal permission flow.
# Must exceed the largest tag wait the guard can take. That is now derived from
# the VPG's checkpoint cadence and capped at 300s (MAX_TAG_TIMEOUT_S), because a
# tag takes ~128s to surface on an AWS-protected VPG. Too small a budget here
# just moves the false denial from the guard into the hook.
GUARD_TIMEOUT_S = float(os.environ.get("ZERTO_HOOK_GUARD_TIMEOUT", "330"))
UNKNOWN_DECISION = os.environ.get("ZERTO_HOOK_UNKNOWN", "prompt") # prompt | allow | deny
def log(msg: str) -> None:
try:
with open(LOG, "a") as fh:
fh.write(f"{time.strftime('%Y-%m-%dT%H:%M:%S')} {msg}\n")
except Exception:
pass
def emit(decision: str | None, reason: str = "", context: str = "") -> None:
"""Write the PreToolUse decision and leave. Never raises."""
if decision is None and not context:
sys.exit(0) # no opinion; normal permission flow applies
out: dict = {"hookEventName": "PreToolUse"}
if decision:
out["permissionDecision"] = decision
out["permissionDecisionReason"] = reason
if context:
out["additionalContext"] = context
print(json.dumps({"hookSpecificOutput": out}))
sys.exit(0)
def split_mcp_name(tool_name: str) -> tuple[str, str] | None:
"""mcp__<server>__<tool> -> (server, tool). Server names may contain _."""
m = re.match(r"^mcp__(.+?)__(.+)$", tool_name or "")
return (m.group(1), m.group(2)) if m else None
def resolve_vm(tool_input: dict, vm_arg: str) -> str:
"""Pull the VM identifier out of the pending call's arguments."""
if vm_arg and tool_input.get(vm_arg):
return str(tool_input[vm_arg])
# vm_arg is the catalog's answer, but fall back to the usual suspects so a
# slightly-wrong catalog entry degrades to a prompt rather than a crash.
for key in (
"host",
"hostname",
"computer_name",
"computerName",
"vm",
"vm_name",
"target",
"limit",
"address",
"server",
):
if tool_input.get(key):
return str(tool_input[key])
return ""
async def run_guard(query: str, change_id: str, action: str) -> dict:
from zerto_rewind_mcp.checkpoints import make_tag, tag_vpgs
from zerto_rewind_mcp.client import ZertoClient, ZertoError
from zerto_rewind_mcp.config import load_config
from zerto_rewind_mcp.protection import find_from_rows
settings = load_config()
client = ZertoClient(
base_url=str(settings["zerto_url"]),
username=str(settings["username"]),
password=str(settings.get("password") or ""),
client_id=str(settings.get("client_id") or "zerto-client"),
verify_tls=bool(settings.get("verify_tls")),
)
try:
# Two different failures hide here. "Zerto has never heard of this VM"
# is not the same as "Zerto refused to tag it", and they get different
# answers, so a lookup miss must not surface as a refusal. Looking up a
# plain hostname by vmIdentifier returns HTTP 400, which is a miss.
rows = []
for kwargs in ({"vm_name": query}, {"vm_identifier": query}):
try:
rows = await client.get_vms(**kwargs)
except ZertoError:
rows = []
if rows:
break
if not rows:
return {
"ok": False,
"protected": False,
"message": f"Zerto has no VM matching {query!r}.",
}
result = find_from_rows(query, rows)
if result.outcome != "ok" or not result.taggable_vpgs:
return {"ok": False, "protected": False, "message": result.message}
tag = make_tag(
"claude-code",
change_id,
action=action,
vm_name=result.vm.vm_name if result.vm else None,
)
out = await tag_vpgs(client, result, tag)
out["protected"] = True
return out
except ZertoError as exc:
# Reached Zerto and it refused, or never reached it at all. Either way
# there is no checkpoint, so this is not a pass.
return {"ok": False, "protected": True, "message": str(exc)}
finally:
try:
await client.aclose()
except Exception:
pass
def main() -> None:
try:
payload = json.loads(sys.stdin.read() or "{}")
except Exception as exc:
log(f"unparseable stdin: {exc}")
emit(None)
tool_name = payload.get("tool_name") or ""
tool_input = payload.get("tool_input") or {}
parts = split_mcp_name(tool_name)
if not parts:
emit(None) # not an MCP tool; this hook has nothing to say
server, tool = parts
try:
from zerto_rewind_mcp.catalog import MUTATING, READ_ONLY
from zerto_rewind_mcp.config import load_catalog, load_config
catalog = load_catalog(load_config())
verdict = catalog.classify(server, tool)
entry = catalog.get(server, tool)
except Exception as exc:
log(f"catalog unavailable ({exc}); staying out of the way")
emit(None)
if verdict == READ_ONLY:
log(f"{tool_name}: read-only, allowed")
emit(None)
if verdict != MUTATING:
log(f"{tool_name}: unknown -> {UNKNOWN_DECISION}")
emit(
UNKNOWN_DECISION,
f"{server}/{tool} is not a known read-only tool and is not in the "
"Zerto mutating catalog, so no checkpoint was taken. If it changes a "
"protected VM, that change cannot be rewound. Allow it?",
)
vm = resolve_vm(tool_input, entry.vm_arg if entry else "")
if not vm:
log(f"{tool_name}: mutating but no VM in args {sorted(tool_input)}")
emit(
"prompt",
f"{server}/{tool} is a guest-mutating tool, but no VM could be read "
f"from its arguments (expected {entry.vm_arg if entry else '?'}). "
"No checkpoint was taken. Allow it?",
)
change_id = f"{payload.get('session_id', 'session')[:8]}-{payload.get('tool_use_id', '')[-8:]}"
action = f"{server}/{tool} on {vm}"
try:
guard = asyncio.run(asyncio.wait_for(run_guard(vm, change_id, action), GUARD_TIMEOUT_S))
except TimeoutError:
log(f"{tool_name}: guard timed out after {GUARD_TIMEOUT_S}s -> deny")
emit(
"deny",
f"Zerto did not confirm a tagged checkpoint for {vm} within "
f"{GUARD_TIMEOUT_S:.0f}s, so the change cannot be rewound. Refusing.",
)
except Exception as exc:
log(f"{tool_name}: guard raised {type(exc).__name__}: {exc} -> deny")
emit("deny", f"The Zerto guard failed for {vm}: {exc}. No checkpoint, so refusing.")
if guard.get("ok"):
tagged = (guard.get("tagged") or [{}])[0]
log(f"{tool_name}: tagged cp {tagged.get('checkpoint_id')} on {vm}, allowed")
emit(
None,
context=(
f"Zerto tagged checkpoint {tagged.get('checkpoint_id')} on VPG "
f"{tagged.get('vpg_name')} for {vm} before this call, tag "
f"{guard.get('tag')!r}. If this change breaks the guest, recover from "
"that checkpoint with zerto_recover_file."
),
)
if not guard.get("protected", True):
log(f"{tool_name}: {vm} not protected -> prompt")
emit(
"prompt",
f"{vm} is not protected by Zerto, so this change cannot be rewound. "
f"({guard.get('message', '')}) Allow it anyway?",
)
log(f"{tool_name}: guard failed on {vm} -> deny :: {guard.get('message')}")
emit(
"deny",
f"Zerto could not tag a checkpoint for {vm}, so this change would not be "
f"recoverable: {guard.get('message')}",
)
if __name__ == "__main__":
main()
+5
View File
@@ -41,3 +41,8 @@ ignore = ["ASYNC240"]
asyncio_mode = "auto"
testpaths = ["tests"]
pythonpath = ["src"]
[tool.ruff.lint.per-file-ignores]
# A PreToolUse hook runs inside the host's tool call. If it raises, it breaks
# the call it was meant to protect, so it swallows everything and logs instead.
"hooks/*.py" = ["BLE001", "S110"]
+73 -11
View File
@@ -4,6 +4,7 @@ from __future__ import annotations
import asyncio
import re
import statistics
from datetime import UTC, datetime
from typing import Any
@@ -23,6 +24,15 @@ _WS = re.compile(r"\s+")
# so the name stays readable in the Zerto UI checkpoint list.
TAG_MAX_LEN = 250
# Tag-wait budget. Checkpoint cadence is set by the VPG's protected site and
# measured 5s (vSphere), 60s (Azure) and 630s (AWS) in one estate, so the wait
# has to be derived rather than fixed.
DEFAULT_TAG_TIMEOUT_S = 45.0 # cadence unmeasurable; the old constant
MIN_TAG_TIMEOUT_S = 45.0
MAX_TAG_TIMEOUT_S = 300.0 # AWS needed 128s; this leaves real headroom
MIN_POLL_INTERVAL_S = 1.5
MAX_POLL_INTERVAL_S = 15.0
def _clean(value: Any, limit: int) -> str:
"""One field of a checkpoint name: single-line, no separator collisions."""
@@ -74,29 +84,81 @@ def checkpoint_id(row: dict[str, Any]) -> str:
return str(value) if value is not None else ""
def checkpoint_gaps(rows: list[dict[str, Any]], sample: int = 20) -> list[float]:
"""Seconds between consecutive checkpoints, newest `sample` of them."""
stamps: list[datetime] = []
for row in rows[-sample:]:
raw = str(pick(row, "TimeStamp", "Timestamp", "timestamp") or "")
try:
stamps.append(datetime.fromisoformat(raw))
except ValueError:
continue
return [
(stamps[i + 1] - stamps[i]).total_seconds()
for i in range(len(stamps) - 1)
if stamps[i + 1] >= stamps[i]
]
def cadence_seconds(rows: list[dict[str, Any]], sample: int = 20) -> float | None:
"""How often this VPG writes a checkpoint. None when it cannot be measured."""
gaps = checkpoint_gaps(rows, sample)
return statistics.median(gaps) if gaps else None
def tag_wait_budget(cadence: float | None) -> tuple[float, float]:
"""How long to wait for a tag, and how often to look, given the cadence.
Cadence is set by the VPG's PROTECTED site, and the spread is enormous:
measured 5s on vSphere, 60s on Azure, 630s on AWS. A single constant cannot
serve all three. The old fixed 45s was generous for vSphere and impossible
for AWS, where a tag took 128s to surface, so the guard reported "no
checkpoint" while Zerto was still creating one and the change was refused
for no reason.
Visibility does not scale linearly with cadence (the insert makes its own
off-cadence checkpoint), so this is 2x cadence plus headroom, clamped.
"""
if cadence is None or cadence <= 0:
return DEFAULT_TAG_TIMEOUT_S, MIN_POLL_INTERVAL_S
timeout = min(max(2 * cadence + 30, MIN_TAG_TIMEOUT_S), MAX_TAG_TIMEOUT_S)
interval = min(max(cadence / 10, MIN_POLL_INTERVAL_S), MAX_POLL_INTERVAL_S)
return timeout, interval
async def wait_for_tag(
client: ZertoClient,
vpg_identifier: str,
tag: str,
*,
timeout_s: float = 45.0,
interval_s: float = 1.5,
timeout_s: float | None = None,
interval_s: float | None = None,
) -> dict[str, Any]:
"""Wait until the tag is listed. Budget derived from the VPG's own cadence."""
rows = await client.list_checkpoints(vpg_identifier)
for row in rows:
if checkpoint_tag(row) == tag:
return row
cadence = cadence_seconds(rows)
budget, poll = tag_wait_budget(cadence)
timeout_s = budget if timeout_s is None else timeout_s
interval_s = poll if interval_s is None else interval_s
deadline = asyncio.get_event_loop().time() + timeout_s
last: list[dict[str, Any]] = []
while asyncio.get_event_loop().time() < deadline:
last = await client.list_checkpoints(vpg_identifier)
for row in last:
await asyncio.sleep(interval_s)
rows = await client.list_checkpoints(vpg_identifier)
for row in rows:
if checkpoint_tag(row) == tag:
return row
await asyncio.sleep(interval_s)
measured = f"{cadence:.0f}s" if cadence else "unknown"
raise ZertoError(
f"Tagged checkpoint {tag!r} did not appear on VPG {vpg_identifier} "
f"within {timeout_s:.0f}s. Do not mutate. "
"On a cloud-protected VPG the tag routinely takes longer than this to appear "
"(measured ~34s on Azure, ~128s on AWS), so this timeout may simply be too "
"short rather than the insert having failed. Check the Zerto task before "
"assuming it did not land."
f"within {timeout_s:.0f}s (this VPG checkpoints about every {measured}). "
"Do not mutate. Check the Zerto task before assuming the insert failed: "
"a completed task with no visible checkpoint means the wait was short, "
"not that the insert was rejected."
)
+49 -12
View File
@@ -2,6 +2,8 @@
from __future__ import annotations
import base64
import hashlib
import json
from pathlib import Path
from typing import Any
@@ -32,6 +34,20 @@ _catalog: MutatingCatalog | None = None
_settings: dict[str, Any] = {}
# Files above this are refused rather than streamed through a tool result.
# FLR is for a config file or a dropped directory; a disk image belongs on the
# whole-VM ladder in docs/recover-ladder.md.
MAX_RECOVER_BYTES = 1_048_576
def _encode_recovered(blob: bytes) -> dict[str, Any]:
"""Text where it is text, base64 otherwise, so the caller can just use it."""
try:
return {"encoding": "text", "content": blob.decode("utf-8")}
except UnicodeDecodeError:
return {"encoding": "base64", "content": base64.b64encode(blob).decode("ascii")}
def _dump(payload: Any) -> str:
return json.dumps(payload, indent=2, default=str)
@@ -447,16 +463,20 @@ async def zerto_recover_file(
checkpoint_identifier: str,
guest_path: str,
confirmed: bool = False,
dest_dir: str | None = None,
) -> str:
"""File-level recovery from a journal checkpoint. The VM stays up.
Returns the file CONTENT, not a path. The caller may be on another machine,
so a path on this host is of no use to them, and letting a caller choose
where bytes land is an arbitrary write once this server is shared.
Requires confirmed=true (human yes). Cannot run during clone/test/live/EJC.
10.9 FLR Operator role fails; use an Administrator account.
Locally replicated VPGs only. FLR is performed at the VPG's recovery site,
so a VPG replicating to a cloud ZCA must be recovered from that ZCA's API.
guest_path is the path on the guest: /home/x/f.conf or C:\\Users\\x\\f.txt.
Files larger than max_recover_bytes are refused; use the whole-VM ladder.
"""
if not confirmed:
return _dump(
@@ -472,7 +492,10 @@ async def zerto_recover_file(
gate = await _flr_site_gate(client, vpg_identifier)
if not gate.get("ok"):
return _dump(gate)
dest = Path(dest_dir or _settings.get("recovery_dir") or "./recovered")
# Server-owned, from config only. This string is also handed to the
# appliance as initialDownloadPath, so a caller-supplied value would let a
# caller point the ZVM at a path of their choosing.
dest = Path(_settings.get("recovery_dir") or "./recovered")
dest.mkdir(parents=True, exist_ok=True)
session_id: str | None = None
before = await _live_session_ids(client)
@@ -491,16 +514,30 @@ async def zerto_recover_file(
token = download_token_from(token_payload)
blob = await client.fetch_download(token)
name = Path(guest_path.replace("\\", "/")).name or "recovered.bin"
out_path = dest / name
out_path.write_bytes(blob)
payload = {
"ok": True,
"path": str(out_path.resolve()),
"bytes": len(blob),
"session_id": session_id,
"flr_path": flr_path,
"message": f"Wrote {len(blob)} bytes to {out_path}",
}
limit = int(_settings.get("max_recover_bytes") or MAX_RECOVER_BYTES)
if len(blob) > limit:
payload = {
"ok": False,
"too_large": True,
"bytes": len(blob),
"limit": limit,
"message": (
f"{name} is {len(blob)} bytes, over the {limit} byte limit for "
"file level recovery. Use a bounded whole-VM operation instead: "
"offsite clone or failover test."
),
}
else:
payload = {
"ok": True,
"name": name,
"bytes": len(blob),
"sha256": hashlib.sha256(blob).hexdigest(),
"session_id": session_id,
"flr_path": flr_path,
**_encode_recovered(blob),
"message": f"Recovered {len(blob)} bytes of {name} from the journal.",
}
except ZertoError as exc:
payload = {"ok": False, "session_id": session_id, "message": str(exc)}
finally:
+92
View File
@@ -1,5 +1,7 @@
from datetime import UTC, datetime
import pytest
from zerto_rewind_mcp.checkpoints import (
TAG_MAX_LEN,
checkpoint_id,
@@ -50,3 +52,93 @@ def test_checkpoint_row_keys():
row2 = {"checkpointId": "cp-2", "tag": "t"}
assert checkpoint_id(row2) == "cp-2"
assert checkpoint_tag(row2) == "t"
def _rows(*offsets_seconds):
from datetime import timedelta
base = datetime(2026, 9, 22, 12, 0, 0, tzinfo=UTC)
return [
{"TimeStamp": (base + timedelta(seconds=o)).isoformat().replace("+00:00", "Z")}
for o in offsets_seconds
]
def test_cadence_measures_the_median_gap():
from zerto_rewind_mcp.checkpoints import cadence_seconds
assert cadence_seconds(_rows(0, 5, 10, 15, 20)) == 5.0
assert cadence_seconds(_rows(0, 60, 120, 180)) == 60.0
# one irregular gap must not drag the answer around
assert cadence_seconds(_rows(0, 5, 10, 400, 405, 410)) == 5.0
def test_cadence_is_none_when_unmeasurable():
from zerto_rewind_mcp.checkpoints import cadence_seconds
assert cadence_seconds([]) is None
assert cadence_seconds([{"TimeStamp": "not-a-date"}]) is None
assert cadence_seconds(_rows(0)) is None # one checkpoint gives no gap
def test_tag_wait_budget_covers_every_measured_platform():
"""The three cadences measured in one estate, and what each actually needed."""
from zerto_rewind_mcp.checkpoints import tag_wait_budget
for cadence, observed_visibility in ((5.0, 4.0), (60.0, 34.0), (630.0, 128.0)):
budget, interval = tag_wait_budget(cadence)
assert budget > observed_visibility, (
f"cadence {cadence}s budgets {budget}s but the tag took {observed_visibility}s"
)
assert interval >= 1.5
def test_tag_wait_budget_is_clamped_at_both_ends():
from zerto_rewind_mcp.checkpoints import (
DEFAULT_TAG_TIMEOUT_S,
MAX_TAG_TIMEOUT_S,
MIN_TAG_TIMEOUT_S,
tag_wait_budget,
)
assert tag_wait_budget(0.1)[0] == MIN_TAG_TIMEOUT_S # absurdly fast VPG
assert tag_wait_budget(100_000)[0] == MAX_TAG_TIMEOUT_S # absurdly slow one
assert tag_wait_budget(None)[0] == DEFAULT_TAG_TIMEOUT_S
assert tag_wait_budget(None)[1] == 1.5
def test_wait_for_tag_returns_immediately_when_already_present():
import asyncio
from zerto_rewind_mcp.checkpoints import wait_for_tag
class Client:
def __init__(self):
self.calls = 0
async def list_checkpoints(self, vpg):
self.calls += 1
return [{"Tag": "ai:x", "CheckpointId": "7"}]
c = Client()
row = asyncio.run(wait_for_tag(c, "vpg", "ai:x"))
assert row["CheckpointId"] == "7"
assert c.calls == 1 # no sleep, no second poll
def test_wait_for_tag_error_names_the_measured_cadence():
import asyncio
from zerto_rewind_mcp.checkpoints import wait_for_tag
from zerto_rewind_mcp.client import ZertoError
class Client:
async def list_checkpoints(self, vpg):
return _rows(0, 60, 120, 180) # 60s cadence, tag never appears
with pytest.raises(ZertoError) as err:
# explicit tiny timeout so the test does not actually wait 150s
asyncio.run(wait_for_tag(Client(), "vpg", "ai:missing", timeout_s=0.01, interval_s=0.01))
msg = str(err.value)
assert "about every 60s" in msg
assert "was short, not that the insert was rejected" in msg
+70
View File
@@ -0,0 +1,70 @@
"""The PreToolUse hook's pure helpers. The decision paths are exercised live."""
import importlib.util
from pathlib import Path
import pytest
spec = importlib.util.spec_from_file_location(
"zerto_guard_hook", Path(__file__).resolve().parent.parent / "hooks" / "zerto_guard_hook.py"
)
hook = importlib.util.module_from_spec(spec)
spec.loader.exec_module(hook)
@pytest.mark.parametrize(
("tool_name", "expected"),
[
("mcp__ssh__exec", ("ssh", "exec")),
("mcp__winrm__run_ps", ("winrm", "run_ps")),
# server names may themselves contain underscores; split on the FIRST __
("mcp__plugin_my_thing__write_file", ("plugin_my_thing", "write_file")),
("mcp__ob1__OB1__search_thoughts", ("ob1", "OB1__search_thoughts")),
("Bash", None),
("", None),
],
)
def test_split_mcp_name(tool_name, expected):
assert hook.split_mcp_name(tool_name) == expected
def test_resolve_vm_prefers_the_catalog_arg():
assert hook.resolve_vm({"host": "a", "computer_name": "b"}, "computer_name") == "b"
def test_resolve_vm_falls_back_when_catalog_arg_is_wrong():
# a slightly wrong catalog entry should degrade to a prompt, not a crash
assert hook.resolve_vm({"computer_name": "web01"}, "host") == "web01"
def test_resolve_vm_returns_empty_when_no_vm_present():
assert hook.resolve_vm({"command": "ls"}, "host") == ""
def test_emit_with_no_decision_and_no_context_exits_silently(capsys):
with pytest.raises(SystemExit) as exc:
hook.emit(None)
assert exc.value.code == 0
assert capsys.readouterr().out == ""
def test_emit_deny_shape(capsys):
import json
with pytest.raises(SystemExit) as exc:
hook.emit("deny", "no checkpoint")
assert exc.value.code == 0 # exit 2 would discard the JSON and lose the reason
out = json.loads(capsys.readouterr().out)["hookSpecificOutput"]
assert out["hookEventName"] == "PreToolUse"
assert out["permissionDecision"] == "deny"
assert out["permissionDecisionReason"] == "no checkpoint"
def test_emit_context_without_decision(capsys):
import json
with pytest.raises(SystemExit):
hook.emit(None, context="checkpoint 7180")
out = json.loads(capsys.readouterr().out)["hookSpecificOutput"]
assert "permissionDecision" not in out
assert out["additionalContext"] == "checkpoint 7180"
+41
View File
@@ -256,3 +256,44 @@ def test_flr_gate_refuses_remote_recovery_site_and_names_it():
# must tell the operator where the operation actually lives
assert out["recovery_site"] == "aws-zca"
assert "aws-zca" in out["message"]
def test_encode_recovered_text_and_binary():
from zerto_rewind_mcp.server import _encode_recovered
text = _encode_recovered(b"listen: 0.0.0.0:8443\n")
assert text["encoding"] == "text"
assert text["content"] == "listen: 0.0.0.0:8443\n"
binary = _encode_recovered(b"\x89PNG\r\n\x1a\n\xff\xfe")
assert binary["encoding"] == "base64"
import base64 as b64
assert b64.b64decode(binary["content"]) == b"\x89PNG\r\n\x1a\n\xff\xfe"
def test_recover_file_takes_no_caller_destination():
"""The caller must not choose where bytes land, nor where the ZVM mounts.
dest_dir used to be a tool parameter whose value was also passed to the
appliance as initialDownloadPath.
"""
import inspect
from zerto_rewind_mcp.server import zerto_recover_file
params = set(inspect.signature(zerto_recover_file).parameters)
assert "dest_dir" not in params
assert params == {
"vpg_identifier",
"vm_identifier",
"checkpoint_identifier",
"guest_path",
"confirmed",
}
def test_recover_byte_cap_is_configurable_and_has_a_default():
from zerto_rewind_mcp.server import MAX_RECOVER_BYTES
assert MAX_RECOVER_BYTES == 1_048_576