OpenVixDiskLib/docs/reverse_engineering_procedure.md

# Reverse-engineering procedure

This is the working method used to replace VDDK’s NBD path with
OpenVixDiskLib. Protocol details live in `docs/nfc_auth.md`,
`docs/nfc_open.md`, `docs/nfc_read.md`, and `docs/nfc_write.md`.
The capture tool is described in `docs/ssl_hook.md`.
This file is the **sequence of steps**, including dead ends, so later
NFC work can follow the same loop instead of rediscovering it.

Scope so far: `VixDiskLib_ConnectEx` + `VixDiskLib_Open` +
`VixDiskLib_Read` + `VixDiskLib_Write` against lab vCenter 8.0.1 /
ESXi 8, transports `nbd` and `nbdssl`. Validation method:
`tests/integration/` (the session-scoped `lab` fixture creates a temporary
empty VM with a 10 GiB disk and destroys it when the pytest session ends).

Rule from `AGENTS.md`: reuse pyVmomi for every public VIM operation.
Only reimplement what pyVmomi does not expose.

## Loop

Each unknown stage (ticket SOAP, authd, NFC binary) went through:

1. **Name it** from VDDK logs and `strings` on the bundled libraries.
2. **See it** on the wire (or prove that tcpdump cannot).
3. **Replay** the smallest working subset in Python against the lab.
4. **Write** findings into a protocol doc and keep the hook out of the
   library path.

Do not skip (2). Log lines such as `SESSIONID` or `useSSL=0` named the
wrong wire command until the intercept existed.

## Lab and artifacts

| Item                    | Where / value                                                        |
| ----------------------- | -------------------------------------------------------------------- |
| VDDK 8.0.2              | `.vddk/` (`libvixDiskLib`, `libvddkVimAccess`, `libvim-types`)       |
| pyVmomi                 | `.venv`                                                              |
| Known-good VDDK client  | `tests/integration/test_vddk.py` / `tests/integration/vixdisklib.py` |
| Verbose NFC logs        | `vixDiskLib.nfc.LogLevel=4` in a temp VDDK config                    |
| ctypes capture drivers  | `docs/probing_samples/` (not library code)                           |
| SSL / `write` hook      | `/tmp/sslhook.c` → `/tmp/sslhook.so`                                 |
| Pickled `LabEnv`        | `/tmp/vddk-write-wire-lab.pkl` during hooked captures only           |

Always set `LD_LIBRARY_PATH` to `.vddk/` so VDDK uses its own
`libssl.so.3`. Unset `LD_PRELOAD` before running OpenVixDiskLib;
a leftover `write` hook will crash pyVmomi’s TLS.

## Tools

tcpdump was the first capture attempt and is the wrong tool for TLS
stages (Step 3). Everything that actually produced protocol bytes or
names is in this table.

| Tool                                     | What it was used for                                              | Limitation                                                         |
| ---------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------ |
| tcpdump on 443 / 902                     | Prove VDDK talks to vCenter then ESXi:902; see TLS record sizes   | Ciphertext only: no SOAP, authd lines, or NFC headers              |
| `strings -a` on `.vddk/*.so`             | Candidate tokens (`SESSION`, `NfcGetVmFiles`, `NFC_AIO_MSG_*`)    | Not command order, spacing, or replies                             |
| `nm -D` / `objdump -T`                   | Which library imports `SSL_write` vs `write`; exported APIs       | Not wire layout                                                    |
| VDDK `vixDiskLib.nfc.LogLevel=4`         | Function names and AIO `opId` / `type` / `size` to label a frame  | Not magic numbers, path placement, or `BANNER \r\n`                |
| `LD_PRELOAD` SSL / `write` hook          | Plaintext of SOAP, authd, and (after PROXY) NFC on fd 902         | Must not stay on the OpenVixDiskLib process; `docs/ssl_hook.md`    |
| `strace -f -x` on `write` / `send*`      | First writable NFC capture without rebuilding the hook (Step 10)  | Noisy; TLS still opaque; `-s` truncates large extras               |
| `pickle` of `LabEnv`                     | Create the temp VM unhooked, then load it under the hook          | `/tmp` only; never commit pickles (lab host and credentials)       |
| ctypes drivers in `docs/probing_samples/` | Repeatable `ConnectEx` / `Open` / `Read` / `Write` under capture | Not library code                                                   |

`ltrace` was considered for OpenSSL and libc `write`. It was not used:
VDDK is stripped enough that `strace` on syscalls plus the `LD_PRELOAD`
hook were enough. An ESXi impersonator (`AGENTS.md`) was also not
needed; the lab already answered VDDK.

## Step 1 — Map the public VDDK calls

`tests/integration/test_vddk.py` is the specification of what
“success” looks like: login, open the temporary VM’s VMDK, write a
known pattern, read it back.

Turn on VDDK verbose logging around `InitEx` / `ConnectEx` / `Open`. The
logs split the work that the Python API hides:

- `ConnectEx` → VIM login only.
- `Open` → NFC ticket, authd, NFC handshake, AIO open, then I/O.
- `transport_modes=nbd` → URL form `vpxa-nfc://[ds] path.vmdk@esxi:902`.
- `snapshot_ref` does not appear on the ticket SOAP call.

That mapping is the table at the top of `docs/nfc_auth.md`. It tells you
which stage to reverse next and which arguments belong there (VM moref
on the ticket, VMDK path on NFC `OPEN_FILE`).

## Step 2 — Strings and pyVmomi before any capture

`strings -a` on `.vddk/*.so` produced candidate tokens before a single
packet was decoded:

- SOAP: `NfcService`, `NfcGetVmFiles`, `nfcService`, `ha-nfc`,
  `HostServiceTicket`.
- authd: `SESSION`, `BANNER`, `THUMBPRINT_SHA2`, `PROXY`, `USER`,
  `PASS`, `SSL Required`.
- NFC: `NFC_HANDSHAKE`, `NFC_CONNECTION_DATA`, `NFC_AIO_MSG_*`,
  `NFC_DISK`.

Then check whether pyVmomi already has the type:

```python
from pyVmomi import vim
hasattr(vim, "NfcService")   # False
hasattr(vim, "HostServiceTicket")  # True
```

`ServiceManager.QueryServiceList` on the live vCenter does **not** list
NFC. `GET /sdk/nfcServiceVersions.xml` does (`urn:nfc` 7.0.3.2). The
moref is hardcoded in VDDK (`nfcService` on vCenter, `ha-nfc` on ESXi).

Anything public (`SmartConnect`, `vim.VirtualMachine`,
`HostServiceTicket`) stays in pyVmomi. Missing managed types are
registered with `CreateManagedType` on the same SOAP stub so cookies
and serialization are not reimplemented.

## Step 3 — Confirm tcpdump is the wrong tool for TLS stages

tcpdump on 443 and 902 shows TLS records only. That is enough to prove
“something talks to vCenter then to ESXi:902”, and not enough for SOAP
bodies, authd lines, or NFC headers.

VDDK logs name functions and AIO `opId` / `type` / `size`. They do not
give magic numbers, path placement, or command spacing (`BANNER \r\n`).

An ESXi-impersonating service was considered (`AGENTS.md`) and not
needed: the lab answers VDDK, so capturing the real client is simpler
than simulating the server.

## Step 4 — Interpose OpenSSL (authd and SOAP)

VDDK 8.0.2 still calls `SSL_write` / `SSL_read`. A small `LD_PRELOAD`
library logs those buffers as hex, tagged with the `SSL *` pointer.

Run a minimal ctypes program (`InitEx`, `ConnectEx`, `Open`, `Read`)
under:

```bash
export LD_LIBRARY_PATH=…/.vddk
export LD_PRELOAD=/tmp/sslhook.so
export SSLHOOK_LOG=/tmp/sslhook-open.log
python /tmp/vddk_open_trace.py
```

Parse offline:

1. Concatenate adjacent same-direction records (authd `SSL_read` is
   often one byte).
2. Split by `SSL *`. vCenter HTTPS contains `POST /sdk` and SOAP.
   ESXi:902 contains `SESSION` / `PROXY`.
3. An early hook without the pointer mixed both streams; always tag.

The vCenter stream identified the ticket as `NfcGetVmFiles` with moref
`nfcService` and `xmlns="urn:vim25"` (not a guess from strings alone).
The ESXi stream gave the authd command order, including the trailing
space on `BANNER` and `THUMBPRINT_SHA2 PlainText`.

Hook implementation notes: `docs/ssl_hook.md`.

## Step 5 — Probe SOAP, then replay only what VDDK sends

With a SmartConnect session, raw SOAP posts were used to learn
parameter names and which moref vCenter accepts:

- `ha-nfc` on vCenter → `ManagedObjectNotFound`.
- `NfcGetServerNfcLibVersion` without `hostForAccess` → invalid
  argument; with a host moref → `11`.
- `NfcGetVmFiles(vm)` → `HostServiceTicket` (VDDK read-only Open).
- `NfcRandomAccessOpenReadonly(vm, diskDeviceKey, host)` → same ticket
  type, disk-scoped read.
- `NfcRandomAccessOpenDisk(vm, diskDeviceKey, host)` → writable
  ticket. `GetVmFiles` plus `OPEN_FILE` flags `0x1a` fails with
  `VIX_E_FILE_READ_ONLY` (`0x0b`).

OpenVixDiskLib registers those methods and calls them through pyVmomi.
It does not ship a hand-rolled SOAP client for login or tickets.

## Step 6 — Probe authd; record dead ends

Plaintext banner (`220 … SSL Required`), then `ssl.wrap_socket`.
Commands before TLS drop the connection.

vCenter UID/password are **not** sent to port 902. Attempts that failed
and must not be retried for this ticket type:

| Attempt                               | Result                                |
| ------------------------------------- | ------------------------------------- |
| `USER` / `PASS` (vCenter account)     | `530 Login incorrect`                 |
| `USER` / `PASS` with `sessionId`      | `530 Login incorrect`                 |
| `SESSIONID <sessionId>`               | `530 Please login with USER and PASS` |
| `CONNECT_VPXA` after TLS              | `530 Please login with USER and PASS` |
| Wait for a reply after `SESSION`      | Hang until `BANNER` / `PROXY` follow  |
| `THUMBPRINT_SHA2` with SHA-1 digest   | `501 Invalid arguments`               |

The working sequence is in `docs/nfc_auth.md`. `useSSL=0` in the VDDK
log means skip a **second** NFCSSL wrap, not skip TLS on 902.

Replay: `openvixdisklib/nfc_auth.py` /
`tests/integration/test_nfc_auth.py`. Stop at `200 Connect`.

## Step 7 — NFC binary: extend the hook to `write` / `read`

After `PROXY`, `SSL_write` on the ESXi `SSL *` goes silent. VDDK logs
`useSSL=0` / “plain-text connection is deprecated” and then NFC
function names. The bytes are `write(SSL_get_fd(ssl), …)` / `read` on
peer port 902.

The hook was extended to those syscalls, filtered with `getpeername`
port 902, and mutex-locked (VDDK is multi-threaded). Skip TLS records
(`16 03` / `17 03`) left over from the authd phase.

Correlate each frame with the verbose log line that has the same
`type` and `size` (`NfcAioSendMessage: opId = … type = … size = …`).
Name the types from the consecutive `NFC_AIO_MSG_*` string table in
`libvixDiskLib.so`. Lengths 4 and 7 on the connection-data message are
the ASCII strings `vddk` and `nbdmode` sent in the next two writes.

Classic NFC uses a 264-byte padded struct; AIO uses a 16-byte header
(`magic 0xA100DA7A`) plus payload; path / DDB key / sector data are
extra writes not included in `size`.

`strace` is a usable second view of this same plaintext NFC path when
the hook is not loaded. It cannot replace the hook for TLS (authd and
SOAP). How it was run, and why pickle sits between VM create and the
hooked VDDK process, is in Step 10.

## Step 8 — Replay the smallest subset, then compare to VDDK

Python must **dup the authd fd** and send NFC as raw TCP.
`SSLSocket.send` would encrypt; `unwrap()` would `SSL_shutdown`. VDDK
does neither.

OpenVixDiskLib (`openvixdisklib/nfc_open.py`) replays handshake + AIO
`OPEN_SESSION` / sockopts / resource pool / `OPEN_FILE`. VDDK’s extra
`DDB_GET` keys were omitted once a file handle was enough to read. Proof
of open: `tests/integration/test_nfc_open.py`.

Do not copy every VDDK message. Copy what the server requires for the
Python API you are replacing.

## Step 9 — Vary `VixDiskLib_Read` until IO fields stop moving

A single-sector read is not enough to decode `NFC_AIO_MSG_IO`. Drive
VDDK with several `(startSector, numSectors)` pairs in one Open
(including `n=128` = 64 KiB and `n=129`) under the `write`/`read` hook.

What that comparison showed:

- Wire units are bytes (`offset = start * 512`, `length = n * 512`).
- Request size stays 44; data is extra after the payload.
- VDDK sends **one** request even when `length > 65536`. The server
  replies with several type-7 messages that share `opId`, each with a
  chunk length at payload offset 32 (max 65536).
- Treating offset 36 as `NFC_DISK` (`2`) was a 1-sector coincidence;
  VDDK repeats the byte length there.
- Zeros on the wire are real transferred zeros, not a sparse skip.

Replay: `NfcDisk.read` places fragments at the byte offset in the
reply (they may arrive out of order) until `length` bytes are filled.
Proof: `tests/integration/test_nfc_read_write.py` writes a known pattern
(including a 129-sector read that must assemble two fragments) and
checks the bytes that came back.

## Step 10 — Writes: `strace`, then the same IO message

The first writable Open was captured with **`strace`**, not the SSL
hook. After `PROXY`, NBD NFC is ordinary `write` / `read` on the authd
fd (`useSSL=0`). `strace` dumps those buffers as hex without compiling
`sslhook.so`. TLS to vCenter and the authd handshake stay ciphertext
in the same log, so this is only useful once Step 7 has already shown
that NFC is plaintext.

```bash
unset LD_PRELOAD
export LD_LIBRARY_PATH=…/.vddk
strace -f -x -s 2048 \
  -e trace=write,writev,send,sendto,sendmsg \
  -o /tmp/vddk_write.strace \
  python docs/probing_samples/vddk_write_trace.py
```

| Flag                 | Why                                                                                  |
| -------------------- | ------------------------------------------------------------------------------------ |
| `-f`                 | VDDK I/O runs on worker threads; without it the NFC `write` is missing               |
| `-x`                 | Hex, so AIO magic and binary payloads are searchable                                 |
| `-s 2048`            | Fits a 264-byte classic frame plus a 44-byte IO header and one 512-byte sector. Truncates 64 KiB extras; use the hook for those |
| `-e trace=write,…`   | Drop `open`/`mmap` noise. Still includes Python logging writes                       |

Parse offline: search for AIO magic `7a da 00 a1` (little-endian
`0xA100DA7A`), then keep the fd that also sent 264-byte frames or
`PROXY`. That stream showed:

- `OPEN_FILE` flags `0x1a` (read-write), not the read-only `0x1e`
- IO direction `0` at payload offset 8 (read is `1`)
- 44-byte IO header and the sector extra in **one** `write`

Later write captures (64 KiB fragments, FastLZ) used the port-902
`write`/`read` hook instead, because `-s` would clip the extra. To keep
pyVmomi’s TLS off that hook, the temp VM was created in a **separate
process** and the `LabEnv` was pickled:

```python
# Process A: no LD_PRELOAD (VIM login + CreateVM)
lab = create_lab_vm()
with open("/tmp/vddk-write-wire-lab.pkl", "wb") as f:
    pickle.dump(lab, f)

# Process B: LD_PRELOAD=/tmp/sslhook.so, SSLHOOK_LOG=…
with open("/tmp/vddk-write-wire-lab.pkl", "rb") as f:
    lab = pickle.load(f)
# VixDiskLib_ConnectEx / Open / Write on lab.disk_path
```

Pickles lived under `/tmp` only. They contain lab host, credentials,
and the VM moref; do not commit them. Destroy the VM in an unhooked
process after the capture (`destroy_lab_vm`).

`VixDiskLib_Write` uses the same 44-byte `NFC_AIO_MSG_IO` layout as
read. The direction field at offset 8 is `0` instead of `1`, and the
sector bytes are sent after the payload (like the path on
`OPEN_FILE`). Writable `OPEN_FILE` flags are `0x1a` (the captured
read-only flags `0x1e` with bit `0x04` cleared, matching
`VIXDISKLIB_FLAG_OPEN_READ_ONLY` in `.vddk/vixDiskLib.h`).

Writable `OPEN_FILE` still failed with `VIX_E_FILE_READ_ONLY` until
the ticket switched from `NfcGetVmFiles` to `NfcRandomAccessOpenDisk`
(`libvim-types.so`: vmodl `randomAccessOpen` ↔ WSDL
`NfcRandomAccessOpenDisk`). Integration tests create a temporary empty
10 GiB VM for the run so writes cannot land on other lab disks.

A write larger than 64 KiB is one AIO `opId` with several type-7
request fragments (same layout as read *replies*: total length, then
fragment offset / length) and a single 44-byte ACK. Separate
`VixDiskLib_Write` calls are not coalesced. Header and extra go in one
`sendall`, with `TCP_NODELAY`. Details: `docs/nfc_write.md`. Proof:
write then read in `tests/integration/test_nfc_read_write.py` and the
VDDK cross-check in `tests/integration/test_crosscheck.py`.

## Step 11 — NBDSSL: second TLS after `PROXY vpxa-nfcssl`

VDDK strings name `nbdssl`, `vpxa-nfcssl://`, and `ha-nfcssl`. The NFC
ticket SOAP call is unchanged (`service` stays `vpxa-nfc`). Transport is
an authd/client choice:

1. Same `SESSION` / `BANNER` / `THUMBPRINT_SHA2 PlainText` as NBD.
2. `PROXY vpxa-nfcssl` → `200 Connect ha-nfcssl`.
3. A new TLS handshake on the **same TCP connection** (not TLS-in-TLS
   and not `THUMBPRINT_SHA2 <sha256>`). The colon thumbprint is still
   `501 Invalid arguments`.
4. Classic NFC handshake type 43 still sends ASCII `PlainText`. I/O
   framing is unchanged.

Replay: `connect_authd(..., nfc_ssl=True)` plus
`nfc_open.wrap_nfcssl_socket`. Sending NFC on the first authd
`SSLSocket` after `ha-nfcssl` fails (`BAD_RECORD_TYPE`); sending
plaintext NFC on the dup'd fd gets EOF. Dup + `wrap_socket` is the
working subset. Proof: `tests/integration/test_nfc_open.py` (`nbdssl`)
and `test_openvixdisklib.py` with `transport_modes="nbdssl"`.

Native `VixDiskLib_ConnectEx(..., transport_modes="nbdssl")` through
the old `VixDiskLibConnectParams` ctypes struct can still log nbdssl
and then fall back to `vpxa-nfc` / `useSSL=0`. Do not treat that log
line as a wire capture of NFCSSL.

## Step 12 — FASTLZ NBD compression

`VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` (`1 << 5`) is an IO codec,
not an OPEN_FILE bit. Capture VDDK with that flag (NBD + the port-902
`write`/`read` hook):

- Handshake stays `PlainText`. `OPEN_FILE` flags stay `0x1a` / `0x1e`.
- VDDK’s URL is `FASTLZ-vpxa-nfc://…`; `PROXY` is still `vpxa-nfc`.
- IO opcode `uint64` = direction in the low half, compression type in
  the high half (`2` = FastLZ). Offset 32 is uncompressed length;
  offset 36 is compressed extra size when type is 2.
- Incompressible chunks fall back to type `0` and raw extra.
- 64 KiB chunks use FastLZ level 2; smaller chunks use level 1.

Replay: pip `pyfastlz` via `openvixdisklib/fastlz.py` (NFC extra is
raw FastLZ, without the wrapper's 4-byte length prefix) plus `NfcDisk`
compression on each IO. Proof:
`tests/integration/test_nfc_read_write.py` (`fastlz`) and
`tests/perf/test_compare.py`.

## What to write down

After a stage works:

| Document                                | Contents                                      |
| --------------------------------------- | --------------------------------------------- |
| `docs/nfc_auth.md`                      | Ticket SOAP + authd wire format               |
| `docs/nfc_open.md`                      | Classic NFC + AIO open                        |
| `docs/nfc_read.md`                      | AIO IO / `VixDiskLib_Read`                    |
| `docs/nfc_write.md`                     | AIO IO / `VixDiskLib_Write`                   |
| `docs/ssl_hook.md`                      | Capture tool only                             |
| `docs/reverse_engineering_procedure.md` | This procedure (update when the method changes) |

Keep the hook and ctypes driver under `/tmp`. They are not part of
OpenVixDiskLib.

## Next stages (same procedure)

Not yet reversed, same loop as above:

- `DDB_GET` / disk geometry, zlib/skipz compression, encrypted disks
- `NFC_DELTA_DISK`, CBT / `QueryAllocatedBlocks`
- `VixDiskLib_GetInfo` capacity
- Host-switch AIO messages
- Direct ESXi `ha-nfc` without vCenter `vpxa-nfc`