# Reverse-engineering procedure This is the working method used to replace VDDK’s NBD path with a Python implementation. Protocol details live in `docs/nfc_auth.md`, `docs/nfc_open.md`, `docs/nfc_read.md`, and `docs/nfc_write.md`. The capture tool is described in `docs/ssl_hook.md`. This file is the **sequence of steps**, including dead ends, so later NFC work can follow the same loop instead of rediscovering it. Scope so far: `VixDiskLib_ConnectEx` + `VixDiskLib_Open` + `VixDiskLib_Read` + `VixDiskLib_Write` against lab vCenter 8.0.1 / ESXi 8, transports `nbd` and `nbdssl`. Validation method: `tests/integration/` (the session-scoped `lab` fixture creates a temporary empty VM with a 10 GiB disk and destroys it when the pytest session ends). Rule from `AGENTS.md`: reuse pyVmomi for every public VIM operation. Only reimplement what pyVmomi does not expose. ## Loop Each unknown stage (ticket SOAP, authd, NFC binary) went through: 1. **Name it** from VDDK logs and `strings` on the bundled libraries. 2. **See it** on the wire (or prove that tcpdump cannot). 3. **Replay** the smallest working subset in Python against the lab. 4. **Write** findings into a protocol doc and keep the hook out of the library path. Do not skip (2). Log lines such as `SESSIONID` or `useSSL=0` named the wrong wire command until the intercept existed. ## Lab and artifacts | Item | Where / value | | ----------------------- | -------------------------------------------------------------------- | | VDDK 8.0.2 | `.vddk/` (`libvixDiskLib`, `libvddkVimAccess`, `libvim-types`) | | pyVmomi | `.venv` | | Known-good VDDK client | `tests/integration/test_vddk.py` / `tests/integration/vixdisklib.py` | | Verbose NFC logs | `vixDiskLib.nfc.LogLevel=4` in a temp VDDK config | | ctypes capture drivers | `docs/probing_samples/` (not library code) | | SSL / `write` hook | `/tmp/sslhook.c` → `/tmp/sslhook.so` | | Pickled `LabEnv` | `/tmp/vddk-write-wire-lab.pkl` during hooked captures only | Always set `LD_LIBRARY_PATH` to `.vddk/` so VDDK uses its own `libssl.so.3`. Unset `LD_PRELOAD` before running the Python replacement; a leftover `write` hook will crash pyVmomi’s TLS. ## Tools tcpdump was the first capture attempt and is the wrong tool for TLS stages (Step 3). Everything that actually produced protocol bytes or names is in this table. | Tool | What it was used for | Limitation | | ---------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------ | | tcpdump on 443 / 902 | Prove VDDK talks to vCenter then ESXi:902; see TLS record sizes | Ciphertext only: no SOAP, authd lines, or NFC headers | | `strings -a` on `.vddk/*.so` | Candidate tokens (`SESSION`, `NfcGetVmFiles`, `NFC_AIO_MSG_*`) | Not command order, spacing, or replies | | `nm -D` / `objdump -T` | Which library imports `SSL_write` vs `write`; exported APIs | Not wire layout | | VDDK `vixDiskLib.nfc.LogLevel=4` | Function names and AIO `opId` / `type` / `size` to label a frame | Not magic numbers, path placement, or `BANNER \r\n` | | `LD_PRELOAD` SSL / `write` hook | Plaintext of SOAP, authd, and (after PROXY) NFC on fd 902 | Must not stay on the replacement process; `docs/ssl_hook.md` | | `strace -f -x` on `write` / `send*` | First writable NFC capture without rebuilding the hook (Step 10) | Noisy; TLS still opaque; `-s` truncates large extras | | `pickle` of `LabEnv` | Create the temp VM unhooked, then load it under the hook | `/tmp` only; never commit pickles (lab host and credentials) | | ctypes drivers in `docs/probing_samples/` | Repeatable `ConnectEx` / `Open` / `Read` / `Write` under capture | Not library code | `ltrace` was considered for OpenSSL and libc `write`. It was not used: VDDK is stripped enough that `strace` on syscalls plus the `LD_PRELOAD` hook were enough. An ESXi impersonator (`AGENTS.md`) was also not needed; the lab already answered VDDK. ## Step 1 — Map the public VDDK calls `tests/integration/test_vddk.py` is the specification of what “success” looks like: login, open the temporary VM’s VMDK, write a known pattern, read it back. Turn on VDDK verbose logging around `InitEx` / `ConnectEx` / `Open`. The logs split the work that the Python API hides: - `ConnectEx` → VIM login only. - `Open` → NFC ticket, authd, NFC handshake, AIO open, then I/O. - `transport_modes=nbd` → URL form `vpxa-nfc://[ds] path.vmdk@esxi:902`. - `snapshot_ref` does not appear on the ticket SOAP call. That mapping is the table at the top of `docs/nfc_auth.md`. It tells you which stage to reverse next and which arguments belong there (VM moref on the ticket, VMDK path on NFC `OPEN_FILE`). ## Step 2 — Strings and pyVmomi before any capture `strings -a` on `.vddk/*.so` produced candidate tokens before a single packet was decoded: - SOAP: `NfcService`, `NfcGetVmFiles`, `nfcService`, `ha-nfc`, `HostServiceTicket`. - authd: `SESSION`, `BANNER`, `THUMBPRINT_SHA2`, `PROXY`, `USER`, `PASS`, `SSL Required`. - NFC: `NFC_HANDSHAKE`, `NFC_CONNECTION_DATA`, `NFC_AIO_MSG_*`, `NFC_DISK`. Then check whether pyVmomi already has the type: ```python from pyVmomi import vim hasattr(vim, "NfcService") # False hasattr(vim, "HostServiceTicket") # True ``` `ServiceManager.QueryServiceList` on the live vCenter does **not** list NFC. `GET /sdk/nfcServiceVersions.xml` does (`urn:nfc` 7.0.3.2). The moref is hardcoded in VDDK (`nfcService` on vCenter, `ha-nfc` on ESXi). Anything public (`SmartConnect`, `vim.VirtualMachine`, `HostServiceTicket`) stays in pyVmomi. Missing managed types are registered with `CreateManagedType` on the same SOAP stub so cookies and serialization are not reimplemented. ## Step 3 — Confirm tcpdump is the wrong tool for TLS stages tcpdump on 443 and 902 shows TLS records only. That is enough to prove “something talks to vCenter then to ESXi:902”, and not enough for SOAP bodies, authd lines, or NFC headers. VDDK logs name functions and AIO `opId` / `type` / `size`. They do not give magic numbers, path placement, or command spacing (`BANNER \r\n`). An ESXi-impersonating service was considered (`AGENTS.md`) and not needed: the lab answers VDDK, so capturing the real client is simpler than simulating the server. ## Step 4 — Interpose OpenSSL (authd and SOAP) VDDK 8.0.2 still calls `SSL_write` / `SSL_read`. A small `LD_PRELOAD` library logs those buffers as hex, tagged with the `SSL *` pointer. Run a minimal ctypes program (`InitEx`, `ConnectEx`, `Open`, `Read`) under: ```bash export LD_LIBRARY_PATH=…/.vddk export LD_PRELOAD=/tmp/sslhook.so export SSLHOOK_LOG=/tmp/sslhook-open.log python /tmp/vddk_open_trace.py ``` Parse offline: 1. Concatenate adjacent same-direction records (authd `SSL_read` is often one byte). 2. Split by `SSL *`. vCenter HTTPS contains `POST /sdk` and SOAP. ESXi:902 contains `SESSION` / `PROXY`. 3. An early hook without the pointer mixed both streams; always tag. The vCenter stream identified the ticket as `NfcGetVmFiles` with moref `nfcService` and `xmlns="urn:vim25"` (not a guess from strings alone). The ESXi stream gave the authd command order, including the trailing space on `BANNER` and `THUMBPRINT_SHA2 PlainText`. Hook implementation notes: `docs/ssl_hook.md`. ## Step 5 — Probe SOAP, then replay only what VDDK sends With a SmartConnect session, raw SOAP posts were used to learn parameter names and which moref vCenter accepts: - `ha-nfc` on vCenter → `ManagedObjectNotFound`. - `NfcGetServerNfcLibVersion` without `hostForAccess` → invalid argument; with a host moref → `11`. - `NfcGetVmFiles(vm)` → `HostServiceTicket` (VDDK read-only Open). - `NfcRandomAccessOpenReadonly(vm, diskDeviceKey, host)` → same ticket type, disk-scoped read. - `NfcRandomAccessOpenDisk(vm, diskDeviceKey, host)` → writable ticket. `GetVmFiles` plus `OPEN_FILE` flags `0x1a` fails with `VIX_E_FILE_READ_ONLY` (`0x0b`). The replacement registers those methods and calls them through pyVmomi. It does not ship a hand-rolled SOAP client for login or tickets. ## Step 6 — Probe authd; record dead ends Plaintext banner (`220 … SSL Required`), then `ssl.wrap_socket`. Commands before TLS drop the connection. vCenter UID/password are **not** sent to port 902. Attempts that failed and must not be retried for this ticket type: | Attempt | Result | | ------------------------------------- | ------------------------------------- | | `USER` / `PASS` (vCenter account) | `530 Login incorrect` | | `USER` / `PASS` with `sessionId` | `530 Login incorrect` | | `SESSIONID ` | `530 Please login with USER and PASS` | | `CONNECT_VPXA` after TLS | `530 Please login with USER and PASS` | | Wait for a reply after `SESSION` | Hang until `BANNER` / `PROXY` follow | | `THUMBPRINT_SHA2` with SHA-1 digest | `501 Invalid arguments` | The working sequence is in `docs/nfc_auth.md`. `useSSL=0` in the VDDK log means skip a **second** NFCSSL wrap, not skip TLS on 902. Replay: `openvixdisklib/nfc_auth.py` / `tests/integration/test_nfc_auth.py`. Stop at `200 Connect`. ## Step 7 — NFC binary: extend the hook to `write` / `read` After `PROXY`, `SSL_write` on the ESXi `SSL *` goes silent. VDDK logs `useSSL=0` / “plain-text connection is deprecated” and then NFC function names. The bytes are `write(SSL_get_fd(ssl), …)` / `read` on peer port 902. The hook was extended to those syscalls, filtered with `getpeername` port 902, and mutex-locked (VDDK is multi-threaded). Skip TLS records (`16 03` / `17 03`) left over from the authd phase. Correlate each frame with the verbose log line that has the same `type` and `size` (`NfcAioSendMessage: opId = … type = … size = …`). Name the types from the consecutive `NFC_AIO_MSG_*` string table in `libvixDiskLib.so`. Lengths 4 and 7 on the connection-data message are the ASCII strings `vddk` and `nbdmode` sent in the next two writes. Classic NFC uses a 264-byte padded struct; AIO uses a 16-byte header (`magic 0xA100DA7A`) plus payload; path / DDB key / sector data are extra writes not included in `size`. `strace` is a usable second view of this same plaintext NFC path when the hook is not loaded. It cannot replace the hook for TLS (authd and SOAP). How it was run, and why pickle sits between VM create and the hooked VDDK process, is in Step 10. ## Step 8 — Replay the smallest subset, then compare to VDDK Python must **dup the authd fd** and send NFC as raw TCP. `SSLSocket.send` would encrypt; `unwrap()` would `SSL_shutdown`. VDDK does neither. `openvixdisklib/nfc_open.py` replays handshake + AIO `OPEN_SESSION` / sockopts / resource pool / `OPEN_FILE`. VDDK’s extra `DDB_GET` keys were omitted once a file handle was enough to read. Proof of open: `tests/integration/test_nfc_open.py`. Do not copy every VDDK message. Copy what the server requires for the Python API you are replacing. ## Step 9 — Vary `VixDiskLib_Read` until IO fields stop moving A single-sector read is not enough to decode `NFC_AIO_MSG_IO`. Drive VDDK with several `(startSector, numSectors)` pairs in one Open (including `n=128` = 64 KiB and `n=129`) under the `write`/`read` hook. What that comparison showed: - Wire units are bytes (`offset = start * 512`, `length = n * 512`). - Request size stays 44; data is extra after the payload. - VDDK sends **one** request even when `length > 65536`. The server replies with several type-7 messages that share `opId`, each with a chunk length at payload offset 32 (max 65536). - Treating offset 36 as `NFC_DISK` (`2`) was a 1-sector coincidence; VDDK repeats the byte length there. - Zeros on the wire are real transferred zeros, not a sparse skip. Replay: `NfcDisk.read` places fragments at the byte offset in the reply (they may arrive out of order) until `length` bytes are filled. Proof: `tests/integration/test_nfc_read_write.py` writes a known pattern (including a 129-sector read that must assemble two fragments) and checks the bytes that came back. ## Step 10 — Writes: `strace`, then the same IO message The first writable Open was captured with **`strace`**, not the SSL hook. After `PROXY`, NBD NFC is ordinary `write` / `read` on the authd fd (`useSSL=0`). `strace` dumps those buffers as hex without compiling `sslhook.so`. TLS to vCenter and the authd handshake stay ciphertext in the same log, so this is only useful once Step 7 has already shown that NFC is plaintext. ```bash unset LD_PRELOAD export LD_LIBRARY_PATH=…/.vddk strace -f -x -s 2048 \ -e trace=write,writev,send,sendto,sendmsg \ -o /tmp/vddk_write.strace \ python docs/probing_samples/vddk_write_trace.py ``` | Flag | Why | | -------------------- | ------------------------------------------------------------------------------------ | | `-f` | VDDK I/O runs on worker threads; without it the NFC `write` is missing | | `-x` | Hex, so AIO magic and binary payloads are searchable | | `-s 2048` | Fits a 264-byte classic frame plus a 44-byte IO header and one 512-byte sector. Truncates 64 KiB extras; use the hook for those | | `-e trace=write,…` | Drop `open`/`mmap` noise. Still includes Python logging writes | Parse offline: search for AIO magic `7a da 00 a1` (little-endian `0xA100DA7A`), then keep the fd that also sent 264-byte frames or `PROXY`. That stream showed: - `OPEN_FILE` flags `0x1a` (read-write), not the read-only `0x1e` - IO direction `0` at payload offset 8 (read is `1`) - 44-byte IO header and the sector extra in **one** `write` Later write captures (64 KiB fragments, FastLZ) used the port-902 `write`/`read` hook instead, because `-s` would clip the extra. To keep pyVmomi’s TLS off that hook, the temp VM was created in a **separate process** and the `LabEnv` was pickled: ```python # Process A: no LD_PRELOAD (VIM login + CreateVM) lab = create_lab_vm() with open("/tmp/vddk-write-wire-lab.pkl", "wb") as f: pickle.dump(lab, f) # Process B: LD_PRELOAD=/tmp/sslhook.so, SSLHOOK_LOG=… with open("/tmp/vddk-write-wire-lab.pkl", "rb") as f: lab = pickle.load(f) # VixDiskLib_ConnectEx / Open / Write on lab.disk_path ``` Pickles lived under `/tmp` only. They contain lab host, credentials, and the VM moref; do not commit them. Destroy the VM in an unhooked process after the capture (`destroy_lab_vm`). `VixDiskLib_Write` uses the same 44-byte `NFC_AIO_MSG_IO` layout as read. The direction field at offset 8 is `0` instead of `1`, and the sector bytes are sent after the payload (like the path on `OPEN_FILE`). Writable `OPEN_FILE` flags are `0x1a` (the captured read-only flags `0x1e` with bit `0x04` cleared, matching `VIXDISKLIB_FLAG_OPEN_READ_ONLY` in `.vddk/vixDiskLib.h`). Writable `OPEN_FILE` still failed with `VIX_E_FILE_READ_ONLY` until the ticket switched from `NfcGetVmFiles` to `NfcRandomAccessOpenDisk` (`libvim-types.so`: vmodl `randomAccessOpen` ↔ WSDL `NfcRandomAccessOpenDisk`). Integration tests create a temporary empty 10 GiB VM for the run so writes cannot land on other lab disks. A write larger than 64 KiB is one AIO `opId` with several type-7 request fragments (same layout as read *replies*: total length, then fragment offset / length) and a single 44-byte ACK. Separate `VixDiskLib_Write` calls are not coalesced. Header and extra go in one `sendall`, with `TCP_NODELAY`. Details: `docs/nfc_write.md`. Proof: write then read in `tests/integration/test_nfc_read_write.py` and the VDDK cross-check in `tests/integration/test_crosscheck.py`. ## Step 11 — NBDSSL: second TLS after `PROXY vpxa-nfcssl` VDDK strings name `nbdssl`, `vpxa-nfcssl://`, and `ha-nfcssl`. The NFC ticket SOAP call is unchanged (`service` stays `vpxa-nfc`). Transport is an authd/client choice: 1. Same `SESSION` / `BANNER` / `THUMBPRINT_SHA2 PlainText` as NBD. 2. `PROXY vpxa-nfcssl` → `200 Connect ha-nfcssl`. 3. A new TLS handshake on the **same TCP connection** (not TLS-in-TLS and not `THUMBPRINT_SHA2 `). The colon thumbprint is still `501 Invalid arguments`. 4. Classic NFC handshake type 43 still sends ASCII `PlainText`. I/O framing is unchanged. Replay: `connect_authd(..., nfc_ssl=True)` plus `nfc_open.wrap_nfcssl_socket`. Sending NFC on the first authd `SSLSocket` after `ha-nfcssl` fails (`BAD_RECORD_TYPE`); sending plaintext NFC on the dup'd fd gets EOF. Dup + `wrap_socket` is the working subset. Proof: `tests/integration/test_nfc_open.py` (`nbdssl`) and `test_openvixdisklib.py` with `transport_modes="nbdssl"`. Native `VixDiskLib_ConnectEx(..., transport_modes="nbdssl")` through the old `VixDiskLibConnectParams` ctypes struct can still log nbdssl and then fall back to `vpxa-nfc` / `useSSL=0`. Do not treat that log line as a wire capture of NFCSSL. ## Step 12 — FASTLZ NBD compression `VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` (`1 << 5`) is an IO codec, not an OPEN_FILE bit. Capture VDDK with that flag (NBD + the port-902 `write`/`read` hook): - Handshake stays `PlainText`. `OPEN_FILE` flags stay `0x1a` / `0x1e`. - VDDK’s URL is `FASTLZ-vpxa-nfc://…`; `PROXY` is still `vpxa-nfc`. - IO opcode `uint64` = direction in the low half, compression type in the high half (`2` = FastLZ). Offset 32 is uncompressed length; offset 36 is compressed extra size when type is 2. - Incompressible chunks fall back to type `0` and raw extra. - 64 KiB chunks use FastLZ level 2; smaller chunks use level 1. Replay: pip `pyfastlz` via `openvixdisklib/fastlz.py` (NFC extra is raw FastLZ, without the wrapper's 4-byte length prefix) plus `NfcDisk` compression on each IO. Proof: `tests/integration/test_nfc_read_write.py` (`fastlz`) and `tests/perf/test_compare.py`. ## What to write down After a stage works: | Document | Contents | | --------------------------------------- | --------------------------------------------- | | `docs/nfc_auth.md` | Ticket SOAP + authd wire format | | `docs/nfc_open.md` | Classic NFC + AIO open | | `docs/nfc_read.md` | AIO IO / `VixDiskLib_Read` | | `docs/nfc_write.md` | AIO IO / `VixDiskLib_Write` | | `docs/ssl_hook.md` | Capture tool only | | `docs/reverse_engineering_procedure.md` | This procedure (update when the method changes) | Keep the hook and ctypes driver under `/tmp`. They are not part of the replacement library. ## Next stages (same procedure) Not yet reversed, same loop as above: - `DDB_GET` / disk geometry, zlib/skipz compression, encrypted disks - `NFC_DELTA_DISK`, CBT / `QueryAllocatedBlocks` - `VixDiskLib_GetInfo` capacity - Host-switch AIO messages - Direct ESXi `ha-nfc` without vCenter `vpxa-nfc`