OpenVixDiskLib/docs/reverse_engineering_procedure.md

Reverse-engineering procedure

This is the working method used to replace VDDK’s NBD path with OpenVixDiskLib. Protocol details live in docs/nfc_auth.md, docs/nfc_open.md, docs/nfc_read.md, and docs/nfc_write.md. The capture tool is described in docs/ssl_hook.md. This file is the sequence of steps, including dead ends, so later NFC work can follow the same loop instead of rediscovering it.

Scope so far: VixDiskLib_ConnectEx + VixDiskLib_Open + VixDiskLib_Read + VixDiskLib_Write against lab vCenter 8.0.1 / ESXi 8, transports nbd and nbdssl. Validation method: tests/integration/ (the session-scoped lab fixture creates a temporary empty VM with a 10 GiB disk and destroys it when the pytest session ends).

Rule from AGENTS.md: reuse pyVmomi for every public VIM operation. Only reimplement what pyVmomi does not expose.

Loop

Each unknown stage (ticket SOAP, authd, NFC binary) went through:

1. Name it from VDDK logs and strings on the bundled libraries.
2. See it on the wire (or prove that tcpdump cannot).
3. Replay the smallest working subset in Python against the lab.
4. Write findings into a protocol doc and keep the hook out of the library path.

Do not skip (2). Log lines such as SESSIONID or useSSL=0 named the wrong wire command until the intercept existed.

Lab and artifacts

Item	Where / value
VDDK 8.0.2	.vddk/ (libvixDiskLib, libvddkVimAccess, libvim-types)
pyVmomi	.venv
Known-good VDDK client	tests/integration/test_vddk.py / tests/integration/vixdisklib.py
Verbose NFC logs	vixDiskLib.nfc.LogLevel=4 in a temp VDDK config
ctypes capture drivers	docs/probing_samples/ (not library code)
SSL / write hook	/tmp/sslhook.c → /tmp/sslhook.so
Pickled LabEnv	/tmp/vddk-write-wire-lab.pkl during hooked captures only

Always set LD_LIBRARY_PATH to .vddk/ so VDDK uses its own libssl.so.3. Unset LD_PRELOAD before running OpenVixDiskLib; a leftover write hook will crash pyVmomi’s TLS.

Tools

tcpdump was the first capture attempt and is the wrong tool for TLS stages (Step 3). Everything that actually produced protocol bytes or names is in this table.

Tool	What it was used for	Limitation
tcpdump on 443 / 902	Prove VDDK talks to vCenter then ESXi:902; see TLS record sizes	Ciphertext only: no SOAP, authd lines, or NFC headers
strings -a on .vddk/*.so	Candidate tokens (SESSION, NfcGetVmFiles, NFC_AIO_MSG_*)	Not command order, spacing, or replies
nm -D / objdump -T	Which library imports SSL_write vs write; exported APIs	Not wire layout
VDDK vixDiskLib.nfc.LogLevel=4	Function names and AIO opId / type / size to label a frame	Not magic numbers, path placement, or BANNER \r\n
LD_PRELOAD SSL / write hook	Plaintext of SOAP, authd, and (after PROXY) NFC on fd 902	Must not stay on the OpenVixDiskLib process; docs/ssl_hook.md
strace -f -x on write / send*	First writable NFC capture without rebuilding the hook (Step 10)	Noisy; TLS still opaque; -s truncates large extras
pickle of LabEnv	Create the temp VM unhooked, then load it under the hook	/tmp only; never commit pickles (lab host and credentials)
ctypes drivers in docs/probing_samples/	Repeatable ConnectEx / Open / Read / Write under capture	Not library code

ltrace was considered for OpenSSL and libc write. It was not used: VDDK is stripped enough that strace on syscalls plus the LD_PRELOAD hook were enough. An ESXi impersonator (AGENTS.md) was also not needed; the lab already answered VDDK.

Step 1 — Map the public VDDK calls

tests/integration/test_vddk.py is the specification of what “success” looks like: login, open the temporary VM’s VMDK, write a known pattern, read it back.

Turn on VDDK verbose logging around InitEx / ConnectEx / Open. The logs split the work that the Python API hides:

• ConnectEx → VIM login only.
• Open → NFC ticket, authd, NFC handshake, AIO open, then I/O.
• transport_modes=nbd → URL form vpxa-nfc://[ds] path.vmdk@esxi:902.
• snapshot_ref does not appear on the ticket SOAP call.

That mapping is the table at the top of docs/nfc_auth.md. It tells you which stage to reverse next and which arguments belong there (VM moref on the ticket, VMDK path on NFC OPEN_FILE).

Step 2 — Strings and pyVmomi before any capture

strings -a on .vddk/*.so produced candidate tokens before a single packet was decoded:

• SOAP: NfcService, NfcGetVmFiles, nfcService, ha-nfc, HostServiceTicket.
• authd: SESSION, BANNER, THUMBPRINT_SHA2, PROXY, USER, PASS, SSL Required.
• NFC: NFC_HANDSHAKE, NFC_CONNECTION_DATA, NFC_AIO_MSG_*, NFC_DISK.

Then check whether pyVmomi already has the type:

from pyVmomi import vim
hasattr(vim, "NfcService")   # False
hasattr(vim, "HostServiceTicket")  # True

ServiceManager.QueryServiceList on the live vCenter does not list NFC. GET /sdk/nfcServiceVersions.xml does (urn:nfc 7.0.3.2). The moref is hardcoded in VDDK (nfcService on vCenter, ha-nfc on ESXi).

Anything public (SmartConnect, vim.VirtualMachine, HostServiceTicket) stays in pyVmomi. Missing managed types are registered with CreateManagedType on the same SOAP stub so cookies and serialization are not reimplemented.

Step 3 — Confirm tcpdump is the wrong tool for TLS stages

tcpdump on 443 and 902 shows TLS records only. That is enough to prove “something talks to vCenter then to ESXi:902”, and not enough for SOAP bodies, authd lines, or NFC headers.

VDDK logs name functions and AIO opId / type / size. They do not give magic numbers, path placement, or command spacing (BANNER \r\n).

An ESXi-impersonating service was considered (AGENTS.md) and not needed: the lab answers VDDK, so capturing the real client is simpler than simulating the server.

Step 4 — Interpose OpenSSL (authd and SOAP)

VDDK 8.0.2 still calls SSL_write / SSL_read. A small LD_PRELOAD library logs those buffers as hex, tagged with the SSL * pointer.

Run a minimal ctypes program (InitEx, ConnectEx, Open, Read) under:

export LD_LIBRARY_PATH=…/.vddk
export LD_PRELOAD=/tmp/sslhook.so
export SSLHOOK_LOG=/tmp/sslhook-open.log
python /tmp/vddk_open_trace.py

Parse offline:

1. Concatenate adjacent same-direction records (authd SSL_read is often one byte).
2. Split by SSL *. vCenter HTTPS contains POST /sdk and SOAP. ESXi:902 contains SESSION / PROXY.
3. An early hook without the pointer mixed both streams; always tag.

The vCenter stream identified the ticket as NfcGetVmFiles with moref nfcService and xmlns="urn:vim25" (not a guess from strings alone). The ESXi stream gave the authd command order, including the trailing space on BANNER and THUMBPRINT_SHA2 PlainText.

Hook implementation notes: docs/ssl_hook.md.

Step 5 — Probe SOAP, then replay only what VDDK sends

With a SmartConnect session, raw SOAP posts were used to learn parameter names and which moref vCenter accepts:

• ha-nfc on vCenter → ManagedObjectNotFound.
• NfcGetServerNfcLibVersion without hostForAccess → invalid argument; with a host moref → 11.
• NfcGetVmFiles(vm) → HostServiceTicket (VDDK read-only Open).
• NfcRandomAccessOpenReadonly(vm, diskDeviceKey, host) → same ticket type, disk-scoped read.
• NfcRandomAccessOpenDisk(vm, diskDeviceKey, host) → writable ticket. GetVmFiles plus OPEN_FILE flags 0x1a fails with VIX_E_FILE_READ_ONLY (0x0b).

OpenVixDiskLib registers those methods and calls them through pyVmomi. It does not ship a hand-rolled SOAP client for login or tickets.

Step 6 — Probe authd; record dead ends

Plaintext banner (220 … SSL Required), then ssl.wrap_socket. Commands before TLS drop the connection.

vCenter UID/password are not sent to port 902. Attempts that failed and must not be retried for this ticket type:

Attempt	Result
USER / PASS (vCenter account)	530 Login incorrect
USER / PASS with sessionId	530 Login incorrect
SESSIONID <sessionId>	530 Please login with USER and PASS
CONNECT_VPXA after TLS	530 Please login with USER and PASS
Wait for a reply after SESSION	Hang until BANNER / PROXY follow
THUMBPRINT_SHA2 with SHA-1 digest	501 Invalid arguments

The working sequence is in docs/nfc_auth.md. useSSL=0 in the VDDK log means skip a second NFCSSL wrap, not skip TLS on 902.

Replay: openvixdisklib/nfc_auth.py / tests/integration/test_nfc_auth.py. Stop at 200 Connect.

Step 7 — NFC binary: extend the hook to write / read

After PROXY, SSL_write on the ESXi SSL * goes silent. VDDK logs useSSL=0 / “plain-text connection is deprecated” and then NFC function names. The bytes are write(SSL_get_fd(ssl), …) / read on peer port 902.

The hook was extended to those syscalls, filtered with getpeername port 902, and mutex-locked (VDDK is multi-threaded). Skip TLS records (16 03 / 17 03) left over from the authd phase.

Correlate each frame with the verbose log line that has the same type and size (NfcAioSendMessage: opId = … type = … size = …). Name the types from the consecutive NFC_AIO_MSG_* string table in libvixDiskLib.so. Lengths 4 and 7 on the connection-data message are the ASCII strings vddk and nbdmode sent in the next two writes.

Classic NFC uses a 264-byte padded struct; AIO uses a 16-byte header (magic 0xA100DA7A) plus payload; path / DDB key / sector data are extra writes not included in size.

strace is a usable second view of this same plaintext NFC path when the hook is not loaded. It cannot replace the hook for TLS (authd and SOAP). How it was run, and why pickle sits between VM create and the hooked VDDK process, is in Step 10.

Step 8 — Replay the smallest subset, then compare to VDDK

Python must dup the authd fd and send NFC as raw TCP. SSLSocket.send would encrypt; unwrap() would SSL_shutdown. VDDK does neither.

OpenVixDiskLib (openvixdisklib/nfc_open.py) replays handshake + AIO OPEN_SESSION / sockopts / resource pool / OPEN_FILE. VDDK’s extra DDB_GET keys were omitted once a file handle was enough to read. Proof of open: tests/integration/test_nfc_open.py.

Do not copy every VDDK message. Copy what the server requires for the Python API you are replacing.

Step 9 — Vary VixDiskLib_Read until IO fields stop moving

A single-sector read is not enough to decode NFC_AIO_MSG_IO. Drive VDDK with several (startSector, numSectors) pairs in one Open (including n=128 = 64 KiB and n=129) under the write/read hook.

What that comparison showed:

• Wire units are bytes (offset = start * 512, length = n * 512).
• Request size stays 44; data is extra after the payload.
• VDDK sends one request even when length > 65536. The server replies with several type-7 messages that share opId, each with a chunk length at payload offset 32 (max = OPEN_SESSION bufSize; default 65536). vixDiskLib.nfcAio.Session.BufSizeIn64KB=32 makes that 2 MiB (docs/probing_samples/vddk_aio_bufsize_probe.py).
• Treating offset 36 as NFC_DISK (2) was a 1-sector coincidence; VDDK repeats the byte length there.
• Zeros on the wire are real transferred zeros, not a sparse skip.

Replay: NfcDisk.read places fragments at the byte offset in the reply (they may arrive out of order) until length bytes are filled. Proof: tests/integration/test_nfc_read_write.py writes a known pattern (including a 129-sector read that must assemble two fragments) and checks the bytes that came back.

Step 10 — Writes: strace, then the same IO message

The first writable Open was captured with strace, not the SSL hook. After PROXY, NBD NFC is ordinary write / read on the authd fd (useSSL=0). strace dumps those buffers as hex without compiling sslhook.so. TLS to vCenter and the authd handshake stay ciphertext in the same log, so this is only useful once Step 7 has already shown that NFC is plaintext.

unset LD_PRELOAD
export LD_LIBRARY_PATH=…/.vddk
strace -f -x -s 2048 \
  -e trace=write,writev,send,sendto,sendmsg \
  -o /tmp/vddk_write.strace \
  python docs/probing_samples/vddk_write_trace.py

Flag	Why
-f	VDDK I/O runs on worker threads; without it the NFC write is missing
-x	Hex, so AIO magic and binary payloads are searchable
-s 2048	Fits a 264-byte classic frame plus a 44-byte IO header and one 512-byte sector. Truncates 64 KiB extras; use the hook for those
-e trace=write,…	Drop open/mmap noise. Still includes Python logging writes

Parse offline: search for AIO magic 7a da 00 a1 (little-endian 0xA100DA7A), then keep the fd that also sent 264-byte frames or PROXY. That stream showed:

• OPEN_FILE flags 0x1a (read-write), not the read-only 0x1e
• IO direction 0 at payload offset 8 (read is 1)
• 44-byte IO header and the sector extra in one write

Later write captures (64 KiB fragments, FastLZ) used the port-902 write/read hook instead, because -s would clip the extra. To keep pyVmomi’s TLS off that hook, the temp VM was created in a separate process and the LabEnv was pickled:

# Process A: no LD_PRELOAD (VIM login + CreateVM)
lab = create_lab_vm()
with open("/tmp/vddk-write-wire-lab.pkl", "wb") as f:
    pickle.dump(lab, f)

# Process B: LD_PRELOAD=/tmp/sslhook.so, SSLHOOK_LOG=…
with open("/tmp/vddk-write-wire-lab.pkl", "rb") as f:
    lab = pickle.load(f)
# VixDiskLib_ConnectEx / Open / Write on lab.disk_path

Pickles lived under /tmp only. They contain lab host, credentials, and the VM moref; do not commit them. Destroy the VM in an unhooked process after the capture (destroy_lab_vm).

VixDiskLib_Write uses the same 44-byte NFC_AIO_MSG_IO layout as read. The direction field at offset 8 is 0 instead of 1, and the sector bytes are sent after the payload (like the path on OPEN_FILE). Writable OPEN_FILE flags are 0x1a (the captured read-only flags 0x1e with bit 0x04 cleared, matching VIXDISKLIB_FLAG_OPEN_READ_ONLY in .vddk/vixDiskLib.h).

Writable OPEN_FILE still failed with VIX_E_FILE_READ_ONLY until the ticket switched from NfcGetVmFiles to NfcRandomAccessOpenDisk (libvim-types.so: vmodl randomAccessOpen ↔ WSDL NfcRandomAccessOpenDisk). Integration tests create a temporary empty 10 GiB VM for the run so writes cannot land on other lab disks.

A write larger than 64 KiB is one AIO opId with several type-7 request fragments (same layout as read replies: total length, then fragment offset / length) and a single 44-byte ACK. Separate VixDiskLib_Write calls are not coalesced. Header and extra go in one sendall, with TCP_NODELAY. Details: docs/nfc_write.md. Proof: write then read in tests/integration/test_nfc_read_write.py and the VDDK cross-check in tests/integration/test_crosscheck.py.

Step 11 — NBDSSL: second TLS after PROXY vpxa-nfcssl

VDDK strings name nbdssl, vpxa-nfcssl://, and ha-nfcssl. The NFC ticket SOAP call is unchanged (service stays vpxa-nfc). Transport is an authd/client choice:

1. Same SESSION / BANNER / THUMBPRINT_SHA2 PlainText as NBD.
2. PROXY vpxa-nfcssl → 200 Connect ha-nfcssl.
3. A new TLS handshake on the same TCP connection (not TLS-in-TLS and not THUMBPRINT_SHA2 <sha256>). The colon thumbprint is still 501 Invalid arguments.
4. Classic NFC handshake type 43 still sends ASCII PlainText. I/O framing is unchanged.

Replay: connect_authd(..., nfc_ssl=True) plus nfc_open.wrap_nfcssl_socket. Sending NFC on the first authd SSLSocket after ha-nfcssl fails (BAD_RECORD_TYPE); sending plaintext NFC on the dup'd fd gets EOF. Dup + wrap_socket is the working subset. Proof: tests/integration/test_nfc_open.py (nbdssl) and test_openvixdisklib.py with transport_modes="nbdssl".

Native VixDiskLib_ConnectEx(..., transport_modes="nbdssl") through the old VixDiskLibConnectParams ctypes struct can still log nbdssl and then fall back to vpxa-nfc / useSSL=0. Do not treat that log line as a wire capture of NFCSSL.

Step 12 — FASTLZ NBD compression

VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ (1 << 5) is an IO codec, not an OPEN_FILE bit. Capture VDDK with that flag (NBD + the port-902 write/read hook):

• Handshake stays PlainText. OPEN_FILE flags stay 0x1a / 0x1e.
• VDDK’s URL is FASTLZ-vpxa-nfc://…; PROXY is still vpxa-nfc.
• IO opcode uint64 = direction in the low half, compression type in the high half (2 = FastLZ). Offset 32 is uncompressed length; offset 36 is compressed extra size when type is 2.
• Incompressible chunks fall back to type 0 and raw extra.
• 64 KiB chunks use FastLZ level 2; smaller chunks use level 1.

Replay: pip pyfastlz via openvixdisklib/fastlz.py (NFC extra is raw FastLZ, without the wrapper's 4-byte length prefix) plus NfcDisk compression on each IO. Proof: tests/integration/test_nfc_read_write.py (fastlz) and tests/perf/test_compare.py.

What to write down

After a stage works:

Document	Contents
docs/nfc_auth.md	Ticket SOAP + authd wire format
docs/nfc_open.md	Classic NFC + AIO open
docs/nfc_read.md	AIO IO / VixDiskLib_Read
docs/nfc_write.md	AIO IO / VixDiskLib_Write
docs/ssl_hook.md	Capture tool only
docs/reverse_engineering_procedure.md	This procedure (update when the method changes)

Keep the hook and ctypes driver under /tmp. They are not part of OpenVixDiskLib.

Next stages (same procedure)

Not yet reversed, same loop as above:

• DDB_GET / disk geometry, zlib/skipz compression, encrypted disks
• NFC_DELTA_DISK, CBT / QueryAllocatedBlocks
• VixDiskLib_GetInfo capacity
• Host-switch AIO messages
• Direct ESXi ha-nfc without vCenter vpxa-nfc