From 0f45cdf4c1c259f3f83dcb24699842401d4725b8 Mon Sep 17 00:00:00 2001 From: Lucian Petrut Date: Thu, 10 Sep 2026 09:36:47 +0000 Subject: [PATCH] Update the docs to use "OpenVixDiskLib" naming --- README.md | 17 ++++++++++------- docs/nfc_auth.md | 18 +++++++++--------- docs/nfc_open.md | 6 +++--- docs/nfc_read.md | 13 +++++++------ docs/nfc_write.md | 12 ++++++------ docs/reverse_engineering_procedure.md | 22 +++++++++++----------- docs/ssl_hook.md | 9 ++++----- 7 files changed, 50 insertions(+), 47 deletions(-) diff --git a/README.md b/README.md index 4135367..7fab225 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,11 @@ -# openvixdisklib +# OpenVixDiskLib -A Python replacement for VMware VDDK's `vixDiskLib` NBD path. It reads -and writes VMDK contents over vSphere NFC without the proprietary VDDK -SDK. +OpenVixDiskLib is an open-source Python replacement for VMware VDDK's +`VixDiskLib` NBD path. It reads and writes VMDK contents over vSphere +NFC without the proprietary VDDK SDK. + +The Python package is `openvixdisklib` (lowercase, following usual +Python naming). VIM login and inventory use [pyVmomi](https://github.com/vmware/pyvmomi). The NFC ticket, ESXi authd handshake, and disk I/O were reverse-engineered @@ -68,13 +71,13 @@ VDDK-shaped handle. | `openvixdisklib/nfc_open.py` | Classic NFC handshake, AIO open, sector read/write | | `openvixdisklib/fastlz.py` | FastLZ NFC adapter (pip `pyfastlz`) | | `tests/integration/` | Live pytest suite against a lab vCenter | -| `tests/perf/` | Throughput comparison of openvixdisklib vs VDDK | +| `tests/perf/` | Throughput comparison of OpenVixDiskLib vs VDDK | | `tests/stress/` | Repeated connect/open/close leak check | | `tests/integration/vixdisklib.py` | Native VDDK wrapper used only to cross-check | | `docs/` | Protocol notes and reverse-engineering steps | VDDK shared libraries, if present for cross-check, belong in `.vddk/` -(gitignored). They are not required to use `openvixdisklib`. +(gitignored). They are not required to use OpenVixDiskLib. ## Tests @@ -107,7 +110,7 @@ Some tests are marked as ``slow`` and skipped unless you pass ``--runslow``: tox -e integration -- --runslow ``` -Compare write/read throughput of openvixdisklib and native VDDK +Compare write/read throughput of OpenVixDiskLib and native VDDK (`64KiB`, 129-sector, and `32MiB` transfers; `nbdssl` and `nbd`; plain and FastLZ): diff --git a/docs/nfc_auth.md b/docs/nfc_auth.md index aa0a207..e19b515 100644 --- a/docs/nfc_auth.md +++ b/docs/nfc_auth.md @@ -1,7 +1,7 @@ # VDDK NFC authentication This document records how VMware VDDK authenticates for NBD/NFC disk -access, and how the Python replacement in `openvixdisklib/nfc_auth.py` +access, and how OpenVixDiskLib (`openvixdisklib/nfc_auth.py`) reproduces that path. Findings come from VDDK 8.0.2 libraries (`libvixDiskLib`, `libvddkVimAccess`, `libvim-types`), live SOAP calls against vCenter @@ -54,7 +54,7 @@ rather than crafting SOAP. - SOAPAction: `"urn:vim25/8.0.1.0"` (negotiated) VDDK logs this as `Connected to VIM Server` / `Authenticating user` / -`Logged in!`. The Python replacement keeps that `ServiceInstance` and +`Logged in!`. OpenVixDiskLib keeps that `ServiceInstance` and its stub for the ticket call. Direct ESXi login is the same SOAP login against hostd, but the NFC @@ -76,10 +76,10 @@ ships. vCenter still implements it: `ServiceManager.QueryServiceList` does **not** list NFC. The moref is hardcoded in VDDK as `nfcService` (vCenter) or `ha-nfc` (ESXi). -`openvixdisklib/nfc_auth.py` registers the missing type with -`pyVmomi.VmomiSupport.CreateManagedType` and invokes it on the existing -SmartConnect stub, so serialization, cookies, and `HostServiceTicket` -stay in pyVmomi. +OpenVixDiskLib (`openvixdisklib/nfc_auth.py`) registers the missing +type with `pyVmomi.VmomiSupport.CreateManagedType` and invokes it on +the existing SmartConnect stub, so serialization, cookies, and +`HostServiceTicket` stay in pyVmomi. ### Methods VDDK actually calls @@ -253,7 +253,7 @@ are for local ESXi credentials. With a vCenter ticket: argument is not what VDDK sends. The SHA-1 value is for verifying the TLS certificate, not for the `THUMBPRINT_SHA2` command. -## Python replacement +## OpenVixDiskLib | Piece | Module | Reuses pyVmomi? | | -------------------- | ---------------------------------------- | ---------------------------------- | @@ -282,6 +282,6 @@ and asserts an established TLS socket on `ticket.host:ticket.port`. Authentication stops at `200 Connect ha-nfc` (NBD) or `200 Connect ha-nfcssl` (NBDSSL). Opening the VMDK and reading or writing sectors is -documented in `docs/nfc_open.md` and implemented in -`openvixdisklib/nfc_open.py`. The datastore path is consumed there +documented in `docs/nfc_open.md` and implemented in OpenVixDiskLib +(`openvixdisklib/nfc_open.py`). The datastore path is consumed there (and, for writes, as `diskDeviceKey` on the ticket). diff --git a/docs/nfc_open.md b/docs/nfc_open.md index 54ff737..6a72962 100644 --- a/docs/nfc_open.md +++ b/docs/nfc_open.md @@ -1,8 +1,8 @@ # VDDK NFC disk open This document records how VMware VDDK opens a VMDK over NBD/NFC after -the authd handshake in `docs/nfc_auth.md`, and how -`openvixdisklib/nfc_open.py` reproduces that path. Findings come from +the authd handshake in `docs/nfc_auth.md`, and how OpenVixDiskLib +(`openvixdisklib/nfc_open.py`) reproduces that path. Findings come from VDDK 8.0.2 verbose logs (`vixDiskLib.nfc.LogLevel=4`) plus an `LD_PRELOAD` intercept of `write` / `read` on the ESXi:902 file descriptor. Capture method: @@ -203,7 +203,7 @@ and `docs/nfc_write.md`. `NfcDisk.read` / `NfcDisk.write` match `CLOSE_FILE` (handle as `uint64`), `CLOSE_SESSION` (`uint32` 0), then classic type 4 `NFC_SESSION_COMPLETE`. -## Python replacement +## OpenVixDiskLib | Piece | Module | | ----------------------------- | ----------------------------------------------- | diff --git a/docs/nfc_read.md b/docs/nfc_read.md index 7f149bc..2afa3c2 100644 --- a/docs/nfc_read.md +++ b/docs/nfc_read.md @@ -1,9 +1,10 @@ # VDDK NFC disk read This document records how VMware VDDK reads VMDK sectors over NBD/NFC -after the open in `docs/nfc_open.md`, and how `NfcDisk.read` in -`openvixdisklib/nfc_open.py` reproduces `VixDiskLib_Read`. Capture -method: `docs/reverse_engineering_procedure.md`. +after the open in `docs/nfc_open.md`, and how OpenVixDiskLib +(`NfcDisk.read` in `openvixdisklib/nfc_open.py`) reproduces +`VixDiskLib_Read`. Capture method: +`docs/reverse_engineering_procedure.md`. ## Mapping from VDDK @@ -51,8 +52,8 @@ Little-endian, after the usual 16-byte AIO header An earlier guess that offset 36 was `NFC_DISK` (`2`) was wrong: a 1-sector VDDK read puts `512` in both `uint32` length fields. A Python -read that sent `(512, 2, 0)` still worked for one sector; the -replacement now matches VDDK. +read that sent `(512, 2, 0)` still worked for one sector; OpenVixDiskLib +now matches VDDK. `VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` does not change OPEN_FILE flags. The IO opcode at offset 8 is a `uint64`: low 32 bits are still @@ -121,7 +122,7 @@ back. An unwritten region is zeros. Writes use the same 44-byte IO payload with opcode `2`; see `docs/nfc_write.md`. -## Python replacement +## OpenVixDiskLib `NfcDisk.read(start_sector, num_sectors)` in `openvixdisklib/nfc_open.py`. Run: diff --git a/docs/nfc_write.md b/docs/nfc_write.md index e992b15..79993ca 100644 --- a/docs/nfc_write.md +++ b/docs/nfc_write.md @@ -1,7 +1,7 @@ # VDDK NFC disk write -This document records how `NfcDisk.write` in -`openvixdisklib/nfc_open.py` implements `VixDiskLib_Write` over NFC AIO. +This document records how OpenVixDiskLib (`NfcDisk.write` in +`openvixdisklib/nfc_open.py`) implements `VixDiskLib_Write` over NFC AIO. The request layout matches the captured `VixDiskLib_Read` IO message in `docs/nfc_read.md` (write requests use the same fragment fields as read replies). Open flags and the IO direction field were taken @@ -75,7 +75,7 @@ sends type `0` and raw extra. Each fragment is compressed on its own; a 32 MiB FastLZ write is 512 independent FastLZ extras, not one. Sector bytes follow the 44-byte payload and are **not** counted in AIO -`size`. The replacement sends header + payload + extra in one +`size`. OpenVixDiskLib sends header + payload + extra in one `sendall` and sets `TCP_NODELAY` on the NFC socket so a small FastLZ extra is not delayed behind Nagle / delayed ACK. Captured VDDK often uses two `write()`s (`60` then `65536`) for a 64 KiB fragment and @@ -98,7 +98,7 @@ C: type=7 opId=14 size=44 total=66048 dest=65536 chunk=512 + 512 data S: type=7 opId=14 size=44 total=66048 dest=0 chunk=66048 ``` -A 32 MiB write is 512 client fragments and one ACK. The Python client +A 32 MiB write is 512 client fragments and one ACK. OpenVixDiskLib does the same. An earlier attempt that used a distinct `opId` per 64 KiB chunk and a sliding window of 4–512 outstanding IOs was waiting for one reply per chunk; raising the window did not match VDDK @@ -109,7 +109,7 @@ client merge of API writes. OPEN_SESSION is 16 zero bytes both ways, so the logged AIO buffer count of 4 is a VDDK client default (`vixDiskLib.nfcAio.Session.BufCount`), not a server cap. -## Python replacement +## OpenVixDiskLib `NfcDisk.write(start_sector, num_sectors, data)` in `openvixdisklib/nfc_open.py`. `open_disk(..., read_only=False)` selects @@ -120,5 +120,5 @@ flags `0x1a`. The drop-in handle exposes the same shape as VDDK: Integration tests create an empty 10 GiB disk, write known patterns, and read them back (`tests/integration/test_nfc_read_write.py`, `tests/integration/test_openvixdisklib.py`). Cross-check tests write -with VDDK and with the replacement and read with both +with VDDK and with OpenVixDiskLib and read with both (`tests/integration/test_crosscheck.py`). diff --git a/docs/reverse_engineering_procedure.md b/docs/reverse_engineering_procedure.md index 50d19c8..0a179e6 100644 --- a/docs/reverse_engineering_procedure.md +++ b/docs/reverse_engineering_procedure.md @@ -1,7 +1,7 @@ # Reverse-engineering procedure -This is the working method used to replace VDDK’s NBD path with a Python -implementation. Protocol details live in `docs/nfc_auth.md`, +This is the working method used to replace VDDK’s NBD path with +OpenVixDiskLib. Protocol details live in `docs/nfc_auth.md`, `docs/nfc_open.md`, `docs/nfc_read.md`, and `docs/nfc_write.md`. The capture tool is described in `docs/ssl_hook.md`. This file is the **sequence of steps**, including dead ends, so later @@ -42,7 +42,7 @@ wrong wire command until the intercept existed. | Pickled `LabEnv` | `/tmp/vddk-write-wire-lab.pkl` during hooked captures only | Always set `LD_LIBRARY_PATH` to `.vddk/` so VDDK uses its own -`libssl.so.3`. Unset `LD_PRELOAD` before running the Python replacement; +`libssl.so.3`. Unset `LD_PRELOAD` before running OpenVixDiskLib; a leftover `write` hook will crash pyVmomi’s TLS. ## Tools @@ -57,7 +57,7 @@ names is in this table. | `strings -a` on `.vddk/*.so` | Candidate tokens (`SESSION`, `NfcGetVmFiles`, `NFC_AIO_MSG_*`) | Not command order, spacing, or replies | | `nm -D` / `objdump -T` | Which library imports `SSL_write` vs `write`; exported APIs | Not wire layout | | VDDK `vixDiskLib.nfc.LogLevel=4` | Function names and AIO `opId` / `type` / `size` to label a frame | Not magic numbers, path placement, or `BANNER \r\n` | -| `LD_PRELOAD` SSL / `write` hook | Plaintext of SOAP, authd, and (after PROXY) NFC on fd 902 | Must not stay on the replacement process; `docs/ssl_hook.md` | +| `LD_PRELOAD` SSL / `write` hook | Plaintext of SOAP, authd, and (after PROXY) NFC on fd 902 | Must not stay on the OpenVixDiskLib process; `docs/ssl_hook.md` | | `strace -f -x` on `write` / `send*` | First writable NFC capture without rebuilding the hook (Step 10) | Noisy; TLS still opaque; `-s` truncates large extras | | `pickle` of `LabEnv` | Create the temp VM unhooked, then load it under the hook | `/tmp` only; never commit pickles (lab host and credentials) | | ctypes drivers in `docs/probing_samples/` | Repeatable `ConnectEx` / `Open` / `Read` / `Write` under capture | Not library code | @@ -172,7 +172,7 @@ parameter names and which moref vCenter accepts: ticket. `GetVmFiles` plus `OPEN_FILE` flags `0x1a` fails with `VIX_E_FILE_READ_ONLY` (`0x0b`). -The replacement registers those methods and calls them through pyVmomi. +OpenVixDiskLib registers those methods and calls them through pyVmomi. It does not ship a hand-rolled SOAP client for login or tickets. ## Step 6 — Probe authd; record dead ends @@ -230,10 +230,10 @@ Python must **dup the authd fd** and send NFC as raw TCP. `SSLSocket.send` would encrypt; `unwrap()` would `SSL_shutdown`. VDDK does neither. -`openvixdisklib/nfc_open.py` replays handshake + AIO `OPEN_SESSION` / -sockopts / resource pool / `OPEN_FILE`. VDDK’s extra `DDB_GET` keys -were omitted once a file handle was enough to read. Proof of open: -`tests/integration/test_nfc_open.py`. +OpenVixDiskLib (`openvixdisklib/nfc_open.py`) replays handshake + AIO +`OPEN_SESSION` / sockopts / resource pool / `OPEN_FILE`. VDDK’s extra +`DDB_GET` keys were omitted once a file handle was enough to read. Proof +of open: `tests/integration/test_nfc_open.py`. Do not copy every VDDK message. Copy what the server requires for the Python API you are replacing. @@ -395,8 +395,8 @@ After a stage works: | `docs/ssl_hook.md` | Capture tool only | | `docs/reverse_engineering_procedure.md` | This procedure (update when the method changes) | -Keep the hook and ctypes driver under `/tmp`. They are not part of the -replacement library. +Keep the hook and ctypes driver under `/tmp`. They are not part of +OpenVixDiskLib. ## Next stages (same procedure) diff --git a/docs/ssl_hook.md b/docs/ssl_hook.md index 080c8d1..cca3a43 100644 --- a/docs/ssl_hook.md +++ b/docs/ssl_hook.md @@ -9,8 +9,7 @@ This project used a small `LD_PRELOAD` library (`sslhook.c`, built to `VixDiskLib_ConnectEx` / `VixDiskLib_Open`. The authd sequence in `docs/nfc_auth.md` was recovered from that log, not from VDDK source. -The hook is a reverse-engineering aid. It is not part of the Python -NFC client. +The hook is a reverse-engineering aid. It is not part of OpenVixDiskLib. ## Why not tcpdump or VDDK logs @@ -124,8 +123,8 @@ Details that only the hex dump made obvious: - `SESSION` has no reply; waiting for a line after it looks like a hang. - Ticket `sessionId` is the UUID string on the `SESSION` line. -Those facts are written up in `docs/nfc_auth.md`. The Python client in -`openvixdisklib/nfc_auth.py` replays this sequence; it does not use +Those facts are written up in `docs/nfc_auth.md`. OpenVixDiskLib +(`openvixdisklib/nfc_auth.py`) replays this sequence; it does not use the hook. After `200 Connect`, NFC is **not** on `SSL_write`. VDDK uses @@ -142,5 +141,5 @@ skipped. `SSL *`. - It does not decode TLS handshakes, certificates, or SOAP envelopes; that is done offline on the hex log. -- It must not ship in a production VDDK replacement. Keep it out of +- It must not ship in OpenVixDiskLib. Keep it out of the library path used by `openvixdisklib/nfc_auth.py`.