Add openvixdisklib as an open NBD replacement for VMware VDDK.

VDDK is no longer publicly distributed, so this library reverse-engineers the vSphere NFC path and exposes ConnectEx, Open, Read, and Write without the proprietary SDK. Co-authored-by: Cursor <[email protected]>
2026-09-07 11:41:01 +00:00
commit 4ddf0c015d
31 changed files with 4089 additions and 0 deletions
@@ -0,0 +1,120 @@
+# VDDK NFC disk read
+
+This document records how VMware VDDK reads VMDK sectors over NBD/NFC
+after the open in `docs/nfc_open.md`, and how `NfcDisk.read` in
+`openvixdisklib/nfc_open.py` reproduces `VixDiskLib_Read`. Capture
+method: `docs/reverse_engineering_procedure.md`.
+
+## Mapping from VDDK
+
+`VixDiskLib_Read(handle, startSector, numSectors, buf)` becomes one
+`NFC_AIO_MSG_IO` (type 7) on the NFC socket. Units on the wire are
+**bytes**, not sectors:
+
+```
+offset = startSector * sectorSize
+length = numSectors * sectorSize
+```
+
+`sectorSize` is 512 from the `OPEN_FILE` reply on this lab disk.
+
+| VDDK call                         | Wire effect                                      |
+| --------------------------------- | ------------------------------------------------ |
+| `VixDiskLib_Read(h, 0, 1, buf)`   | IO offset 0, length 512, one 512-byte fragment   |
+| `VixDiskLib_Read(h, 1, 1, buf)`   | IO offset 512, length 512                        |
+| `VixDiskLib_Read(h, 0, 128, buf)` | IO length 65536 (AIO buffer size), one fragment  |
+| `VixDiskLib_Read(h, 0, 129, buf)` | One request of 66048; **two** reply fragments    |
+
+VDDK does **not** split a `Read` larger than 64 KiB into multiple
+requests. The client sends one AIO message; the server answers with
+one or more same-`opId` replies, each carrying at most
+`NFC_AIO_BUFFER_SIZE` (65536) data bytes. `NfcAioInitSession` logged
+that buffer size and count 4 during open.
+
+Sparse regions are still transferred as zeros. A read of 8 sectors at
+LBA 8 on this disk was 4096 zero bytes on the wire, not a skip.
+
+## Request (44 bytes)
+
+Little-endian, after the usual 16-byte AIO header
+(`magic 0xA100DA7A`, type 7, size 44, monotonic `opId`):
+
+| Offset | Type     | VDDK `Read(start, n)`                          |
+| ------ | -------- | ---------------------------------------------- |
+| 0      | `uint64` | File handle from `OPEN_FILE`                   |
+| 8      | `uint64` | `1` (`NFC_AIO_IO_READ`; write uses `0`)        |
+| 16     | `uint64` | Byte offset                                    |
+| 24     | `uint64` | Byte length                                    |
+| 32     | `uint32` | Byte length (same value)                       |
+| 36     | `uint32` | Byte length (same value)                       |
+| 40     | `uint32` | `0` (flags; uncompressed in this capture)      |
+
+An earlier guess that offset 36 was `NFC_DISK` (`2`) was wrong: a
+1-sector VDDK read puts `512` in both `uint32` length fields. A Python
+read that sent `(512, 2, 0)` still worked for one sector; the
+replacement now matches VDDK.
+
+## Reply
+
+Each fragment is: 16-byte AIO header (same `type` and `opId`) + 44-byte
+payload + `chunkLength` data bytes.
+
+Reply payload (handle is zeroed; lengths describe this fragment):
+
+| Offset | Type     | Meaning                                         |
+| ------ | -------- | ----------------------------------------------- |
+| 0      | `uint64` | `0`                                             |
+| 8      | `uint64` | `1` (read)                                      |
+| 16     | `uint64` | Byte offset of the **request**                  |
+| 24     | `uint32` | Total request length                            |
+| 28     | `uint32` | Fragment index (`0`, `1`, …)                    |
+| 32     | `uint32` | This fragment’s byte length                     |
+| 36     | `uint32` | Same as offset 32                               |
+| 40     | `uint32` | `0`                                             |
+
+When there is a single fragment, offsets 24–31 look like a `uint64`
+length (index is 0). The 129-sector capture shows why they are two
+`uint32`s: fragment 0 has `(66048, 0)` then chunk 65536; fragment 1
+has `(66048, 1)` then chunk 512.
+
+Read loop: receive fragments with that `opId` until the concatenated
+data length equals the request. Use the `uint32` at payload offset 32
+as the extra-data size for that fragment. Do not treat extra data as
+part of AIO `size` (that field stays 44).
+
+129-sector example (one client request, two server fragments):
+
+```
+C: type=7 opId=18 size=44  offset=0 length=66048
+S: type=7 opId=18 size=44  index=0 chunk=65536  + 65536 data
+S: type=7 opId=18 size=44  index=1 chunk=512    + 512 data
+```
+
+## Lab check
+
+Integration tests create an empty 10 GiB thin disk, write a repeating
+pattern at each captured range (including 129 sectors), and read it
+back. An unwritten region is zeros.
+
+Writes use the same 44-byte IO payload with opcode `2`; see
+`docs/nfc_write.md`.
+
+## Python replacement
+
+`NfcDisk.read(start_sector, num_sectors)` in
+`openvixdisklib/nfc_open.py`. Run:
+
+```bash
+.venv/bin/python -m unittest tests.integration.test_nfc_read_write
+```
+
+The integration test writes and then reads the captured VDDK ranges
+(including a 129-sector transfer that must assemble two read
+fragments).
+
+## What is still VDDK-only
+
+- Compression flags on the last `uint32`
+- `VixDiskLib_ReadAsync` (same IO messages, different client threading)
+- `VixDiskLib_QueryAllocatedBlocks` / allocation bitmaps
+- `VixDiskLib_GetInfo` capacity (not required to read a known range)