Lucian Petrut
1e9c2816ef
pytest provides session scoped fixtures, making it much easier to reuse resources across tests.
227 lines
10 KiB
Markdown
227 lines
10 KiB
Markdown
# VDDK NFC disk open
|
||
|
||
This document records how VMware VDDK opens a VMDK over NBD/NFC after
|
||
the authd handshake in `docs/nfc_auth.md`, and how
|
||
`openvixdisklib/nfc_open.py` reproduces that path. Findings come from
|
||
VDDK 8.0.2 verbose logs
|
||
(`vixDiskLib.nfc.LogLevel=4`) plus an `LD_PRELOAD` intercept of
|
||
`write` / `read` on the ESXi:902 file descriptor. Capture method:
|
||
`docs/reverse_engineering_procedure.md`.
|
||
|
||
Authentication is already done: VIM login, NFC ticket (`NfcGetVmFiles`
|
||
for read-only, `NfcRandomAccessOpenDisk` for write), TLS to authd,
|
||
`SESSION` / `BANNER` / `THUMBPRINT_SHA2 PlainText` / `PROXY`. This
|
||
stage starts at `200 Connect ha-nfc` and ends with an open file handle
|
||
that can read and write sectors. Flags `0x1a` require the writable
|
||
ticket; the same flags on a `GetVmFiles` ticket fail with
|
||
`VIX_E_FILE_READ_ONLY`.
|
||
|
||
## Mapping from VDDK
|
||
|
||
| VDDK call / log | Wire effect |
|
||
| ---------------------------------------------------- | -------------------------------------------------------- |
|
||
| `VixDiskLib_Open` | Ticket + authd (see `nfc_auth.md`), then this protocol |
|
||
| `NBD_ClientOpen` `vpxa-nfc://[ds] path.vmdk@esxi:902` | Datastore path is the NFC open argument, not the ticket |
|
||
| `useSSL=0` | NFC bytes are raw TCP, not `SSL_write` |
|
||
| `NfcProcessSessionParams` flags `0x3` | Classic 264-byte session messages |
|
||
| `SendConnectionDataMsg` payloadInfo 4 and 7 | Client name `vddk` (4) and opId `nbdmode` (7) |
|
||
| Server version 11 | Classic version message; 11 on this ESXi 8 lab |
|
||
| `NfcAio_OpenSession` | AIO framing after the classic handshake |
|
||
| `NfcUtil_PrintFileInfoOpenFlag` `NFC_DISK` `0x1e` | `NFC_AIO_MSG_OPEN_FILE` (read-only) |
|
||
| Open without `VIXDISKLIB_FLAG_OPEN_READ_ONLY` | `OPEN_FILE` flags `0x1a` (read-write) |
|
||
| `VixDiskLib_Read` / `VixDiskLib_Write` | `NFC_AIO_MSG_IO` + sector bytes |
|
||
|
||
`snapshot_ref` is still not on the wire. Integration tests pass the
|
||
flat VMDK created with the temporary lab VM.
|
||
|
||
## After PROXY: plaintext on the TLS fd
|
||
|
||
`THUMBPRINT_SHA2 PlainText` tells authd not to wrap NFC in a second
|
||
TLS session. VDDK logs `useSSL=0` and “Plain-text connection is
|
||
deprecated”.
|
||
|
||
On the wire that means:
|
||
|
||
1. Authd commands stay inside the original TLS session (`SSL_write` /
|
||
`SSL_read`).
|
||
2. After `200 Connect ha-nfc`, VDDK calls `write(SSL_get_fd(ssl), …)`
|
||
and `read` on that same descriptor. Those buffers are NFC, not TLS
|
||
records (`0x17 0x03 …`).
|
||
3. ESXi’s `ha-nfc` side does the same: replies are plaintext NFC.
|
||
|
||
An SSL hook that only interposes `SSL_write` / `SSL_read` therefore
|
||
goes silent after PROXY. Interposing `write` / `read` and filtering
|
||
`getpeername` port 902 shows the frames.
|
||
|
||
Python must not use `SSLSocket.send` for this stage: that would
|
||
`SSL_write` and encrypt bytes the server now reads as NFC.
|
||
`nfc_open.takeover_authd_socket` dups `SSL_get_fd` and uses a raw
|
||
`socket.socket`. `unwrap()` / `SSL_shutdown` is not used; VDDK does
|
||
not send `close_notify` before NFC.
|
||
|
||
## Classic 264-byte messages
|
||
|
||
Before AIO, both peers send a **fixed 264-byte** struct, little-endian:
|
||
|
||
| Offset | Type | Meaning |
|
||
| ------ | --------- | -------------------------------------------- |
|
||
| 0 | `uint32` | Message type |
|
||
| 4 | remainder | Type-specific fields, zero-padded to 264 |
|
||
|
||
Types seen in this Open (names from `libvixDiskLib` strings matched to
|
||
the first `uint32`):
|
||
|
||
| Type | Name (inferred) | Body |
|
||
| ---- | ---------------------- | ------------------------------------------------- |
|
||
| 43 | `NFC_HANDSHAKE` | ASCII `PlainText` at offset 4 |
|
||
| 33 | `NFC_SESSION_PARAMS` | zeros |
|
||
| 36 | session-params reply | `uint32` 1 at offset 16 |
|
||
| 51 | version | `uint32` protocol version (11) at offset 4 |
|
||
| 54 | `NFC_CONNECTION_DATA` | `uint32` nameLen, `uint32` opIdLen |
|
||
| 55 | session features | `uint32` `0x3` (interruption \| switch) |
|
||
| 52 | `NFC_AIO_SESSION_OPEN` | zeros |
|
||
| 4 | `NFC_SESSION_COMPLETE` | zeros (sent on close) |
|
||
|
||
After type 54, VDDK writes the two connection-data payloads as **raw
|
||
strings**, not 264-byte frames: `vddk` then `nbdmode`. Lengths 4 and 7
|
||
are the `payloadInfo` values in the VDDK log.
|
||
|
||
Handshake order (client → server unless noted):
|
||
|
||
```
|
||
C: 43 PlainText
|
||
C: 33
|
||
S: 36
|
||
C: 51 version=11
|
||
S: 51 version=11
|
||
C: 54 nameLen=4 opIdLen=7
|
||
C: "vddk"
|
||
C: "nbdmode"
|
||
C: 55 features=3
|
||
C: 52
|
||
S: 52
|
||
```
|
||
|
||
Server version 11 is what this lab returned. VDDK logs that connection
|
||
info requires version ≥ 3.
|
||
|
||
## AIO framing
|
||
|
||
Once type 52 has been acknowledged, I/O uses a 16-byte header:
|
||
|
||
```
|
||
uint32 magic # 0xA100DA7A, wire bytes 7a da 00 a1
|
||
uint32 type # NfcAioSendMessage "type ="
|
||
uint32 size # payload bytes that follow the header
|
||
uint32 opId # monotonic, starting at 0
|
||
```
|
||
|
||
Then `size` bytes of payload. Variable-length extras (VMDK path, DDB
|
||
key name, read data) are **separate** `write`/`read` calls after that
|
||
payload, not counted in `size`.
|
||
|
||
The server echoes the same header (`magic`, `type`, `size`, `opId`)
|
||
and a payload of `size` bytes.
|
||
|
||
Magic mismatch is the `invalid msg hdr magic` string in VDDK. Type 1
|
||
is `NFC_AIO_MSG_ERROR`.
|
||
|
||
AIO types used for Open / Read / Close, correlated with the consecutive
|
||
`NFC_AIO_MSG_*` string table and VDDK logs:
|
||
|
||
| Type | Name | Payload size | Extra on the wire |
|
||
| ---- | -------------------- | ------------ | --------------------------------------------- |
|
||
| 2 | `OPEN_SESSION` | 16 | |
|
||
| 9 | `SET_SOCK_OPTS` | 12 | |
|
||
| 22 | `SET_RES_POOL` | 4 | |
|
||
| 4 | `OPEN_FILE` | 60 | path string |
|
||
| 11 | `DDB_GET` | 16 | key name (VDDK only) |
|
||
| 7 | `IO` | 44 | sector bytes (read reply / write request) |
|
||
| 5 | `CLOSE_FILE` | 8 | |
|
||
| 3 | `CLOSE_SESSION` | 4 | |
|
||
|
||
`opId` increases by one per client message. Replies reuse the request
|
||
`opId`.
|
||
|
||
VDDK Open also issues several `DDB_GET` queries (`resumeConsolidateSector`,
|
||
`isDigest`, `iofilters`, …). The server answered “key is not found”
|
||
(16 zero bytes) on this unencrypted disk. They are not required to
|
||
obtain a file handle or to read sector 0.
|
||
|
||
### OPEN_SESSION / sockopts / resource pool
|
||
|
||
VDDK sends 16 zero bytes (`OPEN_SESSION`), 12 zero bytes
|
||
(`SET_SOCK_OPTS`; server returns send/recv buffer sizes), then
|
||
`uint32` 1 (`SET_RES_POOL`, log: “Setting Resource Pool(1)”).
|
||
|
||
### OPEN_FILE
|
||
|
||
60-byte payload, little-endian:
|
||
|
||
| Offset | Type | Value on a VDDK open |
|
||
| ------ | -------- | -------------------------------------------------------------- |
|
||
| 0 | `uint32` | Path length in bytes |
|
||
| 4 | `uint32` | 0 |
|
||
| 8 | `uint32` | 0 |
|
||
| 12 | `uint32` | 0 |
|
||
| 16 | `uint32` | `2` (`NFC_DISK`) |
|
||
| 20 | `uint32` | `0x0000001e` (read-only) or `0x1a` (read-write) |
|
||
| 24 | 36 bytes | zeros |
|
||
|
||
Immediately afterwards the client writes the path, no NUL terminator
|
||
(for example `[datastore0] ovdl-test-…/ovdl-test-….vmdk`).
|
||
|
||
Reply payload (60 bytes), fields that matter:
|
||
|
||
| Offset | Type | Meaning |
|
||
| ------ | -------- | ------------------------------- |
|
||
| 8 | `uint64` | File handle (opaque, per open) |
|
||
| 16 | `uint32` | File type (`2` = `NFC_DISK`) |
|
||
| 20 | `uint32` | Flags echoed (`0x1e` or `0x1a`) |
|
||
| 36 | `uint32` | Sector size (`512` on this VM) |
|
||
|
||
Later AIO messages pass that handle as a `uint64`.
|
||
|
||
### IO (read / write)
|
||
|
||
Sector reads and writes are `NFC_AIO_MSG_IO` (type 7). Request layout,
|
||
read fragments, and write extras are documented in `docs/nfc_read.md`
|
||
and `docs/nfc_write.md`. `NfcDisk.read` / `NfcDisk.write` match
|
||
`VixDiskLib_Read` / `VixDiskLib_Write`.
|
||
|
||
### Close
|
||
|
||
`CLOSE_FILE` (handle as `uint64`), `CLOSE_SESSION` (`uint32` 0), then
|
||
classic type 4 `NFC_SESSION_COMPLETE`.
|
||
|
||
## Python replacement
|
||
|
||
| Piece | Module |
|
||
| ----------------------------- | ----------------------------------------------- |
|
||
| VIM + authd | `openvixdisklib.nfc_auth.authenticate` |
|
||
| Dup fd, skip TLS for NFC | `openvixdisklib.nfc_open.takeover_authd_socket` |
|
||
| Handshake + AIO + OPEN_FILE | `openvixdisklib.nfc_open.open_disk` |
|
||
| Sector read / write / close | `openvixdisklib.nfc_open.NfcDisk` |
|
||
|
||
Run:
|
||
|
||
```bash
|
||
.venv/bin/pytest tests/integration/test_nfc_open.py
|
||
```
|
||
|
||
The test opens the temporary lab VMDK, asserts an opaque handle and
|
||
`sector_size=512`, writes sector 0, and reads it back. Multi-sector
|
||
I/O: `docs/nfc_read.md`, `docs/nfc_write.md`, and
|
||
`tests/integration/test_nfc_read_write.py`.
|
||
|
||
## What is still VDDK-only
|
||
|
||
- `DDB_GET` / geometry / compression / encryption keys
|
||
- `NFC_DELTA_DISK`, change-block tracking
|
||
- Host-switch (`NFC_AIO_SWITCH_HOST_*`) and a second NFCSSL wrap
|
||
(`useSSL=1`, not what VDDK NBD used here)
|
||
- Direct ESXi `ha-nfc` without vCenter `vpxa-nfc`
|
||
|
||
Reads after open are in `docs/nfc_read.md`. Writes are in
|
||
`docs/nfc_write.md`.
|