Add openvixdisklib as an open NBD replacement for VMware VDDK.
VDDK is no longer publicly distributed, so this library reverse-engineers the vSphere NFC path and exposes ConnectEx, Open, Read, and Write without the proprietary SDK. Co-authored-by: Cursor <[email protected]>
This commit is contained in:
@@ -0,0 +1,265 @@
|
||||
# VDDK NFC authentication
|
||||
|
||||
This document records how VMware VDDK authenticates for NBD/NFC disk
|
||||
access, and how the Python replacement in `openvixdisklib/nfc_auth.py`
|
||||
reproduces that path. Findings come from VDDK 8.0.2 libraries
|
||||
(`libvixDiskLib`, `libvddkVimAccess`, `libvim-types`), live SOAP calls
|
||||
against vCenter
|
||||
8.0.1, and a TLS intercept of `VixDiskLib_ConnectEx` / `VixDiskLib_Open`.
|
||||
The steps used to obtain those findings are in
|
||||
`docs/reverse_engineering_procedure.md`.
|
||||
|
||||
The goal of this stage is authentication only: a logged-in VIM session
|
||||
plus an authd TLS socket that has completed `200 Connect`. Opening a
|
||||
VMDK and reading sectors is `docs/nfc_open.md`.
|
||||
|
||||
## Mapping from VDDK
|
||||
|
||||
The VDDK wrapper in `tests/integration/vixdisklib.py` calls
|
||||
`VixDiskLib_ConnectEx` with UID credentials and `VixDiskLib_Open` on a
|
||||
datastore path. VDDK does **not** send the vCenter username and
|
||||
password to ESXi port 902. It:
|
||||
|
||||
1. Logs into vCenter over HTTPS 443 (SOAP / `urn:vim25`).
|
||||
2. Asks vCenter for a one-time NFC ticket.
|
||||
3. Connects to the ESXi **authd** daemon on TCP 902, upgrades to TLS,
|
||||
and presents that ticket.
|
||||
|
||||
| VDDK call | What actually happens |
|
||||
| --------------------------------- | ---------------------------------------------------------- |
|
||||
| `VixDiskLib_InitEx` | Load plugins, SSL, logging |
|
||||
| `VixDiskLib_ConnectEx` | SOAP `SessionManager.Login` to vCenter |
|
||||
| `VixDiskLib_Open` (read-only) | `NfcGetVmFiles` ticket, then authd handshake, then NFC I/O |
|
||||
| `VixDiskLib_Open` (read-write) | `NfcRandomAccessOpenDisk` ticket (disk key + host) |
|
||||
| `transport_modes="nbd"` | NBD over NFC (`vpxa-nfc://...@esxi:902`) |
|
||||
| `vmxSpec=moref=vm-13098` | VM managed object used as the ticket target |
|
||||
| `snapshot_ref` | Not consumed by the ticket call itself |
|
||||
| `VIXDISKLIB_CRED_UID` | Username/password for VIM only |
|
||||
|
||||
Lab topology used for capture:
|
||||
|
||||
- vCenter: `10.8.1.199` (VirtualCenter 8.0.1)
|
||||
- VM: `vm-13098` on host `host-13001` (`10.8.1.250`)
|
||||
- NFC service moref on vCenter: `nfcService`
|
||||
- Authd: `10.8.1.250:902`
|
||||
|
||||
## Stage 1: VIM login
|
||||
|
||||
This is a public pyVmomi operation. Reuse `pyVim.connect.SmartConnect`
|
||||
rather than crafting SOAP.
|
||||
|
||||
- Endpoint: `https://<vcenter>:443/sdk`
|
||||
- Cookie: `vmware_soap_session`
|
||||
- SOAPAction: `"urn:vim25/8.0.1.0"` (negotiated)
|
||||
|
||||
VDDK logs this as `Connected to VIM Server` / `Authenticating user` /
|
||||
`Logged in!`. The Python replacement keeps that `ServiceInstance` and
|
||||
its stub for the ticket call.
|
||||
|
||||
Direct ESXi login is the same SOAP login against hostd, but the NFC
|
||||
moref and service name differ (`ha-nfc` instead of `nfcService` /
|
||||
`vpxa-nfc`). The lab path is vCenter-mediated.
|
||||
|
||||
## Stage 2: NFC ticket
|
||||
|
||||
### Why this is not public pyVmomi
|
||||
|
||||
`vim.NfcService` is omitted from the public vim25 WSDL that pyVmomi
|
||||
ships. vCenter still implements it:
|
||||
|
||||
- Version document: `GET /sdk/nfcServiceVersions.xml` → namespace
|
||||
`urn:nfc`, version `7.0.3.2`
|
||||
- Methods also accept `urn:vim25` (that is what VDDK uses)
|
||||
- Well-known moref on this vCenter: `nfcService`
|
||||
|
||||
`ServiceManager.QueryServiceList` does **not** list NFC. The moref is
|
||||
hardcoded in VDDK as `nfcService` (vCenter) or `ha-nfc` (ESXi).
|
||||
|
||||
`openvixdisklib/nfc_auth.py` registers the missing type with
|
||||
`pyVmomi.VmomiSupport.CreateManagedType` and invokes it on the existing
|
||||
SmartConnect stub, so serialization, cookies, and `HostServiceTicket`
|
||||
stay in pyVmomi.
|
||||
|
||||
### Methods VDDK actually calls
|
||||
|
||||
Intercepted SOAP for a **read-only** `VixDiskLib_Open` of a datastore
|
||||
path:
|
||||
|
||||
```xml
|
||||
<NfcGetVmFiles xmlns="urn:vim25">
|
||||
<_this type="NfcService">nfcService</_this>
|
||||
<vm type="VirtualMachine">vm-13098</vm>
|
||||
</NfcGetVmFiles>
|
||||
```
|
||||
|
||||
No disk path, snapshot, or host is in this request. The path
|
||||
(`[datastore0] ...-000007.vmdk`) is used later on the NFC channel.
|
||||
|
||||
A `GetVmFiles` ticket is **not** writable. Opening the same path with
|
||||
NFC flags `0x1a` returns AIO error `0x0b` (`VIX_E_FILE_READ_ONLY`).
|
||||
Writable `ConnectEx(readOnly=FALSE)` uses a disk-scoped ticket instead.
|
||||
|
||||
`libvim-types.so` maps vmodl `randomAccessOpen` to WSDL
|
||||
`NfcRandomAccessOpenDisk` (same arguments as the read-only sibling):
|
||||
|
||||
```xml
|
||||
<NfcRandomAccessOpenDisk xmlns="urn:vim25">
|
||||
<_this type="NfcService">nfcService</_this>
|
||||
<vm type="VirtualMachine">vm-13098</vm>
|
||||
<diskDeviceKey>2000</diskDeviceKey>
|
||||
<hostForAccess type="HostSystem">host-13001</hostForAccess>
|
||||
</NfcRandomAccessOpenDisk>
|
||||
```
|
||||
|
||||
A disk-scoped **read** ticket also works and returns the same
|
||||
`HostServiceTicket` type:
|
||||
|
||||
```xml
|
||||
<NfcRandomAccessOpenReadonly xmlns="urn:nfc">
|
||||
<_this type="NfcService">nfcService</_this>
|
||||
<vm type="VirtualMachine">vm-13098</vm>
|
||||
<diskDeviceKey>2000</diskDeviceKey>
|
||||
<hostForAccess type="HostSystem">host-13001</hostForAccess>
|
||||
</NfcRandomAccessOpenReadonly>
|
||||
```
|
||||
|
||||
`diskDeviceKey` is `VirtualDisk.key` from `vm.config.hardware.device`
|
||||
(2000 for Hard disk 1). The replacement resolves it from the datastore
|
||||
path when `open` is given a VMDK rather than a key.
|
||||
|
||||
### Return value: `vim.HostServiceTicket`
|
||||
|
||||
Public pyVmomi type. Example from this lab:
|
||||
|
||||
| Field | Example | Role |
|
||||
| ---------------- | -------------------------------------- | ----------------------------------------- |
|
||||
| `host` | `10.8.1.250` | ESXi management / NFC address |
|
||||
| `port` | `902` | authd TCP port |
|
||||
| `sslThumbprint` | `BE:22:58:...:76:29` | SHA-1 of the ESXi TLS cert |
|
||||
| `service` | `vpxa-nfc` | authd `PROXY` argument |
|
||||
| `serviceVersion` | `1.1` | NFC hosted by hostd (ESX 3.0+ convention) |
|
||||
| `sessionId` | `52cdebc5-b7ee-359a-1dec-76f0bc105ac5` | One-time authd `SESSION` token |
|
||||
|
||||
Tickets are single-use. Calling `GetVmFiles` twice issues two tickets;
|
||||
only the one presented to authd is consumed.
|
||||
|
||||
### Other NfcService methods seen in VDDK
|
||||
|
||||
WSDL names are prefixed with `Nfc`. The vmodl names (from
|
||||
`libvim-types.so`) include:
|
||||
|
||||
| WSDL name | Parameters (observed / from C++) | Notes |
|
||||
| ----------------------------- | -------------------------------------- | ------------------------------ |
|
||||
| `NfcGetVmFiles` | `vm` | VDDK read-only Open path |
|
||||
| `NfcRandomAccessOpenReadonly` | `vm`, `diskDeviceKey`, `hostForAccess` | Disk-scoped read ticket |
|
||||
| `NfcRandomAccessOpenDisk` | `vm`, `diskDeviceKey`, `hostForAccess` | Disk-scoped read-write ticket |
|
||||
| `NfcGetServerNfcLibVersion` | `hostForAccess` | Lab returned `11` |
|
||||
| `NfcFileManagement` | requires `ds` (datastore) | File copy, not NBD |
|
||||
| `NfcSystemManagement` | host moref | Not used for disk open |
|
||||
|
||||
`NfcGetServerNfcLibVersion` without `hostForAccess` fails with
|
||||
`A specified parameter was not correct: hostForAccess`. Using moref
|
||||
`ha-nfc` on vCenter fails with `ManagedObjectNotFound`; `nfcService`
|
||||
is the correct vCenter object.
|
||||
|
||||
## Stage 3: authd handshake (TCP 902)
|
||||
|
||||
authd is the VMware Authentication Daemon. Plaintext banner from ESXi
|
||||
8:
|
||||
|
||||
```
|
||||
220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported, NFCSSL supported/t, SHA256 supported
|
||||
```
|
||||
|
||||
SSL is required. Sending commands before `wrap_socket` closes the
|
||||
connection. After TLS there is **no** `USER` / `PASS` when the client
|
||||
holds a vCenter NFC ticket.
|
||||
|
||||
### Sequence captured from VDDK
|
||||
|
||||
VDDK log line immediately before the socket:
|
||||
|
||||
```
|
||||
Using proxy/session authentication, sessionId=..., useSSL=0
|
||||
Plain-text connection is deprecated; use SSL to connect to NFC server
|
||||
```
|
||||
|
||||
`useSSL=0` does **not** mean skip TLS on 902. It means skip a second
|
||||
NFCSSL wrap after authd TLS (`THUMBPRINT_SHA2 PlainText`). The
|
||||
management channel is still TLS.
|
||||
|
||||
Intercepted writes/reads after the TLS handshake:
|
||||
|
||||
```
|
||||
C -> SESSION <sessionId>\r\n
|
||||
C -> BANNER \r\n
|
||||
S -> 220 VMware Authentication Daemon Version 1.10: ...\r\n
|
||||
C -> THUMBPRINT_SHA2 PlainText\r\n
|
||||
S -> 200 <SHA-256 thumbprint with colons>\r\n
|
||||
C -> PROXY vpxa-nfc\r\n
|
||||
S -> 200 Connect ha-nfc\r\n
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `SESSION` does not get a reply of its own. Waiting for a line after
|
||||
`SESSION` looks like a hang.
|
||||
- `BANNER` is the 7-byte command `BANNER` plus a trailing space. That
|
||||
space is part of the token; authd strips spaces when matching some
|
||||
commands, so `THUMBPRINT_SHA2 <colon-thumbprint>` is parsed as one
|
||||
token and returns `501 Invalid arguments`. `PlainText` has no extra
|
||||
spaces/colons and is the argument VDDK sends.
|
||||
- `PROXY` uses `ticket.service` (`vpxa-nfc` via vCenter). The success
|
||||
line names the host-side NFC endpoint (`ha-nfc`).
|
||||
- After `200 Connect`, the socket speaks binary NFC (not documented
|
||||
here).
|
||||
|
||||
### Commands that are not used for this ticket type
|
||||
|
||||
authd also implements FTP-style `USER` / `PASS` (and `XPAS`). Those
|
||||
are for local ESXi credentials. With a vCenter ticket:
|
||||
|
||||
| Attempt | Result |
|
||||
| -------------------------------------------- | --------------------------------------- |
|
||||
| `USER` / `PASS` (vCenter account) | `530 Login incorrect` |
|
||||
| `USER *` / `PASS <sessionId>` | `530 Login incorrect` |
|
||||
| `USER <sessionId>` / `PASS <sessionId>` | `530 Login incorrect` |
|
||||
| `SESSIONID <sessionId>` | `530 Please login with USER and PASS` |
|
||||
| `CONNECT_VPXA <sessionId>` (after TLS) | `530 Please login with USER and PASS` |
|
||||
| `SESSION <sessionId>` then wait for a reply | No line until `BANNER` / `PROXY` follow |
|
||||
|
||||
`THUMBPRINT` / `THUMBPRINT_SHA2` with the SHA-1 ticket thumbprint as
|
||||
argument is not what VDDK sends. The SHA-1 value is for verifying the
|
||||
TLS certificate, not for the `THUMBPRINT_SHA2` command.
|
||||
|
||||
## Python replacement
|
||||
|
||||
| Piece | Module | Reuses pyVmomi? |
|
||||
| -------------------- | ---------------------------------------- | ---------------------------------- |
|
||||
| VIM login | `openvixdisklib.nfc_auth.connect_vim` | Yes — `SmartConnect` |
|
||||
| VM / host lookup | `vim.VirtualMachine` | Yes |
|
||||
| `HostServiceTicket` | return type of ticket call | Yes — public data object |
|
||||
| NFC ticket | `openvixdisklib.nfc_auth.get_nfc_ticket` | Same stub; type registered locally |
|
||||
| authd TLS + commands | `openvixdisklib.nfc_auth.connect_authd` | No public API |
|
||||
| End-to-end | `openvixdisklib.nfc_auth.authenticate` | `NfcAuthSession` |
|
||||
|
||||
Management SHA-1 thumbprints are read with
|
||||
`openvixdisklib.nfc_auth.get_ssl_cert_thumbprint` (stdlib `ssl` and
|
||||
`hashlib`; no pyOpenSSL). Integration tests call that instead of
|
||||
hard-coding the lab certificate.
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m unittest tests.integration.test_nfc_auth
|
||||
```
|
||||
|
||||
The test completes VIM login and the authd handshake (`200 Connect`)
|
||||
and asserts an established TLS socket on `ticket.host:ticket.port`.
|
||||
|
||||
## What comes after authentication
|
||||
|
||||
Authentication stops at `200 Connect ha-nfc`. Opening the VMDK and
|
||||
reading or writing sectors is documented in `docs/nfc_open.md` and
|
||||
implemented in `openvixdisklib/nfc_open.py`. The datastore path is
|
||||
consumed there (and, for writes, as `diskDeviceKey` on the ticket).
|
||||
@@ -0,0 +1,226 @@
|
||||
# VDDK NFC disk open
|
||||
|
||||
This document records how VMware VDDK opens a VMDK over NBD/NFC after
|
||||
the authd handshake in `docs/nfc_auth.md`, and how
|
||||
`openvixdisklib/nfc_open.py` reproduces that path. Findings come from
|
||||
VDDK 8.0.2 verbose logs
|
||||
(`vixDiskLib.nfc.LogLevel=4`) plus an `LD_PRELOAD` intercept of
|
||||
`write` / `read` on the ESXi:902 file descriptor. Capture method:
|
||||
`docs/reverse_engineering_procedure.md`.
|
||||
|
||||
Authentication is already done: VIM login, NFC ticket (`NfcGetVmFiles`
|
||||
for read-only, `NfcRandomAccessOpenDisk` for write), TLS to authd,
|
||||
`SESSION` / `BANNER` / `THUMBPRINT_SHA2 PlainText` / `PROXY`. This
|
||||
stage starts at `200 Connect ha-nfc` and ends with an open file handle
|
||||
that can read and write sectors. Flags `0x1a` require the writable
|
||||
ticket; the same flags on a `GetVmFiles` ticket fail with
|
||||
`VIX_E_FILE_READ_ONLY`.
|
||||
|
||||
## Mapping from VDDK
|
||||
|
||||
| VDDK call / log | Wire effect |
|
||||
| ---------------------------------------------------- | -------------------------------------------------------- |
|
||||
| `VixDiskLib_Open` | Ticket + authd (see `nfc_auth.md`), then this protocol |
|
||||
| `NBD_ClientOpen` `vpxa-nfc://[ds] path.vmdk@esxi:902` | Datastore path is the NFC open argument, not the ticket |
|
||||
| `useSSL=0` | NFC bytes are raw TCP, not `SSL_write` |
|
||||
| `NfcProcessSessionParams` flags `0x3` | Classic 264-byte session messages |
|
||||
| `SendConnectionDataMsg` payloadInfo 4 and 7 | Client name `vddk` (4) and opId `nbdmode` (7) |
|
||||
| Server version 11 | Classic version message; 11 on this ESXi 8 lab |
|
||||
| `NfcAio_OpenSession` | AIO framing after the classic handshake |
|
||||
| `NfcUtil_PrintFileInfoOpenFlag` `NFC_DISK` `0x1e` | `NFC_AIO_MSG_OPEN_FILE` (read-only) |
|
||||
| Open without `VIXDISKLIB_FLAG_OPEN_READ_ONLY` | `OPEN_FILE` flags `0x1a` (read-write) |
|
||||
| `VixDiskLib_Read` / `VixDiskLib_Write` | `NFC_AIO_MSG_IO` + sector bytes |
|
||||
|
||||
`snapshot_ref` is still not on the wire. Integration tests pass the
|
||||
flat VMDK created with the temporary lab VM.
|
||||
|
||||
## After PROXY: plaintext on the TLS fd
|
||||
|
||||
`THUMBPRINT_SHA2 PlainText` tells authd not to wrap NFC in a second
|
||||
TLS session. VDDK logs `useSSL=0` and “Plain-text connection is
|
||||
deprecated”.
|
||||
|
||||
On the wire that means:
|
||||
|
||||
1. Authd commands stay inside the original TLS session (`SSL_write` /
|
||||
`SSL_read`).
|
||||
2. After `200 Connect ha-nfc`, VDDK calls `write(SSL_get_fd(ssl), …)`
|
||||
and `read` on that same descriptor. Those buffers are NFC, not TLS
|
||||
records (`0x17 0x03 …`).
|
||||
3. ESXi’s `ha-nfc` side does the same: replies are plaintext NFC.
|
||||
|
||||
An SSL hook that only interposes `SSL_write` / `SSL_read` therefore
|
||||
goes silent after PROXY. Interposing `write` / `read` and filtering
|
||||
`getpeername` port 902 shows the frames.
|
||||
|
||||
Python must not use `SSLSocket.send` for this stage: that would
|
||||
`SSL_write` and encrypt bytes the server now reads as NFC.
|
||||
`nfc_open.takeover_authd_socket` dups `SSL_get_fd` and uses a raw
|
||||
`socket.socket`. `unwrap()` / `SSL_shutdown` is not used; VDDK does
|
||||
not send `close_notify` before NFC.
|
||||
|
||||
## Classic 264-byte messages
|
||||
|
||||
Before AIO, both peers send a **fixed 264-byte** struct, little-endian:
|
||||
|
||||
| Offset | Type | Meaning |
|
||||
| ------ | --------- | -------------------------------------------- |
|
||||
| 0 | `uint32` | Message type |
|
||||
| 4 | remainder | Type-specific fields, zero-padded to 264 |
|
||||
|
||||
Types seen in this Open (names from `libvixDiskLib` strings matched to
|
||||
the first `uint32`):
|
||||
|
||||
| Type | Name (inferred) | Body |
|
||||
| ---- | ---------------------- | ------------------------------------------------- |
|
||||
| 43 | `NFC_HANDSHAKE` | ASCII `PlainText` at offset 4 |
|
||||
| 33 | `NFC_SESSION_PARAMS` | zeros |
|
||||
| 36 | session-params reply | `uint32` 1 at offset 16 |
|
||||
| 51 | version | `uint32` protocol version (11) at offset 4 |
|
||||
| 54 | `NFC_CONNECTION_DATA` | `uint32` nameLen, `uint32` opIdLen |
|
||||
| 55 | session features | `uint32` `0x3` (interruption \| switch) |
|
||||
| 52 | `NFC_AIO_SESSION_OPEN` | zeros |
|
||||
| 4 | `NFC_SESSION_COMPLETE` | zeros (sent on close) |
|
||||
|
||||
After type 54, VDDK writes the two connection-data payloads as **raw
|
||||
strings**, not 264-byte frames: `vddk` then `nbdmode`. Lengths 4 and 7
|
||||
are the `payloadInfo` values in the VDDK log.
|
||||
|
||||
Handshake order (client → server unless noted):
|
||||
|
||||
```
|
||||
C: 43 PlainText
|
||||
C: 33
|
||||
S: 36
|
||||
C: 51 version=11
|
||||
S: 51 version=11
|
||||
C: 54 nameLen=4 opIdLen=7
|
||||
C: "vddk"
|
||||
C: "nbdmode"
|
||||
C: 55 features=3
|
||||
C: 52
|
||||
S: 52
|
||||
```
|
||||
|
||||
Server version 11 is what this lab returned. VDDK logs that connection
|
||||
info requires version ≥ 3.
|
||||
|
||||
## AIO framing
|
||||
|
||||
Once type 52 has been acknowledged, I/O uses a 16-byte header:
|
||||
|
||||
```
|
||||
uint32 magic # 0xA100DA7A, wire bytes 7a da 00 a1
|
||||
uint32 type # NfcAioSendMessage "type ="
|
||||
uint32 size # payload bytes that follow the header
|
||||
uint32 opId # monotonic, starting at 0
|
||||
```
|
||||
|
||||
Then `size` bytes of payload. Variable-length extras (VMDK path, DDB
|
||||
key name, read data) are **separate** `write`/`read` calls after that
|
||||
payload, not counted in `size`.
|
||||
|
||||
The server echoes the same header (`magic`, `type`, `size`, `opId`)
|
||||
and a payload of `size` bytes.
|
||||
|
||||
Magic mismatch is the `invalid msg hdr magic` string in VDDK. Type 1
|
||||
is `NFC_AIO_MSG_ERROR`.
|
||||
|
||||
AIO types used for Open / Read / Close, correlated with the consecutive
|
||||
`NFC_AIO_MSG_*` string table and VDDK logs:
|
||||
|
||||
| Type | Name | Payload size | Extra on the wire |
|
||||
| ---- | -------------------- | ------------ | --------------------------------------------- |
|
||||
| 2 | `OPEN_SESSION` | 16 | |
|
||||
| 9 | `SET_SOCK_OPTS` | 12 | |
|
||||
| 22 | `SET_RES_POOL` | 4 | |
|
||||
| 4 | `OPEN_FILE` | 60 | path string |
|
||||
| 11 | `DDB_GET` | 16 | key name (VDDK only) |
|
||||
| 7 | `IO` | 44 | sector bytes (read reply / write request) |
|
||||
| 5 | `CLOSE_FILE` | 8 | |
|
||||
| 3 | `CLOSE_SESSION` | 4 | |
|
||||
|
||||
`opId` increases by one per client message. Replies reuse the request
|
||||
`opId`.
|
||||
|
||||
VDDK Open also issues several `DDB_GET` queries (`resumeConsolidateSector`,
|
||||
`isDigest`, `iofilters`, …). The server answered “key is not found”
|
||||
(16 zero bytes) on this unencrypted disk. They are not required to
|
||||
obtain a file handle or to read sector 0.
|
||||
|
||||
### OPEN_SESSION / sockopts / resource pool
|
||||
|
||||
VDDK sends 16 zero bytes (`OPEN_SESSION`), 12 zero bytes
|
||||
(`SET_SOCK_OPTS`; server returns send/recv buffer sizes), then
|
||||
`uint32` 1 (`SET_RES_POOL`, log: “Setting Resource Pool(1)”).
|
||||
|
||||
### OPEN_FILE
|
||||
|
||||
60-byte payload, little-endian:
|
||||
|
||||
| Offset | Type | Value on a VDDK open |
|
||||
| ------ | -------- | -------------------------------------------------------------- |
|
||||
| 0 | `uint32` | Path length in bytes |
|
||||
| 4 | `uint32` | 0 |
|
||||
| 8 | `uint32` | 0 |
|
||||
| 12 | `uint32` | 0 |
|
||||
| 16 | `uint32` | `2` (`NFC_DISK`) |
|
||||
| 20 | `uint32` | `0x0000001e` (read-only) or `0x1a` (read-write) |
|
||||
| 24 | 36 bytes | zeros |
|
||||
|
||||
Immediately afterwards the client writes the path, no NUL terminator
|
||||
(for example `[datastore0] ovdl-test-…/ovdl-test-….vmdk`).
|
||||
|
||||
Reply payload (60 bytes), fields that matter:
|
||||
|
||||
| Offset | Type | Meaning |
|
||||
| ------ | -------- | ------------------------------- |
|
||||
| 8 | `uint64` | File handle (opaque, per open) |
|
||||
| 16 | `uint32` | File type (`2` = `NFC_DISK`) |
|
||||
| 20 | `uint32` | Flags echoed (`0x1e` or `0x1a`) |
|
||||
| 36 | `uint32` | Sector size (`512` on this VM) |
|
||||
|
||||
Later AIO messages pass that handle as a `uint64`.
|
||||
|
||||
### IO (read / write)
|
||||
|
||||
Sector reads and writes are `NFC_AIO_MSG_IO` (type 7). Request layout,
|
||||
read fragments, and write extras are documented in `docs/nfc_read.md`
|
||||
and `docs/nfc_write.md`. `NfcDisk.read` / `NfcDisk.write` match
|
||||
`VixDiskLib_Read` / `VixDiskLib_Write`.
|
||||
|
||||
### Close
|
||||
|
||||
`CLOSE_FILE` (handle as `uint64`), `CLOSE_SESSION` (`uint32` 0), then
|
||||
classic type 4 `NFC_SESSION_COMPLETE`.
|
||||
|
||||
## Python replacement
|
||||
|
||||
| Piece | Module |
|
||||
| ----------------------------- | ----------------------------------------------- |
|
||||
| VIM + authd | `openvixdisklib.nfc_auth.authenticate` |
|
||||
| Dup fd, skip TLS for NFC | `openvixdisklib.nfc_open.takeover_authd_socket` |
|
||||
| Handshake + AIO + OPEN_FILE | `openvixdisklib.nfc_open.open_disk` |
|
||||
| Sector read / write / close | `openvixdisklib.nfc_open.NfcDisk` |
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m unittest tests.integration.test_nfc_open
|
||||
```
|
||||
|
||||
The test opens the temporary lab VMDK, asserts an opaque handle and
|
||||
`sector_size=512`, writes sector 0, and reads it back. Multi-sector
|
||||
I/O: `docs/nfc_read.md`, `docs/nfc_write.md`, and
|
||||
`tests/integration/test_nfc_read_write.py`.
|
||||
|
||||
## What is still VDDK-only
|
||||
|
||||
- `DDB_GET` / geometry / compression / encryption keys
|
||||
- `NFC_DELTA_DISK`, change-block tracking
|
||||
- Host-switch (`NFC_AIO_SWITCH_HOST_*`) and a second NFCSSL wrap
|
||||
(`useSSL=1`, not what VDDK NBD used here)
|
||||
- Direct ESXi `ha-nfc` without vCenter `vpxa-nfc`
|
||||
|
||||
Reads after open are in `docs/nfc_read.md`. Writes are in
|
||||
`docs/nfc_write.md`.
|
||||
@@ -0,0 +1,120 @@
|
||||
# VDDK NFC disk read
|
||||
|
||||
This document records how VMware VDDK reads VMDK sectors over NBD/NFC
|
||||
after the open in `docs/nfc_open.md`, and how `NfcDisk.read` in
|
||||
`openvixdisklib/nfc_open.py` reproduces `VixDiskLib_Read`. Capture
|
||||
method: `docs/reverse_engineering_procedure.md`.
|
||||
|
||||
## Mapping from VDDK
|
||||
|
||||
`VixDiskLib_Read(handle, startSector, numSectors, buf)` becomes one
|
||||
`NFC_AIO_MSG_IO` (type 7) on the NFC socket. Units on the wire are
|
||||
**bytes**, not sectors:
|
||||
|
||||
```
|
||||
offset = startSector * sectorSize
|
||||
length = numSectors * sectorSize
|
||||
```
|
||||
|
||||
`sectorSize` is 512 from the `OPEN_FILE` reply on this lab disk.
|
||||
|
||||
| VDDK call | Wire effect |
|
||||
| --------------------------------- | ------------------------------------------------ |
|
||||
| `VixDiskLib_Read(h, 0, 1, buf)` | IO offset 0, length 512, one 512-byte fragment |
|
||||
| `VixDiskLib_Read(h, 1, 1, buf)` | IO offset 512, length 512 |
|
||||
| `VixDiskLib_Read(h, 0, 128, buf)` | IO length 65536 (AIO buffer size), one fragment |
|
||||
| `VixDiskLib_Read(h, 0, 129, buf)` | One request of 66048; **two** reply fragments |
|
||||
|
||||
VDDK does **not** split a `Read` larger than 64 KiB into multiple
|
||||
requests. The client sends one AIO message; the server answers with
|
||||
one or more same-`opId` replies, each carrying at most
|
||||
`NFC_AIO_BUFFER_SIZE` (65536) data bytes. `NfcAioInitSession` logged
|
||||
that buffer size and count 4 during open.
|
||||
|
||||
Sparse regions are still transferred as zeros. A read of 8 sectors at
|
||||
LBA 8 on this disk was 4096 zero bytes on the wire, not a skip.
|
||||
|
||||
## Request (44 bytes)
|
||||
|
||||
Little-endian, after the usual 16-byte AIO header
|
||||
(`magic 0xA100DA7A`, type 7, size 44, monotonic `opId`):
|
||||
|
||||
| Offset | Type | VDDK `Read(start, n)` |
|
||||
| ------ | -------- | ---------------------------------------------- |
|
||||
| 0 | `uint64` | File handle from `OPEN_FILE` |
|
||||
| 8 | `uint64` | `1` (`NFC_AIO_IO_READ`; write uses `0`) |
|
||||
| 16 | `uint64` | Byte offset |
|
||||
| 24 | `uint64` | Byte length |
|
||||
| 32 | `uint32` | Byte length (same value) |
|
||||
| 36 | `uint32` | Byte length (same value) |
|
||||
| 40 | `uint32` | `0` (flags; uncompressed in this capture) |
|
||||
|
||||
An earlier guess that offset 36 was `NFC_DISK` (`2`) was wrong: a
|
||||
1-sector VDDK read puts `512` in both `uint32` length fields. A Python
|
||||
read that sent `(512, 2, 0)` still worked for one sector; the
|
||||
replacement now matches VDDK.
|
||||
|
||||
## Reply
|
||||
|
||||
Each fragment is: 16-byte AIO header (same `type` and `opId`) + 44-byte
|
||||
payload + `chunkLength` data bytes.
|
||||
|
||||
Reply payload (handle is zeroed; lengths describe this fragment):
|
||||
|
||||
| Offset | Type | Meaning |
|
||||
| ------ | -------- | ----------------------------------------------- |
|
||||
| 0 | `uint64` | `0` |
|
||||
| 8 | `uint64` | `1` (read) |
|
||||
| 16 | `uint64` | Byte offset of the **request** |
|
||||
| 24 | `uint32` | Total request length |
|
||||
| 28 | `uint32` | Fragment index (`0`, `1`, …) |
|
||||
| 32 | `uint32` | This fragment’s byte length |
|
||||
| 36 | `uint32` | Same as offset 32 |
|
||||
| 40 | `uint32` | `0` |
|
||||
|
||||
When there is a single fragment, offsets 24–31 look like a `uint64`
|
||||
length (index is 0). The 129-sector capture shows why they are two
|
||||
`uint32`s: fragment 0 has `(66048, 0)` then chunk 65536; fragment 1
|
||||
has `(66048, 1)` then chunk 512.
|
||||
|
||||
Read loop: receive fragments with that `opId` until the concatenated
|
||||
data length equals the request. Use the `uint32` at payload offset 32
|
||||
as the extra-data size for that fragment. Do not treat extra data as
|
||||
part of AIO `size` (that field stays 44).
|
||||
|
||||
129-sector example (one client request, two server fragments):
|
||||
|
||||
```
|
||||
C: type=7 opId=18 size=44 offset=0 length=66048
|
||||
S: type=7 opId=18 size=44 index=0 chunk=65536 + 65536 data
|
||||
S: type=7 opId=18 size=44 index=1 chunk=512 + 512 data
|
||||
```
|
||||
|
||||
## Lab check
|
||||
|
||||
Integration tests create an empty 10 GiB thin disk, write a repeating
|
||||
pattern at each captured range (including 129 sectors), and read it
|
||||
back. An unwritten region is zeros.
|
||||
|
||||
Writes use the same 44-byte IO payload with opcode `2`; see
|
||||
`docs/nfc_write.md`.
|
||||
|
||||
## Python replacement
|
||||
|
||||
`NfcDisk.read(start_sector, num_sectors)` in
|
||||
`openvixdisklib/nfc_open.py`. Run:
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m unittest tests.integration.test_nfc_read_write
|
||||
```
|
||||
|
||||
The integration test writes and then reads the captured VDDK ranges
|
||||
(including a 129-sector transfer that must assemble two read
|
||||
fragments).
|
||||
|
||||
## What is still VDDK-only
|
||||
|
||||
- Compression flags on the last `uint32`
|
||||
- `VixDiskLib_ReadAsync` (same IO messages, different client threading)
|
||||
- `VixDiskLib_QueryAllocatedBlocks` / allocation bitmaps
|
||||
- `VixDiskLib_GetInfo` capacity (not required to read a known range)
|
||||
@@ -0,0 +1,91 @@
|
||||
# VDDK NFC disk write
|
||||
|
||||
This document records how `NfcDisk.write` in
|
||||
`openvixdisklib/nfc_open.py` implements `VixDiskLib_Write` over NFC AIO.
|
||||
The request layout matches the captured `VixDiskLib_Read` IO message in
|
||||
`docs/nfc_read.md`. Open flags and the IO direction field were taken
|
||||
from a `strace` of VDDK 8 writing one sector to a temporary 10 GiB
|
||||
disk (`docs/reverse_engineering_procedure.md`).
|
||||
|
||||
The public `VixDiskLib_Write` prototype is in `.vddk/vixDiskLib.h`:
|
||||
|
||||
```
|
||||
VixError VixDiskLib_Write(VixDiskLibHandle diskHandle,
|
||||
VixDiskLibSectorType startSector,
|
||||
VixDiskLibSectorType numSectors,
|
||||
const uint8 *writeBuffer);
|
||||
```
|
||||
|
||||
`ConnectEx(..., Bool readOnly, ...)` with `readOnly=FALSE` and `Open`
|
||||
without `VIXDISKLIB_FLAG_OPEN_READ_ONLY` (that flag is `1 << 2` in the
|
||||
same header) is what produces the writable NFC open below. The ticket
|
||||
must be `NfcRandomAccessOpenDisk` (`docs/nfc_auth.md`); flags `0x1a`
|
||||
on a `NfcGetVmFiles` ticket are rejected as `VIX_E_FILE_READ_ONLY`.
|
||||
|
||||
## Mapping from VDDK
|
||||
|
||||
Units on the wire are **bytes**, as for reads:
|
||||
|
||||
```
|
||||
offset = startSector * sectorSize
|
||||
length = numSectors * sectorSize
|
||||
```
|
||||
|
||||
| VDDK call | Wire effect |
|
||||
| ------------------------------------------- | ------------------------------------------------ |
|
||||
| Open without `VIXDISKLIB_FLAG_OPEN_READ_ONLY` | `OPEN_FILE` flags `0x1a` |
|
||||
| Open with `VIXDISKLIB_FLAG_OPEN_READ_ONLY` | `OPEN_FILE` flags `0x1e` (read-only) |
|
||||
| `VixDiskLib_Write(h, start, n, buf)` | IO opcode `0`, then `n * 512` data bytes |
|
||||
| `VixDiskLib_Read(h, start, n, buf)` | IO opcode `1` |
|
||||
|
||||
`0x1e` vs `0x1a` is bit `0x04`, the same value as
|
||||
`VIXDISKLIB_FLAG_OPEN_READ_ONLY`. Writable opens clear that bit.
|
||||
|
||||
VDDK also issues several `DDB_GET` queries and a type-10
|
||||
`GET_FILE_INFO` (`longContentID`) before the first write. They are not
|
||||
required to write or read sectors.
|
||||
|
||||
## Request (44 bytes + data)
|
||||
|
||||
Little-endian, after the usual 16-byte AIO header
|
||||
(`magic 0xA100DA7A`, type 7, size 44, monotonic `opId`):
|
||||
|
||||
| Offset | Type | `Write(start, n)` |
|
||||
| ------ | -------- | -------------------------------------------------- |
|
||||
| 0 | `uint64` | File handle from `OPEN_FILE` |
|
||||
| 8 | `uint64` | `0` (`NFC_AIO_IO_WRITE`; read uses `1`) |
|
||||
| 16 | `uint64` | Byte offset |
|
||||
| 24 | `uint64` | Byte length |
|
||||
| 32 | `uint32` | Byte length (same value) |
|
||||
| 36 | `uint32` | Byte length (same value) |
|
||||
| 40 | `uint32` | `0` |
|
||||
|
||||
Sector bytes follow the 44-byte payload and are **not** counted in AIO
|
||||
`size`. VDDK sends header + payload + data in one `write()`. The
|
||||
replacement may split that into two `sendall`s; TCP does not care.
|
||||
|
||||
The server replies with a type-7 header and a 44-byte payload for that
|
||||
`opId`. There is no extra data on the write reply (unlike reads).
|
||||
|
||||
A 1-sector VDDK write was 572 bytes on the wire: 16 + 44 + 512.
|
||||
|
||||
## Client-side split
|
||||
|
||||
`NfcAioInitSession` advertises a 64 KiB buffer. VDDK splits writes
|
||||
larger than that into 64 KiB chunks (VDDK programming guide). The
|
||||
Python client does the same: several IO requests of at most
|
||||
`NFC_AIO_BUFFER_SIZE` bytes, each with its own `opId`.
|
||||
|
||||
## Python replacement
|
||||
|
||||
`NfcDisk.write(start_sector, num_sectors, data)` in
|
||||
`openvixdisklib/nfc_open.py`. `open_disk(..., read_only=False)` selects
|
||||
flags `0x1a`. The drop-in handle exposes the same shape as VDDK:
|
||||
`connect(read_only=False)`, `open` without
|
||||
`VIXDISKLIB_FLAG_OPEN_READ_ONLY`, then `write`.
|
||||
|
||||
Integration tests create an empty 10 GiB disk, write known patterns,
|
||||
and read them back (`tests/integration/test_nfc_read_write.py`,
|
||||
`tests/integration/test_openvixdisklib.py`). Cross-check tests write
|
||||
with VDDK and with the replacement and read with both
|
||||
(`tests/integration/test_crosscheck.py`).
|
||||
@@ -0,0 +1,281 @@
|
||||
# Reverse-engineering procedure
|
||||
|
||||
This is the working method used to replace VDDK’s NBD path with Python.
|
||||
Protocol details live in `docs/nfc_auth.md`, `docs/nfc_open.md`,
|
||||
`docs/nfc_read.md`, and `docs/nfc_write.md`. The capture tool is
|
||||
described in `docs/ssl_hook.md`.
|
||||
This file is the **sequence of steps**, including dead ends, so later
|
||||
NFC work can follow the same loop instead of rediscovering it.
|
||||
|
||||
Scope so far: `VixDiskLib_ConnectEx` + `VixDiskLib_Open` +
|
||||
`VixDiskLib_Read` + `VixDiskLib_Write` against lab vCenter 8.0.1 /
|
||||
ESXi 8, transport `nbd`. Driver: `tests/integration/` (`TestBase`
|
||||
creates a temporary empty VM with a 10 GiB disk in `setUpClass` and
|
||||
destroys it in `tearDownClass`).
|
||||
|
||||
Rule from `AGENTS.md`: reuse pyVmomi for every public VIM operation.
|
||||
Only reimplement what pyVmomi does not expose.
|
||||
|
||||
## Loop
|
||||
|
||||
Each unknown stage (ticket SOAP, authd, NFC binary) went through:
|
||||
|
||||
1. **Name it** from VDDK logs and `strings` on the bundled libraries.
|
||||
2. **See it** on the wire (or prove that tcpdump cannot).
|
||||
3. **Replay** the smallest working subset in Python against the lab.
|
||||
4. **Write** findings into a protocol doc and keep the hook out of the
|
||||
library path.
|
||||
|
||||
Do not skip (2). Log lines such as `SESSIONID` or `useSSL=0` named the
|
||||
wrong wire command until the intercept existed.
|
||||
|
||||
## Lab and artifacts
|
||||
|
||||
| Item | Where / value |
|
||||
| ----------------------- | -------------------------------------------------------------------- |
|
||||
| VDDK 8.0.2 | `.vddk/` (`libvixDiskLib`, `libvddkVimAccess`, `libvim-types`) |
|
||||
| pyVmomi | `.venv` |
|
||||
| Known-good VDDK client | `tests/integration/test_vddk.py` / `tests/integration/vixdisklib.py` |
|
||||
| Verbose NFC logs | `vixDiskLib.nfc.LogLevel=4` in a temp VDDK config |
|
||||
| ctypes Open+Read driver | `/tmp/vddk_open_trace.py` (not in the library) |
|
||||
| SSL / `write` hook | `/tmp/sslhook.c` → `/tmp/sslhook.so` |
|
||||
|
||||
Always set `LD_LIBRARY_PATH` to `.vddk/` so VDDK uses its own
|
||||
`libssl.so.3`. Unset `LD_PRELOAD` before running the Python replacement;
|
||||
a leftover `write` hook will crash pyVmomi’s TLS.
|
||||
|
||||
## Step 1 — Map the public VDDK calls
|
||||
|
||||
`tests/integration/test_vddk.py` is the specification of what
|
||||
“success” looks like: login, open the temporary VM’s VMDK, write a
|
||||
known pattern, read it back.
|
||||
|
||||
Turn on VDDK verbose logging around `InitEx` / `ConnectEx` / `Open`. The
|
||||
logs split the work that the Python API hides:
|
||||
|
||||
- `ConnectEx` → VIM login only.
|
||||
- `Open` → NFC ticket, authd, NFC handshake, AIO open, then I/O.
|
||||
- `transport_modes=nbd` → URL form `vpxa-nfc://[ds] path.vmdk@esxi:902`.
|
||||
- `snapshot_ref` does not appear on the ticket SOAP call.
|
||||
|
||||
That mapping is the table at the top of `docs/nfc_auth.md`. It tells you
|
||||
which stage to reverse next and which arguments belong there (VM moref
|
||||
on the ticket, VMDK path on NFC `OPEN_FILE`).
|
||||
|
||||
## Step 2 — Strings and pyVmomi before any capture
|
||||
|
||||
`strings -a` on `.vddk/*.so` produced candidate tokens before a single
|
||||
packet was decoded:
|
||||
|
||||
- SOAP: `NfcService`, `NfcGetVmFiles`, `nfcService`, `ha-nfc`,
|
||||
`HostServiceTicket`.
|
||||
- authd: `SESSION`, `BANNER`, `THUMBPRINT_SHA2`, `PROXY`, `USER`,
|
||||
`PASS`, `SSL Required`.
|
||||
- NFC: `NFC_HANDSHAKE`, `NFC_CONNECTION_DATA`, `NFC_AIO_MSG_*`,
|
||||
`NFC_DISK`.
|
||||
|
||||
Then check whether pyVmomi already has the type:
|
||||
|
||||
```python
|
||||
from pyVmomi import vim
|
||||
hasattr(vim, "NfcService") # False
|
||||
hasattr(vim, "HostServiceTicket") # True
|
||||
```
|
||||
|
||||
`ServiceManager.QueryServiceList` on the live vCenter does **not** list
|
||||
NFC. `GET /sdk/nfcServiceVersions.xml` does (`urn:nfc` 7.0.3.2). The
|
||||
moref is hardcoded in VDDK (`nfcService` on vCenter, `ha-nfc` on ESXi).
|
||||
|
||||
Anything public (`SmartConnect`, `vim.VirtualMachine`,
|
||||
`HostServiceTicket`) stays in pyVmomi. Missing managed types are
|
||||
registered with `CreateManagedType` on the same SOAP stub so cookies
|
||||
and serialization are not reimplemented.
|
||||
|
||||
## Step 3 — Confirm tcpdump is the wrong tool for TLS stages
|
||||
|
||||
tcpdump on 443 and 902 shows TLS records only. That is enough to prove
|
||||
“something talks to vCenter then to ESXi:902”, and not enough for SOAP
|
||||
bodies, authd lines, or NFC headers.
|
||||
|
||||
VDDK logs name functions and AIO `opId` / `type` / `size`. They do not
|
||||
give magic numbers, path placement, or command spacing (`BANNER \r\n`).
|
||||
|
||||
An ESXi-impersonating service was considered (`AGENTS.md`) and not
|
||||
needed: the lab answers VDDK, so capturing the real client is simpler
|
||||
than simulating the server.
|
||||
|
||||
## Step 4 — Interpose OpenSSL (authd and SOAP)
|
||||
|
||||
VDDK 8.0.2 still calls `SSL_write` / `SSL_read`. A small `LD_PRELOAD`
|
||||
library logs those buffers as hex, tagged with the `SSL *` pointer.
|
||||
|
||||
Run a minimal ctypes program (`InitEx`, `ConnectEx`, `Open`, `Read`)
|
||||
under:
|
||||
|
||||
```bash
|
||||
export LD_LIBRARY_PATH=…/.vddk
|
||||
export LD_PRELOAD=/tmp/sslhook.so
|
||||
export SSLHOOK_LOG=/tmp/sslhook-open.log
|
||||
python /tmp/vddk_open_trace.py
|
||||
```
|
||||
|
||||
Parse offline:
|
||||
|
||||
1. Concatenate adjacent same-direction records (authd `SSL_read` is
|
||||
often one byte).
|
||||
2. Split by `SSL *`. vCenter HTTPS contains `POST /sdk` and SOAP.
|
||||
ESXi:902 contains `SESSION` / `PROXY`.
|
||||
3. An early hook without the pointer mixed both streams; always tag.
|
||||
|
||||
The vCenter stream identified the ticket as `NfcGetVmFiles` with moref
|
||||
`nfcService` and `xmlns="urn:vim25"` (not a guess from strings alone).
|
||||
The ESXi stream gave the authd command order, including the trailing
|
||||
space on `BANNER` and `THUMBPRINT_SHA2 PlainText`.
|
||||
|
||||
Hook implementation notes: `docs/ssl_hook.md`.
|
||||
|
||||
## Step 5 — Probe SOAP, then replay only what VDDK sends
|
||||
|
||||
With a SmartConnect session, raw SOAP posts were used to learn
|
||||
parameter names and which moref vCenter accepts:
|
||||
|
||||
- `ha-nfc` on vCenter → `ManagedObjectNotFound`.
|
||||
- `NfcGetServerNfcLibVersion` without `hostForAccess` → invalid
|
||||
argument; with a host moref → `11`.
|
||||
- `NfcGetVmFiles(vm)` → `HostServiceTicket` (VDDK read-only Open).
|
||||
- `NfcRandomAccessOpenReadonly(vm, diskDeviceKey, host)` → same ticket
|
||||
type, disk-scoped read.
|
||||
- `NfcRandomAccessOpenDisk(vm, diskDeviceKey, host)` → writable
|
||||
ticket. `GetVmFiles` plus `OPEN_FILE` flags `0x1a` fails with
|
||||
`VIX_E_FILE_READ_ONLY` (`0x0b`).
|
||||
|
||||
The replacement registers those methods and calls them through pyVmomi.
|
||||
It does not ship a hand-rolled SOAP client for login or tickets.
|
||||
|
||||
## Step 6 — Probe authd; record dead ends
|
||||
|
||||
Plaintext banner (`220 … SSL Required`), then `ssl.wrap_socket`.
|
||||
Commands before TLS drop the connection.
|
||||
|
||||
vCenter UID/password are **not** sent to port 902. Attempts that failed
|
||||
and must not be retried for this ticket type:
|
||||
|
||||
| Attempt | Result |
|
||||
| ------------------------------------- | ------------------------------------- |
|
||||
| `USER` / `PASS` (vCenter account) | `530 Login incorrect` |
|
||||
| `USER` / `PASS` with `sessionId` | `530 Login incorrect` |
|
||||
| `SESSIONID <sessionId>` | `530 Please login with USER and PASS` |
|
||||
| `CONNECT_VPXA` after TLS | `530 Please login with USER and PASS` |
|
||||
| Wait for a reply after `SESSION` | Hang until `BANNER` / `PROXY` follow |
|
||||
| `THUMBPRINT_SHA2` with SHA-1 digest | `501 Invalid arguments` |
|
||||
|
||||
The working sequence is in `docs/nfc_auth.md`. `useSSL=0` in the VDDK
|
||||
log means skip a **second** NFCSSL wrap, not skip TLS on 902.
|
||||
|
||||
Replay: `openvixdisklib/nfc_auth.py` /
|
||||
`tests/integration/test_nfc_auth.py`. Stop at `200 Connect`.
|
||||
|
||||
## Step 7 — NFC binary: extend the hook to `write` / `read`
|
||||
|
||||
After `PROXY`, `SSL_write` on the ESXi `SSL *` goes silent. VDDK logs
|
||||
`useSSL=0` / “plain-text connection is deprecated” and then NFC
|
||||
function names. The bytes are `write(SSL_get_fd(ssl), …)` / `read` on
|
||||
peer port 902.
|
||||
|
||||
The hook was extended to those syscalls, filtered with `getpeername`
|
||||
port 902, and mutex-locked (VDDK is multi-threaded). Skip TLS records
|
||||
(`16 03` / `17 03`) left over from the authd phase.
|
||||
|
||||
Correlate each frame with the verbose log line that has the same
|
||||
`type` and `size` (`NfcAioSendMessage: opId = … type = … size = …`).
|
||||
Name the types from the consecutive `NFC_AIO_MSG_*` string table in
|
||||
`libvixDiskLib.so`. Lengths 4 and 7 on the connection-data message are
|
||||
the ASCII strings `vddk` and `nbdmode` sent in the next two writes.
|
||||
|
||||
Classic NFC uses a 264-byte padded struct; AIO uses a 16-byte header
|
||||
(`magic 0xA100DA7A`) plus payload; path / DDB key / sector data are
|
||||
extra writes not included in `size`.
|
||||
|
||||
## Step 8 — Replay the smallest subset, then compare to VDDK
|
||||
|
||||
Python must **dup the authd fd** and send NFC as raw TCP.
|
||||
`SSLSocket.send` would encrypt; `unwrap()` would `SSL_shutdown`. VDDK
|
||||
does neither.
|
||||
|
||||
`openvixdisklib/nfc_open.py` replays handshake + AIO `OPEN_SESSION` /
|
||||
sockopts / resource pool / `OPEN_FILE`. VDDK’s extra `DDB_GET` keys
|
||||
were omitted once a file handle was enough to read. Proof of open:
|
||||
`tests/integration/test_nfc_open.py`.
|
||||
|
||||
Do not copy every VDDK message. Copy what the server requires for the
|
||||
Python API you are replacing.
|
||||
|
||||
## Step 9 — Vary `VixDiskLib_Read` until IO fields stop moving
|
||||
|
||||
A single-sector read is not enough to decode `NFC_AIO_MSG_IO`. Drive
|
||||
VDDK with several `(startSector, numSectors)` pairs in one Open
|
||||
(including `n=128` = 64 KiB and `n=129`) under the `write`/`read` hook.
|
||||
|
||||
What that comparison showed:
|
||||
|
||||
- Wire units are bytes (`offset = start * 512`, `length = n * 512`).
|
||||
- Request size stays 44; data is extra after the payload.
|
||||
- VDDK sends **one** request even when `length > 65536`. The server
|
||||
replies with several type-7 messages that share `opId`, each with a
|
||||
chunk length at payload offset 32 (max 65536).
|
||||
- Treating offset 36 as `NFC_DISK` (`2`) was a 1-sector coincidence;
|
||||
VDDK repeats the byte length there.
|
||||
- Zeros on the wire are real transferred zeros, not a sparse skip.
|
||||
|
||||
Replay: `NfcDisk.read` loops on fragments until `length` bytes arrive.
|
||||
Proof: `tests/integration/test_nfc_read_write.py` writes a known pattern
|
||||
(including a 129-sector read that must assemble two fragments) and
|
||||
checks the bytes that came back.
|
||||
|
||||
## Step 10 — Writes from the same IO message
|
||||
|
||||
`VixDiskLib_Write` uses the same 44-byte `NFC_AIO_MSG_IO` layout as
|
||||
read. The direction field at offset 8 is `0` instead of `1`, and the
|
||||
sector bytes are sent after the payload (like the path on
|
||||
`OPEN_FILE`). Writable `OPEN_FILE` flags are `0x1a` (the captured
|
||||
read-only flags `0x1e` with bit `0x04` cleared, matching
|
||||
`VIXDISKLIB_FLAG_OPEN_READ_ONLY` in `.vddk/vixDiskLib.h`).
|
||||
|
||||
Writable `OPEN_FILE` still failed with `VIX_E_FILE_READ_ONLY` until
|
||||
the ticket switched from `NfcGetVmFiles` to `NfcRandomAccessOpenDisk`
|
||||
(`libvim-types.so`: vmodl `randomAccessOpen` ↔ WSDL
|
||||
`NfcRandomAccessOpenDisk`). Integration tests create a temporary empty
|
||||
10 GiB VM for the run so writes cannot land on other lab disks.
|
||||
|
||||
The Python client splits writes larger than 64 KiB; it does not send a
|
||||
single oversized write the way VDDK sends an oversized read. Details:
|
||||
`docs/nfc_write.md`. Proof: write then read in
|
||||
`tests/integration/test_nfc_read_write.py` and the VDDK cross-check in
|
||||
`tests/integration/test_crosscheck.py`.
|
||||
|
||||
## What to write down
|
||||
|
||||
After a stage works:
|
||||
|
||||
| Document | Contents |
|
||||
| --------------------------------------- | --------------------------------------------- |
|
||||
| `docs/nfc_auth.md` | Ticket SOAP + authd wire format |
|
||||
| `docs/nfc_open.md` | Classic NFC + AIO open |
|
||||
| `docs/nfc_read.md` | AIO IO / `VixDiskLib_Read` |
|
||||
| `docs/nfc_write.md` | AIO IO / `VixDiskLib_Write` |
|
||||
| `docs/ssl_hook.md` | Capture tool only |
|
||||
| `docs/reverse_engineering_procedure.md` | This procedure (update when the method changes) |
|
||||
|
||||
Keep the hook and ctypes driver under `/tmp`. They are not part of the
|
||||
replacement library.
|
||||
|
||||
## Next stages (same procedure)
|
||||
|
||||
Not yet reversed, same loop as above:
|
||||
|
||||
- `DDB_GET` / disk geometry, compression, encrypted disks
|
||||
- `NFC_DELTA_DISK`, CBT / `QueryAllocatedBlocks`
|
||||
- `VixDiskLib_GetInfo` capacity
|
||||
- Host-switch AIO messages
|
||||
- `useSSL=1` (second NFCSSL wrap)
|
||||
- Direct ESXi `ha-nfc` without vCenter `vpxa-nfc`
|
||||
@@ -0,0 +1,145 @@
|
||||
# SSL hook for VDDK protocol capture
|
||||
|
||||
VDDK’s NBD path is TLS end to end: SOAP to vCenter on 443, then authd/NFC
|
||||
to ESXi on 902. Packet captures on those ports are ciphertext, so they
|
||||
cannot show command names, tickets, or NFC frames.
|
||||
|
||||
This project used a small `LD_PRELOAD` library (`sslhook.c`, built to
|
||||
`sslhook.so`) to log OpenSSL plaintext while a ctypes wrapper ran
|
||||
`VixDiskLib_ConnectEx` / `VixDiskLib_Open`. The authd sequence in
|
||||
`docs/nfc_auth.md` was recovered from that log, not from VDDK source.
|
||||
|
||||
The hook is a reverse-engineering aid. It is not part of the Python
|
||||
NFC client.
|
||||
|
||||
## Why not tcpdump or VDDK logs
|
||||
|
||||
| Approach | What it shows | Gap |
|
||||
| -------------------------------- | -------------------------------------------------- | ------------------------------------------------ |
|
||||
| tcpdump on 443 / 902 | TLS records | No SOAP bodies, no authd lines, no NFC frames |
|
||||
| `vixDiskLib.nfc.LogLevel=4` | Function names, `opId` / `type` / `size` | Not the bytes on the wire |
|
||||
| Strings in `libvixDiskLib.so` | Command tokens (`SESSION`, `PROXY`, `BANNER`) | Not order, spacing, or replies |
|
||||
| SSL hook on `SSL_write`/`read` | Exact buffers before encrypt / after decrypt | Must split connections and reassemble 1-byte I/O |
|
||||
|
||||
VDDK logs were still useful to *name* AIO message types after the hex
|
||||
dump showed `type` and `size`. The hook supplied the actual framing.
|
||||
|
||||
## How `LD_PRELOAD` interposition works
|
||||
|
||||
The hook exports `SSL_write` and `SSL_read` with OpenSSL’s signatures.
|
||||
When the process starts with `LD_PRELOAD=/path/sslhook.so`, the dynamic
|
||||
linker binds VDDK’s calls to those symbols instead of `libssl`.
|
||||
|
||||
Each wrapper:
|
||||
|
||||
1. Resolves the real OpenSSL function with `dlsym(RTLD_NEXT, ...)`.
|
||||
2. Logs the plaintext buffer.
|
||||
3. Calls the real function so the session is unchanged.
|
||||
|
||||
```
|
||||
VixDiskLib --> SSL_write (hook) --> log hex --> SSL_write (libssl)
|
||||
VixDiskLib <-- SSL_read (hook) <-- log hex <-- SSL_read (libssl)
|
||||
```
|
||||
|
||||
`SSL_write` logs **before** encryption. `SSL_read` calls OpenSSL first,
|
||||
then logs `n` decrypted bytes when `n > 0`.
|
||||
|
||||
## Implementation notes
|
||||
|
||||
The working copy lived under `/tmp` during capture (`/tmp/sslhook.c`).
|
||||
Behavior that mattered for parsing:
|
||||
|
||||
- Log path from `SSLHOOK_LOG`, default `/tmp/sslhook-open.log`.
|
||||
- Unbuffered writes (`_IONBF`) so a crash still leaves a complete file.
|
||||
- Each record tagged with the `SSL *` pointer so vCenter HTTPS and
|
||||
ESXi:902 are separable. An earlier version omitted the pointer and
|
||||
mixed both streams into one timeline.
|
||||
- Payload stored as hex, not mixed ASCII, so binary NFC frames stay
|
||||
unambiguous.
|
||||
|
||||
Record layout:
|
||||
|
||||
```
|
||||
==== W 0x7f8a1234 46 ====
|
||||
53455353494f4e2035326364656263352d...0d0a
|
||||
```
|
||||
|
||||
| Field | Meaning |
|
||||
| ------- | ---------------------------------------------------- |
|
||||
| `W`/`R` | Write (plaintext to encrypt) or read (decrypted) |
|
||||
| `%p` | `SSL *` for this socket |
|
||||
| length | Byte count of this OpenSSL call |
|
||||
| hex | Buffer contents |
|
||||
|
||||
`SSL_read` is often **one byte per call**. A 220 banner is therefore
|
||||
dozens of `R 1` records. Adjacent records with the same `SSL *` and
|
||||
direction must be concatenated before parsing lines or NFC headers.
|
||||
|
||||
OpenSSL 3 also has `SSL_write_ex` / `SSL_read_ex`. This VDDK 8.0.2
|
||||
build still used `SSL_write` / `SSL_read`, so those two symbols were
|
||||
enough. If a later library switches APIs, the hook would need matching
|
||||
wrappers.
|
||||
|
||||
## How it was used for authd
|
||||
|
||||
A minimal ctypes program loaded `libvixDiskLib.so`, called
|
||||
`VixDiskLib_InitEx`, `ConnectEx` (UID to vCenter, `nbd`), and `Open` on
|
||||
the lab VMDK. The process was started as:
|
||||
|
||||
```bash
|
||||
export LD_LIBRARY_PATH=/home/ubuntu/workspace/vmware_nbd_tests/.vddk
|
||||
export LD_PRELOAD=/tmp/sslhook.so
|
||||
export SSLHOOK_LOG=/tmp/sslhook-open.log
|
||||
python /tmp/vddk_open_trace.py
|
||||
```
|
||||
|
||||
`LD_LIBRARY_PATH` is required so VDDK uses its bundled `libssl.so.3`.
|
||||
`LD_PRELOAD` still interposes that copy.
|
||||
|
||||
After the run, records were grouped by `SSL *`. The ESXi connection is
|
||||
the one whose writes contain `SESSION ` and `PROXY `. Concatenating
|
||||
that stream after the TLS handshake produced:
|
||||
|
||||
```
|
||||
C -> SESSION <sessionId>\r\n
|
||||
C -> BANNER \r\n
|
||||
S -> 220 VMware Authentication Daemon Version 1.10: ...\r\n
|
||||
C -> THUMBPRINT_SHA2 PlainText\r\n
|
||||
S -> 200 <SHA-256 thumbprint>\r\n
|
||||
C -> PROXY vpxa-nfc\r\n
|
||||
S -> 200 Connect ha-nfc\r\n
|
||||
```
|
||||
|
||||
The same log also showed the SOAP `NfcGetVmFiles` body on the vCenter
|
||||
`SSL *` (`xmlns="urn:vim25"`, moref `nfcService`). That is how the
|
||||
ticket call was identified as `NfcGetVmFiles` rather than guessing
|
||||
from `libvim-types` strings alone.
|
||||
|
||||
Details that only the hex dump made obvious:
|
||||
|
||||
- `BANNER` includes a trailing space (`BANNER \r\n`).
|
||||
- `THUMBPRINT_SHA2` argument is the literal `PlainText`, not the
|
||||
ticket SHA-1 thumbprint.
|
||||
- `SESSION` has no reply; waiting for a line after it looks like a hang.
|
||||
- Ticket `sessionId` is the UUID string on the `SESSION` line.
|
||||
|
||||
Those facts are written up in `docs/nfc_auth.md`. The Python client in
|
||||
`openvixdisklib/nfc_auth.py` replays this sequence; it does not use
|
||||
the hook.
|
||||
|
||||
After `200 Connect`, NFC is **not** on `SSL_write`. VDDK uses
|
||||
`write`/`read` on `SSL_get_fd` (`useSSL=0`). A later hook that also
|
||||
interposed those syscalls, filtered to peer port 902, recovered the
|
||||
264-byte handshake and AIO frames in `docs/nfc_open.md`. TLS record
|
||||
bytes (`16 03` / `17 03`) on that fd are the authd phase and must be
|
||||
skipped.
|
||||
|
||||
## Limits
|
||||
|
||||
- The hook sees every OpenSSL client in the process (VDDK and, if the
|
||||
same interpreter is used, anything else linked to OpenSSL). Filter by
|
||||
`SSL *`.
|
||||
- It does not decode TLS handshakes, certificates, or SOAP envelopes;
|
||||
that is done offline on the hex log.
|
||||
- It must not ship in a production VDDK replacement. Keep it out of
|
||||
the library path used by `openvixdisklib/nfc_auth.py`.
|
||||
Reference in New Issue
Block a user