justin/OpenVixDiskLib
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Allow retrieving compressed chunks

Browse Source 
VDDK always decompresses the chunks it retrieves. However, some
callers may be interested in the compressed chunks, which may
be forwarded to another service involved in the backup process.

We'll add a read flag (skip_decompression). If set, the read
operation will return a "ReadResult" container, containing
a list of fragments and their compressed / decompressed lengths.

If the returned buffer size matches the AIO buffer size (which
now becomes configurable), at most one fragment will be returned.

While at it, we're adding perf tests that check various AIO buffer
sizes, cross checking against VDDK.
This commit is contained in:
Lucian Petrut
2026-09-17 15:29:11 +00:00
parent f0db6e166b
commit 2099591245
12 changed files with 914 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/nfc_open.md
+29 -10
View File
@@ -158,8 +158,26 @@ obtain a file handle or to read sector 0.
### OPEN_SESSION / sockopts / resource pool
VDDK sends 16 zero bytes (`OPEN_SESSION`; server replies with 16 zeros),
12 zero bytes (`SET_SOCK_OPTS`; server returns send/recv buffer sizes
`OPEN_SESSION` payload is 16 bytes, little-endian:
| Offset | Type | Meaning |
| ------ | -------- | ---------------------------------------------------- |
| 0 | `uint32` | 0 (unused in captures) |
| 4 | `uint32` | AIO buffer size in **bytes** (VDDK default 65536) |
| 8 | `uint32` | Buffer count (VDDK ``nfcAio.Session.BufCount``) |
| 12 | `uint32` | 0 |
VDDK config `vixDiskLib.nfcAio.Session.BufSizeIn64KB` is that byte size
divided by 64 KiB (`1` → 65536, `32` → 2097152). The server replies
with 16 zeros; it still **uses** the requested size for IO extras.
A 129-sector read is two fragments at 64 KiB, and one 66048-byte
fragment at 2 MiB. Lab ESXi 8 accepted 2 MiB (`BufCount` 1 and 4) and
rejected 16 MiB and 32 MiB (`OPEN_SESSION` AIO error). Broadcom's 16 MiB
figure is session memory (`size × count`), not a larger extra; the
per-buffer max on the wire is 2 MiB. Probe:
`docs/probing_samples/vddk_aio_bufsize_probe.py`.
`SET_SOCK_OPTS` is 12 zero bytes (server returns send/recv buffer sizes
and a `uint32` flag), then `uint32` 1 (`SET_RES_POOL`, log: “Setting
Resource Pool(1)”).
@@ -205,14 +223,15 @@ classic type 4 `NFC_SESSION_COMPLETE`.
## OpenVixDiskLib
| Piece | Module |
| ----------------------------- | ----------------------------------------------- |
| VIM + authd | `openvixdisklib.nfc_auth.authenticate` |
| Dup fd, skip TLS for NFC | `openvixdisklib.nfc_open.takeover_authd_socket` |
| Second TLS for nbdssl | `openvixdisklib.nfc_open.wrap_nfcssl_socket` |
| FastLZ for NBD compression | `openvixdisklib.fastlz` (pip `pyfastlz`) |
| Handshake + AIO + OPEN_FILE | `openvixdisklib.nfc_open.open_disk` |
| Sector read / write / close | `openvixdisklib.nfc_open.NfcDisk` |
| Piece | Module |
| ------------------------------- | --------------------------------------------------------- |
| VIM + authd | `openvixdisklib.nfc_auth.authenticate` |
| Dup fd, skip TLS for NFC | `openvixdisklib.nfc_open.takeover_authd_socket` |
| Second TLS for nbdssl | `openvixdisklib.nfc_open.wrap_nfcssl_socket` |
| FastLZ for NBD compression | `openvixdisklib.fastlz` (pip `pyfastlz`) |
| Handshake + AIO + OPEN_FILE | `openvixdisklib.nfc_open.open_disk` |
| AIO extra size / pool count | `open_disk(..., aio_buffer_size=, aio_buffer_count=)` |
| Sector read / write / close | `openvixdisklib.nfc_open.NfcDisk` |
Run:
docs/nfc_read.md
+70 -16
View File
@@ -26,11 +26,13 @@ length = numSectors * sectorSize
| `VixDiskLib_Read(h, 0, 128, buf)` | IO length 65536 (AIO buffer size), one fragment |
| `VixDiskLib_Read(h, 0, 129, buf)` | One request of 66048; **two** reply fragments |
VDDK does **not** split a `Read` larger than 64 KiB into multiple
requests. The client sends one AIO message; the server answers with
one or more same-`opId` replies, each carrying at most
`NFC_AIO_BUFFER_SIZE` (65536) data bytes. `NfcAioInitSession` logged
that buffer size and count 4 during open.
VDDK does **not** split a `Read` larger than the AIO buffer into
multiple requests. The client sends one AIO message; the server
answers with one or more same-`opId` replies, each carrying at most
the OPEN_SESSION buffer size (VDDK default 65536).
`vixDiskLib.nfcAio.Session.BufSizeIn64KB=32` advertises 2 MiB; a
129-sector read then returns **one** 66048-byte extra, and a 2 MiB +
512 read returns 2097152 + 512. See `docs/nfc_open.md` (OPEN_SESSION).
Sparse regions are still transferred as zeros. A read of 8 sectors at
LBA 8 on this disk was 4096 zero bytes on the wire, not a skip.
@@ -81,22 +83,23 @@ payload + `chunkLength` data bytes.
Reply payload (handle is zeroed; lengths describe this fragment):
| Offset | Type | Meaning |
| ------ | -------- | ----------------------------------------------- |
| 0 | `uint64` | `0` |
| 8 | `uint64` | `1` (read) |
| 16 | `uint64` | Byte offset of the **request** |
| 24 | `uint32` | Total request length |
| 28 | `uint32` | Byte offset of this fragment (`0`, `65536`, …) |
| 32 | `uint32` | This fragment’s byte length |
| 36 | `uint32` | Same as offset 32 |
| 40 | `uint32` | `0` |
| Offset | Type | Meaning |
| ------ | -------- | -------------------------------------------------------------------- |
| 0 | `uint64` | `0` |
| 8 | `uint64` | `1` (read) |
| 16 | `uint64` | Byte offset of the **request** on disk |
| 24 | `uint32` | Total request length |
| 28 | `uint32` | Fragment byte offset **in this request** (`0`, `65536`, …), not disk |
| 32 | `uint32` | This fragment’s uncompressed byte length |
| 36 | `uint32` | Same as offset 32, or compressed extra size when type is FastLZ |
| 40 | `uint32` | `0` |
When there is a single fragment, offsets 24–31 look like a `uint64`
length (the fragment offset is 0). The 129-sector capture shows why
they are two `uint32`s: fragment 0 has `(66048, 0)` then chunk 65536;
fragment 1 has `(66048, 65536)` then chunk 512. `0x00010000` at offset
28 is the byte offset, not a 0-based index.
28 is the byte offset, not a 0-based index. Disk byte address of a
fragment is request offset (payload 16) plus payload 28.
Read loop: receive fragments with that `opId` until the concatenated
data length equals the request. Use the `uint32` at payload offset 32
@@ -113,6 +116,9 @@ S: type=7 opId=18 size=44 dest=0 chunk=65536 + 65536 data
S: type=7 opId=18 size=44 dest=65536 chunk=512 + 512 data
```
`dest` in that dump is payload offset 28 (`ReadFragment.dest`): 0 and
65536 are positions in this 66048-byte read, not sector numbers.
## Lab check
Integration tests create an empty 10 GiB thin disk, write a repeating
@@ -135,6 +141,54 @@ The integration test writes and then reads the captured VDDK ranges
(including a 129-sector transfer that must assemble two read
fragments).
## Skip decompression (OpenVixDiskLib extension)
`VixDiskLib_Read` always fills `buf` with uncompressed sector bytes.
OpenVixDiskLib can skip FastLZ decode so a backup application can
forward the compressed data as-is, avoiding unnecessary re-compression.
`NfcDisk.readinto(..., skip_decompression=True)` and
`VixDiskLibHandle.read(..., skip_decompression=True)` still send one
IO request and wait until uncompressed `filled == length`. They do
**not** decompress. Extras are packed densely from offset 0 of `buf`.
`ReadResult.fragments` describes each extra. Type `2` extras are
FastLZ; type `0` fallbacks are raw. Concatenating extras is not a
valid FastLZ stream; the caller must use the table to split them.
| Field | Meaning |
| ---------------------- | ------------------------------------------------------------------------------------------------ |
| `dest` | Byte offset **in this uncompressed read** (NFC payload 28). Not a disk LBA or VMDK file offset. |
| `uncompressed_length` | Uncompressed fragment size (NFC payload 32). |
| `compression_type` | `NFC_COMPRESSION_NONE` (0) or `NFC_COMPRESSION_FASTLZ` (2). |
| `offset` | Start of this extra in packed `buf` (receive order, densely from 0). |
| `length` | Extra size on the wire. |
Disk byte address of a fragment is `start_sector * 512 + dest`. A
129-sector `read` from sector 0 or from sector 1000 still reports
`dest=0` and `dest=65536` when extras are 64 KiB.
`buf` is sized for the uncompressed request, so it is always large
enough. Default `read` still decompresses; `fragments` is empty and
`compressed_length` is still the extra bytes on the wire.
`skip_decompression` with a plain (no FASTLZ) open only records raw
extras (`compressed_length == uncompressed_length`).
This is not `VixDiskLib_Read`. Do not add an open flag for it;
compression on the wire is already the FASTLZ open flag.
A 32 MiB read at 64 KiB extras is 512 fragments in **one** result. A
2 MiB OPEN_SESSION extra (`aio_buffer_size=2097152`) is 16 fragments
for the same read. One dest PUT per extra is not viable.
```
uncompressed request (offsets in this read, not on disk)
|---------------- 64KiB --|-- 64KiB --|-- ... --|
dest=0 dest=65536
extra (FastLZ or raw) extra (FastLZ or raw)
buf when skip_decompression=True: extras packed densely from offset 0
```
## What is still VDDK-only
- zlib and skipz NBD compression flags
docs/nfc_write.md
+6 -5
View File
@@ -85,8 +85,9 @@ A 1-sector VDDK write was 572 bytes on the wire: 16 + 44 + 512.
## Fragments and the single reply
`NfcAioInitSession` advertises a 64 KiB buffer. Extra per type-7
message is at most that size. VDDK does **not** issue a new `opId` per
`OPEN_SESSION` advertises the AIO buffer size (VDDK default 64 KiB;
`BufSizeIn64KB` can raise it). Extra per type-7 message is at most
that size. VDDK does **not** issue a new `opId` per
chunk, and it does **not** coalesce separate `VixDiskLib_Write` calls
(eight 8 KiB writes stayed eight IOs). One public write becomes N
client type-7 messages with the **same** `opId`, then **one** 44-byte
@@ -105,9 +106,9 @@ for one reply per chunk; raising the window did not match VDDK
throughput because VDDK pays one RTT per `Write`, not per fragment.
`NfcAioFlushCoalescedWrites` is server-side (`nfcAioServer.c`), not a
client merge of API writes. OPEN_SESSION is 16 zero bytes both ways, so
the logged AIO buffer count of 4 is a VDDK client default
(`vixDiskLib.nfcAio.Session.BufCount`), not a server cap.
client merge of API writes. Buffer count is
`vixDiskLib.nfcAio.Session.BufCount` (OPEN_SESSION offset 8); size is
`BufSizeIn64KB` (offset 4, in bytes). `docs/nfc_open.md`.
## OpenVixDiskLib
docs/probing_samples/vddk_aio_bufsize_probe.py
+224
View File
@@ -0,0 +1,224 @@
#!/usr/bin/env python3
"""Probe VDDK ``vixDiskLib.nfcAio.Session.BufSizeIn64KB`` vs NFC read extras.
Not part of the library. Creates a temp lab VM, runs native VDDK over ``nbd``
under ``strace``, and prints OPEN_SESSION payloads plus IO reply chunk
lengths. BufSizeIn64KB=1 is 64 KiB; 32 is 2 MiB.
"""
from __future__ import annotations
import os
import pickle
import re
import struct
import subprocess
import sys
import tempfile
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
os.environ.pop("LD_PRELOAD", None)
from tests.integration import vixdisklib # noqa: E402
from tests.integration.base import ( # noqa: E402
SECTOR_SIZE,
create_lab_vm,
destroy_lab_vm,
ensure_vddk_library_path,
)
_REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), "../.."))
_VDDK = os.path.join(_REPO, ".vddk")
_AIO_MAGIC = 0xA100DA7A
_AIO_HDR = 16
_NFC_AIO_MSG_OPEN_SESSION = 2
_NFC_AIO_MSG_IO = 7
_NFC_AIO_IO_READ = 1
# 129 sectors: two 64 KiB-class fragments today. 4097: 2 MiB + 512.
_READS = ((129, "129s"), (4097, "2MiB+512"))
def _vddk_config(directory: str, buf_size_in_64kb: int, buf_count: int) -> str:
path = os.path.join(directory, "vddk.config")
log = os.path.join(directory, "vddk.log")
with open(path, "w", encoding="utf-8") as config:
config.write(f"tmpDirectory={directory}\n")
config.write(f"log.fileName={log}\n")
config.write("log.fileLevel=verbose\n")
config.write("vixDiskLib.nfc.LogLevel=4\n")
config.write("vixDiskLib.transport.LogLevel=4\n")
config.write(f"vixDiskLib.nfcAio.Session.BufSizeIn64KB={buf_size_in_64kb}\n")
config.write(f"vixDiskLib.nfcAio.Session.BufCount={buf_count}\n")
return path
def _worker(lab_pkl: str, work_dir: str, buf_size_in_64kb: int, buf_count: int) -> None:
ensure_vddk_library_path()
with open(lab_pkl, "rb") as pickle_file:
lab = pickle.load(pickle_file)
config_path = _vddk_config(work_dir, buf_size_in_64kb, buf_count)
handle = vixdisklib.VixDiskLibHandle(
vixdisklib_compatibility_version="8.0", config_path=config_path
)
kwargs = {
"server_name": lab.host,
"port": lab.port,
"thumbprint": lab.thumbprint,
"username": lab.username,
"password": lab.password,
"vmx_spec": lab.vmx_spec,
"transport_modes": "nbd",
"read_only": True,
}
with (
handle.connect(**kwargs) as conn,
handle.open(
conn, lab.disk_path, flags=vixdisklib.VIXDISKLIB_FLAG_OPEN_READ_ONLY
) as disk,
):
print("transport", handle.get_transport_mode(disk), flush=True)
for n_sectors, label in _READS:
buf = vixdisklib.get_buffer(n_sectors * SECTOR_SIZE)
handle.read(disk, 0, n_sectors, buf)
print(f"read {label} ok", flush=True)
handle.exit()
def _decode_strace_hex(quoted: str) -> bytes:
parts = re.findall(r"\\x([0-9a-fA-F]{2})", quoted)
return bytes(int(part, 16) for part in parts)
def parse_strace(path: str) -> tuple[list[bytes], list[tuple[int, int, int]]]:
"""Return client OPEN_SESSION payloads and **server** read-reply chunks.
VDDK reads the 16-byte AIO header in one syscall and the payload in
the next, so bytes are concatenated per fd before parsing.
"""
syscall_re = re.compile(r'(read|write|recv|send)\((\d+),\s*"(.*?)"')
writes: dict[int, bytearray] = {}
reads: dict[int, bytearray] = {}
with open(path, encoding="utf-8", errors="replace") as strace_file:
for line in strace_file:
match = syscall_re.search(line)
if not match:
continue
op, fd_s, quoted = match.group(1), match.group(2), match.group(3)
buf = _decode_strace_hex(quoted)
if not buf:
continue
fd = int(fd_s)
bucket = writes if op in ("write", "send") else reads
bucket.setdefault(fd, bytearray()).extend(buf)
def walk(buf: bytes, collect_open: bool, collect_io: bool) -> None:
offset = 0
while offset + _AIO_HDR <= len(buf):
magic, msg_type, size, _opid = struct.unpack_from("<IIII", buf, offset)
if magic != _AIO_MAGIC:
offset += 1
continue
payload = buf[offset + _AIO_HDR : offset + _AIO_HDR + size]
if collect_open and msg_type == _NFC_AIO_MSG_OPEN_SESSION and len(payload) >= 16:
open_sessions.append(payload[:16])
if collect_io and msg_type == _NFC_AIO_MSG_IO and len(payload) >= 40:
opcode = struct.unpack_from("<Q", payload, 8)[0]
if opcode & 0xFFFFFFFF == _NFC_AIO_IO_READ:
dest, chunk_len, extra_len = struct.unpack_from("<III", payload, 28)
io_reads.append((dest, chunk_len, extra_len))
offset += _AIO_HDR + size
open_sessions: list[bytes] = []
io_reads: list[tuple[int, int, int]] = []
for buf in writes.values():
walk(bytes(buf), collect_open=True, collect_io=False)
for buf in reads.values():
walk(bytes(buf), collect_open=False, collect_io=True)
return open_sessions, io_reads
def _interesting_log_lines(log_path: str) -> list[str]:
keys = (
"Buffer Size",
"BufCount",
"BufSize",
"AIO session",
"Aio Session",
"maximum session",
"Req. buffer",
)
lines: list[str] = []
if not os.path.isfile(log_path):
return lines
with open(log_path, encoding="utf-8", errors="replace") as log_file:
for line in log_file:
if any(key in line for key in keys):
lines.append(line.rstrip())
return lines
def _run_traced(lab_pkl: str, buf_size_in_64kb: int, buf_count: int) -> None:
work_dir = tempfile.mkdtemp(prefix=f"vddk-aio-bufsize-{buf_size_in_64kb}-")
strace_path = os.path.join(work_dir, "nfc.strace")
python = sys.executable
cmd = [
"strace",
"-f",
"-x",
"-s",
"96",
"-e",
"trace=read,write,readv,writev,send,recv,sendto,recvfrom",
"-o",
strace_path,
python,
__file__,
"--worker",
lab_pkl,
work_dir,
str(buf_size_in_64kb),
str(buf_count),
]
env = os.environ.copy()
env.pop("LD_PRELOAD", None)
lib_path = env.get("LD_LIBRARY_PATH", "")
env["LD_LIBRARY_PATH"] = _VDDK if not lib_path else f"{_VDDK}:{lib_path}"
print(f"\n=== BufSizeIn64KB={buf_size_in_64kb} BufCount={buf_count} ===")
print("work_dir", work_dir)
proc = subprocess.run(cmd, env=env, check=False, text=True, capture_output=True)
sys.stdout.write(proc.stdout)
sys.stderr.write(proc.stderr)
print("worker exit", proc.returncode)
for line in _interesting_log_lines(os.path.join(work_dir, "vddk.log")):
print("LOG", line)
open_sessions, io_reads = parse_strace(strace_path)
for payload in open_sessions:
ints = struct.unpack("<IIII", payload)
print("OPEN_SESSION hex", payload.hex(), "u32", ints)
print("read fragments (dest, chunk_len, extra_len):")
for dest, chunk_len, extra_len in io_reads:
print(f" dest={dest} chunk_len={chunk_len} extra_len={extra_len}")
if io_reads:
print("max chunk_len", max(item[1] for item in io_reads))
def main() -> None:
if "--worker" in sys.argv:
_, lab_pkl, work_dir, buf_size, buf_count = sys.argv[1:]
_worker(lab_pkl, work_dir, int(buf_size), int(buf_count))
return
ensure_vddk_library_path()
lab = create_lab_vm()
lab_pkl = "/tmp/vddk-aio-bufsize-lab.pkl"
try:
with open(lab_pkl, "wb") as pickle_file:
pickle.dump(lab, pickle_file)
print("lab", lab.disk_path, lab.vm_moref)
for buf_size, buf_count in ((1, 1), (32, 1)):
_run_traced(lab_pkl, buf_size, buf_count)
finally:
destroy_lab_vm(lab)
if __name__ == "__main__":
main()
docs/reverse_engineering_procedure.md
+3 -1
View File
@@ -250,7 +250,9 @@ What that comparison showed:
- Request size stays 44; data is extra after the payload.
- VDDK sends **one** request even when `length > 65536`. The server
replies with several type-7 messages that share `opId`, each with a
chunk length at payload offset 32 (max 65536).
chunk length at payload offset 32 (max = OPEN_SESSION bufSize;
default 65536). `vixDiskLib.nfcAio.Session.BufSizeIn64KB=32` makes
that 2 MiB (`docs/probing_samples/vddk_aio_bufsize_probe.py`).
- Treating offset 36 as `NFC_DISK` (`2`) was a 1-sector coincidence;
VDDK repeats the byte length there.
- Zeros on the wire are real transferred zeros, not a sparse skip.