diff --git a/README.md b/README.md index 445b948..a84b483 100644 --- a/README.md +++ b/README.md @@ -18,9 +18,9 @@ Implemented against vCenter 8 / ESXi 8. Default transport is `nbdssl` - `VixDiskLib_Read` - `VixDiskLib_Write` -Not implemented: compression open flags, CBT / allocated-block -queries, disk geometry (`DDB_GET`), encrypted disks, and direct ESXi -`ha-nfc` without vCenter `vpxa-nfc`. +Not implemented: compression open flags other than FastLZ, CBT / +allocated-block queries, disk geometry (`DDB_GET`), encrypted disks, +and direct ESXi `ha-nfc` without vCenter `vpxa-nfc`. ## Install @@ -64,6 +64,7 @@ VDDK-shaped handle. | `openvixdisklib/openvixdisklib.py` | Drop-in handle (`connect` / `open` / `read` / `write`) | | `openvixdisklib/nfc_auth.py` | VIM login, NFC ticket, authd on 902 | | `openvixdisklib/nfc_open.py` | Classic NFC handshake, AIO open, sector read/write | +| `openvixdisklib/fastlz.py` | FastLZ for `VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` | | `tests/integration/` | Live pytest suite against a lab vCenter | | `tests/perf/` | Throughput comparison of openvixdisklib vs VDDK | | `tests/integration/vixdisklib.py` | Native VDDK wrapper used only to cross-check | diff --git a/docs/nfc_open.md b/docs/nfc_open.md index 569cade..97cee27 100644 --- a/docs/nfc_open.md +++ b/docs/nfc_open.md @@ -31,6 +31,7 @@ sectors. Flags `0x1a` require the writable ticket; the same flags on a | `NfcUtil_PrintFileInfoOpenFlag` `NFC_DISK` `0x1e` | `NFC_AIO_MSG_OPEN_FILE` (read-only) | | Open without `VIXDISKLIB_FLAG_OPEN_READ_ONLY` | `OPEN_FILE` flags `0x1a` (read-write) | | `VixDiskLib_Read` / `VixDiskLib_Write` | `NFC_AIO_MSG_IO` + sector bytes | +| `VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` | IO opcode high bits `2`; extra data is FastLZ | `snapshot_ref` is still not on the wire. Integration tests pass the flat VMDK created with the temporary lab VM. @@ -208,6 +209,7 @@ classic type 4 `NFC_SESSION_COMPLETE`. | VIM + authd | `openvixdisklib.nfc_auth.authenticate` | | Dup fd, skip TLS for NFC | `openvixdisklib.nfc_open.takeover_authd_socket` | | Second TLS for nbdssl | `openvixdisklib.nfc_open.wrap_nfcssl_socket` | +| FastLZ for NBD compression | `openvixdisklib.fastlz` | | Handshake + AIO + OPEN_FILE | `openvixdisklib.nfc_open.open_disk` | | Sector read / write / close | `openvixdisklib.nfc_open.NfcDisk` | @@ -224,7 +226,7 @@ I/O: `docs/nfc_read.md`, `docs/nfc_write.md`, and ## What is still VDDK-only -- `DDB_GET` / geometry / compression / encryption keys +- `DDB_GET` / geometry / zlib and skipz compression / encryption keys - `NFC_DELTA_DISK`, change-block tracking - Host-switch (`NFC_AIO_SWITCH_HOST_*`) - Direct ESXi `ha-nfc` without vCenter `vpxa-nfc` diff --git a/docs/nfc_read.md b/docs/nfc_read.md index f2158bf..7f149bc 100644 --- a/docs/nfc_read.md +++ b/docs/nfc_read.md @@ -39,21 +39,40 @@ LBA 8 on this disk was 4096 zero bytes on the wire, not a skip. Little-endian, after the usual 16-byte AIO header (`magic 0xA100DA7A`, type 7, size 44, monotonic `opId`): -| Offset | Type | VDDK `Read(start, n)` | -| ------ | -------- | ---------------------------------------------- | -| 0 | `uint64` | File handle from `OPEN_FILE` | -| 8 | `uint64` | `1` (`NFC_AIO_IO_READ`; write uses `0`) | -| 16 | `uint64` | Byte offset | -| 24 | `uint64` | Byte length | -| 32 | `uint32` | Byte length (same value) | -| 36 | `uint32` | Byte length (same value) | -| 40 | `uint32` | `0` (flags; uncompressed in this capture) | +| Offset | Type | VDDK `Read(start, n)` | +| ------ | -------- | ---------------------------------------------------------- | +| 0 | `uint64` | File handle from `OPEN_FILE` | +| 8 | `uint64` | Direction in low 32 bits; FastLZ type `2` in high 32 bits | +| 16 | `uint64` | Byte offset | +| 24 | `uint64` | Byte length | +| 32 | `uint32` | Byte length (same value) | +| 36 | `uint32` | Byte length, or compressed extra size when type is FastLZ | +| 40 | `uint32` | `0` | An earlier guess that offset 36 was `NFC_DISK` (`2`) was wrong: a 1-sector VDDK read puts `512` in both `uint32` length fields. A Python read that sent `(512, 2, 0)` still worked for one sector; the replacement now matches VDDK. +`VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` does not change OPEN_FILE +flags. The IO opcode at offset 8 is a `uint64`: low 32 bits are still +`0`/`1` (write/read), high 32 bits are the NFC compression type +(`2` = FastLZ). OPEN still uses handshake `PlainText`. + +| Open flag / wire | Request extra | Reply extra | +| ---------------------------------------- | ------------------------------------- | ------------------------------------------------ | +| No compression flag | Raw `length` bytes on write | Raw fragment at offset 32 | +| FASTLZ, data that shrinks | FastLZ bytes; offset 36 = packed size | Opcode type `2`; extra is FastLZ of offset 32 | +| FASTLZ, incompressible | Raw bytes; opcode type `0` | Opcode type `0`; extra is raw | + +Reads with FASTLZ always *request* type `2`. The server may answer type +`2` or fall back to type `0`. Decompress into the uncompressed fragment +length at offset 32 and copy to the dest at offset 28. + +64 KiB chunks use FastLZ level 2 (first byte has bit 5 set). Smaller +chunks use level 1. VDDK’s URL form is `FASTLZ-vpxa-nfc://…`; authd +`PROXY` is unchanged. + ## Reply Each fragment is: 16-byte AIO header (same `type` and `opId`) + 44-byte @@ -117,7 +136,7 @@ fragments). ## What is still VDDK-only -- Compression flags on the last `uint32` +- zlib and skipz NBD compression flags - `VixDiskLib_ReadAsync` (same IO messages, different client threading) - `VixDiskLib_QueryAllocatedBlocks` / allocation bitmaps - `VixDiskLib_GetInfo` capacity (not required to read a known range) diff --git a/docs/nfc_write.md b/docs/nfc_write.md index f8ab3ec..fd4ed80 100644 --- a/docs/nfc_write.md +++ b/docs/nfc_write.md @@ -60,6 +60,11 @@ Little-endian, after the usual 16-byte AIO header | 36 | `uint32` | Byte length (same value) | | 40 | `uint32` | `0` | +FASTLZ writes use the same 44-byte header. The opcode `uint64` high +half is `2`, offset 36 is the compressed size, and FastLZ bytes follow +instead of raw sectors. If compression does not shrink the chunk, VDDK +sends type `0` and raw extra (same as an uncompressed write). + Sector bytes follow the 44-byte payload and are **not** counted in AIO `size`. VDDK sends header + payload + data in one `write()`. The replacement may split that into two `sendall`s; TCP does not care. diff --git a/docs/reverse_engineering_procedure.md b/docs/reverse_engineering_procedure.md index eddaf44..d432203 100644 --- a/docs/reverse_engineering_procedure.md +++ b/docs/reverse_engineering_procedure.md @@ -280,6 +280,24 @@ the old `VixDiskLibConnectParams` ctypes struct can still log nbdssl and then fall back to `vpxa-nfc` / `useSSL=0`. Do not treat that log line as a wire capture of NFCSSL. +## Step 12 — FASTLZ NBD compression + +`VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` (`1 << 5`) is an IO codec, +not an OPEN_FILE bit. Capture VDDK with that flag (NBD + the port-902 +`write`/`read` hook): + +- Handshake stays `PlainText`. `OPEN_FILE` flags stay `0x1a` / `0x1e`. +- VDDK’s URL is `FASTLZ-vpxa-nfc://…`; `PROXY` is still `vpxa-nfc`. +- IO opcode `uint64` = direction in the low half, compression type in + the high half (`2` = FastLZ). Offset 32 is uncompressed length; + offset 36 is compressed extra size when type is 2. +- Incompressible chunks fall back to type `0` and raw extra. +- 64 KiB chunks use FastLZ level 2; smaller chunks use level 1. + +Replay: `openvixdisklib/fastlz.py` plus `NfcDisk` compression on each +IO. Proof: `tests/integration/test_nfc_read_write.py` (`fastlz`) and +`tests/perf/test_compare.py`. + ## What to write down After a stage works: @@ -300,7 +318,7 @@ replacement library. Not yet reversed, same loop as above: -- `DDB_GET` / disk geometry, compression, encrypted disks +- `DDB_GET` / disk geometry, zlib/skipz compression, encrypted disks - `NFC_DELTA_DISK`, CBT / `QueryAllocatedBlocks` - `VixDiskLib_GetInfo` capacity - Host-switch AIO messages diff --git a/openvixdisklib/fastlz.py b/openvixdisklib/fastlz.py new file mode 100644 index 0000000..21cfd18 --- /dev/null +++ b/openvixdisklib/fastlz.py @@ -0,0 +1,296 @@ +# Copyright 2026 Cloudbase Solutions Srl +# All Rights Reserved. +# +# Python port of FastLZ (Byte-aligned LZ77), used by VMware NFC +# VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ. +# +# FastLZ - Copyright (C) 2005-2020 Ariya Hidayat +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +# THE SOFTWARE. + +"""FastLZ compress / decompress used by NFC NBD compression.""" + +from __future__ import annotations + +MAX_COPY = 32 +MAX_LEN = 264 +MAX_L1_DISTANCE = 8192 +MAX_L2_DISTANCE = 8191 +MAX_FARDISTANCE = 65535 + MAX_L2_DISTANCE - 1 +HASH_LOG = 13 +HASH_SIZE = 1 << HASH_LOG +HASH_MASK = HASH_SIZE - 1 + + +def _readu32(buf: bytes | bytearray, offset: int) -> int: + return buf[offset] | (buf[offset + 1] << 8) | ( + buf[offset + 2] << 16) | (buf[offset + 3] << 24) + + +def _hash(value: int) -> int: + return ((value * 2654435769) >> (32 - HASH_LOG)) & HASH_MASK + + +def _cmp(src: bytes, p: int, q: int, q_end: int) -> int: + start = p + if _readu32(src, p) == _readu32(src, q): + p += 4 + q += 4 + while q < q_end: + mismatch = src[p] != src[q] + p += 1 + q += 1 + if mismatch: + break + return p - start + + +def _extend_match(out: bytearray, ref: int, length: int) -> None: + while length: + take = min(length, len(out) - ref) + out.extend(out[ref:ref + take]) + length -= take + + +def _literals(src: bytes, src_off: int, runs: int, out: bytearray) -> None: + while runs >= MAX_COPY: + out.append(MAX_COPY - 1) + out.extend(src[src_off:src_off + MAX_COPY]) + src_off += MAX_COPY + runs -= MAX_COPY + if runs > 0: + out.append(runs - 1) + out.extend(src[src_off:src_off + runs]) + + +def _flz1_match(length: int, distance: int, out: bytearray) -> None: + distance -= 1 + if length > MAX_LEN - 2: + while length > MAX_LEN - 2: + out.append((7 << 5) + (distance >> 8)) + out.append(MAX_LEN - 2 - 7 - 2) + out.append(distance & 255) + length -= MAX_LEN - 2 + if length < 7: + out.append((length << 5) + (distance >> 8)) + out.append(distance & 255) + else: + out.append((7 << 5) + (distance >> 8)) + out.append(length - 7) + out.append(distance & 255) + + +def _flz2_match(length: int, distance: int, out: bytearray) -> None: + distance -= 1 + if distance < MAX_L2_DISTANCE: + if length < 7: + out.append((length << 5) + (distance >> 8)) + out.append(distance & 255) + else: + out.append((7 << 5) + (distance >> 8)) + length -= 7 + while length >= 255: + out.append(255) + length -= 255 + out.append(length) + out.append(distance & 255) + elif length < 7: + distance -= MAX_L2_DISTANCE + out.append((length << 5) + 31) + out.append(255) + out.append(distance >> 8) + out.append(distance & 255) + else: + distance -= MAX_L2_DISTANCE + out.append((7 << 5) + 31) + length -= 7 + while length >= 255: + out.append(255) + length -= 255 + out.append(length) + out.append(255) + out.append(distance >> 8) + out.append(distance & 255) + + +def _compress_level(src: bytes, level: int) -> bytes: + length = len(src) + ip = 0 + ip_bound = length - 4 + ip_limit = length - 12 - 1 + out = bytearray() + htab = [0] * HASH_SIZE + max_distance = MAX_L1_DISTANCE if level == 1 else MAX_FARDISTANCE + anchor = 0 + ip = 2 + while ip < ip_limit: + while True: + seq = _readu32(src, ip) & 0xffffff + h = _hash(seq) + ref = htab[h] + htab[h] = ip + distance = ip - ref + cmp_val = ( + _readu32(src, ref) & 0xffffff + if 0 <= ref < ip and distance < max_distance + else 0x1000000) + if ip >= ip_limit: + break + ip += 1 + if seq == cmp_val: + break + if ip >= ip_limit: + break + ip -= 1 + if level == 2 and distance >= MAX_L2_DISTANCE: + if src[ref + 3] != src[ip + 3] or src[ref + 4] != src[ip + 4]: + ip += 1 + continue + if ip > anchor: + _literals(src, anchor, ip - anchor, out) + match_len = _cmp(src, ref + 3, ip + 3, ip_bound) + if level == 1: + _flz1_match(match_len, distance, out) + else: + _flz2_match(match_len, distance, out) + ip += match_len + seq = _readu32(src, ip) + htab[_hash(seq & 0xffffff)] = ip + ip += 1 + htab[_hash(seq >> 8)] = ip + ip += 1 + anchor = ip + _literals(src, anchor, length - anchor, out) + if level == 2 and out: + out[0] |= 1 << 5 + return bytes(out) + + +def compress(data: bytes) -> bytes: + """Compress ``data`` with FastLZ (level 1 below 64 KiB, else level 2). + + Args: + data: Uncompressed bytes. FastLZ needs at least 16 bytes. + """ + if len(data) < 65536: + return _compress_level(data, 1) + return _compress_level(data, 2) + + +def decompress(data: bytes, maxout: int) -> bytes: + """Decompress FastLZ ``data`` into at most ``maxout`` bytes. + + Args: + data: Compressed FastLZ buffer. + maxout: Expected uncompressed length (output cap). + """ + if not data: + raise ValueError("FastLZ input is empty") + level = (data[0] >> 5) + 1 + if level == 1: + return _decompress_level1(data, maxout) + if level == 2: + return _decompress_level2(data, maxout) + raise ValueError(f"unsupported FastLZ level {level}") + + +def _decompress_level1(src: bytes, maxout: int) -> bytes: + ip = 0 + ip_limit = len(src) + ip_bound = ip_limit - 2 + out = bytearray() + ctrl = src[ip] & 31 + ip += 1 + while True: + if ctrl >= 32: + length = (ctrl >> 5) - 1 + ofs = (ctrl & 31) << 8 + if length == 6: + if ip > ip_bound: + raise ValueError("truncated FastLZ match") + length += src[ip] + ip += 1 + if ip >= ip_limit: + raise ValueError("truncated FastLZ match distance") + ofs += src[ip] + ip += 1 + length += 3 + ref = len(out) - ofs - 1 + if ref < 0 or len(out) + length > maxout: + raise ValueError("FastLZ match out of range") + _extend_match(out, ref, length) + else: + ctrl += 1 + if ip + ctrl > ip_limit or len(out) + ctrl > maxout: + raise ValueError("truncated FastLZ literals") + out.extend(src[ip:ip + ctrl]) + ip += ctrl + if ip > ip_bound: + break + ctrl = src[ip] + ip += 1 + return bytes(out) + + +def _decompress_level2(src: bytes, maxout: int) -> bytes: + ip = 0 + ip_limit = len(src) + ip_bound = ip_limit - 2 + out = bytearray() + ctrl = src[ip] & 31 + ip += 1 + while True: + if ctrl >= 32: + length = (ctrl >> 5) - 1 + ofs = (ctrl & 31) << 8 + if length == 6: + while True: + if ip > ip_bound: + raise ValueError("truncated FastLZ match") + code = src[ip] + ip += 1 + length += code + if code != 255: + break + if ip >= ip_limit: + raise ValueError("truncated FastLZ match distance") + code = src[ip] + ip += 1 + ofs += code + length += 3 + ref = len(out) - ofs - 1 + if code == 255 and ofs == (31 << 8): + if ip >= ip_bound: + raise ValueError("truncated FastLZ far match") + ofs = (src[ip] << 8) + src[ip + 1] + ip += 2 + ref = len(out) - ofs - MAX_L2_DISTANCE - 1 + if ref < 0 or len(out) + length > maxout: + raise ValueError("FastLZ match out of range") + _extend_match(out, ref, length) + else: + ctrl += 1 + if ip + ctrl > ip_limit or len(out) + ctrl > maxout: + raise ValueError("truncated FastLZ literals") + out.extend(src[ip:ip + ctrl]) + ip += ctrl + if ip >= ip_limit: + break + ctrl = src[ip] + ip += 1 + return bytes(out) diff --git a/openvixdisklib/nfc_open.py b/openvixdisklib/nfc_open.py index 0f9f389..37b6257 100644 --- a/openvixdisklib/nfc_open.py +++ b/openvixdisklib/nfc_open.py @@ -26,6 +26,7 @@ import socket import ssl import struct +from openvixdisklib import fastlz from openvixdisklib.nfc_auth import NfcAuthSession, _ssl_client_context NFC_MSG_SIZE = 264 @@ -69,6 +70,12 @@ NFC_OPEN_FLAGS_READ_WRITE = 0x1A NFC_AIO_IO_WRITE = 0 NFC_AIO_IO_READ = 1 +# High 32 bits of the IO opcode uint64. Captured from VDDK FASTLZ: +# writes that shrink go on the wire as type 2; incompressible writes +# fall back to type 0 with raw extra data. +NFC_COMPRESSION_NONE = 0 +NFC_COMPRESSION_FASTLZ = 2 + class NfcProtocolError(ConnectionError): """Raised when an NFC message is malformed or reports failure.""" @@ -168,7 +175,8 @@ class NfcDisk: sock: socket.socket, path: str, handle: int, - sector_size: int) -> None: + sector_size: int, + compression: int = NFC_COMPRESSION_NONE) -> None: """Wrap an AIO session that already has ``path`` open. Args: @@ -176,12 +184,15 @@ class NfcDisk: path: Datastore path that was opened. handle: Server file handle from OPEN_FILE. sector_size: Sector size from the OPEN_FILE reply. + compression: NFC IO compression type (``NFC_COMPRESSION_NONE`` + or ``NFC_COMPRESSION_FASTLZ``). """ self._sock = sock self._op_id = 0 self.path = path self.handle = handle self.sector_size = sector_size + self.compression = compression self._closed = False def _next_op_id(self) -> int: @@ -226,7 +237,8 @@ class NfcDisk: byte units. If the length exceeds the AIO buffer (64 KiB) the server replies with several same-``opId`` fragments, which are placed by the fragment byte offset in the reply (they may arrive - out of order). + out of order). FASTLZ open requests compression in the opcode; + each reply fragment may be compressed (type 2) or raw (type 0). Args: start_sector: Sector offset from the start of the disk. @@ -236,10 +248,11 @@ class NfcDisk: raise ValueError("num_sectors must be at least 1") length = num_sectors * self.sector_size offset = start_sector * self.sector_size + opcode = NFC_AIO_IO_READ | (self.compression << 32) payload = struct.pack( "> 32 + if ctype == NFC_COMPRESSION_FASTLZ: + comp_len = struct.unpack_from(" None: length = num_sectors * self.sector_size offset = start_sector * self.sector_size + extra = data + ctype = NFC_COMPRESSION_NONE + extra_len = length + if self.compression == NFC_COMPRESSION_FASTLZ and length >= 16: + compressed = fastlz.compress(data) + if compressed and len(compressed) < length: + extra = compressed + ctype = NFC_COMPRESSION_FASTLZ + extra_len = len(compressed) + opcode = NFC_AIO_IO_WRITE | (ctype << 32) payload = struct.pack( " None: """Close the VMDK, the AIO session, and the classic NFC session.""" @@ -422,7 +467,8 @@ def open_disk( client_name: str = "vddk", op_id: str = "nbdmode", version: int = NFC_PROTOCOL_VERSION, - read_only: bool = True) -> NfcDisk: + read_only: bool = True, + compression: int = NFC_COMPRESSION_NONE) -> NfcDisk: """Open ``disk_path`` over the authenticated authd socket. Matches VDDK ``VixDiskLib_Open`` of a datastore path after the NFC @@ -439,7 +485,13 @@ def open_disk( op_id: NFC operation id; VDDK NBD sends ``nbdmode``. version: Client NFC protocol version (lab ESXi answered 11). read_only: When True, open with VDDK's read-only NFC flags. + compression: ``NFC_COMPRESSION_NONE`` or ``NFC_COMPRESSION_FASTLZ``. + OPEN_FILE flags are unchanged; compression is per IO message. """ + if compression not in ( + NFC_COMPRESSION_NONE, NFC_COMPRESSION_FASTLZ): + raise NotImplementedError( + f"NFC compression type {compression} is not supported") if session.nfc_ssl: sock = wrap_nfcssl_socket( session.authd_sock, session.ticket.host) @@ -447,7 +499,9 @@ def open_disk( sock = takeover_authd_socket(session.authd_sock) try: _handshake(sock, client_name, op_id, version) - disk = NfcDisk(sock, disk_path, handle=0, sector_size=NFC_SECTOR_SIZE) + disk = NfcDisk( + sock, disk_path, handle=0, sector_size=NFC_SECTOR_SIZE, + compression=compression) _aio_prepare(disk) path_b = disk_path.encode("utf-8") open_flags = ( diff --git a/openvixdisklib/openvixdisklib.py b/openvixdisklib/openvixdisklib.py index 759ab29..9e6e582 100644 --- a/openvixdisklib/openvixdisklib.py +++ b/openvixdisklib/openvixdisklib.py @@ -44,10 +44,24 @@ VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB = 16 VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ = 32 VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ = 64 -_COMPRESSION_FLAGS = ( - VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB - | VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ - | VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ) + +def _nfc_compression(flags: int) -> int: + """Return the NFC IO compression type for VixDiskLib open ``flags``.""" + alg = flags & ( + VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB + | VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ + | VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ) + if alg == 0: + return nfc_open.NFC_COMPRESSION_NONE + if alg == VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ: + return nfc_open.NFC_COMPRESSION_FASTLZ + if alg & (alg - 1): + raise NotImplementedError( + "Cannot set two or more NBD compression algorithms at the " + "same time") + raise NotImplementedError( + f"NBD compression open flag 0x{alg:x} is not supported") + VIX_SUPPORTED_COMPATIBILITY_MODES = [ "6.0", "6.5", "6.7", "7.0", "8.0"] @@ -243,13 +257,12 @@ class VixDiskLibHandle: conn: Connection from ``connect``. disk_path: Datastore path of the VMDK. flags: Open flags. ``VIXDISKLIB_FLAG_OPEN_READ_ONLY`` opens - the disk read-only; omit it for write. Compression flags - are not implemented. + the disk read-only; omit it for write. + ``VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ`` compresses + NFC IO. zlib and skipz are not implemented. """ LOG.debug("Openning VixDiskLib disk: %s", disk_path) - if flags & _COMPRESSION_FLAGS: - raise NotImplementedError( - "NBD compression open flags are not supported") + compression = _nfc_compression(flags) read_only = bool(flags & VIXDISKLIB_FLAG_OPEN_READ_ONLY) if not read_only and conn.read_only: raise NotImplementedError( @@ -265,7 +278,8 @@ class VixDiskLibHandle: conn.si, ticket, authd_sock, nfc_ssl=nfc_ssl) try: disk = nfc_open.open_disk( - session, disk_path, read_only=read_only) + session, disk_path, read_only=read_only, + compression=compression) except Exception: authd_sock.close() raise diff --git a/tests/integration/test_crosscheck.py b/tests/integration/test_crosscheck.py index 1820832..4a48b57 100644 --- a/tests/integration/test_crosscheck.py +++ b/tests/integration/test_crosscheck.py @@ -5,6 +5,8 @@ from typing import Any, Optional +import pytest + from openvixdisklib import openvixdisklib as open_vix from tests.integration import vixdisklib from tests.integration.base import ( @@ -21,7 +23,8 @@ def _connect_extra(lab: LabEnv, module: Any) -> Optional[dict[str, Any]]: def _write_sectors( lab: LabEnv, module: Any, - payloads: dict[int, bytes]) -> None: + payloads: dict[int, bytes], + flags: int = 0) -> None: """Write one sector at each index using a vixdisklib-compatible module.""" handle = module.VixDiskLibHandle( vixdisklib_compatibility_version="8.0", @@ -29,7 +32,7 @@ def _write_sectors( buf = module.get_buffer(SECTOR_SIZE) kwargs = lab.vixdisklib_connect_kwargs(_connect_extra(lab, module)) with handle.connect(**kwargs) as conn: - with handle.open(conn, lab.disk_path, flags=0) as disk: + with handle.open(conn, lab.disk_path, flags=flags) as disk: for start, data in payloads.items(): buf[:SECTOR_SIZE] = data handle.write(disk, start, 1, buf) @@ -38,7 +41,8 @@ def _write_sectors( def _read_sectors( lab: LabEnv, module: Any, - sectors: tuple[int, ...]) -> dict[int, bytes]: + sectors: tuple[int, ...], + flags: int = 0) -> dict[int, bytes]: """Read one sector at each index using a vixdisklib-compatible module.""" handle = module.VixDiskLibHandle( vixdisklib_compatibility_version="8.0", @@ -47,7 +51,7 @@ def _read_sectors( result: dict[int, bytes] = {} kwargs = lab.vixdisklib_connect_kwargs(_connect_extra(lab, module)) with handle.connect(**kwargs) as conn: - with handle.open(conn, lab.disk_path, flags=0) as disk: + with handle.open(conn, lab.disk_path, flags=flags) as disk: for start in sectors: buf[:SECTOR_SIZE] = b"\xa5" * SECTOR_SIZE handle.read(disk, start, 1, buf) @@ -58,9 +62,10 @@ def _read_sectors( def _assert_both_read( lab: LabEnv, sectors: tuple[int, ...], - expected: dict[int, bytes]) -> None: - vddk_data = _read_sectors(lab, vixdisklib, sectors) - replacement = _read_sectors(lab, open_vix, sectors) + expected: dict[int, bytes], + flags: int = 0) -> None: + vddk_data = _read_sectors(lab, vixdisklib, sectors, flags=flags) + replacement = _read_sectors(lab, open_vix, sectors, flags=flags) for start in sectors: assert vddk_data[start] == expected[start], ( f"VDDK mismatch at sector {start}") @@ -69,8 +74,12 @@ def _assert_both_read( class TestCrosscheck: + @pytest.mark.parametrize( + "open_flags", + [0, vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ], + ids=["plain", "fastlz"]) def test_openvixdisklib_matches_vddk_sectors( - self, lab: LabEnv, vddk: None) -> None: + self, lab: LabEnv, vddk: None, open_flags: int) -> None: """Writes from either library must be visible to both readers.""" sectors = (0, 1, SECTOR_AT_1GB) vddk_payloads = { @@ -78,13 +87,13 @@ class TestCrosscheck: 1: pattern_bytes(SECTOR_SIZE, b"XCHK-VDDK-S1"), SECTOR_AT_1GB: pattern_bytes(SECTOR_SIZE, b"XCHK-VDDK-1G"), } - _write_sectors(lab, vixdisklib, vddk_payloads) - _assert_both_read(lab, sectors, vddk_payloads) + _write_sectors(lab, vixdisklib, vddk_payloads, flags=open_flags) + _assert_both_read(lab, sectors, vddk_payloads, flags=open_flags) ovdl_payloads = { 0: pattern_bytes(SECTOR_SIZE, b"XCHK-OVDL-S0"), 1: pattern_bytes(SECTOR_SIZE, b"XCHK-OVDL-S1"), SECTOR_AT_1GB: pattern_bytes(SECTOR_SIZE, b"XCHK-OVDL-1G"), } - _write_sectors(lab, open_vix, ovdl_payloads) - _assert_both_read(lab, sectors, ovdl_payloads) + _write_sectors(lab, open_vix, ovdl_payloads, flags=open_flags) + _assert_both_read(lab, sectors, ovdl_payloads, flags=open_flags) diff --git a/tests/integration/test_nfc_open.py b/tests/integration/test_nfc_open.py index 20d6c34..abb3db3 100644 --- a/tests/integration/test_nfc_open.py +++ b/tests/integration/test_nfc_open.py @@ -12,13 +12,18 @@ from tests.integration.base import LabEnv, SECTOR_SIZE, pattern_bytes class TestNfcOpen: @pytest.mark.parametrize( "nfc_ssl", [True, False], ids=["nbdssl", "nbd"]) + @pytest.mark.parametrize( + "compression", + [nfc_open.NFC_COMPRESSION_NONE, nfc_open.NFC_COMPRESSION_FASTLZ], + ids=["plain", "fastlz"]) def test_open_disk_and_read_first_sector( - self, lab: LabEnv, nfc_ssl: bool) -> None: + self, lab: LabEnv, nfc_ssl: bool, compression: int) -> None: """Open the temp VMDK, write sector 0, and read it back.""" expected = pattern_bytes(SECTOR_SIZE, b"NFC-OPEN-S0") with lab.authenticate(read_only=False, nfc_ssl=nfc_ssl) as session: with nfc_open.open_disk( - session, lab.disk_path, read_only=False) as disk: + session, lab.disk_path, read_only=False, + compression=compression) as disk: assert disk.path == lab.disk_path assert disk.handle > 0 assert disk.sector_size == SECTOR_SIZE diff --git a/tests/integration/test_nfc_read_write.py b/tests/integration/test_nfc_read_write.py index 2667d33..1d796e6 100644 --- a/tests/integration/test_nfc_read_write.py +++ b/tests/integration/test_nfc_read_write.py @@ -5,6 +5,8 @@ import os +import pytest + from openvixdisklib import nfc_open from tests.integration.base import LabEnv, SECTOR_SIZE, pattern_bytes @@ -12,7 +14,12 @@ _32MIB = 32 * 1024 * 1024 class TestNfcReadWrite: - def test_sector_writes_and_reads(self, lab: LabEnv) -> None: + @pytest.mark.parametrize( + "compression", + [nfc_open.NFC_COMPRESSION_NONE, nfc_open.NFC_COMPRESSION_FASTLZ], + ids=["plain", "fastlz"]) + def test_sector_writes_and_reads( + self, lab: LabEnv, compression: int) -> None: """Write known patterns and read them back at several ranges.""" ranges = [ (0, 1), @@ -25,7 +32,8 @@ class TestNfcReadWrite: ] with lab.authenticate(read_only=False) as session: with nfc_open.open_disk( - session, lab.disk_path, read_only=False) as disk: + session, lab.disk_path, read_only=False, + compression=compression) as disk: for start, n_sectors in ranges: length = n_sectors * SECTOR_SIZE seed = f"NFC-R{start}:{n_sectors}-".encode() @@ -55,13 +63,19 @@ class TestNfcReadWrite: big_got[SECTOR_SIZE:2 * SECTOR_SIZE] == big_to_write[SECTOR_SIZE:2 * SECTOR_SIZE]) - def test_write_and_read_32mb(self, lab: LabEnv) -> None: + @pytest.mark.parametrize( + "compression", + [nfc_open.NFC_COMPRESSION_NONE, nfc_open.NFC_COMPRESSION_FASTLZ], + ids=["plain", "fastlz"]) + def test_write_and_read_32mb( + self, lab: LabEnv, compression: int) -> None: """Write 32 MiB (512 AIO chunks) and read it back in one request.""" n_sectors = _32MIB // SECTOR_SIZE to_write = os.urandom(_32MIB) with lab.authenticate(read_only=False) as session: with nfc_open.open_disk( - session, lab.disk_path, read_only=False) as disk: + session, lab.disk_path, read_only=False, + compression=compression) as disk: disk.write(0, n_sectors, to_write) got = disk.read(0, n_sectors) assert got is not to_write diff --git a/tests/integration/test_openvixdisklib.py b/tests/integration/test_openvixdisklib.py index c006b35..5212170 100644 --- a/tests/integration/test_openvixdisklib.py +++ b/tests/integration/test_openvixdisklib.py @@ -12,8 +12,12 @@ from tests.integration.base import ( class TestOpenvixdisklib: @pytest.mark.parametrize("transport_mode", ["nbdssl", "nbd"]) + @pytest.mark.parametrize( + "open_flags", + [0, vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ], + ids=["plain", "fastlz"]) def test_write_and_read_sector_zero_and_one_gib( - self, lab: LabEnv, transport_mode: str) -> None: + self, lab: LabEnv, transport_mode: str, open_flags: int) -> None: """Write then read sector 0 and the sector at a 1 GiB offset.""" handle = vixdisklib.VixDiskLibHandle( vixdisklib_compatibility_version="8.0", @@ -30,7 +34,7 @@ class TestOpenvixdisklib: } assert handle.get_transport_modes() == ["nbdssl", "nbd"] with handle.connect(**connect_kwargs) as conn: - with handle.open(conn, lab.disk_path, flags=0) as disk: + with handle.open(conn, lab.disk_path, flags=open_flags) as disk: assert handle.get_transport_mode(disk) == transport_mode for start, expected in patterns.items(): write_buf[:SECTOR_SIZE] = expected diff --git a/tests/perf/test_compare.py b/tests/perf/test_compare.py index 2a2ab8c..ce47f22 100644 --- a/tests/perf/test_compare.py +++ b/tests/perf/test_compare.py @@ -29,7 +29,8 @@ def _connect_extra(lab: LabEnv, module: Any) -> Optional[dict[str, Any]]: def _time_write_read( lab: LabEnv, module: Any, - payload: bytes) -> tuple[float, float]: + payload: bytes, + flags: int = 0) -> tuple[float, float]: """Write ``payload`` at sector 0, read it back, and return durations.""" n_sectors = len(payload) // SECTOR_SIZE handle = module.VixDiskLibHandle( @@ -40,7 +41,7 @@ def _time_write_read( write_buf[:len(payload)] = payload kwargs = lab.vixdisklib_connect_kwargs(_connect_extra(lab, module)) with handle.connect(**kwargs) as conn: - with handle.open(conn, lab.disk_path, flags=0) as disk: + with handle.open(conn, lab.disk_path, flags=flags) as disk: started = time.perf_counter() handle.write(disk, 0, n_sectors, write_buf) write_s = time.perf_counter() - started @@ -65,24 +66,33 @@ class TestCompare: ("vddk", vixdisklib), ("openvixdisklib", open_vix), ) - rows: list[tuple[str, str, float, float, float, float]] = [] + open_modes = ( + ("plain", 0), + ("fastlz", vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ), + ) + rows: list[tuple[str, str, str, float, float, float, float]] = [] for label, nbytes in _SIZES: payload = pattern_bytes(nbytes, f"PERF-{label}-".encode()) - for name, module in libraries: - write_s, read_s = _time_write_read(lab, module, payload) - rows.append(( - label, - name, - write_s, - read_s, - _mib_per_s(nbytes, write_s), - _mib_per_s(nbytes, read_s), - )) + for mode_name, flags in open_modes: + for name, module in libraries: + write_s, read_s = _time_write_read( + lab, module, payload, flags=flags) + rows.append(( + label, + mode_name, + name, + write_s, + read_s, + _mib_per_s(nbytes, write_s), + _mib_per_s(nbytes, read_s), + )) print() print( - f"{'size':<14} {'library':<16} {'write_s':>10} {'read_s':>10} " + f"{'size':<14} {'flags':<8} {'library':<16} " + f"{'write_s':>10} {'read_s':>10} " f"{'write_MiB/s':>12} {'read_MiB/s':>12}") - for label, name, write_s, read_s, write_r, read_r in rows: + for label, mode_name, name, write_s, read_s, write_r, read_r in rows: print( - f"{label:<14} {name:<16} {write_s:10.3f} {read_s:10.3f} " + f"{label:<14} {mode_name:<8} {name:<16} " + f"{write_s:10.3f} {read_s:10.3f} " f"{write_r:12.1f} {read_r:12.1f}")