Add fastlz compression support

README.md

@@ -18,9 +18,9 @@ Implemented against vCenter 8 / ESXi 8. Default transport is `nbdssl`
 - `VixDiskLib_Read`
 - `VixDiskLib_Write`
 
-Not implemented: compression open flags, CBT / allocated-block
+Not implemented: compression open flags other than FastLZ, CBT /
-queries, disk geometry (`DDB_GET`), encrypted disks, and direct ESXi
+allocated-block queries, disk geometry (`DDB_GET`), encrypted disks,
-`ha-nfc` without vCenter `vpxa-nfc`.
+and direct ESXi `ha-nfc` without vCenter `vpxa-nfc`.
 
 ## Install
 
@@ -64,6 +64,7 @@ VDDK-shaped handle.
 | `openvixdisklib/openvixdisklib.py` | Drop-in handle (`connect` / `open` / `read` / `write`) |
 | `openvixdisklib/nfc_auth.py`       | VIM login, NFC ticket, authd on 902                    |
 | `openvixdisklib/nfc_open.py`       | Classic NFC handshake, AIO open, sector read/write     |
+| `openvixdisklib/fastlz.py`         | FastLZ for `VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ`   |
 | `tests/integration/`               | Live pytest suite against a lab vCenter                |
 | `tests/perf/`                      | Throughput comparison of openvixdisklib vs VDDK        |
 | `tests/integration/vixdisklib.py`  | Native VDDK wrapper used only to cross-check           |

docs/nfc_open.md

@@ -31,6 +31,7 @@ sectors. Flags `0x1a` require the writable ticket; the same flags on a
 | `NfcUtil_PrintFileInfoOpenFlag` `NFC_DISK` `0x1e`             | `NFC_AIO_MSG_OPEN_FILE` (read-only)                        |
 | Open without `VIXDISKLIB_FLAG_OPEN_READ_ONLY`                 | `OPEN_FILE` flags `0x1a` (read-write)                      |
 | `VixDiskLib_Read` / `VixDiskLib_Write`                        | `NFC_AIO_MSG_IO` + sector bytes                            |
+| `VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ`                     | IO opcode high bits `2`; extra data is FastLZ              |
 
 `snapshot_ref` is still not on the wire. Integration tests pass the
 flat VMDK created with the temporary lab VM.
@@ -208,6 +209,7 @@ classic type 4 `NFC_SESSION_COMPLETE`.
 | VIM + authd                   | `openvixdisklib.nfc_auth.authenticate`          |
 | Dup fd, skip TLS for NFC      | `openvixdisklib.nfc_open.takeover_authd_socket` |
 | Second TLS for nbdssl         | `openvixdisklib.nfc_open.wrap_nfcssl_socket`    |
+| FastLZ for NBD compression    | `openvixdisklib.fastlz`                         |
 | Handshake + AIO + OPEN_FILE   | `openvixdisklib.nfc_open.open_disk`             |
 | Sector read / write / close   | `openvixdisklib.nfc_open.NfcDisk`               |
 
@@ -224,7 +226,7 @@ I/O: `docs/nfc_read.md`, `docs/nfc_write.md`, and
 
 ## What is still VDDK-only
 
-- `DDB_GET` / geometry / compression / encryption keys
+- `DDB_GET` / geometry / zlib and skipz compression / encryption keys
 - `NFC_DELTA_DISK`, change-block tracking
 - Host-switch (`NFC_AIO_SWITCH_HOST_*`)
 - Direct ESXi `ha-nfc` without vCenter `vpxa-nfc`

docs/nfc_read.md

@@ -39,21 +39,40 @@ LBA 8 on this disk was 4096 zero bytes on the wire, not a skip.
 Little-endian, after the usual 16-byte AIO header
 (`magic 0xA100DA7A`, type 7, size 44, monotonic `opId`):
 
-| Offset | Type     | VDDK `Read(start, n)`                          |
+| Offset | Type     | VDDK `Read(start, n)`                                      |
-| ------ | -------- | ---------------------------------------------- |
+| ------ | -------- | ---------------------------------------------------------- |
-| 0      | `uint64` | File handle from `OPEN_FILE`                   |
+| 0      | `uint64` | File handle from `OPEN_FILE`                               |
-| 8      | `uint64` | `1` (`NFC_AIO_IO_READ`; write uses `0`)        |
+| 8      | `uint64` | Direction in low 32 bits; FastLZ type `2` in high 32 bits  |
-| 16     | `uint64` | Byte offset                                    |
+| 16     | `uint64` | Byte offset                                                |
-| 24     | `uint64` | Byte length                                    |
+| 24     | `uint64` | Byte length                                                |
-| 32     | `uint32` | Byte length (same value)                       |
+| 32     | `uint32` | Byte length (same value)                                   |
-| 36     | `uint32` | Byte length (same value)                       |
+| 36     | `uint32` | Byte length, or compressed extra size when type is FastLZ  |
-| 40     | `uint32` | `0` (flags; uncompressed in this capture)      |
+| 40     | `uint32` | `0`                                                        |
 
 An earlier guess that offset 36 was `NFC_DISK` (`2`) was wrong: a
 1-sector VDDK read puts `512` in both `uint32` length fields. A Python
 read that sent `(512, 2, 0)` still worked for one sector; the
 replacement now matches VDDK.
 
+`VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` does not change OPEN_FILE
+flags. The IO opcode at offset 8 is a `uint64`: low 32 bits are still
+`0`/`1` (write/read), high 32 bits are the NFC compression type
+(`2` = FastLZ). OPEN still uses handshake `PlainText`.
+
+| Open flag / wire                         | Request extra                         | Reply extra                                      |
+| ---------------------------------------- | ------------------------------------- | ------------------------------------------------ |
+| No compression flag                      | Raw `length` bytes on write           | Raw fragment at offset 32                        |
+| FASTLZ, data that shrinks                | FastLZ bytes; offset 36 = packed size | Opcode type `2`; extra is FastLZ of offset 32    |
+| FASTLZ, incompressible                   | Raw bytes; opcode type `0`            | Opcode type `0`; extra is raw                    |
+
+Reads with FASTLZ always *request* type `2`. The server may answer type
+`2` or fall back to type `0`. Decompress into the uncompressed fragment
+length at offset 32 and copy to the dest at offset 28.
+
+64 KiB chunks use FastLZ level 2 (first byte has bit 5 set). Smaller
+chunks use level 1. VDDK’s URL form is `FASTLZ-vpxa-nfc://…`; authd
+`PROXY` is unchanged.
+
 ## Reply
 
 Each fragment is: 16-byte AIO header (same `type` and `opId`) + 44-byte
@@ -117,7 +136,7 @@ fragments).
 
 ## What is still VDDK-only
 
-- Compression flags on the last `uint32`
+- zlib and skipz NBD compression flags
 - `VixDiskLib_ReadAsync` (same IO messages, different client threading)
 - `VixDiskLib_QueryAllocatedBlocks` / allocation bitmaps
 - `VixDiskLib_GetInfo` capacity (not required to read a known range)

docs/nfc_write.md

@@ -60,6 +60,11 @@ Little-endian, after the usual 16-byte AIO header
 | 36     | `uint32` | Byte length (same value)                           |
 | 40     | `uint32` | `0`                                                |
 
+FASTLZ writes use the same 44-byte header. The opcode `uint64` high
+half is `2`, offset 36 is the compressed size, and FastLZ bytes follow
+instead of raw sectors. If compression does not shrink the chunk, VDDK
+sends type `0` and raw extra (same as an uncompressed write).
+
 Sector bytes follow the 44-byte payload and are **not** counted in AIO
 `size`. VDDK sends header + payload + data in one `write()`. The
 replacement may split that into two `sendall`s; TCP does not care.

docs/reverse_engineering_procedure.md

@@ -280,6 +280,24 @@ the old `VixDiskLibConnectParams` ctypes struct can still log nbdssl
 and then fall back to `vpxa-nfc` / `useSSL=0`. Do not treat that log
 line as a wire capture of NFCSSL.
 
+## Step 12 — FASTLZ NBD compression
+
+`VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` (`1 << 5`) is an IO codec,
+not an OPEN_FILE bit. Capture VDDK with that flag (NBD + the port-902
+`write`/`read` hook):
+
+- Handshake stays `PlainText`. `OPEN_FILE` flags stay `0x1a` / `0x1e`.
+- VDDK’s URL is `FASTLZ-vpxa-nfc://…`; `PROXY` is still `vpxa-nfc`.
+- IO opcode `uint64` = direction in the low half, compression type in
+  the high half (`2` = FastLZ). Offset 32 is uncompressed length;
+  offset 36 is compressed extra size when type is 2.
+- Incompressible chunks fall back to type `0` and raw extra.
+- 64 KiB chunks use FastLZ level 2; smaller chunks use level 1.
+
+Replay: `openvixdisklib/fastlz.py` plus `NfcDisk` compression on each
+IO. Proof: `tests/integration/test_nfc_read_write.py` (`fastlz`) and
+`tests/perf/test_compare.py`.
+
 ## What to write down
 
 After a stage works:
@@ -300,7 +318,7 @@ replacement library.
 
 Not yet reversed, same loop as above:
 
-- `DDB_GET` / disk geometry, compression, encrypted disks
+- `DDB_GET` / disk geometry, zlib/skipz compression, encrypted disks
 - `NFC_DELTA_DISK`, CBT / `QueryAllocatedBlocks`
 - `VixDiskLib_GetInfo` capacity
 - Host-switch AIO messages

openvixdisklib/fastlz.py

@@ -0,0 +1,296 @@
+# Copyright 2026 Cloudbase Solutions Srl
+# All Rights Reserved.
+#
+# Python port of FastLZ (Byte-aligned LZ77), used by VMware NFC
+# VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ.
+#
+# FastLZ - Copyright (C) 2005-2020 Ariya Hidayat <[email protected]>
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+"""FastLZ compress / decompress used by NFC NBD compression."""
+
+from __future__ import annotations
+
+MAX_COPY = 32
+MAX_LEN = 264
+MAX_L1_DISTANCE = 8192
+MAX_L2_DISTANCE = 8191
+MAX_FARDISTANCE = 65535 + MAX_L2_DISTANCE - 1
+HASH_LOG = 13
+HASH_SIZE = 1 << HASH_LOG
+HASH_MASK = HASH_SIZE - 1
+
+
+def _readu32(buf: bytes | bytearray, offset: int) -> int:
+    return buf[offset] | (buf[offset + 1] << 8) | (
+        buf[offset + 2] << 16) | (buf[offset + 3] << 24)
+
+
+def _hash(value: int) -> int:
+    return ((value * 2654435769) >> (32 - HASH_LOG)) & HASH_MASK
+
+
+def _cmp(src: bytes, p: int, q: int, q_end: int) -> int:
+    start = p
+    if _readu32(src, p) == _readu32(src, q):
+        p += 4
+        q += 4
+    while q < q_end:
+        mismatch = src[p] != src[q]
+        p += 1
+        q += 1
+        if mismatch:
+            break
+    return p - start
+
+
+def _extend_match(out: bytearray, ref: int, length: int) -> None:
+    while length:
+        take = min(length, len(out) - ref)
+        out.extend(out[ref:ref + take])
+        length -= take
+
+
+def _literals(src: bytes, src_off: int, runs: int, out: bytearray) -> None:
+    while runs >= MAX_COPY:
+        out.append(MAX_COPY - 1)
+        out.extend(src[src_off:src_off + MAX_COPY])
+        src_off += MAX_COPY
+        runs -= MAX_COPY
+    if runs > 0:
+        out.append(runs - 1)
+        out.extend(src[src_off:src_off + runs])
+
+
+def _flz1_match(length: int, distance: int, out: bytearray) -> None:
+    distance -= 1
+    if length > MAX_LEN - 2:
+        while length > MAX_LEN - 2:
+            out.append((7 << 5) + (distance >> 8))
+            out.append(MAX_LEN - 2 - 7 - 2)
+            out.append(distance & 255)
+            length -= MAX_LEN - 2
+    if length < 7:
+        out.append((length << 5) + (distance >> 8))
+        out.append(distance & 255)
+    else:
+        out.append((7 << 5) + (distance >> 8))
+        out.append(length - 7)
+        out.append(distance & 255)
+
+
+def _flz2_match(length: int, distance: int, out: bytearray) -> None:
+    distance -= 1
+    if distance < MAX_L2_DISTANCE:
+        if length < 7:
+            out.append((length << 5) + (distance >> 8))
+            out.append(distance & 255)
+        else:
+            out.append((7 << 5) + (distance >> 8))
+            length -= 7
+            while length >= 255:
+                out.append(255)
+                length -= 255
+            out.append(length)
+            out.append(distance & 255)
+    elif length < 7:
+        distance -= MAX_L2_DISTANCE
+        out.append((length << 5) + 31)
+        out.append(255)
+        out.append(distance >> 8)
+        out.append(distance & 255)
+    else:
+        distance -= MAX_L2_DISTANCE
+        out.append((7 << 5) + 31)
+        length -= 7
+        while length >= 255:
+            out.append(255)
+            length -= 255
+        out.append(length)
+        out.append(255)
+        out.append(distance >> 8)
+        out.append(distance & 255)
+
+
+def _compress_level(src: bytes, level: int) -> bytes:
+    length = len(src)
+    ip = 0
+    ip_bound = length - 4
+    ip_limit = length - 12 - 1
+    out = bytearray()
+    htab = [0] * HASH_SIZE
+    max_distance = MAX_L1_DISTANCE if level == 1 else MAX_FARDISTANCE
+    anchor = 0
+    ip = 2
+    while ip < ip_limit:
+        while True:
+            seq = _readu32(src, ip) & 0xffffff
+            h = _hash(seq)
+            ref = htab[h]
+            htab[h] = ip
+            distance = ip - ref
+            cmp_val = (
+                _readu32(src, ref) & 0xffffff
+                if 0 <= ref < ip and distance < max_distance
+                else 0x1000000)
+            if ip >= ip_limit:
+                break
+            ip += 1
+            if seq == cmp_val:
+                break
+        if ip >= ip_limit:
+            break
+        ip -= 1
+        if level == 2 and distance >= MAX_L2_DISTANCE:
+            if src[ref + 3] != src[ip + 3] or src[ref + 4] != src[ip + 4]:
+                ip += 1
+                continue
+        if ip > anchor:
+            _literals(src, anchor, ip - anchor, out)
+        match_len = _cmp(src, ref + 3, ip + 3, ip_bound)
+        if level == 1:
+            _flz1_match(match_len, distance, out)
+        else:
+            _flz2_match(match_len, distance, out)
+        ip += match_len
+        seq = _readu32(src, ip)
+        htab[_hash(seq & 0xffffff)] = ip
+        ip += 1
+        htab[_hash(seq >> 8)] = ip
+        ip += 1
+        anchor = ip
+    _literals(src, anchor, length - anchor, out)
+    if level == 2 and out:
+        out[0] |= 1 << 5
+    return bytes(out)
+
+
+def compress(data: bytes) -> bytes:
+    """Compress ``data`` with FastLZ (level 1 below 64 KiB, else level 2).
+
+    Args:
+        data: Uncompressed bytes. FastLZ needs at least 16 bytes.
+    """
+    if len(data) < 65536:
+        return _compress_level(data, 1)
+    return _compress_level(data, 2)
+
+
+def decompress(data: bytes, maxout: int) -> bytes:
+    """Decompress FastLZ ``data`` into at most ``maxout`` bytes.
+
+    Args:
+        data: Compressed FastLZ buffer.
+        maxout: Expected uncompressed length (output cap).
+    """
+    if not data:
+        raise ValueError("FastLZ input is empty")
+    level = (data[0] >> 5) + 1
+    if level == 1:
+        return _decompress_level1(data, maxout)
+    if level == 2:
+        return _decompress_level2(data, maxout)
+    raise ValueError(f"unsupported FastLZ level {level}")
+
+
+def _decompress_level1(src: bytes, maxout: int) -> bytes:
+    ip = 0
+    ip_limit = len(src)
+    ip_bound = ip_limit - 2
+    out = bytearray()
+    ctrl = src[ip] & 31
+    ip += 1
+    while True:
+        if ctrl >= 32:
+            length = (ctrl >> 5) - 1
+            ofs = (ctrl & 31) << 8
+            if length == 6:
+                if ip > ip_bound:
+                    raise ValueError("truncated FastLZ match")
+                length += src[ip]
+                ip += 1
+            if ip >= ip_limit:
+                raise ValueError("truncated FastLZ match distance")
+            ofs += src[ip]
+            ip += 1
+            length += 3
+            ref = len(out) - ofs - 1
+            if ref < 0 or len(out) + length > maxout:
+                raise ValueError("FastLZ match out of range")
+            _extend_match(out, ref, length)
+        else:
+            ctrl += 1
+            if ip + ctrl > ip_limit or len(out) + ctrl > maxout:
+                raise ValueError("truncated FastLZ literals")
+            out.extend(src[ip:ip + ctrl])
+            ip += ctrl
+        if ip > ip_bound:
+            break
+        ctrl = src[ip]
+        ip += 1
+    return bytes(out)
+
+
+def _decompress_level2(src: bytes, maxout: int) -> bytes:
+    ip = 0
+    ip_limit = len(src)
+    ip_bound = ip_limit - 2
+    out = bytearray()
+    ctrl = src[ip] & 31
+    ip += 1
+    while True:
+        if ctrl >= 32:
+            length = (ctrl >> 5) - 1
+            ofs = (ctrl & 31) << 8
+            if length == 6:
+                while True:
+                    if ip > ip_bound:
+                        raise ValueError("truncated FastLZ match")
+                    code = src[ip]
+                    ip += 1
+                    length += code
+                    if code != 255:
+                        break
+            if ip >= ip_limit:
+                raise ValueError("truncated FastLZ match distance")
+            code = src[ip]
+            ip += 1
+            ofs += code
+            length += 3
+            ref = len(out) - ofs - 1
+            if code == 255 and ofs == (31 << 8):
+                if ip >= ip_bound:
+                    raise ValueError("truncated FastLZ far match")
+                ofs = (src[ip] << 8) + src[ip + 1]
+                ip += 2
+                ref = len(out) - ofs - MAX_L2_DISTANCE - 1
+            if ref < 0 or len(out) + length > maxout:
+                raise ValueError("FastLZ match out of range")
+            _extend_match(out, ref, length)
+        else:
+            ctrl += 1
+            if ip + ctrl > ip_limit or len(out) + ctrl > maxout:
+                raise ValueError("truncated FastLZ literals")
+            out.extend(src[ip:ip + ctrl])
+            ip += ctrl
+        if ip >= ip_limit:
+            break
+        ctrl = src[ip]
+        ip += 1
+    return bytes(out)

openvixdisklib/nfc_open.py

@@ -26,6 +26,7 @@ import socket
 import ssl
 import struct
 
+from openvixdisklib import fastlz
 from openvixdisklib.nfc_auth import NfcAuthSession, _ssl_client_context
 
 NFC_MSG_SIZE = 264
@@ -69,6 +70,12 @@ NFC_OPEN_FLAGS_READ_WRITE = 0x1A
 NFC_AIO_IO_WRITE = 0
 NFC_AIO_IO_READ = 1
 
+# High 32 bits of the IO opcode uint64. Captured from VDDK FASTLZ:
+# writes that shrink go on the wire as type 2; incompressible writes
+# fall back to type 0 with raw extra data.
+NFC_COMPRESSION_NONE = 0
+NFC_COMPRESSION_FASTLZ = 2
+
 
 class NfcProtocolError(ConnectionError):
     """Raised when an NFC message is malformed or reports failure."""
@@ -168,7 +175,8 @@ class NfcDisk:
             sock: socket.socket,
             path: str,
             handle: int,
-            sector_size: int) -> None:
+            sector_size: int,
+            compression: int = NFC_COMPRESSION_NONE) -> None:
         """Wrap an AIO session that already has ``path`` open.
 
         Args:
@@ -176,12 +184,15 @@ class NfcDisk:
             path: Datastore path that was opened.
             handle: Server file handle from OPEN_FILE.
             sector_size: Sector size from the OPEN_FILE reply.
+            compression: NFC IO compression type (``NFC_COMPRESSION_NONE``
+                or ``NFC_COMPRESSION_FASTLZ``).
         """
         self._sock = sock
         self._op_id = 0
         self.path = path
         self.handle = handle
         self.sector_size = sector_size
+        self.compression = compression
         self._closed = False
 
     def _next_op_id(self) -> int:
@@ -226,7 +237,8 @@ class NfcDisk:
         byte units. If the length exceeds the AIO buffer (64 KiB) the
         server replies with several same-``opId`` fragments, which are
         placed by the fragment byte offset in the reply (they may arrive
-        out of order).
+        out of order). FASTLZ open requests compression in the opcode;
+        each reply fragment may be compressed (type 2) or raw (type 0).
 
         Args:
             start_sector: Sector offset from the start of the disk.
@@ -236,10 +248,11 @@ class NfcDisk:
             raise ValueError("num_sectors must be at least 1")
         length = num_sectors * self.sector_size
         offset = start_sector * self.sector_size
+        opcode = NFC_AIO_IO_READ | (self.compression << 32)
         payload = struct.pack(
             "<QQQQIII",
             self.handle,
-            NFC_AIO_IO_READ,
+            opcode,
             offset,
             length,
             length,
@@ -264,7 +277,10 @@ class NfcDisk:
                     f"AIO IO reply payload too short: {rsize}")
             # Fragments may arrive out of order. Offset 28 is the byte
             # offset of this chunk within the request (0, 65536, …),
-            # not a 0-based index.
+            # not a 0-based index. Offset 32 is the uncompressed
+            # fragment length. When the opcode high bits are FastLZ
+            # (2), extra data is compressed and offset 36 is its size.
+            opcode = struct.unpack_from("<Q", body, 8)[0]
             dest, chunk_len = struct.unpack_from("<II", body, 28)
             if (
                     dest in seen
@@ -274,7 +290,25 @@ class NfcDisk:
                     f"AIO IO chunk offset={dest} length={chunk_len} invalid, "
                     f"request {length}")
             seen.add(dest)
-            data[dest:dest + chunk_len] = _recvn(self._sock, chunk_len)
+            ctype = opcode >> 32
+            if ctype == NFC_COMPRESSION_FASTLZ:
+                comp_len = struct.unpack_from("<I", body, 36)[0]
+                extra = _recvn(self._sock, comp_len)
+                try:
+                    chunk = fastlz.decompress(extra, chunk_len)
+                except ValueError as exc:
+                    raise NfcProtocolError(
+                        f"FastLZ read fragment failed: {exc}") from exc
+                if len(chunk) != chunk_len:
+                    raise NfcProtocolError(
+                        f"FastLZ read got {len(chunk)} bytes, "
+                        f"expected {chunk_len}")
+            elif ctype == NFC_COMPRESSION_NONE:
+                chunk = _recvn(self._sock, chunk_len)
+            else:
+                raise NfcProtocolError(
+                    f"unsupported NFC IO compression type {ctype}")
+            data[dest:dest + chunk_len] = chunk
             filled += chunk_len
         return bytes(data)
 
@@ -288,6 +322,7 @@ class NfcDisk:
         Matches ``VixDiskLib_Write``: one ``NFC_AIO_MSG_IO`` request per
         chunk in byte units, with sector bytes sent after the 44-byte
         payload. Chunks larger than the AIO buffer (64 KiB) are split.
+        FASTLZ open compresses each chunk when that shrinks it.
 
         Args:
             start_sector: Sector offset from the start of the disk.
@@ -317,16 +352,26 @@ class NfcDisk:
             data: bytes) -> None:
         length = num_sectors * self.sector_size
         offset = start_sector * self.sector_size
+        extra = data
+        ctype = NFC_COMPRESSION_NONE
+        extra_len = length
+        if self.compression == NFC_COMPRESSION_FASTLZ and length >= 16:
+            compressed = fastlz.compress(data)
+            if compressed and len(compressed) < length:
+                extra = compressed
+                ctype = NFC_COMPRESSION_FASTLZ
+                extra_len = len(compressed)
+        opcode = NFC_AIO_IO_WRITE | (ctype << 32)
         payload = struct.pack(
             "<QQQQIII",
             self.handle,
-            NFC_AIO_IO_WRITE,
+            opcode,
             offset,
             length,
             length,
-            length,
+            extra_len,
             0)
-        self._aio_roundtrip(NFC_AIO_MSG_IO, payload, extra=data)
+        self._aio_roundtrip(NFC_AIO_MSG_IO, payload, extra=extra)
 
     def close(self) -> None:
         """Close the VMDK, the AIO session, and the classic NFC session."""
@@ -422,7 +467,8 @@ def open_disk(
         client_name: str = "vddk",
         op_id: str = "nbdmode",
         version: int = NFC_PROTOCOL_VERSION,
-        read_only: bool = True) -> NfcDisk:
+        read_only: bool = True,
+        compression: int = NFC_COMPRESSION_NONE) -> NfcDisk:
     """Open ``disk_path`` over the authenticated authd socket.
 
     Matches VDDK ``VixDiskLib_Open`` of a datastore path after the NFC
@@ -439,7 +485,13 @@ def open_disk(
         op_id: NFC operation id; VDDK NBD sends ``nbdmode``.
         version: Client NFC protocol version (lab ESXi answered 11).
         read_only: When True, open with VDDK's read-only NFC flags.
+        compression: ``NFC_COMPRESSION_NONE`` or ``NFC_COMPRESSION_FASTLZ``.
+            OPEN_FILE flags are unchanged; compression is per IO message.
     """
+    if compression not in (
+            NFC_COMPRESSION_NONE, NFC_COMPRESSION_FASTLZ):
+        raise NotImplementedError(
+            f"NFC compression type {compression} is not supported")
     if session.nfc_ssl:
         sock = wrap_nfcssl_socket(
             session.authd_sock, session.ticket.host)
@@ -447,7 +499,9 @@ def open_disk(
         sock = takeover_authd_socket(session.authd_sock)
     try:
         _handshake(sock, client_name, op_id, version)
-        disk = NfcDisk(sock, disk_path, handle=0, sector_size=NFC_SECTOR_SIZE)
+        disk = NfcDisk(
+            sock, disk_path, handle=0, sector_size=NFC_SECTOR_SIZE,
+            compression=compression)
         _aio_prepare(disk)
         path_b = disk_path.encode("utf-8")
         open_flags = (

openvixdisklib/openvixdisklib.py

@@ -44,10 +44,24 @@ VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB = 16
 VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ = 32
 VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ = 64
 
-_COMPRESSION_FLAGS = (
+
-    VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB
+def _nfc_compression(flags: int) -> int:
-    | VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ
+    """Return the NFC IO compression type for VixDiskLib open ``flags``."""
-    | VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ)
+    alg = flags & (
+        VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB
+        | VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ
+        | VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ)
+    if alg == 0:
+        return nfc_open.NFC_COMPRESSION_NONE
+    if alg == VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ:
+        return nfc_open.NFC_COMPRESSION_FASTLZ
+    if alg & (alg - 1):
+        raise NotImplementedError(
+            "Cannot set two or more NBD compression algorithms at the "
+            "same time")
+    raise NotImplementedError(
+        f"NBD compression open flag 0x{alg:x} is not supported")
+
 
 VIX_SUPPORTED_COMPATIBILITY_MODES = [
     "6.0", "6.5", "6.7", "7.0", "8.0"]
@@ -243,13 +257,12 @@ class VixDiskLibHandle:
             conn: Connection from ``connect``.
             disk_path: Datastore path of the VMDK.
             flags: Open flags. ``VIXDISKLIB_FLAG_OPEN_READ_ONLY`` opens
-                the disk read-only; omit it for write. Compression flags
+                the disk read-only; omit it for write.
-                are not implemented.
+                ``VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ`` compresses
+                NFC IO. zlib and skipz are not implemented.
         """
         LOG.debug("Openning VixDiskLib disk: %s", disk_path)
-        if flags & _COMPRESSION_FLAGS:
+        compression = _nfc_compression(flags)
-            raise NotImplementedError(
-                "NBD compression open flags are not supported")
         read_only = bool(flags & VIXDISKLIB_FLAG_OPEN_READ_ONLY)
         if not read_only and conn.read_only:
             raise NotImplementedError(
@@ -265,7 +278,8 @@ class VixDiskLibHandle:
             conn.si, ticket, authd_sock, nfc_ssl=nfc_ssl)
         try:
             disk = nfc_open.open_disk(
-                session, disk_path, read_only=read_only)
+                session, disk_path, read_only=read_only,
+                compression=compression)
         except Exception:
             authd_sock.close()
             raise

tests/integration/test_crosscheck.py

@@ -5,6 +5,8 @@
 
 from typing import Any, Optional
 
+import pytest
+
 from openvixdisklib import openvixdisklib as open_vix
 from tests.integration import vixdisklib
 from tests.integration.base import (
@@ -21,7 +23,8 @@ def _connect_extra(lab: LabEnv, module: Any) -> Optional[dict[str, Any]]:
 def _write_sectors(
         lab: LabEnv,
         module: Any,
-        payloads: dict[int, bytes]) -> None:
+        payloads: dict[int, bytes],
+        flags: int = 0) -> None:
     """Write one sector at each index using a vixdisklib-compatible module."""
     handle = module.VixDiskLibHandle(
         vixdisklib_compatibility_version="8.0",
@@ -29,7 +32,7 @@ def _write_sectors(
     buf = module.get_buffer(SECTOR_SIZE)
     kwargs = lab.vixdisklib_connect_kwargs(_connect_extra(lab, module))
     with handle.connect(**kwargs) as conn:
-        with handle.open(conn, lab.disk_path, flags=0) as disk:
+        with handle.open(conn, lab.disk_path, flags=flags) as disk:
             for start, data in payloads.items():
                 buf[:SECTOR_SIZE] = data
                 handle.write(disk, start, 1, buf)
@@ -38,7 +41,8 @@ def _write_sectors(
 def _read_sectors(
         lab: LabEnv,
         module: Any,
-        sectors: tuple[int, ...]) -> dict[int, bytes]:
+        sectors: tuple[int, ...],
+        flags: int = 0) -> dict[int, bytes]:
     """Read one sector at each index using a vixdisklib-compatible module."""
     handle = module.VixDiskLibHandle(
         vixdisklib_compatibility_version="8.0",
@@ -47,7 +51,7 @@ def _read_sectors(
     result: dict[int, bytes] = {}
     kwargs = lab.vixdisklib_connect_kwargs(_connect_extra(lab, module))
     with handle.connect(**kwargs) as conn:
-        with handle.open(conn, lab.disk_path, flags=0) as disk:
+        with handle.open(conn, lab.disk_path, flags=flags) as disk:
             for start in sectors:
                 buf[:SECTOR_SIZE] = b"\xa5" * SECTOR_SIZE
                 handle.read(disk, start, 1, buf)
@@ -58,9 +62,10 @@ def _read_sectors(
 def _assert_both_read(
         lab: LabEnv,
         sectors: tuple[int, ...],
-        expected: dict[int, bytes]) -> None:
+        expected: dict[int, bytes],
-    vddk_data = _read_sectors(lab, vixdisklib, sectors)
+        flags: int = 0) -> None:
-    replacement = _read_sectors(lab, open_vix, sectors)
+    vddk_data = _read_sectors(lab, vixdisklib, sectors, flags=flags)
+    replacement = _read_sectors(lab, open_vix, sectors, flags=flags)
     for start in sectors:
         assert vddk_data[start] == expected[start], (
             f"VDDK mismatch at sector {start}")
@@ -69,8 +74,12 @@ def _assert_both_read(
 
 
 class TestCrosscheck:
+    @pytest.mark.parametrize(
+        "open_flags",
+        [0, vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ],
+        ids=["plain", "fastlz"])
     def test_openvixdisklib_matches_vddk_sectors(
-            self, lab: LabEnv, vddk: None) -> None:
+            self, lab: LabEnv, vddk: None, open_flags: int) -> None:
         """Writes from either library must be visible to both readers."""
         sectors = (0, 1, SECTOR_AT_1GB)
         vddk_payloads = {
@@ -78,13 +87,13 @@ class TestCrosscheck:
             1: pattern_bytes(SECTOR_SIZE, b"XCHK-VDDK-S1"),
             SECTOR_AT_1GB: pattern_bytes(SECTOR_SIZE, b"XCHK-VDDK-1G"),
         }
-        _write_sectors(lab, vixdisklib, vddk_payloads)
+        _write_sectors(lab, vixdisklib, vddk_payloads, flags=open_flags)
-        _assert_both_read(lab, sectors, vddk_payloads)
+        _assert_both_read(lab, sectors, vddk_payloads, flags=open_flags)
 
         ovdl_payloads = {
             0: pattern_bytes(SECTOR_SIZE, b"XCHK-OVDL-S0"),
             1: pattern_bytes(SECTOR_SIZE, b"XCHK-OVDL-S1"),
             SECTOR_AT_1GB: pattern_bytes(SECTOR_SIZE, b"XCHK-OVDL-1G"),
         }
-        _write_sectors(lab, open_vix, ovdl_payloads)
+        _write_sectors(lab, open_vix, ovdl_payloads, flags=open_flags)
-        _assert_both_read(lab, sectors, ovdl_payloads)
+        _assert_both_read(lab, sectors, ovdl_payloads, flags=open_flags)

tests/integration/test_nfc_open.py

@@ -12,13 +12,18 @@ from tests.integration.base import LabEnv, SECTOR_SIZE, pattern_bytes
 class TestNfcOpen:
     @pytest.mark.parametrize(
         "nfc_ssl", [True, False], ids=["nbdssl", "nbd"])
+    @pytest.mark.parametrize(
+        "compression",
+        [nfc_open.NFC_COMPRESSION_NONE, nfc_open.NFC_COMPRESSION_FASTLZ],
+        ids=["plain", "fastlz"])
     def test_open_disk_and_read_first_sector(
-            self, lab: LabEnv, nfc_ssl: bool) -> None:
+            self, lab: LabEnv, nfc_ssl: bool, compression: int) -> None:
         """Open the temp VMDK, write sector 0, and read it back."""
         expected = pattern_bytes(SECTOR_SIZE, b"NFC-OPEN-S0")
         with lab.authenticate(read_only=False, nfc_ssl=nfc_ssl) as session:
             with nfc_open.open_disk(
-                    session, lab.disk_path, read_only=False) as disk:
+                    session, lab.disk_path, read_only=False,
+                    compression=compression) as disk:
                 assert disk.path == lab.disk_path
                 assert disk.handle > 0
                 assert disk.sector_size == SECTOR_SIZE

tests/integration/test_nfc_read_write.py

@@ -5,6 +5,8 @@
 
 import os
 
+import pytest
+
 from openvixdisklib import nfc_open
 from tests.integration.base import LabEnv, SECTOR_SIZE, pattern_bytes
 
@@ -12,7 +14,12 @@ _32MIB = 32 * 1024 * 1024
 
 
 class TestNfcReadWrite:
-    def test_sector_writes_and_reads(self, lab: LabEnv) -> None:
+    @pytest.mark.parametrize(
+        "compression",
+        [nfc_open.NFC_COMPRESSION_NONE, nfc_open.NFC_COMPRESSION_FASTLZ],
+        ids=["plain", "fastlz"])
+    def test_sector_writes_and_reads(
+            self, lab: LabEnv, compression: int) -> None:
         """Write known patterns and read them back at several ranges."""
         ranges = [
             (0, 1),
@@ -25,7 +32,8 @@ class TestNfcReadWrite:
         ]
         with lab.authenticate(read_only=False) as session:
             with nfc_open.open_disk(
-                    session, lab.disk_path, read_only=False) as disk:
+                    session, lab.disk_path, read_only=False,
+                    compression=compression) as disk:
                 for start, n_sectors in ranges:
                     length = n_sectors * SECTOR_SIZE
                     seed = f"NFC-R{start}:{n_sectors}-".encode()
@@ -55,13 +63,19 @@ class TestNfcReadWrite:
                     big_got[SECTOR_SIZE:2 * SECTOR_SIZE]
                     == big_to_write[SECTOR_SIZE:2 * SECTOR_SIZE])
 
-    def test_write_and_read_32mb(self, lab: LabEnv) -> None:
+    @pytest.mark.parametrize(
+        "compression",
+        [nfc_open.NFC_COMPRESSION_NONE, nfc_open.NFC_COMPRESSION_FASTLZ],
+        ids=["plain", "fastlz"])
+    def test_write_and_read_32mb(
+            self, lab: LabEnv, compression: int) -> None:
         """Write 32 MiB (512 AIO chunks) and read it back in one request."""
         n_sectors = _32MIB // SECTOR_SIZE
         to_write = os.urandom(_32MIB)
         with lab.authenticate(read_only=False) as session:
             with nfc_open.open_disk(
-                    session, lab.disk_path, read_only=False) as disk:
+                    session, lab.disk_path, read_only=False,
+                    compression=compression) as disk:
                 disk.write(0, n_sectors, to_write)
                 got = disk.read(0, n_sectors)
                 assert got is not to_write

tests/integration/test_openvixdisklib.py

@@ -12,8 +12,12 @@ from tests.integration.base import (
 
 class TestOpenvixdisklib:
     @pytest.mark.parametrize("transport_mode", ["nbdssl", "nbd"])
+    @pytest.mark.parametrize(
+        "open_flags",
+        [0, vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ],
+        ids=["plain", "fastlz"])
     def test_write_and_read_sector_zero_and_one_gib(
-            self, lab: LabEnv, transport_mode: str) -> None:
+            self, lab: LabEnv, transport_mode: str, open_flags: int) -> None:
         """Write then read sector 0 and the sector at a 1 GiB offset."""
         handle = vixdisklib.VixDiskLibHandle(
             vixdisklib_compatibility_version="8.0",
@@ -30,7 +34,7 @@ class TestOpenvixdisklib:
         }
         assert handle.get_transport_modes() == ["nbdssl", "nbd"]
         with handle.connect(**connect_kwargs) as conn:
-            with handle.open(conn, lab.disk_path, flags=0) as disk:
+            with handle.open(conn, lab.disk_path, flags=open_flags) as disk:
                 assert handle.get_transport_mode(disk) == transport_mode
                 for start, expected in patterns.items():
                     write_buf[:SECTOR_SIZE] = expected

tests/perf/test_compare.py

@@ -29,7 +29,8 @@ def _connect_extra(lab: LabEnv, module: Any) -> Optional[dict[str, Any]]:
 def _time_write_read(
         lab: LabEnv,
         module: Any,
-        payload: bytes) -> tuple[float, float]:
+        payload: bytes,
+        flags: int = 0) -> tuple[float, float]:
     """Write ``payload`` at sector 0, read it back, and return durations."""
     n_sectors = len(payload) // SECTOR_SIZE
     handle = module.VixDiskLibHandle(
@@ -40,7 +41,7 @@ def _time_write_read(
     write_buf[:len(payload)] = payload
     kwargs = lab.vixdisklib_connect_kwargs(_connect_extra(lab, module))
     with handle.connect(**kwargs) as conn:
-        with handle.open(conn, lab.disk_path, flags=0) as disk:
+        with handle.open(conn, lab.disk_path, flags=flags) as disk:
             started = time.perf_counter()
             handle.write(disk, 0, n_sectors, write_buf)
             write_s = time.perf_counter() - started
@@ -65,24 +66,33 @@ class TestCompare:
             ("vddk", vixdisklib),
             ("openvixdisklib", open_vix),
         )
-        rows: list[tuple[str, str, float, float, float, float]] = []
+        open_modes = (
+            ("plain", 0),
+            ("fastlz", vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ),
+        )
+        rows: list[tuple[str, str, str, float, float, float, float]] = []
         for label, nbytes in _SIZES:
             payload = pattern_bytes(nbytes, f"PERF-{label}-".encode())
-            for name, module in libraries:
+            for mode_name, flags in open_modes:
-                write_s, read_s = _time_write_read(lab, module, payload)
+                for name, module in libraries:
-                rows.append((
+                    write_s, read_s = _time_write_read(
-                    label,
+                        lab, module, payload, flags=flags)
-                    name,
+                    rows.append((
-                    write_s,
+                        label,
-                    read_s,
+                        mode_name,
-                    _mib_per_s(nbytes, write_s),
+                        name,
-                    _mib_per_s(nbytes, read_s),
+                        write_s,
-                ))
+                        read_s,
+                        _mib_per_s(nbytes, write_s),
+                        _mib_per_s(nbytes, read_s),
+                    ))
         print()
         print(
-            f"{'size':<14} {'library':<16} {'write_s':>10} {'read_s':>10} "
+            f"{'size':<14} {'flags':<8} {'library':<16} "
+            f"{'write_s':>10} {'read_s':>10} "
             f"{'write_MiB/s':>12} {'read_MiB/s':>12}")
-        for label, name, write_s, read_s, write_r, read_r in rows:
+        for label, mode_name, name, write_s, read_s, write_r, read_r in rows:
             print(
-                f"{label:<14} {name:<16} {write_s:10.3f} {read_s:10.3f} "
+                f"{label:<14} {mode_name:<8} {name:<16} "
+                f"{write_s:10.3f} {read_s:10.3f} "
                 f"{write_r:12.1f} {read_r:12.1f}")