Add fastlz compression support
This commit is contained in:
Lucian Petrut
@@ -18,9 +18,9 @@ Implemented against vCenter 8 / ESXi 8. Default transport is `nbdssl`
|
||||
- `VixDiskLib_Read`
|
||||
- `VixDiskLib_Write`
|
||||
|
||||
Not implemented: compression open flags, CBT / allocated-block
|
||||
queries, disk geometry (`DDB_GET`), encrypted disks, and direct ESXi
|
||||
`ha-nfc` without vCenter `vpxa-nfc`.
|
||||
Not implemented: compression open flags other than FastLZ, CBT /
|
||||
allocated-block queries, disk geometry (`DDB_GET`), encrypted disks,
|
||||
and direct ESXi `ha-nfc` without vCenter `vpxa-nfc`.
|
||||
|
||||
## Install
|
||||
|
||||
@@ -64,6 +64,7 @@ VDDK-shaped handle.
|
||||
| `openvixdisklib/openvixdisklib.py` | Drop-in handle (`connect` / `open` / `read` / `write`) |
|
||||
| `openvixdisklib/nfc_auth.py` | VIM login, NFC ticket, authd on 902 |
|
||||
| `openvixdisklib/nfc_open.py` | Classic NFC handshake, AIO open, sector read/write |
|
||||
| `openvixdisklib/fastlz.py` | FastLZ for `VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` |
|
||||
| `tests/integration/` | Live pytest suite against a lab vCenter |
|
||||
| `tests/perf/` | Throughput comparison of openvixdisklib vs VDDK |
|
||||
| `tests/integration/vixdisklib.py` | Native VDDK wrapper used only to cross-check |
|
||||
|
||||
+3
-1
@@ -31,6 +31,7 @@ sectors. Flags `0x1a` require the writable ticket; the same flags on a
|
||||
| `NfcUtil_PrintFileInfoOpenFlag` `NFC_DISK` `0x1e` | `NFC_AIO_MSG_OPEN_FILE` (read-only) |
|
||||
| Open without `VIXDISKLIB_FLAG_OPEN_READ_ONLY` | `OPEN_FILE` flags `0x1a` (read-write) |
|
||||
| `VixDiskLib_Read` / `VixDiskLib_Write` | `NFC_AIO_MSG_IO` + sector bytes |
|
||||
| `VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` | IO opcode high bits `2`; extra data is FastLZ |
|
||||
|
||||
`snapshot_ref` is still not on the wire. Integration tests pass the
|
||||
flat VMDK created with the temporary lab VM.
|
||||
@@ -208,6 +209,7 @@ classic type 4 `NFC_SESSION_COMPLETE`.
|
||||
| VIM + authd | `openvixdisklib.nfc_auth.authenticate` |
|
||||
| Dup fd, skip TLS for NFC | `openvixdisklib.nfc_open.takeover_authd_socket` |
|
||||
| Second TLS for nbdssl | `openvixdisklib.nfc_open.wrap_nfcssl_socket` |
|
||||
| FastLZ for NBD compression | `openvixdisklib.fastlz` |
|
||||
| Handshake + AIO + OPEN_FILE | `openvixdisklib.nfc_open.open_disk` |
|
||||
| Sector read / write / close | `openvixdisklib.nfc_open.NfcDisk` |
|
||||
|
||||
@@ -224,7 +226,7 @@ I/O: `docs/nfc_read.md`, `docs/nfc_write.md`, and
|
||||
|
||||
## What is still VDDK-only
|
||||
|
||||
- `DDB_GET` / geometry / compression / encryption keys
|
||||
- `DDB_GET` / geometry / zlib and skipz compression / encryption keys
|
||||
- `NFC_DELTA_DISK`, change-block tracking
|
||||
- Host-switch (`NFC_AIO_SWITCH_HOST_*`)
|
||||
- Direct ESXi `ha-nfc` without vCenter `vpxa-nfc`
|
||||
|
||||
+29
-10
@@ -39,21 +39,40 @@ LBA 8 on this disk was 4096 zero bytes on the wire, not a skip.
|
||||
Little-endian, after the usual 16-byte AIO header
|
||||
(`magic 0xA100DA7A`, type 7, size 44, monotonic `opId`):
|
||||
|
||||
| Offset | Type | VDDK `Read(start, n)` |
|
||||
| ------ | -------- | ---------------------------------------------- |
|
||||
| 0 | `uint64` | File handle from `OPEN_FILE` |
|
||||
| 8 | `uint64` | `1` (`NFC_AIO_IO_READ`; write uses `0`) |
|
||||
| 16 | `uint64` | Byte offset |
|
||||
| 24 | `uint64` | Byte length |
|
||||
| 32 | `uint32` | Byte length (same value) |
|
||||
| 36 | `uint32` | Byte length (same value) |
|
||||
| 40 | `uint32` | `0` (flags; uncompressed in this capture) |
|
||||
| Offset | Type | VDDK `Read(start, n)` |
|
||||
| ------ | -------- | ---------------------------------------------------------- |
|
||||
| 0 | `uint64` | File handle from `OPEN_FILE` |
|
||||
| 8 | `uint64` | Direction in low 32 bits; FastLZ type `2` in high 32 bits |
|
||||
| 16 | `uint64` | Byte offset |
|
||||
| 24 | `uint64` | Byte length |
|
||||
| 32 | `uint32` | Byte length (same value) |
|
||||
| 36 | `uint32` | Byte length, or compressed extra size when type is FastLZ |
|
||||
| 40 | `uint32` | `0` |
|
||||
|
||||
An earlier guess that offset 36 was `NFC_DISK` (`2`) was wrong: a
|
||||
1-sector VDDK read puts `512` in both `uint32` length fields. A Python
|
||||
read that sent `(512, 2, 0)` still worked for one sector; the
|
||||
replacement now matches VDDK.
|
||||
|
||||
`VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` does not change OPEN_FILE
|
||||
flags. The IO opcode at offset 8 is a `uint64`: low 32 bits are still
|
||||
`0`/`1` (write/read), high 32 bits are the NFC compression type
|
||||
(`2` = FastLZ). OPEN still uses handshake `PlainText`.
|
||||
|
||||
| Open flag / wire | Request extra | Reply extra |
|
||||
| ---------------------------------------- | ------------------------------------- | ------------------------------------------------ |
|
||||
| No compression flag | Raw `length` bytes on write | Raw fragment at offset 32 |
|
||||
| FASTLZ, data that shrinks | FastLZ bytes; offset 36 = packed size | Opcode type `2`; extra is FastLZ of offset 32 |
|
||||
| FASTLZ, incompressible | Raw bytes; opcode type `0` | Opcode type `0`; extra is raw |
|
||||
|
||||
Reads with FASTLZ always *request* type `2`. The server may answer type
|
||||
`2` or fall back to type `0`. Decompress into the uncompressed fragment
|
||||
length at offset 32 and copy to the dest at offset 28.
|
||||
|
||||
64 KiB chunks use FastLZ level 2 (first byte has bit 5 set). Smaller
|
||||
chunks use level 1. VDDK’s URL form is `FASTLZ-vpxa-nfc://…`; authd
|
||||
`PROXY` is unchanged.
|
||||
|
||||
## Reply
|
||||
|
||||
Each fragment is: 16-byte AIO header (same `type` and `opId`) + 44-byte
|
||||
@@ -117,7 +136,7 @@ fragments).
|
||||
|
||||
## What is still VDDK-only
|
||||
|
||||
- Compression flags on the last `uint32`
|
||||
- zlib and skipz NBD compression flags
|
||||
- `VixDiskLib_ReadAsync` (same IO messages, different client threading)
|
||||
- `VixDiskLib_QueryAllocatedBlocks` / allocation bitmaps
|
||||
- `VixDiskLib_GetInfo` capacity (not required to read a known range)
|
||||
|
||||
@@ -60,6 +60,11 @@ Little-endian, after the usual 16-byte AIO header
|
||||
| 36 | `uint32` | Byte length (same value) |
|
||||
| 40 | `uint32` | `0` |
|
||||
|
||||
FASTLZ writes use the same 44-byte header. The opcode `uint64` high
|
||||
half is `2`, offset 36 is the compressed size, and FastLZ bytes follow
|
||||
instead of raw sectors. If compression does not shrink the chunk, VDDK
|
||||
sends type `0` and raw extra (same as an uncompressed write).
|
||||
|
||||
Sector bytes follow the 44-byte payload and are **not** counted in AIO
|
||||
`size`. VDDK sends header + payload + data in one `write()`. The
|
||||
replacement may split that into two `sendall`s; TCP does not care.
|
||||
|
||||
@@ -280,6 +280,24 @@ the old `VixDiskLibConnectParams` ctypes struct can still log nbdssl
|
||||
and then fall back to `vpxa-nfc` / `useSSL=0`. Do not treat that log
|
||||
line as a wire capture of NFCSSL.
|
||||
|
||||
## Step 12 — FASTLZ NBD compression
|
||||
|
||||
`VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` (`1 << 5`) is an IO codec,
|
||||
not an OPEN_FILE bit. Capture VDDK with that flag (NBD + the port-902
|
||||
`write`/`read` hook):
|
||||
|
||||
- Handshake stays `PlainText`. `OPEN_FILE` flags stay `0x1a` / `0x1e`.
|
||||
- VDDK’s URL is `FASTLZ-vpxa-nfc://…`; `PROXY` is still `vpxa-nfc`.
|
||||
- IO opcode `uint64` = direction in the low half, compression type in
|
||||
the high half (`2` = FastLZ). Offset 32 is uncompressed length;
|
||||
offset 36 is compressed extra size when type is 2.
|
||||
- Incompressible chunks fall back to type `0` and raw extra.
|
||||
- 64 KiB chunks use FastLZ level 2; smaller chunks use level 1.
|
||||
|
||||
Replay: `openvixdisklib/fastlz.py` plus `NfcDisk` compression on each
|
||||
IO. Proof: `tests/integration/test_nfc_read_write.py` (`fastlz`) and
|
||||
`tests/perf/test_compare.py`.
|
||||
|
||||
## What to write down
|
||||
|
||||
After a stage works:
|
||||
@@ -300,7 +318,7 @@ replacement library.
|
||||
|
||||
Not yet reversed, same loop as above:
|
||||
|
||||
- `DDB_GET` / disk geometry, compression, encrypted disks
|
||||
- `DDB_GET` / disk geometry, zlib/skipz compression, encrypted disks
|
||||
- `NFC_DELTA_DISK`, CBT / `QueryAllocatedBlocks`
|
||||
- `VixDiskLib_GetInfo` capacity
|
||||
- Host-switch AIO messages
|
||||
|
||||
@@ -0,0 +1,296 @@
|
||||
# Copyright 2026 Cloudbase Solutions Srl
|
||||
# All Rights Reserved.
|
||||
#
|
||||
# Python port of FastLZ (Byte-aligned LZ77), used by VMware NFC
|
||||
# VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ.
|
||||
#
|
||||
# FastLZ - Copyright (C) 2005-2020 Ariya Hidayat <[email protected]>
|
||||
# Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
# of this software and associated documentation files (the "Software"), to deal
|
||||
# in the Software without restriction, including without limitation the rights
|
||||
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
# copies of the Software, and to permit persons to whom the Software is
|
||||
# furnished to do so, subject to the following conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be included in
|
||||
# all copies or substantial portions of the Software.
|
||||
#
|
||||
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||
# THE SOFTWARE.
|
||||
|
||||
"""FastLZ compress / decompress used by NFC NBD compression."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
MAX_COPY = 32
|
||||
MAX_LEN = 264
|
||||
MAX_L1_DISTANCE = 8192
|
||||
MAX_L2_DISTANCE = 8191
|
||||
MAX_FARDISTANCE = 65535 + MAX_L2_DISTANCE - 1
|
||||
HASH_LOG = 13
|
||||
HASH_SIZE = 1 << HASH_LOG
|
||||
HASH_MASK = HASH_SIZE - 1
|
||||
|
||||
|
||||
def _readu32(buf: bytes | bytearray, offset: int) -> int:
|
||||
return buf[offset] | (buf[offset + 1] << 8) | (
|
||||
buf[offset + 2] << 16) | (buf[offset + 3] << 24)
|
||||
|
||||
|
||||
def _hash(value: int) -> int:
|
||||
return ((value * 2654435769) >> (32 - HASH_LOG)) & HASH_MASK
|
||||
|
||||
|
||||
def _cmp(src: bytes, p: int, q: int, q_end: int) -> int:
|
||||
start = p
|
||||
if _readu32(src, p) == _readu32(src, q):
|
||||
p += 4
|
||||
q += 4
|
||||
while q < q_end:
|
||||
mismatch = src[p] != src[q]
|
||||
p += 1
|
||||
q += 1
|
||||
if mismatch:
|
||||
break
|
||||
return p - start
|
||||
|
||||
|
||||
def _extend_match(out: bytearray, ref: int, length: int) -> None:
|
||||
while length:
|
||||
take = min(length, len(out) - ref)
|
||||
out.extend(out[ref:ref + take])
|
||||
length -= take
|
||||
|
||||
|
||||
def _literals(src: bytes, src_off: int, runs: int, out: bytearray) -> None:
|
||||
while runs >= MAX_COPY:
|
||||
out.append(MAX_COPY - 1)
|
||||
out.extend(src[src_off:src_off + MAX_COPY])
|
||||
src_off += MAX_COPY
|
||||
runs -= MAX_COPY
|
||||
if runs > 0:
|
||||
out.append(runs - 1)
|
||||
out.extend(src[src_off:src_off + runs])
|
||||
|
||||
|
||||
def _flz1_match(length: int, distance: int, out: bytearray) -> None:
|
||||
distance -= 1
|
||||
if length > MAX_LEN - 2:
|
||||
while length > MAX_LEN - 2:
|
||||
out.append((7 << 5) + (distance >> 8))
|
||||
out.append(MAX_LEN - 2 - 7 - 2)
|
||||
out.append(distance & 255)
|
||||
length -= MAX_LEN - 2
|
||||
if length < 7:
|
||||
out.append((length << 5) + (distance >> 8))
|
||||
out.append(distance & 255)
|
||||
else:
|
||||
out.append((7 << 5) + (distance >> 8))
|
||||
out.append(length - 7)
|
||||
out.append(distance & 255)
|
||||
|
||||
|
||||
def _flz2_match(length: int, distance: int, out: bytearray) -> None:
|
||||
distance -= 1
|
||||
if distance < MAX_L2_DISTANCE:
|
||||
if length < 7:
|
||||
out.append((length << 5) + (distance >> 8))
|
||||
out.append(distance & 255)
|
||||
else:
|
||||
out.append((7 << 5) + (distance >> 8))
|
||||
length -= 7
|
||||
while length >= 255:
|
||||
out.append(255)
|
||||
length -= 255
|
||||
out.append(length)
|
||||
out.append(distance & 255)
|
||||
elif length < 7:
|
||||
distance -= MAX_L2_DISTANCE
|
||||
out.append((length << 5) + 31)
|
||||
out.append(255)
|
||||
out.append(distance >> 8)
|
||||
out.append(distance & 255)
|
||||
else:
|
||||
distance -= MAX_L2_DISTANCE
|
||||
out.append((7 << 5) + 31)
|
||||
length -= 7
|
||||
while length >= 255:
|
||||
out.append(255)
|
||||
length -= 255
|
||||
out.append(length)
|
||||
out.append(255)
|
||||
out.append(distance >> 8)
|
||||
out.append(distance & 255)
|
||||
|
||||
|
||||
def _compress_level(src: bytes, level: int) -> bytes:
|
||||
length = len(src)
|
||||
ip = 0
|
||||
ip_bound = length - 4
|
||||
ip_limit = length - 12 - 1
|
||||
out = bytearray()
|
||||
htab = [0] * HASH_SIZE
|
||||
max_distance = MAX_L1_DISTANCE if level == 1 else MAX_FARDISTANCE
|
||||
anchor = 0
|
||||
ip = 2
|
||||
while ip < ip_limit:
|
||||
while True:
|
||||
seq = _readu32(src, ip) & 0xffffff
|
||||
h = _hash(seq)
|
||||
ref = htab[h]
|
||||
htab[h] = ip
|
||||
distance = ip - ref
|
||||
cmp_val = (
|
||||
_readu32(src, ref) & 0xffffff
|
||||
if 0 <= ref < ip and distance < max_distance
|
||||
else 0x1000000)
|
||||
if ip >= ip_limit:
|
||||
break
|
||||
ip += 1
|
||||
if seq == cmp_val:
|
||||
break
|
||||
if ip >= ip_limit:
|
||||
break
|
||||
ip -= 1
|
||||
if level == 2 and distance >= MAX_L2_DISTANCE:
|
||||
if src[ref + 3] != src[ip + 3] or src[ref + 4] != src[ip + 4]:
|
||||
ip += 1
|
||||
continue
|
||||
if ip > anchor:
|
||||
_literals(src, anchor, ip - anchor, out)
|
||||
match_len = _cmp(src, ref + 3, ip + 3, ip_bound)
|
||||
if level == 1:
|
||||
_flz1_match(match_len, distance, out)
|
||||
else:
|
||||
_flz2_match(match_len, distance, out)
|
||||
ip += match_len
|
||||
seq = _readu32(src, ip)
|
||||
htab[_hash(seq & 0xffffff)] = ip
|
||||
ip += 1
|
||||
htab[_hash(seq >> 8)] = ip
|
||||
ip += 1
|
||||
anchor = ip
|
||||
_literals(src, anchor, length - anchor, out)
|
||||
if level == 2 and out:
|
||||
out[0] |= 1 << 5
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def compress(data: bytes) -> bytes:
|
||||
"""Compress ``data`` with FastLZ (level 1 below 64 KiB, else level 2).
|
||||
|
||||
Args:
|
||||
data: Uncompressed bytes. FastLZ needs at least 16 bytes.
|
||||
"""
|
||||
if len(data) < 65536:
|
||||
return _compress_level(data, 1)
|
||||
return _compress_level(data, 2)
|
||||
|
||||
|
||||
def decompress(data: bytes, maxout: int) -> bytes:
|
||||
"""Decompress FastLZ ``data`` into at most ``maxout`` bytes.
|
||||
|
||||
Args:
|
||||
data: Compressed FastLZ buffer.
|
||||
maxout: Expected uncompressed length (output cap).
|
||||
"""
|
||||
if not data:
|
||||
raise ValueError("FastLZ input is empty")
|
||||
level = (data[0] >> 5) + 1
|
||||
if level == 1:
|
||||
return _decompress_level1(data, maxout)
|
||||
if level == 2:
|
||||
return _decompress_level2(data, maxout)
|
||||
raise ValueError(f"unsupported FastLZ level {level}")
|
||||
|
||||
|
||||
def _decompress_level1(src: bytes, maxout: int) -> bytes:
|
||||
ip = 0
|
||||
ip_limit = len(src)
|
||||
ip_bound = ip_limit - 2
|
||||
out = bytearray()
|
||||
ctrl = src[ip] & 31
|
||||
ip += 1
|
||||
while True:
|
||||
if ctrl >= 32:
|
||||
length = (ctrl >> 5) - 1
|
||||
ofs = (ctrl & 31) << 8
|
||||
if length == 6:
|
||||
if ip > ip_bound:
|
||||
raise ValueError("truncated FastLZ match")
|
||||
length += src[ip]
|
||||
ip += 1
|
||||
if ip >= ip_limit:
|
||||
raise ValueError("truncated FastLZ match distance")
|
||||
ofs += src[ip]
|
||||
ip += 1
|
||||
length += 3
|
||||
ref = len(out) - ofs - 1
|
||||
if ref < 0 or len(out) + length > maxout:
|
||||
raise ValueError("FastLZ match out of range")
|
||||
_extend_match(out, ref, length)
|
||||
else:
|
||||
ctrl += 1
|
||||
if ip + ctrl > ip_limit or len(out) + ctrl > maxout:
|
||||
raise ValueError("truncated FastLZ literals")
|
||||
out.extend(src[ip:ip + ctrl])
|
||||
ip += ctrl
|
||||
if ip > ip_bound:
|
||||
break
|
||||
ctrl = src[ip]
|
||||
ip += 1
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def _decompress_level2(src: bytes, maxout: int) -> bytes:
|
||||
ip = 0
|
||||
ip_limit = len(src)
|
||||
ip_bound = ip_limit - 2
|
||||
out = bytearray()
|
||||
ctrl = src[ip] & 31
|
||||
ip += 1
|
||||
while True:
|
||||
if ctrl >= 32:
|
||||
length = (ctrl >> 5) - 1
|
||||
ofs = (ctrl & 31) << 8
|
||||
if length == 6:
|
||||
while True:
|
||||
if ip > ip_bound:
|
||||
raise ValueError("truncated FastLZ match")
|
||||
code = src[ip]
|
||||
ip += 1
|
||||
length += code
|
||||
if code != 255:
|
||||
break
|
||||
if ip >= ip_limit:
|
||||
raise ValueError("truncated FastLZ match distance")
|
||||
code = src[ip]
|
||||
ip += 1
|
||||
ofs += code
|
||||
length += 3
|
||||
ref = len(out) - ofs - 1
|
||||
if code == 255 and ofs == (31 << 8):
|
||||
if ip >= ip_bound:
|
||||
raise ValueError("truncated FastLZ far match")
|
||||
ofs = (src[ip] << 8) + src[ip + 1]
|
||||
ip += 2
|
||||
ref = len(out) - ofs - MAX_L2_DISTANCE - 1
|
||||
if ref < 0 or len(out) + length > maxout:
|
||||
raise ValueError("FastLZ match out of range")
|
||||
_extend_match(out, ref, length)
|
||||
else:
|
||||
ctrl += 1
|
||||
if ip + ctrl > ip_limit or len(out) + ctrl > maxout:
|
||||
raise ValueError("truncated FastLZ literals")
|
||||
out.extend(src[ip:ip + ctrl])
|
||||
ip += ctrl
|
||||
if ip >= ip_limit:
|
||||
break
|
||||
ctrl = src[ip]
|
||||
ip += 1
|
||||
return bytes(out)
|
||||
+64
-10
@@ -26,6 +26,7 @@ import socket
|
||||
import ssl
|
||||
import struct
|
||||
|
||||
from openvixdisklib import fastlz
|
||||
from openvixdisklib.nfc_auth import NfcAuthSession, _ssl_client_context
|
||||
|
||||
NFC_MSG_SIZE = 264
|
||||
@@ -69,6 +70,12 @@ NFC_OPEN_FLAGS_READ_WRITE = 0x1A
|
||||
NFC_AIO_IO_WRITE = 0
|
||||
NFC_AIO_IO_READ = 1
|
||||
|
||||
# High 32 bits of the IO opcode uint64. Captured from VDDK FASTLZ:
|
||||
# writes that shrink go on the wire as type 2; incompressible writes
|
||||
# fall back to type 0 with raw extra data.
|
||||
NFC_COMPRESSION_NONE = 0
|
||||
NFC_COMPRESSION_FASTLZ = 2
|
||||
|
||||
|
||||
class NfcProtocolError(ConnectionError):
|
||||
"""Raised when an NFC message is malformed or reports failure."""
|
||||
@@ -168,7 +175,8 @@ class NfcDisk:
|
||||
sock: socket.socket,
|
||||
path: str,
|
||||
handle: int,
|
||||
sector_size: int) -> None:
|
||||
sector_size: int,
|
||||
compression: int = NFC_COMPRESSION_NONE) -> None:
|
||||
"""Wrap an AIO session that already has ``path`` open.
|
||||
|
||||
Args:
|
||||
@@ -176,12 +184,15 @@ class NfcDisk:
|
||||
path: Datastore path that was opened.
|
||||
handle: Server file handle from OPEN_FILE.
|
||||
sector_size: Sector size from the OPEN_FILE reply.
|
||||
compression: NFC IO compression type (``NFC_COMPRESSION_NONE``
|
||||
or ``NFC_COMPRESSION_FASTLZ``).
|
||||
"""
|
||||
self._sock = sock
|
||||
self._op_id = 0
|
||||
self.path = path
|
||||
self.handle = handle
|
||||
self.sector_size = sector_size
|
||||
self.compression = compression
|
||||
self._closed = False
|
||||
|
||||
def _next_op_id(self) -> int:
|
||||
@@ -226,7 +237,8 @@ class NfcDisk:
|
||||
byte units. If the length exceeds the AIO buffer (64 KiB) the
|
||||
server replies with several same-``opId`` fragments, which are
|
||||
placed by the fragment byte offset in the reply (they may arrive
|
||||
out of order).
|
||||
out of order). FASTLZ open requests compression in the opcode;
|
||||
each reply fragment may be compressed (type 2) or raw (type 0).
|
||||
|
||||
Args:
|
||||
start_sector: Sector offset from the start of the disk.
|
||||
@@ -236,10 +248,11 @@ class NfcDisk:
|
||||
raise ValueError("num_sectors must be at least 1")
|
||||
length = num_sectors * self.sector_size
|
||||
offset = start_sector * self.sector_size
|
||||
opcode = NFC_AIO_IO_READ | (self.compression << 32)
|
||||
payload = struct.pack(
|
||||
"<QQQQIII",
|
||||
self.handle,
|
||||
NFC_AIO_IO_READ,
|
||||
opcode,
|
||||
offset,
|
||||
length,
|
||||
length,
|
||||
@@ -264,7 +277,10 @@ class NfcDisk:
|
||||
f"AIO IO reply payload too short: {rsize}")
|
||||
# Fragments may arrive out of order. Offset 28 is the byte
|
||||
# offset of this chunk within the request (0, 65536, …),
|
||||
# not a 0-based index.
|
||||
# not a 0-based index. Offset 32 is the uncompressed
|
||||
# fragment length. When the opcode high bits are FastLZ
|
||||
# (2), extra data is compressed and offset 36 is its size.
|
||||
opcode = struct.unpack_from("<Q", body, 8)[0]
|
||||
dest, chunk_len = struct.unpack_from("<II", body, 28)
|
||||
if (
|
||||
dest in seen
|
||||
@@ -274,7 +290,25 @@ class NfcDisk:
|
||||
f"AIO IO chunk offset={dest} length={chunk_len} invalid, "
|
||||
f"request {length}")
|
||||
seen.add(dest)
|
||||
data[dest:dest + chunk_len] = _recvn(self._sock, chunk_len)
|
||||
ctype = opcode >> 32
|
||||
if ctype == NFC_COMPRESSION_FASTLZ:
|
||||
comp_len = struct.unpack_from("<I", body, 36)[0]
|
||||
extra = _recvn(self._sock, comp_len)
|
||||
try:
|
||||
chunk = fastlz.decompress(extra, chunk_len)
|
||||
except ValueError as exc:
|
||||
raise NfcProtocolError(
|
||||
f"FastLZ read fragment failed: {exc}") from exc
|
||||
if len(chunk) != chunk_len:
|
||||
raise NfcProtocolError(
|
||||
f"FastLZ read got {len(chunk)} bytes, "
|
||||
f"expected {chunk_len}")
|
||||
elif ctype == NFC_COMPRESSION_NONE:
|
||||
chunk = _recvn(self._sock, chunk_len)
|
||||
else:
|
||||
raise NfcProtocolError(
|
||||
f"unsupported NFC IO compression type {ctype}")
|
||||
data[dest:dest + chunk_len] = chunk
|
||||
filled += chunk_len
|
||||
return bytes(data)
|
||||
|
||||
@@ -288,6 +322,7 @@ class NfcDisk:
|
||||
Matches ``VixDiskLib_Write``: one ``NFC_AIO_MSG_IO`` request per
|
||||
chunk in byte units, with sector bytes sent after the 44-byte
|
||||
payload. Chunks larger than the AIO buffer (64 KiB) are split.
|
||||
FASTLZ open compresses each chunk when that shrinks it.
|
||||
|
||||
Args:
|
||||
start_sector: Sector offset from the start of the disk.
|
||||
@@ -317,16 +352,26 @@ class NfcDisk:
|
||||
data: bytes) -> None:
|
||||
length = num_sectors * self.sector_size
|
||||
offset = start_sector * self.sector_size
|
||||
extra = data
|
||||
ctype = NFC_COMPRESSION_NONE
|
||||
extra_len = length
|
||||
if self.compression == NFC_COMPRESSION_FASTLZ and length >= 16:
|
||||
compressed = fastlz.compress(data)
|
||||
if compressed and len(compressed) < length:
|
||||
extra = compressed
|
||||
ctype = NFC_COMPRESSION_FASTLZ
|
||||
extra_len = len(compressed)
|
||||
opcode = NFC_AIO_IO_WRITE | (ctype << 32)
|
||||
payload = struct.pack(
|
||||
"<QQQQIII",
|
||||
self.handle,
|
||||
NFC_AIO_IO_WRITE,
|
||||
opcode,
|
||||
offset,
|
||||
length,
|
||||
length,
|
||||
length,
|
||||
extra_len,
|
||||
0)
|
||||
self._aio_roundtrip(NFC_AIO_MSG_IO, payload, extra=data)
|
||||
self._aio_roundtrip(NFC_AIO_MSG_IO, payload, extra=extra)
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close the VMDK, the AIO session, and the classic NFC session."""
|
||||
@@ -422,7 +467,8 @@ def open_disk(
|
||||
client_name: str = "vddk",
|
||||
op_id: str = "nbdmode",
|
||||
version: int = NFC_PROTOCOL_VERSION,
|
||||
read_only: bool = True) -> NfcDisk:
|
||||
read_only: bool = True,
|
||||
compression: int = NFC_COMPRESSION_NONE) -> NfcDisk:
|
||||
"""Open ``disk_path`` over the authenticated authd socket.
|
||||
|
||||
Matches VDDK ``VixDiskLib_Open`` of a datastore path after the NFC
|
||||
@@ -439,7 +485,13 @@ def open_disk(
|
||||
op_id: NFC operation id; VDDK NBD sends ``nbdmode``.
|
||||
version: Client NFC protocol version (lab ESXi answered 11).
|
||||
read_only: When True, open with VDDK's read-only NFC flags.
|
||||
compression: ``NFC_COMPRESSION_NONE`` or ``NFC_COMPRESSION_FASTLZ``.
|
||||
OPEN_FILE flags are unchanged; compression is per IO message.
|
||||
"""
|
||||
if compression not in (
|
||||
NFC_COMPRESSION_NONE, NFC_COMPRESSION_FASTLZ):
|
||||
raise NotImplementedError(
|
||||
f"NFC compression type {compression} is not supported")
|
||||
if session.nfc_ssl:
|
||||
sock = wrap_nfcssl_socket(
|
||||
session.authd_sock, session.ticket.host)
|
||||
@@ -447,7 +499,9 @@ def open_disk(
|
||||
sock = takeover_authd_socket(session.authd_sock)
|
||||
try:
|
||||
_handshake(sock, client_name, op_id, version)
|
||||
disk = NfcDisk(sock, disk_path, handle=0, sector_size=NFC_SECTOR_SIZE)
|
||||
disk = NfcDisk(
|
||||
sock, disk_path, handle=0, sector_size=NFC_SECTOR_SIZE,
|
||||
compression=compression)
|
||||
_aio_prepare(disk)
|
||||
path_b = disk_path.encode("utf-8")
|
||||
open_flags = (
|
||||
|
||||
@@ -44,10 +44,24 @@ VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB = 16
|
||||
VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ = 32
|
||||
VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ = 64
|
||||
|
||||
_COMPRESSION_FLAGS = (
|
||||
VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB
|
||||
| VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ
|
||||
| VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ)
|
||||
|
||||
def _nfc_compression(flags: int) -> int:
|
||||
"""Return the NFC IO compression type for VixDiskLib open ``flags``."""
|
||||
alg = flags & (
|
||||
VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB
|
||||
| VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ
|
||||
| VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ)
|
||||
if alg == 0:
|
||||
return nfc_open.NFC_COMPRESSION_NONE
|
||||
if alg == VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ:
|
||||
return nfc_open.NFC_COMPRESSION_FASTLZ
|
||||
if alg & (alg - 1):
|
||||
raise NotImplementedError(
|
||||
"Cannot set two or more NBD compression algorithms at the "
|
||||
"same time")
|
||||
raise NotImplementedError(
|
||||
f"NBD compression open flag 0x{alg:x} is not supported")
|
||||
|
||||
|
||||
VIX_SUPPORTED_COMPATIBILITY_MODES = [
|
||||
"6.0", "6.5", "6.7", "7.0", "8.0"]
|
||||
@@ -243,13 +257,12 @@ class VixDiskLibHandle:
|
||||
conn: Connection from ``connect``.
|
||||
disk_path: Datastore path of the VMDK.
|
||||
flags: Open flags. ``VIXDISKLIB_FLAG_OPEN_READ_ONLY`` opens
|
||||
the disk read-only; omit it for write. Compression flags
|
||||
are not implemented.
|
||||
the disk read-only; omit it for write.
|
||||
``VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ`` compresses
|
||||
NFC IO. zlib and skipz are not implemented.
|
||||
"""
|
||||
LOG.debug("Openning VixDiskLib disk: %s", disk_path)
|
||||
if flags & _COMPRESSION_FLAGS:
|
||||
raise NotImplementedError(
|
||||
"NBD compression open flags are not supported")
|
||||
compression = _nfc_compression(flags)
|
||||
read_only = bool(flags & VIXDISKLIB_FLAG_OPEN_READ_ONLY)
|
||||
if not read_only and conn.read_only:
|
||||
raise NotImplementedError(
|
||||
@@ -265,7 +278,8 @@ class VixDiskLibHandle:
|
||||
conn.si, ticket, authd_sock, nfc_ssl=nfc_ssl)
|
||||
try:
|
||||
disk = nfc_open.open_disk(
|
||||
session, disk_path, read_only=read_only)
|
||||
session, disk_path, read_only=read_only,
|
||||
compression=compression)
|
||||
except Exception:
|
||||
authd_sock.close()
|
||||
raise
|
||||
|
||||
@@ -5,6 +5,8 @@
|
||||
|
||||
from typing import Any, Optional
|
||||
|
||||
import pytest
|
||||
|
||||
from openvixdisklib import openvixdisklib as open_vix
|
||||
from tests.integration import vixdisklib
|
||||
from tests.integration.base import (
|
||||
@@ -21,7 +23,8 @@ def _connect_extra(lab: LabEnv, module: Any) -> Optional[dict[str, Any]]:
|
||||
def _write_sectors(
|
||||
lab: LabEnv,
|
||||
module: Any,
|
||||
payloads: dict[int, bytes]) -> None:
|
||||
payloads: dict[int, bytes],
|
||||
flags: int = 0) -> None:
|
||||
"""Write one sector at each index using a vixdisklib-compatible module."""
|
||||
handle = module.VixDiskLibHandle(
|
||||
vixdisklib_compatibility_version="8.0",
|
||||
@@ -29,7 +32,7 @@ def _write_sectors(
|
||||
buf = module.get_buffer(SECTOR_SIZE)
|
||||
kwargs = lab.vixdisklib_connect_kwargs(_connect_extra(lab, module))
|
||||
with handle.connect(**kwargs) as conn:
|
||||
with handle.open(conn, lab.disk_path, flags=0) as disk:
|
||||
with handle.open(conn, lab.disk_path, flags=flags) as disk:
|
||||
for start, data in payloads.items():
|
||||
buf[:SECTOR_SIZE] = data
|
||||
handle.write(disk, start, 1, buf)
|
||||
@@ -38,7 +41,8 @@ def _write_sectors(
|
||||
def _read_sectors(
|
||||
lab: LabEnv,
|
||||
module: Any,
|
||||
sectors: tuple[int, ...]) -> dict[int, bytes]:
|
||||
sectors: tuple[int, ...],
|
||||
flags: int = 0) -> dict[int, bytes]:
|
||||
"""Read one sector at each index using a vixdisklib-compatible module."""
|
||||
handle = module.VixDiskLibHandle(
|
||||
vixdisklib_compatibility_version="8.0",
|
||||
@@ -47,7 +51,7 @@ def _read_sectors(
|
||||
result: dict[int, bytes] = {}
|
||||
kwargs = lab.vixdisklib_connect_kwargs(_connect_extra(lab, module))
|
||||
with handle.connect(**kwargs) as conn:
|
||||
with handle.open(conn, lab.disk_path, flags=0) as disk:
|
||||
with handle.open(conn, lab.disk_path, flags=flags) as disk:
|
||||
for start in sectors:
|
||||
buf[:SECTOR_SIZE] = b"\xa5" * SECTOR_SIZE
|
||||
handle.read(disk, start, 1, buf)
|
||||
@@ -58,9 +62,10 @@ def _read_sectors(
|
||||
def _assert_both_read(
|
||||
lab: LabEnv,
|
||||
sectors: tuple[int, ...],
|
||||
expected: dict[int, bytes]) -> None:
|
||||
vddk_data = _read_sectors(lab, vixdisklib, sectors)
|
||||
replacement = _read_sectors(lab, open_vix, sectors)
|
||||
expected: dict[int, bytes],
|
||||
flags: int = 0) -> None:
|
||||
vddk_data = _read_sectors(lab, vixdisklib, sectors, flags=flags)
|
||||
replacement = _read_sectors(lab, open_vix, sectors, flags=flags)
|
||||
for start in sectors:
|
||||
assert vddk_data[start] == expected[start], (
|
||||
f"VDDK mismatch at sector {start}")
|
||||
@@ -69,8 +74,12 @@ def _assert_both_read(
|
||||
|
||||
|
||||
class TestCrosscheck:
|
||||
@pytest.mark.parametrize(
|
||||
"open_flags",
|
||||
[0, vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ],
|
||||
ids=["plain", "fastlz"])
|
||||
def test_openvixdisklib_matches_vddk_sectors(
|
||||
self, lab: LabEnv, vddk: None) -> None:
|
||||
self, lab: LabEnv, vddk: None, open_flags: int) -> None:
|
||||
"""Writes from either library must be visible to both readers."""
|
||||
sectors = (0, 1, SECTOR_AT_1GB)
|
||||
vddk_payloads = {
|
||||
@@ -78,13 +87,13 @@ class TestCrosscheck:
|
||||
1: pattern_bytes(SECTOR_SIZE, b"XCHK-VDDK-S1"),
|
||||
SECTOR_AT_1GB: pattern_bytes(SECTOR_SIZE, b"XCHK-VDDK-1G"),
|
||||
}
|
||||
_write_sectors(lab, vixdisklib, vddk_payloads)
|
||||
_assert_both_read(lab, sectors, vddk_payloads)
|
||||
_write_sectors(lab, vixdisklib, vddk_payloads, flags=open_flags)
|
||||
_assert_both_read(lab, sectors, vddk_payloads, flags=open_flags)
|
||||
|
||||
ovdl_payloads = {
|
||||
0: pattern_bytes(SECTOR_SIZE, b"XCHK-OVDL-S0"),
|
||||
1: pattern_bytes(SECTOR_SIZE, b"XCHK-OVDL-S1"),
|
||||
SECTOR_AT_1GB: pattern_bytes(SECTOR_SIZE, b"XCHK-OVDL-1G"),
|
||||
}
|
||||
_write_sectors(lab, open_vix, ovdl_payloads)
|
||||
_assert_both_read(lab, sectors, ovdl_payloads)
|
||||
_write_sectors(lab, open_vix, ovdl_payloads, flags=open_flags)
|
||||
_assert_both_read(lab, sectors, ovdl_payloads, flags=open_flags)
|
||||
|
||||
@@ -12,13 +12,18 @@ from tests.integration.base import LabEnv, SECTOR_SIZE, pattern_bytes
|
||||
class TestNfcOpen:
|
||||
@pytest.mark.parametrize(
|
||||
"nfc_ssl", [True, False], ids=["nbdssl", "nbd"])
|
||||
@pytest.mark.parametrize(
|
||||
"compression",
|
||||
[nfc_open.NFC_COMPRESSION_NONE, nfc_open.NFC_COMPRESSION_FASTLZ],
|
||||
ids=["plain", "fastlz"])
|
||||
def test_open_disk_and_read_first_sector(
|
||||
self, lab: LabEnv, nfc_ssl: bool) -> None:
|
||||
self, lab: LabEnv, nfc_ssl: bool, compression: int) -> None:
|
||||
"""Open the temp VMDK, write sector 0, and read it back."""
|
||||
expected = pattern_bytes(SECTOR_SIZE, b"NFC-OPEN-S0")
|
||||
with lab.authenticate(read_only=False, nfc_ssl=nfc_ssl) as session:
|
||||
with nfc_open.open_disk(
|
||||
session, lab.disk_path, read_only=False) as disk:
|
||||
session, lab.disk_path, read_only=False,
|
||||
compression=compression) as disk:
|
||||
assert disk.path == lab.disk_path
|
||||
assert disk.handle > 0
|
||||
assert disk.sector_size == SECTOR_SIZE
|
||||
|
||||
@@ -5,6 +5,8 @@
|
||||
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
from openvixdisklib import nfc_open
|
||||
from tests.integration.base import LabEnv, SECTOR_SIZE, pattern_bytes
|
||||
|
||||
@@ -12,7 +14,12 @@ _32MIB = 32 * 1024 * 1024
|
||||
|
||||
|
||||
class TestNfcReadWrite:
|
||||
def test_sector_writes_and_reads(self, lab: LabEnv) -> None:
|
||||
@pytest.mark.parametrize(
|
||||
"compression",
|
||||
[nfc_open.NFC_COMPRESSION_NONE, nfc_open.NFC_COMPRESSION_FASTLZ],
|
||||
ids=["plain", "fastlz"])
|
||||
def test_sector_writes_and_reads(
|
||||
self, lab: LabEnv, compression: int) -> None:
|
||||
"""Write known patterns and read them back at several ranges."""
|
||||
ranges = [
|
||||
(0, 1),
|
||||
@@ -25,7 +32,8 @@ class TestNfcReadWrite:
|
||||
]
|
||||
with lab.authenticate(read_only=False) as session:
|
||||
with nfc_open.open_disk(
|
||||
session, lab.disk_path, read_only=False) as disk:
|
||||
session, lab.disk_path, read_only=False,
|
||||
compression=compression) as disk:
|
||||
for start, n_sectors in ranges:
|
||||
length = n_sectors * SECTOR_SIZE
|
||||
seed = f"NFC-R{start}:{n_sectors}-".encode()
|
||||
@@ -55,13 +63,19 @@ class TestNfcReadWrite:
|
||||
big_got[SECTOR_SIZE:2 * SECTOR_SIZE]
|
||||
== big_to_write[SECTOR_SIZE:2 * SECTOR_SIZE])
|
||||
|
||||
def test_write_and_read_32mb(self, lab: LabEnv) -> None:
|
||||
@pytest.mark.parametrize(
|
||||
"compression",
|
||||
[nfc_open.NFC_COMPRESSION_NONE, nfc_open.NFC_COMPRESSION_FASTLZ],
|
||||
ids=["plain", "fastlz"])
|
||||
def test_write_and_read_32mb(
|
||||
self, lab: LabEnv, compression: int) -> None:
|
||||
"""Write 32 MiB (512 AIO chunks) and read it back in one request."""
|
||||
n_sectors = _32MIB // SECTOR_SIZE
|
||||
to_write = os.urandom(_32MIB)
|
||||
with lab.authenticate(read_only=False) as session:
|
||||
with nfc_open.open_disk(
|
||||
session, lab.disk_path, read_only=False) as disk:
|
||||
session, lab.disk_path, read_only=False,
|
||||
compression=compression) as disk:
|
||||
disk.write(0, n_sectors, to_write)
|
||||
got = disk.read(0, n_sectors)
|
||||
assert got is not to_write
|
||||
|
||||
@@ -12,8 +12,12 @@ from tests.integration.base import (
|
||||
|
||||
class TestOpenvixdisklib:
|
||||
@pytest.mark.parametrize("transport_mode", ["nbdssl", "nbd"])
|
||||
@pytest.mark.parametrize(
|
||||
"open_flags",
|
||||
[0, vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ],
|
||||
ids=["plain", "fastlz"])
|
||||
def test_write_and_read_sector_zero_and_one_gib(
|
||||
self, lab: LabEnv, transport_mode: str) -> None:
|
||||
self, lab: LabEnv, transport_mode: str, open_flags: int) -> None:
|
||||
"""Write then read sector 0 and the sector at a 1 GiB offset."""
|
||||
handle = vixdisklib.VixDiskLibHandle(
|
||||
vixdisklib_compatibility_version="8.0",
|
||||
@@ -30,7 +34,7 @@ class TestOpenvixdisklib:
|
||||
}
|
||||
assert handle.get_transport_modes() == ["nbdssl", "nbd"]
|
||||
with handle.connect(**connect_kwargs) as conn:
|
||||
with handle.open(conn, lab.disk_path, flags=0) as disk:
|
||||
with handle.open(conn, lab.disk_path, flags=open_flags) as disk:
|
||||
assert handle.get_transport_mode(disk) == transport_mode
|
||||
for start, expected in patterns.items():
|
||||
write_buf[:SECTOR_SIZE] = expected
|
||||
|
||||
+26
-16
@@ -29,7 +29,8 @@ def _connect_extra(lab: LabEnv, module: Any) -> Optional[dict[str, Any]]:
|
||||
def _time_write_read(
|
||||
lab: LabEnv,
|
||||
module: Any,
|
||||
payload: bytes) -> tuple[float, float]:
|
||||
payload: bytes,
|
||||
flags: int = 0) -> tuple[float, float]:
|
||||
"""Write ``payload`` at sector 0, read it back, and return durations."""
|
||||
n_sectors = len(payload) // SECTOR_SIZE
|
||||
handle = module.VixDiskLibHandle(
|
||||
@@ -40,7 +41,7 @@ def _time_write_read(
|
||||
write_buf[:len(payload)] = payload
|
||||
kwargs = lab.vixdisklib_connect_kwargs(_connect_extra(lab, module))
|
||||
with handle.connect(**kwargs) as conn:
|
||||
with handle.open(conn, lab.disk_path, flags=0) as disk:
|
||||
with handle.open(conn, lab.disk_path, flags=flags) as disk:
|
||||
started = time.perf_counter()
|
||||
handle.write(disk, 0, n_sectors, write_buf)
|
||||
write_s = time.perf_counter() - started
|
||||
@@ -65,24 +66,33 @@ class TestCompare:
|
||||
("vddk", vixdisklib),
|
||||
("openvixdisklib", open_vix),
|
||||
)
|
||||
rows: list[tuple[str, str, float, float, float, float]] = []
|
||||
open_modes = (
|
||||
("plain", 0),
|
||||
("fastlz", vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ),
|
||||
)
|
||||
rows: list[tuple[str, str, str, float, float, float, float]] = []
|
||||
for label, nbytes in _SIZES:
|
||||
payload = pattern_bytes(nbytes, f"PERF-{label}-".encode())
|
||||
for name, module in libraries:
|
||||
write_s, read_s = _time_write_read(lab, module, payload)
|
||||
rows.append((
|
||||
label,
|
||||
name,
|
||||
write_s,
|
||||
read_s,
|
||||
_mib_per_s(nbytes, write_s),
|
||||
_mib_per_s(nbytes, read_s),
|
||||
))
|
||||
for mode_name, flags in open_modes:
|
||||
for name, module in libraries:
|
||||
write_s, read_s = _time_write_read(
|
||||
lab, module, payload, flags=flags)
|
||||
rows.append((
|
||||
label,
|
||||
mode_name,
|
||||
name,
|
||||
write_s,
|
||||
read_s,
|
||||
_mib_per_s(nbytes, write_s),
|
||||
_mib_per_s(nbytes, read_s),
|
||||
))
|
||||
print()
|
||||
print(
|
||||
f"{'size':<14} {'library':<16} {'write_s':>10} {'read_s':>10} "
|
||||
f"{'size':<14} {'flags':<8} {'library':<16} "
|
||||
f"{'write_s':>10} {'read_s':>10} "
|
||||
f"{'write_MiB/s':>12} {'read_MiB/s':>12}")
|
||||
for label, name, write_s, read_s, write_r, read_r in rows:
|
||||
for label, mode_name, name, write_s, read_s, write_r, read_r in rows:
|
||||
print(
|
||||
f"{label:<14} {name:<16} {write_s:10.3f} {read_s:10.3f} "
|
||||
f"{label:<14} {mode_name:<8} {name:<16} "
|
||||
f"{write_s:10.3f} {read_s:10.3f} "
|
||||
f"{write_r:12.1f} {read_r:12.1f}")
|
||||
|
||||
Reference in New Issue
Block a user