Add fastlz compression support
This commit is contained in:
Lucian Petrut
+3
-1
@@ -31,6 +31,7 @@ sectors. Flags `0x1a` require the writable ticket; the same flags on a
|
||||
| `NfcUtil_PrintFileInfoOpenFlag` `NFC_DISK` `0x1e` | `NFC_AIO_MSG_OPEN_FILE` (read-only) |
|
||||
| Open without `VIXDISKLIB_FLAG_OPEN_READ_ONLY` | `OPEN_FILE` flags `0x1a` (read-write) |
|
||||
| `VixDiskLib_Read` / `VixDiskLib_Write` | `NFC_AIO_MSG_IO` + sector bytes |
|
||||
| `VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` | IO opcode high bits `2`; extra data is FastLZ |
|
||||
|
||||
`snapshot_ref` is still not on the wire. Integration tests pass the
|
||||
flat VMDK created with the temporary lab VM.
|
||||
@@ -208,6 +209,7 @@ classic type 4 `NFC_SESSION_COMPLETE`.
|
||||
| VIM + authd | `openvixdisklib.nfc_auth.authenticate` |
|
||||
| Dup fd, skip TLS for NFC | `openvixdisklib.nfc_open.takeover_authd_socket` |
|
||||
| Second TLS for nbdssl | `openvixdisklib.nfc_open.wrap_nfcssl_socket` |
|
||||
| FastLZ for NBD compression | `openvixdisklib.fastlz` |
|
||||
| Handshake + AIO + OPEN_FILE | `openvixdisklib.nfc_open.open_disk` |
|
||||
| Sector read / write / close | `openvixdisklib.nfc_open.NfcDisk` |
|
||||
|
||||
@@ -224,7 +226,7 @@ I/O: `docs/nfc_read.md`, `docs/nfc_write.md`, and
|
||||
|
||||
## What is still VDDK-only
|
||||
|
||||
- `DDB_GET` / geometry / compression / encryption keys
|
||||
- `DDB_GET` / geometry / zlib and skipz compression / encryption keys
|
||||
- `NFC_DELTA_DISK`, change-block tracking
|
||||
- Host-switch (`NFC_AIO_SWITCH_HOST_*`)
|
||||
- Direct ESXi `ha-nfc` without vCenter `vpxa-nfc`
|
||||
|
||||
+29
-10
@@ -39,21 +39,40 @@ LBA 8 on this disk was 4096 zero bytes on the wire, not a skip.
|
||||
Little-endian, after the usual 16-byte AIO header
|
||||
(`magic 0xA100DA7A`, type 7, size 44, monotonic `opId`):
|
||||
|
||||
| Offset | Type | VDDK `Read(start, n)` |
|
||||
| ------ | -------- | ---------------------------------------------- |
|
||||
| 0 | `uint64` | File handle from `OPEN_FILE` |
|
||||
| 8 | `uint64` | `1` (`NFC_AIO_IO_READ`; write uses `0`) |
|
||||
| 16 | `uint64` | Byte offset |
|
||||
| 24 | `uint64` | Byte length |
|
||||
| 32 | `uint32` | Byte length (same value) |
|
||||
| 36 | `uint32` | Byte length (same value) |
|
||||
| 40 | `uint32` | `0` (flags; uncompressed in this capture) |
|
||||
| Offset | Type | VDDK `Read(start, n)` |
|
||||
| ------ | -------- | ---------------------------------------------------------- |
|
||||
| 0 | `uint64` | File handle from `OPEN_FILE` |
|
||||
| 8 | `uint64` | Direction in low 32 bits; FastLZ type `2` in high 32 bits |
|
||||
| 16 | `uint64` | Byte offset |
|
||||
| 24 | `uint64` | Byte length |
|
||||
| 32 | `uint32` | Byte length (same value) |
|
||||
| 36 | `uint32` | Byte length, or compressed extra size when type is FastLZ |
|
||||
| 40 | `uint32` | `0` |
|
||||
|
||||
An earlier guess that offset 36 was `NFC_DISK` (`2`) was wrong: a
|
||||
1-sector VDDK read puts `512` in both `uint32` length fields. A Python
|
||||
read that sent `(512, 2, 0)` still worked for one sector; the
|
||||
replacement now matches VDDK.
|
||||
|
||||
`VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` does not change OPEN_FILE
|
||||
flags. The IO opcode at offset 8 is a `uint64`: low 32 bits are still
|
||||
`0`/`1` (write/read), high 32 bits are the NFC compression type
|
||||
(`2` = FastLZ). OPEN still uses handshake `PlainText`.
|
||||
|
||||
| Open flag / wire | Request extra | Reply extra |
|
||||
| ---------------------------------------- | ------------------------------------- | ------------------------------------------------ |
|
||||
| No compression flag | Raw `length` bytes on write | Raw fragment at offset 32 |
|
||||
| FASTLZ, data that shrinks | FastLZ bytes; offset 36 = packed size | Opcode type `2`; extra is FastLZ of offset 32 |
|
||||
| FASTLZ, incompressible | Raw bytes; opcode type `0` | Opcode type `0`; extra is raw |
|
||||
|
||||
Reads with FASTLZ always *request* type `2`. The server may answer type
|
||||
`2` or fall back to type `0`. Decompress into the uncompressed fragment
|
||||
length at offset 32 and copy to the dest at offset 28.
|
||||
|
||||
64 KiB chunks use FastLZ level 2 (first byte has bit 5 set). Smaller
|
||||
chunks use level 1. VDDK’s URL form is `FASTLZ-vpxa-nfc://…`; authd
|
||||
`PROXY` is unchanged.
|
||||
|
||||
## Reply
|
||||
|
||||
Each fragment is: 16-byte AIO header (same `type` and `opId`) + 44-byte
|
||||
@@ -117,7 +136,7 @@ fragments).
|
||||
|
||||
## What is still VDDK-only
|
||||
|
||||
- Compression flags on the last `uint32`
|
||||
- zlib and skipz NBD compression flags
|
||||
- `VixDiskLib_ReadAsync` (same IO messages, different client threading)
|
||||
- `VixDiskLib_QueryAllocatedBlocks` / allocation bitmaps
|
||||
- `VixDiskLib_GetInfo` capacity (not required to read a known range)
|
||||
|
||||
@@ -60,6 +60,11 @@ Little-endian, after the usual 16-byte AIO header
|
||||
| 36 | `uint32` | Byte length (same value) |
|
||||
| 40 | `uint32` | `0` |
|
||||
|
||||
FASTLZ writes use the same 44-byte header. The opcode `uint64` high
|
||||
half is `2`, offset 36 is the compressed size, and FastLZ bytes follow
|
||||
instead of raw sectors. If compression does not shrink the chunk, VDDK
|
||||
sends type `0` and raw extra (same as an uncompressed write).
|
||||
|
||||
Sector bytes follow the 44-byte payload and are **not** counted in AIO
|
||||
`size`. VDDK sends header + payload + data in one `write()`. The
|
||||
replacement may split that into two `sendall`s; TCP does not care.
|
||||
|
||||
@@ -280,6 +280,24 @@ the old `VixDiskLibConnectParams` ctypes struct can still log nbdssl
|
||||
and then fall back to `vpxa-nfc` / `useSSL=0`. Do not treat that log
|
||||
line as a wire capture of NFCSSL.
|
||||
|
||||
## Step 12 — FASTLZ NBD compression
|
||||
|
||||
`VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` (`1 << 5`) is an IO codec,
|
||||
not an OPEN_FILE bit. Capture VDDK with that flag (NBD + the port-902
|
||||
`write`/`read` hook):
|
||||
|
||||
- Handshake stays `PlainText`. `OPEN_FILE` flags stay `0x1a` / `0x1e`.
|
||||
- VDDK’s URL is `FASTLZ-vpxa-nfc://…`; `PROXY` is still `vpxa-nfc`.
|
||||
- IO opcode `uint64` = direction in the low half, compression type in
|
||||
the high half (`2` = FastLZ). Offset 32 is uncompressed length;
|
||||
offset 36 is compressed extra size when type is 2.
|
||||
- Incompressible chunks fall back to type `0` and raw extra.
|
||||
- 64 KiB chunks use FastLZ level 2; smaller chunks use level 1.
|
||||
|
||||
Replay: `openvixdisklib/fastlz.py` plus `NfcDisk` compression on each
|
||||
IO. Proof: `tests/integration/test_nfc_read_write.py` (`fastlz`) and
|
||||
`tests/perf/test_compare.py`.
|
||||
|
||||
## What to write down
|
||||
|
||||
After a stage works:
|
||||
@@ -300,7 +318,7 @@ replacement library.
|
||||
|
||||
Not yet reversed, same loop as above:
|
||||
|
||||
- `DDB_GET` / disk geometry, compression, encrypted disks
|
||||
- `DDB_GET` / disk geometry, zlib/skipz compression, encrypted disks
|
||||
- `NFC_DELTA_DISK`, CBT / `QueryAllocatedBlocks`
|
||||
- `VixDiskLib_GetInfo` capacity
|
||||
- Host-switch AIO messages
|
||||
|
||||
Reference in New Issue
Block a user