[Copilot]

Add direct Parquet loading into one NumPy array per column, accepting filesystem paths or seekable binary files and optional column-name selection.

• API: parquet_to_arrays(file, *, columns_select=None) returns immutable 1D arrays. Selection preserves requested order and decodes only selected top-level columns.
• Execution: Calls PyArrow’s compiled reader and NumPy conversion methods from C, bypassing Python’s read_table/dataset layer. PyArrow remains required; this is not a standalone decoder.
• Compatibility: Keeps PyArrow optional through arraykit[parquet] and lazy imports. Caller-owned files remain open.
• Conversion: Uses Arrow’s dtype/null semantics; supports dictionary and nested columns. Rejects missing, ambiguous, or duplicate selections.

from arraykit import parquet_to_arrays

prices, volumes = parquet_to_arrays(
    "trades.parquet",
    columns_select=["price", "volume"],
)

• Fixes Go directly form parquet files to a NumPy array per column #229

[flexatone]

@copilot : can this be done without depending on the Python PyArrow library at all, but by directly reding bytes from the file?

[Copilot]

@copilot : can this be done without depending on the Python PyArrow library at all, but by directly reding bytes from the file?

Yes. The current implementation is a C wrapper around PyArrow, not an independent decoder.

We could remove PyArrow by linking directly to the native Arrow/Parquet C++ libraries, or implement a reader in C that decodes file bytes into NumPy-owned buffers. The latter must handle Parquet metadata, page encodings, compression, nulls, and logical types—it is substantially more than reading raw column bytes.

For direct-to-NumPy decoding without Arrow intermediates, a native C reader is the better fit, but it is a larger replacement and would need compatibility tests and benchmarks; outperforming PyArrow is not guaranteed. Would you prefer a standalone C implementation, or are native C/C++ library dependencies acceptable?

[flexatone]

@copilot : lets see if we can link directly to the native Arrow/Parquet C++ libraries

[flexatone]

@copilot : are you able to resume work on this?

[Copilot]

@copilot : are you able to resume work on this?

Resumed and completed in e2a2e18. The reader now links directly to the native Arrow/Parquet C++ libraries, with no Python PyArrow or arrow_python dependency. Paths, open files, column projection, and immutable NumPy results are preserved.

Enable it when building with ARRAYKIT_WITH_PARQUET=1; native library setup is documented in the README. Default builds remain NumPy-only.

Validation: 799 tests passed, including reads with every PyArrow import blocked; native linkage excludes arrow_python. CodeQL and secret scans are clean. The local benchmark did not show a speedup over Python PyArrow, so no performance advantage is claimed.