# rypipe Documentation

> Format-agnostic columnar engine. Parse row-oriented byte streams into Apache Arrow record batches with a Rust core, Python bindings, parallel scheduling, memory-bounded execution, and query pushdown.

## Overview
- [rypipe](https://rypipe.emiliano-go.com/): Format-agnostic columnar engine that turns byte streams into Arrow record batches

## API
- [Python API](https://rypipe.emiliano-go.com/python-api/): `Adapter`/`Source` pipeline API, stages, sinks, and the `rypipe.read` registry
- [Rust Core](https://rypipe.emiliano-go.com/rust-api/): Use the pure-Rust `rypipe-core` crate directly

## Extending
- [Writing Adapters](https://rypipe.emiliano-go.com/writing-adapters/): Add a format by implementing `Splitter`/`RecordParser` or subclassing `rypipe.Adapter`

## Examples
- [Examples](https://rypipe.emiliano-go.com/examples/): Python and Rust API recipes for reading, pipelines, adapters, and sinks
- [crxml adapter](https://rypipe.emiliano-go.com/crxml-adapter/): Real-world adapter example; 3.0 GB/s Crystal Reports XML parser built on rypipe-core (714 MB/s single, 2.6–3.0 GB/s parallel, 508 MB/s streaming)

## Advanced
- [Advanced overview](https://rypipe.emiliano-go.com/advanced/): Roadmap and checklist for the advanced section
- [Pushdown fusion](https://rypipe.emiliano-go.com/advanced/fusion/): How RenameFields/DropFields/CastTypes/FilterRows collapse into an ExecutionPlan
- [Execution modes](https://rypipe.emiliano-go.com/advanced/execution-modes/): stream, columnar, parallel, and auto engine selection
- [Memory and chunking](https://rypipe.emiliano-go.com/advanced/memory-and-chunking/): Memory budgets, BoundedExecutor, and sizing chunks
- [Parallelism](https://rypipe.emiliano-go.com/advanced/parallelism/): rayon internals, num_chunks, and measuring speedup
- [Dictionary encoding](https://rypipe.emiliano-go.com/advanced/dictionary-encoding/): Arrow dictionaries, auto_dict, and the merge path
- [Schema and types](https://rypipe.emiliano-go.com/advanced/schema-and-types/): schema_order, field_types, and numeric compare filters
- [I/O tuning](https://rypipe.emiliano-go.com/advanced/io-tuning/): mmap vs buffered reads, prefault, and storage classes
- [Adapter design](https://rypipe.emiliano-go.com/advanced/adapter-design/): Splitter, RecordParser, memchr, and sink.wants
- [Profiling](https://rypipe.emiliano-go.com/advanced/profiling/): perf, flamegraph, bench_throughput, and RSS measurement
- [Anti-patterns](https://rypipe.emiliano-go.com/advanced/anti-patterns/): Common mistakes that hurt throughput and memory
- [Case study: crxml](https://rypipe.emiliano-go.com/advanced/case-study-crxml/): How crxml reaches 3.0 GB/s (scanner + row_dirty + streaming RowSink)

## Internals
- [Architecture](https://rypipe.emiliano-go.com/architecture/): Splitting, row extraction, typed builders, and engine design
- [Performance](https://rypipe.emiliano-go.com/performance/): Parallel scheduling, memory budgets, and optimization notes

For the full documentation in a single file, see [llms-full.txt](https://rypipe.emiliano-go.com/llms-full.txt).