How to choose an in-process analytics engine

An in-process, or embedded, analytics engine runs as a library inside the application, script or notebook that calls it — there is no server to install, no port to open, no account to create. That single fact explains almost everything else about this category: no multi-user concurrency model to configure, no cluster to size, and no operations team required to start using one. It also explains the limits: these engines assume one process, usually one machine, and usually one primary user at a time. If your requirement is many people querying the same shared, durable dataset concurrently, you want a server-based database or warehouse, not this category.

Who needs one: analysts and engineers doing local exploration on files, data scientists replacing slow single-threaded Python with something faster, and developers who need to embed real SQL or DataFrame capability directly inside an application or CLI tool. Who does not: teams that need one dataset shared live across a company, or workloads that are fundamentally transactional with many concurrent writers — that is a job for a conventional operational database.

Row-oriented versus columnar changes what "fast" means

SQLite sits slightly apart from the rest of this category. It is row-oriented and built for transactional workloads — lots of small reads and writes to individual records — which is a different job from analytics. It belongs in the same conversation only because the newer generation of embedded analytical engines explicitly copies its model: one library, one file, no server. DuckDB, chDB, clickhouse-local, Apache DataFusion and Polars apply that same embedded, serverless idea to columnar, vectorized execution instead — scanning and aggregating millions of rows fast rather than looking up or writing one row at a time. If your workload is "find and update this one record," you want SQLite. If it is "aggregate this whole file," you want one of the columnar engines.

Which language and API you already use

The fastest way to shortlist within the columnar group is to start from what your team already writes:

• Python and pandas users wanting more speed and less memory pressure should look first at Polars — a ready-to-use DataFrame library with a lazy query planner, or DuckDB, which adds a full SQL interface over the same kind of local files.
• Teams who specifically want ClickHouse's SQL dialect and function library, without running a ClickHouse cluster, should look at chDB (embedded directly in a Python process) or clickhouse-local (a single binary for shell pipelines and one-off jobs).
• Engineers building a new database, DataFrame library or streaming engine from scratch should look at Apache DataFusion — it is a toolkit rather than a finished product, providing the SQL parser, planner and vectorized executor that other projects embed rather than a tool an analyst opens directly.

None of this is exclusive. It is common to see Polars or DuckDB used for exploration and DataFusion used inside a purpose-built internal tool, on the same team.

Apache Arrow is the format, not the engine

Apache Arrow does not answer queries itself. It defines the columnar in-memory layout that most of the engines above share, so passing a dataset between two of them — pandas to Polars to DuckDB, for instance — happens without a serialize-and-copy step. Its relevance to a buying decision is indirect: if your team is stitching several of these tools together in one pipeline, favor engines that already speak Arrow natively (all of the ones in this guide do) so zero-copy data sharing actually holds in practice, rather than silently falling back to a slower conversion.

What each engine reads, and what it does not manage for you

All of these engines can query Parquet, CSV and JSON directly, without a separate load step, which is most of their appeal over a traditional database. Where they differ is in file-format and storage breadth: DuckDB and Polars both read Iceberg and Delta tables; chDB and clickhouse-local inherit ClickHouse's broad format support (Parquet, CSV, JSON, ORC, Arrow and more); DataFusion is a library, so its supported formats depend on which table providers the embedding application wires in. None of them manage concurrent writers, replication or backup for you — that responsibility sits with whatever application embeds them, which is fine for a notebook and a real constraint for a production service.

When you outgrow a single process

The moment two people, or two services, need to see the same up-to-date data at once, "in-process" stops being enough. MotherDuck is the direct answer for teams already using DuckDB: it runs the same engine in the cloud with persistent, shareable storage and a hybrid mode where small queries still execute locally while larger ones scale out, so the transition doesn't mean re-learning a new SQL dialect. Teams that have outgrown clickhouse-local's single-file model have a similar path into full ClickHouse or ClickHouse Cloud, covered in the real-time OLAP database guide — same SQL dialect, now on a server built for concurrent access.

Questions to ask before committing

1. Does it read the file formats and remote storage (S3, Iceberg, Delta) you actually have, or only the ones in the demo?
2. What happens with a dataset larger than available memory — does it spill to disk or stream, or does it simply fail?
3. What is the realistic migration path once a second user or service needs the same data?
4. Does it have first-class bindings for the language your team writes in daily, not just a community wrapper?
5. For the newer projects here (chDB, DataFusion as an end-user tool), how active is development, and what platforms are actually supported today?

Common mistakes

Treating an in-process engine as a drop-in replacement for a shared warehouse is the most common one — it works in a demo with one user and breaks down the day a second person needs live access to the same numbers. A close second is assuming "embedded" means no operational cost at all: these engines still need memory and disk capacity planning, and someone still owns the files they read from. The third is choosing by a raw speed benchmark rather than matching the tool to the actual shape of the workload — a transactional access pattern will not get faster by switching to a columnar engine built for aggregation, and vice versa.

Where to go next

Two comparisons look at the closest pairings in this category: chDB vs SQLite, for teams deciding between an analytical and a transactional embedded engine, and Apache DataFusion vs Polars, for teams choosing between a toolkit and a ready-to-use DataFrame library. See every tool in this category for the full list.