DuckDB Core Architecture & Internals

Overview

DuckDB is a high-performance analytical database system designed for speed, reliability, portability, and ease of use. It provides a rich SQL dialect with support for advanced features like window functions, complex types (arrays, structs, maps), and nested correlated subqueries. The system is available as a standalone CLI and through clients for Python, R, Java, Wasm, and other languages.

Architecture Overview

DuckDB follows a modular pipeline architecture that transforms SQL queries into optimized execution plans. The system is organized into distinct layers, each responsible for a specific phase of query processing:

Loading diagram...

Core Components

Parser transforms SQL strings into an abstract syntax tree (AST). DuckDB uses PostgreSQL's parser (libpg_query) and converts tokens into a custom representation based on SQLStatements, Expressions, and TableRefs.

Planner converts the AST into a Logical Query Plan represented as a tree of LogicalOperator nodes. The Binder resolves symbols (table names, columns) using the Catalog to ensure all references are valid.

Optimizer takes the logical plan and applies both cost-based and rule-based optimizations including predicate pushdown, expression rewriting, and join ordering. The result is a logically equivalent but faster logical plan.

Executor converts the optimized logical plan into a Physical Query Plan consisting of PhysicalOperators. These operators are executed using a push-based execution model with support for parallelization.

Catalog maintains metadata about tables, schemas, and functions. It is consulted during planning to resolve symbols and validate query structure.

Storage manages physical data both in memory and on disk. It handles base table scans, data updates (INSERT, UPDATE), and persistence.

Transaction Manager handles ACID compliance, managing open transactions and processing COMMIT/ROLLBACK commands.

Query Execution Flow

1. User submits a SQL query through a Connection
2. Parser tokenizes and validates the SQL syntax
3. Planner creates a logical query plan with the Binder resolving all symbols
4. Optimizer applies transformations to improve performance
5. Executor generates a physical plan and executes it
6. Storage layer retrieves or updates data as needed
7. Results are returned to the user

Key Classes

DuckDB is the main database object that holds the catalog and database-specific metadata. DatabaseInstance manages the actual database state including the buffer manager, storage, and transaction management.

Connection represents a client connection to the database and provides methods like Query() and SendQuery() to execute SQL statements.

ClientContext holds session-specific information during query execution, including transaction state, configuration, and client data.

QueryResult represents the output of a query execution, available as either StreamQueryResult (for streaming results) or MaterializedQueryResult (for fully materialized results).

Architecture & Query Processing Pipeline

DuckDB's query processing follows a modular pipeline architecture that transforms SQL queries into optimized execution plans. Each stage is responsible for a specific transformation, enabling clean separation of concerns and extensibility.

Query Processing Pipeline

Loading diagram...

Stage 1: Parsing

The Parser transforms SQL strings into an abstract syntax tree (AST). DuckDB leverages PostgreSQL's parser (libpg_query) for tokenization, then converts tokens into a custom representation using SQLStatements, Expressions, and TableRefs. This approach provides robust SQL compatibility while maintaining DuckDB's extensibility.

Stage 2: Planning & Binding

The Planner converts the AST into a Logical Query Plan represented as a tree of LogicalOperator nodes. The Binder resolves symbols (table names, columns, functions) using the Catalog to ensure all references are valid. This stage also performs type inference and validates query structure.

Stage 3: Optimization

The Optimizer applies both cost-based and rule-based transformations to improve performance:

• Expression Rewriting - Simplifies expressions through constant folding, algebraic simplification, and rule-based transformations
• Predicate Pushdown - Moves filter conditions closer to data sources to reduce intermediate data
• Join Ordering - Uses dynamic programming to find optimal join orders based on cardinality estimates
• Filter Pullup - Elevates filters above operators when beneficial
• CTE Inlining - Materializes or inlines common table expressions based on cost

The result is a logically equivalent but faster logical plan.

Stage 4: Physical Planning

The PhysicalPlanGenerator converts the optimized logical plan into a Physical Query Plan consisting of PhysicalOperators. This stage:

1. Resolves operator types and cardinality estimates
2. Binds column references to physical positions
3. Creates the physical operator tree

Physical operators include scans (TABLE_SCAN, CTE_SCAN), joins (HASH_JOIN, NESTED_LOOP_JOIN), aggregations (HASH_GROUP_BY), and sinks (result collection).

Stage 5: Execution

The Executor executes the physical plan using a push-based execution model with parallelization support. Key concepts:

• Pipelines - Chains of operators that process data together. A pipeline consists of a source (scan), intermediate operators (filters, projections), and a sink (aggregation, result collection)
• MetaPipelines - Groups of pipelines with dependencies, scheduled as atomic units
• Parallel Execution - Multiple threads process different chunks of data concurrently through the same pipeline
• Events - Synchronization points between pipelines (initialization, completion)

Data flows through operators as DataChunk objects (columnar batches), enabling vectorized processing and cache efficiency.

Key Integration Points

The Catalog maintains metadata about tables, schemas, and functions, consulted during planning for symbol resolution. The Storage layer handles physical data access and persistence. The Transaction Manager ensures ACID compliance throughout execution.

Core Data Structures & Vectorization

DuckDB uses a columnar vectorized execution model where data flows through the system in batches called DataChunks, each containing multiple Vectors. This design enables efficient SIMD processing and cache utilization.

Vector Types

Vectors represent columns of data with different physical storage formats:

• FLAT_VECTOR - Standard uncompressed data, stored contiguously in memory
• CONSTANT_VECTOR - Single repeated value (memory-efficient for constants)
• DICTIONARY_VECTOR - Compressed representation using a selection vector pointing to a base vector
• FSST_VECTOR - String data compressed using FSST (Fast Static Symbol Table) compression
• SEQUENCE_VECTOR - Arithmetic sequences defined by start, increment, and count

Each vector maintains:

• data - Pointer to the actual values
• validity - Bitset mask tracking NULL values (1 bit per row)
• buffer - Main memory buffer holding the data
• auxiliary - Secondary buffer for nested types or string heaps

UnifiedVectorFormat: The Vectorization Abstraction

The UnifiedVectorFormat struct is DuckDB's key abstraction for efficient vectorized operations. It normalizes different vector types into a canonical format:

struct UnifiedVectorFormat {
    const SelectionVector *sel;      // Indirection indices
    data_ptr_t data;                 // Pointer to actual data
    ValidityMask validity;           // NULL tracking bitset
    SelectionVector owned_sel;       // Owned selection vector
    PhysicalType physical_type;      // Data type
};

Key insight: Most vector types (flat, constant, dictionary) convert to UnifiedVectorFormat with zero-copy overhead. Operations access elements via data[sel[i]] with validity checked at validity[sel[i]].

DataChunk: The Execution Unit

A DataChunk is a columnar batch containing multiple vectors of the same row count:

class DataChunk {
    vector<Vector> data;        // Columns
    idx_t count;                // Current row count
    idx_t capacity;             // Max capacity (typically 2048)
    vector<VectorCache> vector_caches;  // Reusable buffers
};

DataChunks flow through the execution engine, with operators transforming them. The capacity is typically STANDARD_VECTOR_SIZE (2048 rows), balancing memory efficiency with cache locality.

Vectorization Benefits

Loading diagram...

The vectorized approach enables:

• SIMD parallelism - Process multiple values per CPU cycle
• Cache efficiency - Columnar layout improves cache hit rates
• Lazy evaluation - Selection vectors defer materialization
• Compression - Dictionary and FSST vectors reduce memory bandwidth

Selection Vectors and Lazy Filtering

Selection vectors enable lazy filtering without materializing intermediate results. A filter operation creates a dictionary vector with a selection vector pointing to matching rows, avoiding data copies until necessary.

Nested Type Handling

For complex types (lists, structs, arrays), RecursiveToUnifiedFormat recursively normalizes child vectors, enabling uniform processing of nested data structures.

Execution Engine & Physical Operators

Overview

The execution engine transforms a physical query plan into actual data processing through a push-based pipeline model. Physical operators form a tree where data flows from sources (table scans) through intermediate operators (filters, projections) to sinks (aggregations, result collection). Each operator implements one or more of three interfaces: Source (produces data), Operator (transforms data), and Sink (consumes data).

Physical Operator Architecture

A PhysicalOperator is the base class for all execution operators. Each operator has:

• Type: Identifies the operator (FILTER, PROJECTION, HASH_JOIN, etc.)
• Output types: The schema of data it produces
• Estimated cardinality: Used for parallelism decisions
• State objects: Global state (shared across threads) and local state (per-thread)

Operators can implement multiple interfaces simultaneously. For example, a recursive CTE is both a source and a sink.

Three Operator Interfaces

Source Interface (IsSource() returns true):

• Generates data from external sources (table scans, CTEs, constants)
• Implements GetDataInternal() to produce DataChunk objects
• Supports parallel sources via ParallelSource() and partitioning

Operator Interface (default):

• Transforms input chunks to output chunks
• Implements Execute() to process data row-by-row or vectorized
• Examples: FILTER, PROJECTION, LIMIT
• Can support parallelism via ParallelOperator()

Sink Interface (IsSink() returns true):

• Consumes data and accumulates state
• Implements Sink() (parallel, called per chunk) and Finalize() (single-threaded, called once)
• Examples: HASH_GROUP_BY, HASH_JOIN, result collection
• Supports parallel sinks via ParallelSink()

Pipeline Execution Model

Loading diagram...

A Pipeline chains operators together: one source, zero or more intermediate operators, and one sink. The PipelineExecutor drives execution:

1. Source phase: Calls GetData() to fetch chunks
2. Push phase: Passes chunks through operators via Execute()
3. Sink phase: Calls Sink() to accumulate state
4. Finalize phase: Calls Finalize() for final aggregation

Parallelism and State Management

Each operator maintains:

• GlobalOperatorState: Shared across all threads (requires locking)
• LocalOperatorState: Per-thread state (no locking needed)

The executor checks ParallelOperator(), ParallelSource(), and ParallelSink() to determine if a pipeline can run in parallel. If any operator returns false, the pipeline executes sequentially.

Caching Operators

CachingPhysicalOperator buffers small chunks to improve efficiency. When a chunk is smaller than CACHE_THRESHOLD (64 rows), it accumulates chunks until reaching STANDARD_VECTOR_SIZE (typically 2048). This reduces overhead from processing many tiny chunks. The FinalExecute() method flushes remaining cached data.

Example: PhysicalFilter

The filter operator demonstrates the operator interface:

• Inherits from CachingPhysicalOperator for buffering
• ExecuteInternal() evaluates the filter expression using ExpressionExecutor
• Returns a selection vector marking matching rows
• Supports parallel execution (ParallelOperator() returns true)
• Uses FilterState to maintain per-thread expression executor and selection vector

Storage & Persistence

DuckDB's storage system is built on a layered architecture that manages both in-memory and persistent data. The system ensures durability through write-ahead logging (WAL) and periodic checkpoints, while optimizing performance through intelligent buffer management and block allocation.

Core Components

StorageManager is the top-level orchestrator that manages the physical storage of a persistent database. It coordinates between the buffer manager, block manager, WAL, and checkpoint system. For single-file databases, SingleFileStorageManager handles all storage operations within a single database file.

BufferManager (specifically StandardBufferManager) handles memory allocation and eviction for the database. It cooperatively shares a BufferPool across multiple databases and manages:

• In-memory buffer allocation and pinning
• Temporary file swapping to disk when memory pressure occurs
• Memory limits and query-specific memory constraints
• Eviction policies for unpinned blocks

BlockManager abstracts the physical block storage strategy. It manages block allocation, reading, writing, and metadata tracking. Concrete implementations include SingleFileBlockManager for persistent storage and InMemoryBlockManager for temporary data.

Durability: Write-Ahead Log (WAL)

The WriteAheadLog provides crash recovery by recording all database modifications before they are applied. Key features:

• Pre-commit logging: All INSERT, UPDATE, DELETE, and DDL operations are written to the WAL before transaction commit
• Metadata tracking: Schema changes, table creation, sequences, and indexes are logged
• Checksum protection: Each WAL entry includes checksums for integrity verification
• Encryption support: WAL entries can be encrypted using GCM or CTR ciphers
• Replay mechanism: Upon startup, the WAL is replayed to restore the database to its last consistent state

// WAL records transaction changes before commit
void WriteAheadLog::WriteInsert(DataChunk &chunk);
void WriteAheadLog::WriteDelete(DataChunk &chunk);
void WriteAheadLog::WriteUpdate(DataChunk &chunk, const vector<column_t> &column_path);

Checkpointing

CheckpointManager periodically consolidates WAL entries into the main database file, reducing recovery time and WAL size. The checkpoint process:

1. Initiates checkpoint: WALStartCheckpoint() marks the current transaction boundary
2. Writes metadata: Schema, catalog entries, and table structures are serialized
3. Writes table data: Row groups and column segments are written to disk via TableDataWriter
4. Finalizes: WALFinishCheckpoint() records the checkpoint completion in the WAL

Checkpoints can be triggered automatically based on WAL size or manually via CreateCheckpoint().

Memory Management Flow

Loading diagram...

Key Design Patterns

Lazy WAL Initialization: The WAL is only created when needed, reducing overhead for in-memory databases.

Partial Block Sharing: During checkpoints, the PartialBlockManager shares partial blocks across multiple tables to optimize disk space.

Transactional Consistency: The storage system integrates with the transaction manager to ensure ACID properties. Checkpoints capture a consistent snapshot at a specific transaction boundary.

Configurable Storage Options: Block size, encryption, compression, and row group size can be customized per database attachment.

Catalog & Metadata Management

DuckDB's catalog system manages all database metadata including schemas, tables, views, functions, and types. It provides a hierarchical structure with transaction-aware versioning to support MVCC (Multi-Version Concurrency Control) semantics.

Core Architecture

The catalog is organized into three main layers:

1. Catalog - The top-level entry point managing schemas and global operations
2. SchemaCatalogEntry - Containers for related objects (tables, functions, views)
3. CatalogEntry - Individual metadata objects (tables, columns, functions, etc.)

Each entry type (table, view, function, sequence) extends CatalogEntry and implements type-specific behavior. The DuckCatalog class is the concrete implementation for DuckDB's persistent catalog.

Transaction-Aware Versioning

The catalog uses a versioned entry chain to support concurrent transactions without locking:

// Each entry has a timestamp and optional child (newer version)
class CatalogEntry {
    atomic<transaction_t> timestamp;  // When created
    unique_ptr<CatalogEntry> child;   // Newer version
    optional_ptr<CatalogEntry> parent; // Older version
};

When a transaction modifies an entry, a new version is created and linked as a child. The CatalogSet::GetEntryForTransaction() method traverses this chain to find the appropriate version for each transaction's snapshot.

Entry Visibility Rules

Entry visibility is determined by comparing timestamps:

• Committed entries have timestamps < TRANSACTION_ID_START
• Active transaction entries have timestamps >= TRANSACTION_ID_START
• A transaction sees an entry if: created by itself OR committed before it started

The UseTimestamp() method implements this logic, allowing transactions to see consistent snapshots without blocking.

Dependency Management

The DependencyManager tracks relationships between catalog entries:

• Blocking dependencies - Prevent dropping an entry if dependents exist
• Non-blocking dependencies - Informational relationships
• Ownership dependencies - Track object ownership

When dropping an entry with cascade=true, dependent entries are recursively dropped. The dependency graph is stored as mangled entry names in a special catalog set.

Default Entries

The DefaultGenerator system provides built-in entries (schemas, functions, types) that are lazily created on first access. This allows DuckDB to support extensive built-in functionality without materializing all entries upfront.

Key Operations

Creating an entry:

1. Create a dummy deleted entry to signal other transactions
2. Verify no write-write conflicts exist
3. Add the new entry to the chain
4. Register dependencies

Dropping an entry:

1. Create a tombstone entry marked as deleted
2. Recursively drop dependents if cascade=true
3. Verify no active transactions reference the entry

Altering an entry:

1. Create a new version with modified metadata
2. Link as child of current entry
3. On commit, the new version becomes visible to future transactions

Query Optimization

DuckDB's query optimizer transforms logical plans into faster, logically equivalent plans through a series of coordinated optimization passes. The optimizer runs approximately 30 different optimization techniques in a carefully orchestrated sequence.

Optimization Pipeline

The optimization process follows this general flow:

1. Expression Rewriting - Simplifies expressions through constant folding, algebraic simplification, and pattern-based rules (e.g., x + 0 → x, CASE simplification)
2. CTE Inlining - Decides whether to inline or materialize common table expressions
3. Filter Pullup & Pushdown - Moves filter conditions to optimal positions in the plan tree
4. Join Ordering - Uses dynamic programming to find the best join order based on cardinality estimates
5. Column Lifetime Analysis - Determines which columns are needed at each stage
6. Statistics Propagation - Estimates cardinality through the plan for cost-based decisions

Key Optimization Techniques

Filter Pushdown moves predicates closer to data sources to reduce intermediate rows. The FilterPushdown class recursively traverses the logical plan, collecting filters and pushing them down through projections, joins, and aggregates. For example, a filter on a joined table can be pushed below the join if it only references one side.

Join Order Optimization uses a query graph representation and dynamic programming to enumerate join orderings. The JoinOrderOptimizer builds a hypergraph of relations and filters, then uses a CostModel and PlanEnumerator to find the lowest-cost join sequence. This is critical for multi-join queries where join order dramatically affects performance.

Expression Rewriting applies pattern-matching rules to simplify expressions without changing the plan structure. Rules include constant folding, distributivity, comparison simplification, and function-specific optimizations. The ExpressionRewriter recursively applies rules until no further changes occur.

Optimization Passes

Expression Rewriter → CTE Inlining → Filter Pullup → Filter Pushdown
→ Join Order → Join Elimination → Unused Columns → Common Subexpressions
→ Column Lifetime → Build/Probe Side Selection → Limit Pushdown
→ Row Group Pruning → Top-N Transformation → Late Materialization
→ Statistics Propagation → Filter Reordering → Join Filter Pushdown

Each pass can be individually disabled via configuration. The optimizer verifies plan correctness after each pass and profiles execution time for performance monitoring.

Cost-Based Decisions

The join order optimizer uses cardinality estimates to guide decisions. Statistics are propagated through the plan to estimate row counts at each operator. The cost model evaluates different join orders and selects the one with the lowest estimated cost, considering factors like table size, selectivity, and join type.

Functions & Extensions System

DuckDB provides a flexible function system that supports multiple function types and a powerful extension mechanism for registering custom functions. The system is designed to be both performant and extensible.

Function Types

DuckDB supports four primary function types:

1. Scalar Functions - Process individual rows and return a single value per input row. Examples include arithmetic operations, string functions, and type conversions.

2. Aggregate Functions - Combine multiple rows into a single result. They maintain state across rows using initialize, update, combine, and finalize callbacks.

3. Table Functions - Return entire tables as results. Used for operations like reading files, generating sequences, or scanning external data sources.

4. Copy Functions - Handle data import/export operations with custom bind, initialization, and sink callbacks.

Scalar Function Architecture

Scalar functions are defined by the ScalarFunction class, which encapsulates:

• Function callback (scalar_function_t) - The core execution logic operating on DataChunk inputs
• Bind callback - Validates arguments and creates bind data at query planning time
• Statistics callback - Propagates cardinality and value range information for optimization
• Local state initialization - Sets up thread-local state for stateful operations
• Null handling - Configurable behavior for NULL values (default, special, or strict)

ScalarFunction add_func(
    {LogicalType::INTEGER, LogicalType::INTEGER},
    LogicalType::INTEGER,
    &AddFunction,  // execution callback
    &BindAddFunction  // optional bind callback
);

Extension System

Extensions are dynamically loaded libraries that register functions and types with DuckDB. The system supports two ABI types:

• C++ ABI - Direct C++ function pointers, tightly coupled to DuckDB version
• C ABI - Stable C API with versioned function pointer structs, compatible across versions

Extension Loading Pipeline

Loading diagram...

Registering Functions via Extensions

Extensions use the ExtensionLoader class to register functions with the system catalog:

void MyExtension::Load(ExtensionLoader &loader) {
    loader.RegisterFunction(my_scalar_function);
    loader.RegisterFunction(my_table_function);
    loader.RegisterType(my_custom_type);
}

The loader handles catalog integration, conflict resolution, and system transaction management automatically.

Function Execution

When a function is executed:

1. Binding - The bind callback validates arguments and creates bind data
2. Initialization - Local state is initialized per thread if needed
3. Execution - The function callback processes DataChunk batches
4. Finalization - Results are verified and returned

This architecture enables efficient vectorized execution while maintaining flexibility for complex operations.

Planning & Binding

Overview

Planning and binding are the core phases that transform parsed SQL statements into executable logical query plans. The Planner orchestrates the process, while the Binder performs semantic analysis and converts parsed expressions into bound expressions that reference actual catalog objects.

The Planning Pipeline

The planning process follows these stages:

1. Parsing (handled by parser) produces an abstract syntax tree (AST)
2. Binding (Binder) validates and resolves references to catalog objects
3. Logical Plan Generation converts bound statements into logical operators
4. Optimization (separate phase) improves the logical plan
5. Physical Plan Generation converts logical operators to executable physical operators

Planner Class

The Planner class (in src/planner/planner.cpp) is the entry point for query planning:

class Planner {
  unique_ptr<LogicalOperator> plan;
  shared_ptr<Binder> binder;
  vector<string> names;
  vector<LogicalType> types;
  
  void CreatePlan(SQLStatement &statement);
};

When CreatePlan() is called, it:

1. Starts the binding phase via binder->Bind(statement)
2. Receives a BoundStatement containing the logical operator tree
3. Validates tree depth to prevent stack overflow
4. Applies dependent join flattening for subquery optimization
5. Verifies the plan integrity

Binder Class

The Binder class (in src/planner/binder.cpp) performs semantic analysis:

• Resolves column references to actual table columns via BindContext
• Validates function calls against registered functions
• Handles subqueries by creating child binders
• Manages scope through parent-child binder relationships
• Tracks correlated columns for subquery correlation detection

Key methods:

• Bind(SQLStatement &statement) - Main entry point, dispatches to statement-specific binders
• BindNode(QueryNode &node) - Binds SELECT, set operations, and CTEs
• CreatePlan(BoundSelectNode &) - Converts bound query nodes to logical operators

Binding Context

The BindContext maintains the scope of available tables and columns during binding:

• Tracks table bindings and their columns
• Resolves column references to specific tables
• Handles aliases and qualified names
• Supports generated columns and CTEs

Logical Operators

The LogicalOperator class represents nodes in the logical query tree:

class LogicalOperator {
  LogicalOperatorType type;
  vector<unique_ptr<LogicalOperator>> children;
  vector<unique_ptr<Expression>> expressions;
  vector<LogicalType> types;
};

Common logical operators include:

• LogicalGet - Table scans
• LogicalFilter - WHERE clauses
• LogicalProjection - SELECT list
• LogicalJoin - JOIN operations
• LogicalAggregate - GROUP BY and aggregates
• LogicalOrder - ORDER BY
• LogicalLimit - LIMIT/OFFSET

Expression Binding

Expressions are bound through specialized ExpressionBinder subclasses:

• SelectBinder - SELECT list expressions
• WhereBinder - WHERE clause conditions
• GroupBinder - GROUP BY expressions
• HavingBinder - HAVING clause conditions
• OrderBinder - ORDER BY expressions

Each binder validates expressions in their specific context and resolves column references.

Error Handling

Binding errors are caught and reported with context:

• BinderException - Semantic errors (unknown column, type mismatch)
• ParserException - Syntax errors (caught earlier)
• Depth tracking prevents infinite recursion in nested expressions

Key Data Structures

• BoundStatement - Result of binding: contains logical plan, column names, and types
• ColumnBinding - Identifies a column by table index and column index
• CorrelatedColumnInfo - Tracks columns referenced from outer scopes

Testing & Development

DuckDB maintains a comprehensive testing and benchmarking infrastructure to ensure code quality and performance. The system includes unit tests, SQL logic tests, and performance benchmarks across multiple domains.

Test Framework Architecture

The testing infrastructure uses Catch2 as the C++ unit testing framework and a custom SQL test runner for SQL logic tests. The main entry point is test/unittest.cpp, which orchestrates test discovery and execution. Tests are organized into categories: API tests, appender tests, optimizer tests, storage tests, and SQL tests.

SQL tests use a .test file format with a simple declarative syntax:

statement ok
CREATE TABLE tbl (a VARCHAR);

query I
SELECT COUNT(*) FROM tbl
----
0

Running Tests

Build with tests enabled:

make

Run all tests:

./build/release/unittest

Run specific test categories:

./build/release/unittest "[api]"
./build/release/unittest "[sql]"

Run SQL tests from a specific directory:

./build/release/unittest test/sql/select

Test Configuration

The test runner accepts several command-line arguments:

• --test-dir <path> — Override the working directory (default: project root)
• --test-temp-dir <path> — Set temporary directory for test artifacts
• --require <env_var> — Require specific environment variables

Environment variables available to tests:

• TEST_NAME — Current test file path
• TEST_UUID — Unique identifier per test invocation
• WORKING_DIR — Project root directory
• DATA_DIR — Test data directory (default: {WORKING_DIR}/data)
• TEMP_DIR — Temporary directory for test outputs

Benchmark Infrastructure

The benchmark system measures query performance across micro and macro benchmarks. Benchmarks are defined in .benchmark files and executed by benchmark_runner.

List all benchmarks:

./build/release/benchmark/benchmark_runner --list

Run a single benchmark:

./build/release/benchmark/benchmark_runner benchmark/micro/nulls/no_nulls_addition.benchmark

Run benchmarks matching a regex pattern:

./build/release/benchmark/benchmark_runner "benchmark/micro/nulls/.*"

Output options:

• --info — Display benchmark metadata
• --query — Print the SQL query being benchmarked
• --profile — Show query execution tree with timing breakdown
• --out <file> — Write timing results to file

Test Categories

• API Tests (test/api/) — C++ API functionality, connection management, prepared statements
• SQL Tests (test/sql/) — SQL parsing, execution, and correctness across all features
• Optimizer Tests (test/optimizer/) — Query optimization rules and plan generation
• Storage Tests (test/storage/) — Persistence, transactions, and data integrity
• Extension Tests (test/extension/) — Extension loading and custom functionality
• Fuzzer Tests (test/fuzzer/) — Fuzz-based testing for robustness

Benchmark Categories

• Micro Benchmarks (benchmark/micro/) — Individual operations (aggregates, joins, filters, casts)
• Macro Benchmarks — Full workloads (TPC-H, TPC-DS, ClickBench, IMDB)
• Ingestion Benchmarks — Data loading performance
• Specialized Benchmarks — Type casting, compression, JSON processing