Redis Open Source

Overview

Redis is a high-performance, in-memory data structure server that serves as a cache, database, and message broker. Written in C, it prioritizes speed, simplicity, and efficiency while supporting a rich ecosystem of data types and operations.

Core Philosophy

Redis is built on ten core principles outlined in its manifesto. The project emphasizes memory-first storage, fundamental data structures with clear semantics, code efficiency, and fighting complexity. Rather than using threading, Redis employs a single-threaded event-driven architecture, with horizontal scaling achieved through Redis Cluster and replication.

Key Capabilities

Data Structures: Redis provides native support for strings, lists, sets, sorted sets, hashes, streams, JSON documents, and vector sets. Each data type is optimized for specific use cases—from leaderboards (sorted sets) to real-time messaging (streams).

Processing Engines: The Redis Query Engine enables full-text search, vector similarity search, geospatial queries, and aggregations on indexed documents. Probabilistic data structures like HyperLogLog, Bloom filters, and t-digest support analytics and cardinality estimation.

Persistence: Redis offers two persistence mechanisms—RDB (point-in-time snapshots) and AOF (append-only file)—allowing data durability without sacrificing performance.

Replication & Clustering: Master-replica replication enables high availability, while Redis Cluster provides horizontal scaling across multiple nodes with automatic slot management.

Architecture Highlights

Loading diagram...

The server core (server.c, server.h) manages the global state, client connections, and the main event loop. Commands are dispatched through a command table, with each operation manipulating data stored in memory-resident hash tables and specialized structures.

Performance Characteristics

Redis achieves sub-millisecond latency through in-memory storage and efficient data structures. The event-driven architecture handles thousands of concurrent clients without blocking. Memory management uses jemalloc by default on Linux to minimize fragmentation, and the codebase includes sophisticated eviction policies (LRU, LFU) when memory limits are reached.

Use Cases

• Caching: Fast access to frequently used data with configurable expiration
• Session Storage: Flexible session data modeling with multiple data types
• Real-Time Analytics: Counters, leaderboards, and rate limiting
• Message Queues: Lists, streams, and pub/sub for inter-service communication
• Vector Search: Semantic similarity and RAG applications with vector sets
• Search Engine: Full-text and geospatial indexing via Redis Query Engine

Architecture & Core Components

Redis is built on a single-threaded, event-driven architecture that efficiently handles thousands of concurrent connections. The core design revolves around three key components: the event loop, the connection abstraction, and the networking layer.

Event Loop (ae.c/ae.h)

The event loop is the heart of Redis. It's a multiplexing system that monitors file descriptors and timers, dispatching events as they occur. The aeEventLoop structure tracks:

• File events: Read/write readiness on sockets (registered via aeCreateFileEvent)
• Time events: Periodic tasks like key expiration and background operations
• Fired events: Events ready to process in the current iteration

The loop uses the best available multiplexing API for the platform (epoll on Linux, kqueue on BSD, evport on Solaris, select as fallback). Each iteration calls aeProcessEvents(), which polls for ready file descriptors, executes callbacks, and processes timers.

void aeMain(aeEventLoop *eventLoop) {
    eventLoop->stop = 0;
    while (!eventLoop->stop) {
        aeProcessEvents(eventLoop, AE_ALL_EVENTS|
                                   AE_CALL_BEFORE_SLEEP|
                                   AE_CALL_AFTER_SLEEP);
    }
}

Connection Abstraction (connection.h)

The connection structure provides a protocol-agnostic interface for TCP, Unix sockets, and TLS. Each connection holds:

• A ConnectionType function pointer table (vtable pattern)
• State tracking (connecting, accepting, connected, closed, error)
• Bound event loop reference and read/write handlers
• Private data for protocol-specific state

This abstraction allows Redis to support multiple connection types without duplicating I/O logic. Connection types register themselves at startup via connTypeRegister().

Networking Layer (networking.c)

The networking layer handles client protocol parsing, command execution, and response buffering. Key responsibilities:

• Client management: Linking/unlinking clients, tracking connection state
• Protocol parsing: RESP (Redis Serialization Protocol) parsing
• Output buffering: Queuing responses for efficient batch writes
• Memory tracking: Monitoring client output buffer sizes

Integration Flow

Loading diagram...

When a client connects, the accept handler creates a connection object and registers read/write handlers with the event loop. As data arrives, the read handler parses commands and queues responses. The write handler flushes buffered output when the socket becomes writable. This non-blocking, event-driven model allows a single thread to serve thousands of clients efficiently.

Key Design Patterns

• Callback-based: Handlers are registered and invoked asynchronously
• Barrier writes: The AE_BARRIER flag ensures writes happen after reads in the same iteration, useful for persistence
• Before/after sleep hooks: beforeSleep() performs maintenance (expiration, client cleanup) before blocking on I/O
• Lazy event processing: Events are only processed when needed, minimizing CPU usage

Data Structures & Encodings

Core Object Model

Every Redis value is wrapped in a redisObject structure (16 bytes on 64-bit systems) that stores metadata:

struct redisObject {
    unsigned type:4;           /* OBJ_STRING, OBJ_LIST, OBJ_SET, etc. */
    unsigned encoding:4;       /* Internal representation format */
    unsigned lru:24;           /* LRU/LFU eviction data */
    unsigned iskvobj:1;        /* Embedded key flag */
    unsigned expirable:1;      /* Has expiration time */
    unsigned refcount:30;      /* Reference counting */
    void *ptr;                 /* Pointer to actual data */
};

The type field identifies the Redis data type (String, List, Set, Hash, Sorted Set, Stream, or Module). The encoding field specifies how that type is internally represented, allowing Redis to optimize storage based on data characteristics.

Encoding Strategies

Redis uses multiple encodings per type to balance memory efficiency and performance:

• Strings: RAW (raw SDS), EMBSTR (embedded <44 bytes), INT (integer)
• Lists: QUICKLIST (linked list of compressed listpacks), LISTPACK (compact sequential storage)
• Sets: INTSET (compact integer storage), LISTPACK (small sets), HT (hash table for large sets)
• Hashes: LISTPACK (small hashes), HT (hash table for large hashes)
• Sorted Sets: LISTPACK (small sets), SKIPLIST (skip list + hash table for large sets)
• Streams: STREAM (radix tree of listpacks)

Compact Data Structures

Listpack is a memory-efficient sequential container storing strings and integers with variable-length encoding. Each entry includes length metadata, enabling O(1) random access while maintaining compact storage.

Intset stores integers in a sorted array with adaptive encoding (16-bit, 32-bit, or 64-bit) based on value ranges. When a value outside the current encoding range is added, the entire intset is upgraded.

Quicklist combines linked lists with listpacks: a doubly-linked list of listpack nodes with optional LZF compression. This balances list operation efficiency with memory usage, configurable via list-max-listpack-size and list-compress-depth.

Skip List powers sorted sets, providing O(log N) insertion, deletion, and range queries. Combined with a hash table for O(1) member lookups, sorted sets achieve both fast ranking and membership testing.

Stream Encoding

Streams use a radix tree of listpacks indexed by stream IDs (128-bit: milliseconds + sequence). Each listpack stores multiple entries with delta-encoded IDs and field deduplication. Consumer groups track delivery state via pending entry lists (PELs) stored as radix trees.

Loading diagram...

Memory Optimization

Redis automatically converts between encodings as data grows. For example, a set starts as an intset, converts to a listpack when it exceeds set-max-intset-entries, then to a hash table when it exceeds set-max-listpack-entries. This lazy conversion minimizes memory overhead for small collections while maintaining performance for large ones.

Embedded keys in kvobj structures reduce allocation overhead by storing keys directly within the object metadata when possible, particularly beneficial for large keys.

Command Processing Pipeline

The command processing pipeline transforms raw network data into executed Redis commands. It consists of three main stages: protocol parsing, command lookup and validation, and command execution.

Protocol Parsing

When a client sends data, it arrives in the query buffer. The pipeline first determines the protocol type:

• RESP (REdis Serialization Protocol): Starts with * for multibulk arrays
• Inline Protocol: Plain text commands separated by spaces

processInputBuffer() routes to the appropriate parser:

• processMultibulkBuffer() handles RESP format, parsing the array length, then each bulk string argument
• processInlineBuffer() handles inline commands, splitting on whitespace

Both parsers populate a pendingCommand structure with argc (argument count) and argv (argument array).

Command Lookup and Validation

Once a complete command is parsed, preprocessCommand() performs early validation:

1. Command Lookup: lookupCommand() searches the command dictionary using the first argument as the key. It handles both top-level commands and subcommands (e.g., CONFIG GET).

2. Arity Check: Verifies argument count matches the command's arity specification.

3. Reuse Optimization: If the previous command matches, it reuses the cached command pointer to avoid dictionary lookups.

Command Execution

processCommand() performs comprehensive checks before execution:

• Authentication: Verifies client is authenticated (unless command has CMD_NO_AUTH flag)
• ACL Permissions: Checks access control lists
• State Validation: Ensures server state allows the command (not loading, not in readonly mode, etc.)
• Transaction Handling: Queues commands in MULTI blocks instead of executing immediately

Finally, call() invokes the command's handler function (cmd->proc), which implements the actual logic. The handler receives the client structure and reads arguments from c->argv.

Data Flow Diagram

Loading diagram...

Key Optimizations

• Command Reuse: Caches the last command to avoid repeated dictionary lookups for pipelined commands
• Pending Commands Queue: Buffers multiple commands for efficient batch processing
• Lazy Parsing: Only parses complete commands; incomplete data waits for more input
• Per-Slot Metrics: Tracks network bytes and command statistics per cluster slot

Persistence: RDB & AOF

Redis provides two complementary persistence mechanisms: RDB snapshots for point-in-time backups and AOF (Append-Only File) for command-level durability. Both can run independently or together.

RDB Snapshots

RDB creates a binary snapshot of the entire dataset at a specific moment. The format is optimized for fast loading and compression.

Key characteristics:

• Format: Binary with version header (REDIS0012), metadata, all databases, and CRC64 checksum
• Encoding: Supports multiple object types (strings, lists, sets, hashes, sorted sets, streams) with compression (LZF)
• Process: Uses rdbSave() for synchronous saves or rdbSaveBackground() for background saves via fork
• Atomic writes: Writes to temporary file, then renames atomically to final location
• Opcodes: Special markers like RDB_OPCODE_SELECTDB, RDB_OPCODE_EXPIRETIME_MS, RDB_OPCODE_EOF structure the file

Loading happens via rdbLoad() or rdbLoadRio(), which validates the magic header, reads all databases, and reconstructs objects in memory.

AOF (Append-Only File)

AOF logs every write command in Redis protocol format, enabling recovery to any point in time.

File structure (managed via manifest):

• BASE: RDB snapshot from last rewrite (optional, can use RDB preamble)
• INCR: Incremental command logs since last rewrite (multiple files possible)
• HISTORY: Previous BASE/INCR files marked for deletion after successful rewrite

Key operations:

• feedAppendOnlyFile() buffers commands in server.aof_buf before flushing to disk
• flushAppendOnlyFile() writes buffered commands with configurable fsync policies (always, everysec, no)
• rewriteAppendOnlyFileBackground() forks a child to compact the AOF via rewriteAppendOnlyFileRio()

AOF rewrite (triggered by BGREWRITEAOF or automatic thresholds):

1. Child process writes new BASE file (RDB or command format)
2. Parent opens new INCR file for incoming commands
3. On success, manifest updates atomically; old files become HISTORY
4. Background job deletes HISTORY files asynchronously

RIO (Redis I/O)

The rio abstraction layer handles I/O to files, buffers, file descriptors, or sockets with:

• Automatic checksumming (CRC64 for RDB)
• Incremental fsync for large writes
• Page cache reclamation to reduce memory overhead

Hybrid Approach

When aof_use_rdb_preamble is enabled, AOF rewrites use RDB format for the BASE file, combining RDB's compression efficiency with AOF's command-level granularity for INCR files.

Loading diagram...

Replication & Clustering

Redis provides three complementary mechanisms for high availability and scalability: replication for data synchronization, clustering for horizontal scaling, and Sentinel for automatic failover.

Replication: Master-Replica Synchronization

Replication enables data synchronization between a master and one or more replicas. The master maintains a replication offset tracking all commands sent, while replicas track their applied offset.

Key Components:

• Replication Backlog: A circular buffer storing recent commands. When a replica reconnects, it can request a partial resync (PSYNC) to receive only missing commands instead of a full RDB snapshot.
• Replication Buffer Blocks: Shared memory blocks referenced by both the backlog and replica clients, reducing memory overhead.
• Full Sync: When partial resync fails, the master generates an RDB snapshot and streams it to the replica, followed by command stream.
• RDB Channel: An optimized dual-channel mechanism where one connection streams the RDB and another buffers commands, improving throughput during full sync.

Synchronization Flow:

Replica connects → PSYNC request → Master checks offset
  ├─ Offset in backlog → Partial resync (fast)
  └─ Offset lost → Full resync (RDB + commands)

Cluster: Distributed Hash Slots

Redis Cluster partitions data across multiple nodes using 16,384 hash slots. Each key is mapped to a slot via CRC16 hashing, with support for hash tags ({tag}) to co-locate related keys.

Architecture:

• Slot Distribution: Each master node owns a range of slots; replicas replicate their master's data.
• Key Routing: Clients compute slot IDs and route requests to the correct node. Redirects (MOVED, ASK) guide clients to proper nodes.
• Atomic Slot Migration (ASM): Moves slots between nodes using dual channels (main + RDB) to minimize downtime and ensure consistency.
• Cluster State: Maintained via gossip protocol; nodes exchange configuration epochs to detect stale information.

Slot Migration States:

Loading diagram...

Sentinel: Automatic Failover Orchestration

Sentinel monitors master-replica topologies and automatically promotes replicas when masters fail. Multiple Sentinels use quorum-based voting to prevent split-brain scenarios.

Failure Detection:

• Subjective Down (S-DOWN): A single Sentinel detects no response within down-after-period.
• Objective Down (O-DOWN): Quorum of Sentinels agree the master is down.
• Failover Trigger: Once O-DOWN is confirmed, Sentinels elect a leader to orchestrate failover.

Failover State Machine:

1. Wait Start: Random delay to desynchronize Sentinels
2. Select Slave: Choose replica with best priority and replication offset
3. Send SLAVEOF NO ONE: Promote selected replica to master
4. Wait Promotion: Verify replica became master
5. Reconf Slaves: Reconfigure other replicas to replicate from new master
6. Update Config: Update Sentinel configuration with new topology

Sentinel also handles configuration propagation via pub/sub and script execution for custom notifications.

Modules & Scripting

Redis provides two complementary extension mechanisms: modules for compiled extensions and scripting for dynamic Lua code execution. Both enable custom functionality while maintaining atomicity and isolation.

Module System

Modules are dynamically loaded C libraries that extend Redis with custom commands, data types, and event hooks. The module lifecycle begins with RedisModule_OnLoad(), the entry point called when a module is loaded via the MODULE LOAD command.

Key Components:

• Module Context (RedisModuleCtx): Opaque handle passed to all module API calls, managing memory, command registration, and server interaction
• Module API (redismodule.h): Comprehensive C API with 200+ functions for memory management, key operations, command creation, and server events
• Module Types: Custom data types registered via RedisModule_CreateDataType(), with callbacks for RDB persistence, AOF rewriting, and memory reporting
• Shared APIs: Modules can export functions for other modules via RedisModule_ExportSharedAPI()

Modules integrate deeply with Redis: they can subscribe to server events (client changes, flushes, replication), implement command filters, handle authentication, and participate in cluster operations.

Lua Scripting

Redis executes Lua scripts atomically via EVAL and EVALSHA commands. Scripts are cached by SHA1 hash to enable efficient re-execution without recompilation.

Script Execution Flow:

1. Script is compiled to a Lua function named f_<SHA1>
2. Function is stored in the Lua registry and cached in lctx.lua_scripts dictionary
3. On execution, the script receives KEYS and ARGV arrays and can call redis.call() or redis.pcall()
4. Results are converted from Lua types to Redis protocol and returned to the client

Script Caching & Management:

• LRU eviction via lua_scripts_lru_list when memory limits are exceeded
• SCRIPT LOAD pre-compiles scripts without execution
• SCRIPT EXISTS checks cache status; SCRIPT FLUSH clears all scripts
• Scripts support shebang headers (#!lua flags=...) for declaring execution flags

Functions System

Functions extend scripting with persistent, reusable libraries. Unlike ephemeral EVAL scripts, functions are stored in RDB and replicated to replicas.

Function Architecture:

• Libraries: Named collections of functions, loaded via FUNCTION LOAD with metadata (engine, name, description)
• Engines: Pluggable execution environments; currently Lua is the primary engine
• Function Flags: Control execution behavior (no-writes, allow-stale, allow-oom, no-cluster, allow-cross-slot-keys)
• Registration: Functions register themselves via redis.register_function() during library load

Invocation:

Functions are called via FCALL <name> <numkeys> [keys...] [args...] or FCALL_RO for read-only variants. The execution context (scriptRunCtx) manages isolation, timeout handling, and replication.

Lua API & Sandboxing

Scripts access Redis via redis.call() (raises errors) and redis.pcall() (returns errors). The Lua environment is sandboxed with an allowlist of safe globals and libraries (string, math, table, cjson, bit, struct, os). Dangerous functions like setfenv and getfenv are blocked.

-- Example: Atomic increment with expiry
return redis.call('SET', KEYS[1], redis.call('INCR', KEYS[1]))

Execution Guarantees

Both modules and scripts execute atomically: no other commands or scripts run concurrently. The scriptRunCtx structure tracks execution state, enforcing timeouts (default 5 seconds), preventing writes in read-only mode, and managing cluster key restrictions. Timeout violations trigger script termination and client disconnection.

Memory Management & Eviction

Memory Allocation & Tracking

Redis uses a custom memory allocator wrapper (zmalloc) that tracks all allocations across the system. The wrapper supports multiple underlying allocators: jemalloc (default, recommended), tcmalloc, or libc. Thread-local counters track memory usage per thread to avoid contention, with zmalloc_used_memory() aggregating totals across all threads.

Memory allocation includes safety mechanisms: zmalloc() panics on failure, while ztrymalloc() returns NULL. The allocator also provides usable size information to optimize memory utilization beyond requested sizes.

Eviction Policies & Maxmemory

When memory usage exceeds maxmemory, Redis evicts keys according to the configured policy:

• LRU-based: volatile-lru (TTL keys only), allkeys-lru (any key)
• LFU-based: volatile-lfu, allkeys-lfu (frequency-based, with decay)
• TTL-based: volatile-ttl (evict keys expiring soonest)
• Random: volatile-random, allkeys-random
• No eviction: Reject writes when over limit

The eviction pool maintains 16 candidate keys sampled from the dataset. During eviction, Redis samples maxmemory-samples keys (default 5) and selects the best candidate from the pool. This approximates true LRU/LFU in constant memory.

LFU Implementation

LFU uses 24 bits per object: 16 bits for last access time (in minutes) and 8 bits for a logarithmic frequency counter. The counter saturates at 255 and decays over time based on lfu-decay-time. New keys start at LFU_INIT_VAL to avoid immediate eviction. Logarithmic increment (LFULogIncr) makes frequent keys harder to increment, creating a natural frequency distribution.

Expiration & Active Cleanup

Keys with TTL are stored in a separate expires dictionary. Redis uses two expiration mechanisms:

1. Passive expiration: Keys expire on access
2. Active expiration cycle: Runs at server.hz frequency (default 10 Hz), sampling keys and removing expired ones

The active cycle adapts effort based on expiration density. If many keys are stale, it increases sampling; if few expire, it reduces work to avoid CPU waste.

Lazy Freeing

Large objects are freed asynchronously in a background thread via lazyfree.c. This prevents blocking the main thread during eviction or deletion of large structures (hashes, lists, sets). The background thread decrements reference counts and updates memory statistics.

Active Defragmentation

When using jemalloc with fragmentation hints, Redis can actively defragment memory. The defrag process scans the keyspace in stages, asking jemalloc if moving allocations would reduce fragmentation. Defragmentation runs in bounded time windows to maintain responsiveness. It's controlled by activedefrag config and fragmentation thresholds.

Memory Overhead Exclusions

Replication buffers and AOF buffers are excluded from eviction calculations to prevent feedback loops: evicting keys generates DEL commands that fill buffers, triggering more evictions. Client connection buffers can also be evicted via maxmemory-clients to prevent client memory from consuming all available memory.

Loading diagram...