OpenSSL Cryptography & TLS Toolkit

Overview

OpenSSL is a robust, commercial-grade cryptographic toolkit providing secure communication protocols and general-purpose cryptographic operations. It is descended from the SSLeay library and is licensed under the Apache License 2.0.

Core Libraries

OpenSSL consists of two main libraries:

libcrypto provides general-purpose cryptographic functionality including symmetric encryption, public key cryptography, key agreement, certificate handling, hash functions, message authentication codes (MACs), and key derivation functions (KDFs). It can be used independently or as the foundation for TLS implementations.

libssl implements secure network communication protocols: TLS (versions 1.0 through 1.3), DTLS (versions 1.0 and 1.2), and QUIC (version 1). It depends on libcrypto for all cryptographic operations and provides a unified API across all supported protocols.

Provider Architecture

OpenSSL 3.0 introduced a modular provider system where algorithm implementations are organized into pluggable providers. Each provider contains implementations for various cryptographic operations. The standard providers include:

• Default Provider: Built-in implementations of modern algorithms (AES, SHA, RSA, ECDSA, etc.)
• Legacy Provider: Older algorithms (DES, MD5, RC4) for backward compatibility
• FIPS Provider: FIPS 140-2 validated algorithm subset
• Base Provider: Non-cryptographic utilities for key serialization
• Null Provider: Empty provider to prevent automatic loading

Applications access algorithms through high-level APIs (EVP functions) which automatically select appropriate provider implementations.

Command-Line Tool

The openssl command-line utility provides a comprehensive interface for cryptographic operations including key generation, certificate creation, encryption/decryption, message digests, SSL/TLS testing, and S/MIME handling.

Key Features

• Multi-protocol support (TLS, DTLS, QUIC)
• Extensible provider-based architecture
• FIPS-validated cryptographic module
• Comprehensive certificate and PKI support
• Cross-platform compatibility (Unix, Windows, VMS, Android, iOS)
• Thread-safe operations with atomic primitives
• Async I/O capabilities via BIO abstraction layer

Architecture & Core Design

OpenSSL 3.x is built on a provider-based architecture that separates algorithm implementations from the core library. This design enables modularity, FIPS compliance, and third-party algorithm support.

Library Context (OSSL_LIB_CTX)

The library context is the central hub for all cryptographic operations. Represented by OSSL_LIB_CTX, it manages:

• Provider store: Collection of loaded providers
• Method stores: EVP cipher, digest, key exchange, and signature implementations
• Property system: Algorithm selection via property queries
• Name mapping: Algorithm name resolution
• Thread-local state: DRBG, compression methods, and thread pools

Each application can create multiple contexts for isolation. A default global context is automatically created on first use and destroyed at exit.

struct ossl_lib_ctx_st {
    CRYPTO_RWLOCK *lock;
    void *provider_store;
    void *evp_method_store;
    void *namemap;
    void *property_defns;
    void *drbg;
    /* ... additional stores and state ... */
};

Provider System

Providers are containers for algorithm implementations. OpenSSL ships with five standard providers:

• Default: Standard built-in algorithms (RSA, AES, SHA, etc.)
• Legacy: Deprecated algorithms (MD2, RC4, DES, etc.)
• FIPS: FIPS 140-2 validated algorithms
• Base: Non-cryptographic utilities (encoding, key serialization)
• Null: Empty provider to prevent auto-loading

Providers are loaded into a library context and expose algorithms via dispatch tables. Each provider has:

• Initialization function: Called when provider is activated
• Dispatch table: Function pointers for core operations
• Algorithm table: Available algorithms with properties
• Reference counting: Tracks active usage

Algorithm Fetching & Method Construction

When an application requests an algorithm (e.g., EVP_CIPHER_fetch()), OpenSSL:

1. Queries providers in the library context for matching algorithms
2. Applies property filters to select the best implementation
3. Constructs the method by calling provider functions
4. Caches the result in the method store for performance

Loading diagram...

Dispatch Tables & Core Interface

The dispatch table is the contract between core and providers. It defines:

• Function IDs: Numeric identifiers for operations
• Function pointers: Implementation addresses
• Terminator: Array ends with {0, NULL}

Providers implement OSSL_ALGORITHM structures that map algorithm names to implementations:

struct ossl_algorithm_st {
    const char *algorithm_names;      /* e.g., "AES-256-GCM" */
    const char *property_definition;  /* e.g., "fips=yes" */
    const OSSL_DISPATCH *implementation;
};

Thread Safety & Locking

The architecture uses multiple locks to prevent deadlocks:

• Context lock: Protects library context state
• Provider store lock: Protects provider collection
• Provider flag lock: Protects initialization/activation flags
• Method store locks: Protect algorithm caches

Lock acquisition order is strictly enforced: store lock → flag lock → activation lock. No locks are held during provider upcalls to prevent deadlock.

SSL/TLS Integration

The SSL layer (ssl/ssl_lib.c) integrates with the provider system through:

• EVP interfaces: Cipher, digest, and signature operations
• Key management: Private/public key operations via providers
• Algorithm negotiation: Cipher suite selection using provider capabilities
• Context binding: Each SSL connection uses the library context for algorithm selection

TLS & DTLS Protocol Implementation

Overview

OpenSSL implements both TLS (Transport Layer Security) and DTLS (Datagram TLS) protocols through a unified state machine architecture. The implementation separates concerns into two independent layers: the message flow state machine (handling I/O and buffering) and the handshake state machine (tracking protocol state). This design eliminates code duplication between TLS/DTLS and client/server variants.

State Machine Architecture

The state machine consists of two primary components:

1. Message Flow State Machine - Controls reading and writing of messages, handles non-blocking I/O, buffer flushing, and unexpected message handling. It has two sub-state machines: one for reading and one for writing.

2. Handshake State Machine - Tracks the current SSL/TLS handshake state and transitions based on events from the message flow state machine.

The architecture is hierarchical:

Loading diagram...

TLS Record Layer

The record layer (rec_layer_s3.c) handles:

• Encryption/Decryption - Uses EVP (envelope) functions for cipher operations
• MAC/Authentication - Computes and verifies message authentication codes
• Compression - Optional compression before encryption
• Fragmentation - Splits large messages into <16KB records

Record processing follows: plaintext → compress → MAC → encrypt → send.

DTLS-Specific Implementation

DTLS differs from TLS in handling unreliable UDP transport:

Message Reassembly (statem_dtls.c):

• Fragments are numbered with sequence numbers and offsets
• Bitmasks track which fragments have been received
• Out-of-order fragments are buffered and reassembled
• Retransmission timers trigger resending of lost messages

Record Layer (rec_layer_d1.c):

• Maintains a priority queue (pqueue) for buffered out-of-order records
• Implements DTLS-specific record format with epoch and sequence numbers
• Handles application data buffering during handshake

Initialization (d1_lib.c):

• Defines DTLSv1_enc_data and DTLSv1_2_enc_data encryption method structures
• Sets default timeout to 2 hours (vs. indefinite for TLS)
• Initializes DTLS record layer with priority queue support

Protocol Methods

The methods.c file defines protocol entry points for each TLS/DTLS version:

• TLS_method() - Negotiates highest supported TLS version
• tlsv1_3_method(), tlsv1_2_method() - Version-specific methods
• DTLS_method() - Negotiates highest supported DTLS version
• Each method specifies accept/connect handlers and encryption data

Methods are implemented via macros (IMPLEMENT_tls_meth_func) that register the appropriate state machine handlers.

Key Differences: TLS vs DTLS

Handshake Flow

Both TLS and DTLS follow similar handshake patterns:

1. ClientHello - Client initiates with supported ciphers and extensions
2. ServerHello - Server selects cipher and parameters
3. Certificate Exchange - Server sends certificate (if needed)
4. Key Exchange - Parties establish shared secret
5. Finished - Both sides verify handshake integrity

DTLS adds retransmission logic: if no response within timeout, messages are resent with exponential backoff.

QUIC Protocol Support

OpenSSL provides comprehensive QUIC protocol support starting from version 3.2 for clients and 3.5 for servers. QUIC is a modern, UDP-based secure transport protocol that serves as the foundation for HTTP/3 and offers advantages like connection migration, multiple concurrent streams, and faster connection establishment compared to TLS over TCP.

Architecture Overview

The QUIC implementation follows a layered architecture with clear separation between the API personality layer (APL) and the QUIC core:

Loading diagram...

Core Components

QUIC_ENGINE is the top-level object representing an event processing domain. It owns zero or more QUIC_PORT instances, each representing a UDP socket. The engine manages the asynchronous reactor that drives all QUIC processing.

QUIC_PORT corresponds to a listening port or network BIO. It owns zero or more QUIC_CHANNEL instances and handles packet demultiplexing via the QUIC demuxer (QUIC_DEMUX), which routes incoming packets to the correct connection based on connection IDs.

QUIC_CHANNEL represents a single QUIC connection and ties together all connection-level components: TLS handshake layer, stream management, flow control, congestion control, and packet processing. It maintains separate send and receive crypto streams for each protection level (Initial, Handshake, 1-RTT).

QUIC_STREAM tracks individual QUIC streams. Send streams (QUIC_SSTREAM) manage buffering and frame generation, while receive streams (QUIC_RSTREAM) handle reassembly and flow control. Streams are exposed to applications via QUIC_XSO (QUIC Stream SSL Object) wrappers.

TLS Integration

QUIC uses TLS 1.3 for its handshake, but with modifications for the QUIC transport. The QUIC_TLS object wraps an SSL connection and provides:

• Crypto stream handling: TLS handshake data flows through dedicated crypto streams rather than TLS records
• Record layer adaptation: Custom record layer implementation (quic_tls.c) bridges TLS and QUIC framing
• Transport parameters: QUIC-specific parameters exchanged during the handshake via TLS extensions
• Key derivation: Yields encryption keys at each protection level for packet protection

Packet Processing Pipeline

Incoming packets flow through the RX path: network BIO → QUIC_DEMUX → QUIC_CHANNEL → RX depacketizer → frame handlers. Outgoing data flows through the TX path: application writes → stream buffers → TX packetizer → encryption → network BIO.

API Usage

Applications interact with QUIC through SSL-like methods:

const SSL_METHOD *method = OSSL_QUIC_client_method();
SSL_CTX *ctx = SSL_CTX_new(method);
SSL *conn = SSL_new(ctx);
SSL_set_connect_state(conn);
SSL_do_handshake(conn);

For servers (OpenSSL 3.5+), use OSSL_QUIC_server_method(). Thread-assisted mode is available via OSSL_QUIC_client_thread_method() for non-blocking operation.

Key Features

• Multiple streams: Bidirectional and unidirectional streams with independent flow control
• Connection migration: Connections survive IP address changes via connection ID mechanism
• 0-RTT support: Early data transmission before handshake completion
• Congestion control: NewReno algorithm with extensible framework
• Flow control: Per-stream and connection-level limits
• Error handling: QUIC-specific error codes (RFC 9000) for transport and application errors

Cryptographic Algorithms & EVP

The EVP (Envelope) API provides a unified, high-level interface for cryptographic operations. It abstracts away algorithm-specific details, allowing applications to work with symmetric encryption, hashing, digital signatures, and key management through consistent function calls.

EVP Architecture

The EVP layer sits above algorithm-specific implementations. Applications call EVP functions like EVP_EncryptInit_ex(), EVP_DigestInit_ex(), and EVP_PKEY_sign(), which dispatch to the appropriate underlying algorithm. This design enables algorithm agility—applications can switch ciphers or hash functions without code changes.

Symmetric Encryption

EVP supports multiple cipher modes for block ciphers like AES:

• Block modes: ECB, CBC, CFB, OFB, CTR, XTS
• AEAD modes: GCM, CCM, OCB, SIV, GCM-SIV (authenticated encryption with associated data)
• Stream ciphers: ChaCha20, ChaCha20-Poly1305

Each mode is initialized via EVP_CipherInit_ex() with a cipher object (e.g., EVP_aes_256_gcm()). The context maintains state across EVP_CipherUpdate() calls for incremental processing. AEAD modes require additional authenticated data via control operations before finalization.

Message Digests

The digest interface (EVP_MD_CTX) handles cryptographic hashing. Functions like EVP_DigestInit_ex(), EVP_DigestUpdate(), and EVP_DigestFinal_ex() process data incrementally. Supported algorithms include SHA-256, SHA-512, SHA-3, and others. The EVP_Digest() convenience function performs one-shot hashing.

Asymmetric Cryptography

EVP_PKEY abstracts RSA, ECDSA, DSA, and other asymmetric algorithms. Key operations include:

• RSA: Public/private encryption, PKCS#1 padding, PSS padding
• ECDSA: Deterministic and randomized signing, curve-specific operations
• Key management: Loading from PKCS#8, encoding/decoding via providers

Digital signatures use EVP_DigestSignInit() and EVP_DigestVerifyInit() for combined digest-and-sign workflows.

Hardware Acceleration

The implementation includes platform-specific optimizations:

• x86-64: AES-NI, PCLMULQDQ (GCM), AVX
• ARM: NEON for AES and GCM
• s390x: Dedicated crypto instructions for AES, GCM, CCM
• RISC-V: Vector extensions for AES and SM4

Algorithm selection happens at runtime via CPU capability detection, ensuring optimal performance without manual configuration.

Loading diagram...

Key Derivation & Encoding

EVP supports PKCS#5/PBKDF2 for password-based key derivation and PKCS#8 for private key encoding. The EVP_PKEY2PKCS8() and EVP_PKCS82PKEY() functions convert between internal and standard formats, enabling interoperability with other cryptographic libraries.

Provider Architecture & Implementations

OpenSSL 3.0+ uses a modular provider architecture where cryptographic algorithms are organized into pluggable providers. Each provider is a container of algorithm implementations that can be loaded dynamically or built-in.

Core Provider Types

Default Provider (defltprov.c) is the primary provider containing all standard OpenSSL algorithms. It is automatically loaded unless another provider is explicitly loaded first. This is a built-in provider compiled directly into libcrypto.

FIPS Provider (fips/fipsprov.c) contains a validated subset of algorithms conforming to FIPS 140-2 standards. It includes self-test capabilities and security checks. The FIPS provider can operate independently or alongside other providers.

Legacy Provider (legacyprov.c) contains deprecated algorithms like MD2, MD4, RC2, RC4, DES, and Blowfish. It is not loaded by default, allowing applications to explicitly opt-in to legacy algorithms for backward compatibility.

Base Provider (baseprov.c) supplies non-cryptographic utilities like key encoders and decoders. It should always be loaded when not using the default provider.

Implementation Organization

Algorithm implementations are organized by type in providers/implementations/:

• Ciphers (ciphers/) - Block ciphers (AES, ChaCha20, 3DES) and stream ciphers with various modes (CBC, GCM, CCM, XTS)
• Digests (digests/) - Hash functions (SHA-2, SHA-3, BLAKE2, MD5)
• Signature (signature/) - Digital signatures (RSA, ECDSA, EdDSA, ML-DSA)
• Key Management (keymgmt/) - Key generation and manipulation for RSA, EC, DH, DSA
• Key Exchange (exchange/) - ECDH, DH, and other key agreement protocols
• KDFs (kdfs/) - Key derivation functions (PBKDF2, HKDF, Argon2, Scrypt)
• MACs (macs/) - Message authentication codes (HMAC, CMAC, Poly1305)
• Encode/Decode (encode_decode/) - Key serialization (PEM, DER, PKCS12)

Provider Context & Initialization

Each provider maintains a context (PROV_CTX) that holds:

• Reference to the library context (OSSL_LIB_CTX)
• Core function pointers for memory allocation and error handling
• BIO method for provider I/O operations
• Handle to the core library

Providers implement the OSSL_provider_init() function which:

1. Receives core function pointers from OpenSSL
2. Creates and initializes provider context
3. Registers available algorithms via OSSL_ALGORITHM structures
4. Returns dispatch table with provider callbacks

Algorithm Query & Selection

When an application requests an algorithm, the core library:

1. Queries each loaded provider via query_operation() callback
2. Receives algorithm metadata including names, properties, and function pointers
3. Selects the best match based on algorithm name and property filters
4. Calls the provider's implementation function

Properties enable fine-grained algorithm selection. For example, provider=fips,fips=yes selects FIPS-approved algorithms from the FIPS provider.

/* Example: Query for AES-256-GCM */
EVP_CIPHER *cipher = EVP_CIPHER_fetch(libctx, "AES-256-GCM", "provider=default");

Hardware Acceleration

Implementations support platform-specific optimizations through conditional compilation:

• x86-64 - AES-NI, AVX-512 for AES-GCM
• ARM - NEON/ARMv8 for AES, SHA
• RISC-V - RV32I/RV64I extensions
• s390x - Native hardware crypto instructions

Each cipher/digest has generic C code plus optional hardware-specific .inc files that override performance-critical sections.

X.509 Certificates & PKI

Certificate Structure

X.509 certificates in OpenSSL are represented by the X509 structure, which contains the certificate information (X509_CINF), signature algorithm, and the signature itself. The certificate info includes the version, serial number, issuer and subject names, validity period, public key, and optional extensions. Certificates are encoded in ASN.1 DER format and can be serialized to PEM (Privacy Enhanced Mail) format for transport.

Certificate Verification Pipeline

The verification process is orchestrated by X509_STORE_CTX and follows a multi-stage pipeline:

1. Chain Building (build_chain()) - Constructs a certificate chain from the leaf certificate to a trusted root by searching the untrusted certificate stack and trust store.
2. Extension Validation (check_extensions()) - Validates critical extensions and enforces constraints like name constraints and basic constraints.
3. Revocation Checking (check_revocation()) - Checks certificate status via CRL or OCSP.
4. Signature Verification (internal_verify()) - Verifies each certificate's signature using its issuer's public key.
5. Trust Evaluation (check_trust()) - Confirms the root certificate is trusted.

Certificate Extensions (X.509v3)

Extensions provide additional certificate properties and constraints. The v3_lib.c module manages extension registration and lookup. Common extensions include:

• Basic Constraints - Indicates if the certificate is a CA and maximum path length
• Key Usage - Restricts key usage (signing, encryption, etc.)
• Extended Key Usage - Specifies purposes (TLS server auth, code signing, etc.)
• Subject Alternative Name - Additional identities for the certificate
• Authority Key Identifier - Links to the issuer's public key
• CRL Distribution Points - URLs for certificate revocation lists

ASN.1 Encoding & Decoding

Certificates use ASN.1 DER encoding. The asn1_lib.c module provides low-level encoding/decoding primitives. tasn_dec.c and tasn_enc.c implement generic ASN.1 template-based serialization. PEM encoding wraps DER data in Base64 with header/footer markers.

CRL & Revocation Handling

Certificate Revocation Lists (CRLs) are managed by the X509_CRL structure. The verification pipeline scores CRLs based on issuer match, time validity, and scope. Delta CRLs (incremental updates) are supported. OCSP (Online Certificate Status Protocol) provides real-time revocation checking as an alternative to CRLs.

Trust Store & Lookup

The X509_STORE maintains trusted root certificates and lookup methods. Lookup can be file-based (by_file.c), directory-based (by_dir.c), or store-based (by_store.c). The trust evaluation checks explicit trust/reject settings in certificate auxiliary data before falling back to purpose-based checks.

Loading diagram...

Key Comparison & Hashing

The x509_cmp.c module provides certificate comparison functions. X509_issuer_and_serial_cmp() compares certificates by issuer name and serial number. Hash functions compute digests of issuer/serial pairs for efficient certificate lookup in stores.

Key Generation & Management

Overview

OpenSSL provides a unified key generation framework through the EVP (Envelope) API, which abstracts algorithm-specific implementations behind a provider-based architecture. Key generation involves three main phases: initialization, parameter configuration, and key material generation using cryptographically secure random numbers.

EVP_PKEY_CTX: The Generation Context

The EVP_PKEY_CTX structure is the central control point for key generation. It manages:

• Algorithm selection via EVP_PKEY_CTX_new_from_name() - fetches the appropriate key manager (keymgmt) from providers
• Generation state - tracks whether the context is initialized for parameter or key generation
• Provider integration - delegates actual generation to provider implementations
• Callbacks - supports progress callbacks via EVP_PKEY_CTX_set_cb()

Generation Workflow

// 1. Create context for algorithm
EVP_PKEY_CTX *ctx = EVP_PKEY_CTX_new_from_name(libctx, "RSA", propq);

// 2. Initialize for key generation
EVP_PKEY_keygen_init(ctx);

// 3. Set algorithm parameters (e.g., key size)
EVP_PKEY_CTX_set_int(ctx, "bits", 2048);

// 4. Generate the key
EVP_PKEY *pkey = NULL;
EVP_PKEY_keygen(ctx, &pkey);

// 5. Cleanup
EVP_PKEY_CTX_free(ctx);

Algorithm-Specific Generation

RSA: Uses ossl_rsa_generate_multi_prime_key() which generates prime numbers using BN_generate_prime_ex2(). Supports multi-prime keys (more than 2 primes) and includes pairwise consistency tests in FIPS mode.

Elliptic Curve: EC_KEY_generate_key() generates a random private key scalar and computes the corresponding public key point via scalar multiplication: pub_key = priv_key * G.

DH: DH_generate_parameters_ex() generates safe primes and a generator. In FIPS mode, only named groups are allowed; legacy mode supports custom parameter generation.

Post-Quantum: ML-KEM and ML-DSA use deterministic generation from seeds via RAND_priv_bytes_ex() for entropy.

Random Number Generation Integration

Key generation depends on RAND_priv_bytes_ex() for cryptographically secure random material:

• Private bytes - used for sensitive key material (private keys, seeds)
• Strength parameter - specifies required entropy bits (typically 128 or 256)
• DRBG backend - uses deterministic random bit generators with entropy sources
• Provider support - can delegate to custom random providers

// Generate 32 bytes of private random material
RAND_priv_bytes_ex(libctx, buffer, 32, 256);

Provider Architecture

The provider system enables pluggable key generation:

1. gen_init() - initializes generation context with algorithm parameters
2. gen_set_params() - configures generation options (key size, curve, etc.)
3. gen() - performs actual key generation
4. gen_cleanup() - releases generation context

Providers implement these callbacks in their keymgmt modules (e.g., rsa_kmgmt.c, ec_kmgmt.c).

FIPS Compliance

FIPS mode enforces stricter requirements:

• Pairwise consistency tests - verify generated keys work correctly
• SP 800-56B compliance - RSA generation follows NIST standards
• Named groups only - DH restricted to approved parameter sets
• Entropy validation - health tests on random sources

Command-Line Usage

The genpkey application demonstrates practical key generation:

# Generate 2048-bit RSA key
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out key.pem

# Generate EC key on P-256 curve
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out key.pem

# Generate parameters separately, then key
openssl genpkey -genparam -algorithm DH -pkeyopt dh_paramgen_prime_len:2048 -out params.pem
openssl genpkey -paramfile params.pem -out key.pem

Command-Line Tools & Applications

OpenSSL provides a comprehensive suite of command-line tools for cryptographic operations, certificate management, and TLS/SSL testing. The command dispatcher in openssl.c serves as the entry point, routing user commands to specialized implementations.

Command Registration & Dispatch

The command system uses a dynamic registration mechanism. Each command implements a *_main() function and an OPTIONS array for help text. The progs.pl Perl script scans all command source files to generate progs.h and progs.c, which register commands in a FUNCTION structure array.

typedef struct function_st {
    FUNC_TYPE type;           /* FT_general, FT_md, FT_cipher, etc. */
    const char *name;         /* Command name */
    int (*func)(int argc, char *argv[]);  /* Main function pointer */
    const OPTIONS *help;      /* Help/options definition */
    const char *deprecated_alternative;
    const char *deprecated_version;
} FUNCTION;

Commands are stored in an LHASH (hash table) for O(1) lookup. The dispatcher (do_cmd()) retrieves the appropriate command and invokes its main function.

Core Command Categories

General Commands include ca, req, x509, verify, and crl for certificate lifecycle management. The ca command manages a certificate authority, handling certificate signing, revocation, and database operations. The req command generates certificate signing requests and self-signed certificates.

TLS Testing Tools (s_client, s_server) enable SSL/TLS protocol testing. s_client connects to remote servers for debugging handshakes and cipher negotiation. s_server listens for incoming connections, useful for testing client implementations.

Cryptographic Operations include dgst (message digests), enc (symmetric encryption), pkey (key manipulation), and genpkey (key generation). These commands leverage the provider architecture for algorithm selection.

Shared Infrastructure

The apps/lib/apps.c module provides common utilities: BIO management for input/output, configuration file loading, password callbacks, and X.509 certificate utilities. Global BIO pointers (bio_in, bio_out, bio_err) handle standard I/O streams.

The opt.c module implements command-line option parsing with a declarative OPTIONS array format, supporting flags, arguments, and help generation.

Initialization & Cleanup

apps_startup() initializes OpenSSL, loads configuration, and sets up the UI method for password prompts. apps_shutdown() cleans up providers and library contexts. Signal handling (SIGPIPE) prevents crashes on broken pipes during network operations.

Dynamic Algorithm Discovery

Commands like dgst and enc support dynamic algorithm discovery. If a command name matches a registered digest or cipher algorithm, the dispatcher automatically routes it to the appropriate handler (dgst_main or enc_main), enabling transparent support for new algorithms added via providers.

FIPS Validation & Security Module

Overview

The FIPS module is a cryptographic provider that implements FIPS 140-3 validated algorithms and enforces strict security requirements. It operates as a dynamically loadable shared library (fips.so on Unix, fips.dll on Windows) and must be built from FIPS-validated source code to maintain compliance.

Module Initialization & Self-Tests

When the FIPS provider loads, it executes critical self-tests to verify module integrity:

1. Module Integrity Check: HMAC-SHA256 verification of the provider binary itself
2. Known Answer Tests (KATs): Cryptographic algorithm validation tests
3. DRBG Tests: Random number generator self-tests
4. Deferred Tests: Algorithm-specific tests run on first use (OpenSSL 3.5+)

The SELF_TEST_post() function orchestrates these tests. It maintains a state machine (FIPS_state) tracking whether the module is initializing, running, or in error state. Thread-safe locking ensures only one test sequence executes at a time.

FIPS Indicators & Approved Mode Tracking

FIPS indicators track whether operations use only approved algorithms and parameters. The OSSL_FIPS_IND structure maintains approval status for each algorithm context:

typedef struct ossl_fips_ind_st {
    unsigned char approved;
    signed char settable[OSSL_FIPS_IND_SETTABLE_MAX];
} OSSL_FIPS_IND;

When an operation fails FIPS compliance checks (e.g., insufficient key size, unapproved digest), the indicator is marked unapproved. Applications can register callbacks via OSSL_INDICATOR_set_callback() to log or handle non-approved operations.

Security Checks

The security check layer validates algorithm parameters against FIPS requirements:

• RSA Key Checks: Minimum key size enforcement (2048-bit minimum for signing)
• EC Curve Validation: Only approved curves allowed (P-256, P-384, P-521, etc.)
• Digest Restrictions: SHA-1 disallowed for signatures; SHA-512 truncation restricted
• HMAC/KMAC Key Validation: Minimum key length enforcement
• KDF Digest Checks: Approved digests required for HKDF, TLS-PRF, SSHKDF

These checks are configurable via FIPS module configuration parameters, allowing tolerant mode for testing while maintaining strict mode by default.

Configuration & Installation

The FIPS module requires a configuration file (fipsmodule.cnf) containing:

• Module checksum for integrity verification
• Self-test status (for OpenSSL 3.1.2)
• Indicator configuration parameters
• Security check settings

The openssl fipsinstall command generates this configuration by running self-tests on the target machine. This ensures the module operates correctly in its deployment environment.

Deferred Testing Architecture

Modern FIPS implementations defer non-critical tests until algorithms are actually used, improving startup performance. The deferred test system:

• Tracks test dependencies (some tests must run before others)
• Executes tests on-demand with thread-safe locking
• Maintains test state (pending, in-progress, passed, failed)
• Supports both immediate and deferred KAT execution

This allows faster provider initialization while maintaining full FIPS compliance.