Pub/Sub Messaging in Redis

Redis provides a high-performance messaging system implementing the Publisher/Subscriber pattern. This pattern decouples message senders (publishers) from receivers (subscribers), enabling real-time, asynchronous communication between different components of a distributed system.

However, Redis Pub/Sub operates on a strict fire-and-forget model, making its internal architecture, limitations, and use cases fundamentally different from persistent message queues.

1. Internal Architecture: How Redis Manages Subscriptions

Under the hood, Redis uses two distinct structures inside the global redisServer struct to manage subscriptions.

A. Direct Channel Subscriptions (pubsub_channels)

Redis maintains a dictionary called pubsub_channels.

• Key: The channel name (string).
• Value: A linked list of client pointers currently subscribed to that channel.
• Performance:
  • Subscribing (SUBSCRIBE): Adds the client to the channel's list. Operates in $O(1)$ complexity.
  • Publishing (PUBLISH): Performs a dictionary lookup to find the channel list and iterates over the clients to write the message into their output buffers. Operates in $O(1)$ lookup and $O(M)$ write where $M$ is the number of subscribers.

B. Pattern Matching Subscriptions (pubsub_patterns)

Clients can subscribe to multiple channels matching a glob-style pattern using PSUBSCRIBE (e.g., PSUBSCRIBE news_*).

• Instead of a dictionary, Redis stores these in a flat linked list called pubsub_patterns.
• Each node in the list stores the client pointer and the pattern string.
• Performance Alert: When a message is published, Redis must iterate through every single node in the pubsub_patterns list to check if the published channel name matches the pattern.
• Warning: Pattern matching runs in $O(N)$ where $N$ is the total number of pattern subscriptions across all clients. Heavy usage of PSUBSCRIBE can degrade Redis performance under high message rates.

2. Technical Constraints & Gotchas

A. The "Blocking Client" Mode

Once a client connection executes a SUBSCRIBE or PSUBSCRIBE command, it enters subscription mode.

• In this mode, the client is blocked from executing regular commands (e.g., GET, SET, INCR). Trying to do so results in a protocol error.
• The only allowed commands are: SUBSCRIBE, UNSUBSCRIBE, PSUBSCRIBE, PUNSUBSCRIBE, PING, and QUIT.
• Developer Rule: You must allocate at least two separate Redis connection clients in your application code: one dedicated to listening for messages (subscribing) and another for regular queries and publishing.

B. Fire-and-Forget (No Persistence)

Redis Pub/Sub does not store messages. If a publisher sends a message to channel_A and no subscriber is connected, the message is instantly discarded.

• If a subscriber disconnects due to a brief network blip, all messages sent during that outage are permanently lost.
• If you need durability, replay history, or delivery guarantees, you should use Redis Streams or Redis Lists instead.

C. Buffer Bloat & Connection Drops (The Slow Consumer Problem)

When a message is published, Redis pushes it directly into the output buffer of each subscribed client's TCP socket.

• If a subscriber is slow to read (e.g., due to Node.js event loop lag or heavy processing), the output buffer inside Redis RAM grows.
• To prevent Redis from running out of memory, it enforces strict client output buffer limits:

  client-output-buffer-limit pubsub 32mb 8mb 60
• Result: If a subscriber's buffer exceeds 32MB immediately, or remains above 8MB for 60 consecutive seconds, Redis force-kills the subscriber's TCP connection, leading to lost messages and client-side reconnect loops.

3. Node.js Implementation (using ioredis)

Because subscribing blocks the connection, the following example demonstrates how to implement Redis Pub/Sub using separate publisher and subscriber connections:

import Redis from 'ioredis';
 
// 1. Separate connections are mandatory
const pubClient = new Redis({ host: '127.0.0.1', port: 6379 });
const subClient = new Redis({ host: '127.0.0.1', port: 6379 });
 
const CHANNEL = 'notifications:orders';
 
// 2. Setup Subscriber
subClient.subscribe(CHANNEL, (err, count) => {
  if (err) {
    console.error('Failed to subscribe:', err);
    return;
  }
  console.log(`Subscribed successfully! Listening on ${CHANNEL}. Total channel subscriptions: ${count}`);
});
 
// Listen for incoming messages
subClient.on('message', (channel, message) => {
  console.log(`Received message from channel "${channel}":`, message);
  
  // Note: Do NOT attempt "pubClient.get()" or other commands here on "subClient"!
  // Always use the regular client "pubClient" for query operations.
});
 
// 3. Setup Publisher (can execute regular queries and publish messages)
async function publishEvent() {
  const payload = JSON.stringify({
    orderId: 'ORD-9872',
    status: 'SHIPPED',
    timestamp: new Date().toISOString()
  });
 
  // Publish to channel
  const activeSubscribers = await pubClient.publish(CHANNEL, payload);
  console.log(`Event published. Received by ${activeSubscribers} active subscribers.`);
  
  // Regular operations can run on this client
  await pubClient.set('last_published_order', 'ORD-9872');
}
 
setTimeout(publishEvent, 2000);

4. Advanced Patterns: Distributed Traceability & Compile-Time Type Safety

As systems scale and complexity increases, raw Pub/Sub mechanisms present two major challenges: silent flow breaks (losing request visibility across network boundaries) and silent syntax bugs (data shape mismatch between publisher and subscriber).

Two patterns address these issues in production.

A. Distributed Traceability (Context Propagation)

In distributed systems, a request might be processed synchronously by an API before publishing an event to notify downstream workers asynchronously. Normally, when you publish a raw message, the trace parent context is lost. Downstream consumers will execute the task under a new, disconnected Trace ID, breaking the end-to-end observability pipeline.

The Solution: The Metadata Envelope

To maintain context, publishers wrap the actual transaction payload in an envelope containing tracking metadata (like traceId and spanId), which is then unpacked by the subscriber to start a linked child span.

// 1. Publisher wraps payload with OpenTelemetry context
async function publishOrderCreated(orderPayload: any) {
  const messageEnvelope = {
    metadata: {
      traceId: activeSpan.context().traceId, // Passes current Trace ID
      spanId: activeSpan.context().spanId,   // Passes Parent Span ID
      timestamp: new Date().toISOString()
    },
    payload: orderPayload
  };
 
  await pubClient.publish('order:created', JSON.stringify(messageEnvelope));
}
 
// 2. Subscriber unpacks metadata and propagates tracing context
subClient.on('message', (channel, messageJson) => {
  const { metadata, payload } = JSON.parse(messageJson);
  
  // Reconstruct telemetry context from the metadata envelope
  const parentSpanContext = extractSpanContext(metadata.traceId, metadata.spanId);
  
  startActiveSpan('process_order_event', { parent: parentSpanContext }, (childSpan) => {
    // Process payload (logs generated here will share the same Trace ID!)
    processOrder(payload);
    childSpan.end();
  });
});

B. Compile-Time Type Safety (Avoid Runtime Mismatch)

By default, Redis Pub/Sub channels and payloads are plain strings. A publisher might change a field name (e.g. changing orderId to id), compiling without error but immediately breaking the subscriber logic at runtime.

The Solution: Typed Mappings and TypeScript Generics

You can enforce strict type safety by defining a schema contract mapping channel names to their exact payload shapes, wrapping the standard client inside a type-safe wrapper class.

// 1. Define the Global Pub/Sub Schema Contract
interface PubSubSchema {
  "order:created": { orderId: string; totalAmount: number };
  "user:registered": { email: string; signupTimestamp: string };
}
 
// 2. Build the Type-Safe Wrapper
class TypeSafePubSub {
  constructor(private pub: Redis, private sub: Redis) {}
 
  // Enforces that channel and payload must strictly match the schema at compile time
  async publish<K extends keyof PubSubSchema>(
    channel: K,
    payload: PubSubSchema[K]
  ): Promise<number> {
    return this.pub.publish(channel, JSON.stringify(payload));
  }
 
  // Enforces callback payload typing based on the subscribed channel
  subscribe<K extends keyof PubSubSchema>(
    channel: K,
    callback: (payload: PubSubSchema[K]) => void
  ): void {
    this.sub.subscribe(channel);
    this.sub.on('message', (chan, message) => {
      if (chan === channel) {
        callback(JSON.parse(message));
      }
    });
  }
}
 
// 3. Execution (Compiler enforces safety)
const typedPubSub = new TypeSafePubSub(pubClient, subClient);
 
// SUCCESS: Compiles perfectly
typedPubSub.publish("order:created", { orderId: "ORD-12", totalAmount: 45.99 });
 
// COMPILE ERROR: Typo in channel name ("order:creatd") or invalid payload shape
// typedPubSub.publish("order:creatd", { orderId: "ORD-12" }); 

5. Comparing Pub/Sub with Alternative Patterns