C++20 for real-time services: a processing pipeline and WebSocket streaming

“Real-time” is one of the most abused words in backend engineering. In most job descriptions it means “fast, ideally”. In systems work it means something narrower and more demanding: bounded latency, and behaviour you can predict at the worst case rather than the average.

A service that answers in 2 ms on the median and 400 ms at the 99.9th percentile is not real-time. A service that answers in a reliable 20 ms, always, is.

That distinction drives every decision below.

What makes C++ the right tool — and when it is not

C++ earns its place here for one reason: deterministic resource management. No garbage collector means no pause you did not schedule. RAII means a socket closes when the scope ends, not when a finaliser feels like it.

That is worth a lot when your latency budget is measured in single-digit milliseconds and you care about the tail, not the mean.

It is worth much less if your pipeline spends most of its time waiting on a database or an HTTP call. If the dominant cost is I/O, a managed runtime will get you there with a fraction of the effort, and the GC pauses will disappear into the network jitter. Choosing C++ for a service whose p99 is dominated by a SQL round-trip is choosing the hard tool for the easy part of the problem.

How do you structure a processing pipeline?

The shape that works is a sequence of stages, each with a single responsibility, connected by explicit queues. In a vital-signs monitoring service that looks like:

sensor → [Filter] → [Validate] → [Analyze] → [Alert] → sink

Each stage does one thing, and the boundaries between them are where you get to make decisions — about buffering, about threading, about what happens when the next stage is slower than this one.

The temptation is to write it as a chain of direct calls. It is simpler, and it is right up to the moment one stage becomes slow, at which point the whole pipeline runs at the speed of its worst member with no way to observe where the time went. Explicit queues cost you a little indirection and buy you the ability to answer “which stage is behind?” — which, in production, is the only question you will be asking.

Threads per stage, or one executor?

This is the first real architectural fork.

Model	Latency behaviour	Cost
One thread per stage	Predictable; stages run concurrently	Context switches; over-subscription if stages > cores
One executor, tasks per item	Excellent throughput, higher latency variance	Harder to reason about ordering
strand per connection over a shared pool	Ordering guaranteed per stream, work shared across cores	Needs discipline about what runs where

For a pipeline with four stages and many concurrent streams, the third is usually right. Boost.Asio’s strand gives you the property that actually matters: handlers on the same strand never run concurrently, so per-connection state needs no mutex, while a single io_context with a thread pool keeps all cores busy.

Getting this wrong in the other direction is the classic mistake — a thread per connection, which is fine at 50 connections and collapses at 5000.

std::jthread (C++20) is the right primitive for the long-lived stage threads you do keep: it joins in its destructor and carries a stop_token, which removes an entire category of shutdown bugs where a thread outlives the object it is reading from.

What happens when the queue is full?

This is the question that separates a design from a demo, and it has exactly three honest answers:

1. Block the producer. Correct when the producer can be slowed — a file reader, a replay. Wrong when it is a sensor: physics does not wait for your queue.
2. Drop the oldest. Correct for continuous signals where the newest reading supersedes the old one. A monitoring display wants the current value, not a faithful replay of the last four seconds.
3. Drop the newest / reject. Correct when every item must be processed or explicitly accounted for.

There is no fourth option. An unbounded queue is not a way of avoiding the choice — it is choice 1 with the blocking deferred until you run out of memory, which is the worst possible moment for it to happen.

Whatever you pick, count the drops. A pipeline that silently discards under load is indistinguishable from one that is working, right up until someone asks why the numbers do not add up.

Where does the latency actually go?

Rarely where people look first. From profiling this kind of service, the recurring offenders:

• Allocation in the hot path. A std::string built per message, a std::function capturing by value, a vector that grows one element at a time. Each is small; at 1000 messages/second each is 1000 allocations/second competing for the allocator lock.
• Serialisation. JSON encoding is often the single most expensive step in a streaming service. It is also the easiest to fix — build into a reused buffer instead of a fresh string per message.
• Logging. A synchronous log line in the hot path serialises your whole pipeline behind a file handle. Asynchronous logging is not an optimisation here, it is a correctness requirement for the latency budget.
• False sharing. Two atomics that happen to live on the same cache line will destroy the scaling you expected from adding threads. alignas(std::hardware_destructive_interference_size) is the textbook fix — but be aware that GCC warns about using it in headers (-Winterference-size), because its value is an ABI commitment that can differ between translation units built for different targets. A project-defined constant is often the more honest choice. Either way, measure before and after rather than applying it by reflex.

The order matters: measure before you touch any of them. Every one of these is real, and on any given service two of them will be irrelevant.

What does C++20 actually change here?

Less than the marketing suggests, and more than nothing.

Concepts replace SFINAE for constraining a stage’s interface, and — more usefully — turn a page of template instantiation errors into one line naming the requirement that failed. On a pipeline that is generic over sample types, that alone justifies the standard bump.

Ranges make the transformation stages read like what they are, and the views compose lazily rather than materialising an intermediate vector per step.

std::span is the one I would keep if I could keep only one. Passing a buffer as span<const std::byte> instead of a pointer and a length removes a whole class of mismatch bugs at zero runtime cost.

std::jthread and stop_token for cooperative shutdown, as above.

Coroutines are the genuinely transformative feature for asynchronous I/O — Asio’s co_spawn with use_awaitable turns callback chains into linear code. But they are also where you can quietly reintroduce allocation, since a coroutine frame is heap-allocated unless the compiler elides it. Worth adopting; worth measuring after you do.

What C++20 does not change: none of it answers what happens when the queue is full.

What about the WebSocket end?

Streaming to connected clients has its own failure mode, and it is the one people miss: a slow client must not slow the pipeline.

If your write path is “for each client, send”, one client on a bad mobile connection applies backpressure all the way back to the sensor. The fix is a per-connection outbound queue with its own bounded policy — usually “drop the oldest”, since a live monitor wants the current value and has no use for a stale one.

Two implementation details that bite with Boost.Beast specifically:

• one outstanding async_write per stream. Beast requires this; issuing a second write before the first completes is undefined behaviour, not a queueing mechanism. You need your own outbound queue and a single in-flight write drained from it.
• serialise once, send many. Encoding the same payload separately for each of 200 connected clients is 200× the work for one result. Encode into a shared immutable buffer, then hand each connection a reference to it.

And keep a ping/idle timeout. A TCP connection to a client that vanished without a FIN will sit there consuming a slot indefinitely.

How do you test something with timing in it?

Timing-dependent tests are the flakiest thing you can put in CI, and the fix is to remove the timing rather than to add sleeps.

• Inject the clock. A stage that takes a clock as a dependency can be tested with a fake one, deterministically, at whatever rate you like.
• Test stages in isolation. Each stage is a pure-ish transformation; those tests are ordinary unit tests with no concurrency at all.
• Test the queue policy explicitly. Fill it, overfill it, assert the drop counter. This is the behaviour most likely to be wrong and least likely to be covered.
• Run the integration test under sanitizers. ThreadSanitizer finds the data race that shows up once a fortnight in production; no amount of running the test normally will.
• Never assert on wall-clock durations. EXPECT_LT(elapsed, 50ms) will pass on your machine and fail on a loaded CI runner, and you will end up deleting it. Assert on ordering and on counters instead.

The through-line

Most of this is not about C++. Bounded queues, an explicit policy at every boundary, keeping a slow consumer from infecting the producer, measuring before optimising — those apply equally to a Go service or a .NET one.

What C++ gives you is the ability to make the resulting latency predictable, because nothing runs that you did not schedule. What it charges you is that every one of those decisions is yours, explicitly, including the ones another runtime would have made silently on your behalf. That trade is worth taking when the tail latency is the requirement. It is a poor trade when it is not.

If you are building a service where the p99 is the specification rather than a nice-to-have, that is the kind of problem I enjoy talking through.