Seng-Wei Chieh

Same GPU, Same Impossible Job: One Engine Crashed, the Other Slowed Down 8%

schieh710 — Sat, 05 Sep 2026 18:13:04 GMT

I gave two inference engines the same job on the same 8 GB GPU: 64 requests, each with a 2,000-word prompt. Neither had enough memory for it — the job needed about a third more cache than the card could hold.

One returned nothing. It asked for 2.19 GB it didn't have, and stopped.

The other returned all 64 answers, complete, 8% slower than its own best pace.

That pairing is the most useful thing this benchmark produced, and it isn't really a story about speed. I'm building a paged memory manager for LLM inference and wanted a baseline I could defend first.

Setup: Qwen2.5-1.5B on an RTX 4060 laptop GPU — 8 GB, measured bandwidth 219.8 GB/s. Same model and settings both sides, prompts fixed by seed so nothing could be cached and reused, median of three runs. Because this 105 W part overheats, every cell waits for the GPU to cool to 70 °C first — otherwise whichever engine ran second would be measured on a hot card.

1. At batch 1, the gap has nothing to do with memory

Prompt length	HF	vLLM	vLLM/HF	HF first token	vLLM first token
64 words	35.6 tok/s	66.8 tok/s	1.88×	31 ms	32 ms
512	36.1 tok/s	65.7 tok/s	1.82×	71 ms	80 ms
2048	33.6 tok/s	63.1 tok/s	1.88×	337 ms	340 ms

I expected these to be close. With one request in flight there is no queue and nothing to schedule — yet vLLM is a stable 1.86× faster. That gap is code quality, not memory management, which makes it the number to subtract before crediting anything clever:

Where the advantage comes from	Size	Shows up when
Better code	~1.86×	Always, even at one request
Smarter memory + scheduling	~1.8× more	Only as requests pile up
Not dying under pressure	—	At 64 requests

vLLM's first token is never faster, and is 13% slower mid-range — the throughput-for-latency trade, visible before any load.

2. Under load: where each one breaks

Batch	HF tok/s	vLLM tok/s	vLLM/HF	HF peak VRAM	vLLM peak VRAM
1	31.4	61.4	1.96×	3484 MiB	7050 MiB
2	56.5	111.0	1.96×	3676 MiB	7260 MiB
4	88.8	192.4	2.17×	4094 MiB	7260 MiB
8	107.3	299.9	2.79×	4914 MiB	7260 MiB
16	122.5	413.9	3.38×	6826 MiB	7260 MiB
32	81.6	494.8	6.06×	7960 MiB	7260 MiB
64	CUDA OOM	454.4	—	—	7260 MiB

HF doesn't stop scaling — it reverses. Batch 32 is 33% slower than 16, at 7960 of 8188 MiB. Reproduced across four passes.

At 64 the job fits in neither engine — 147,456 tokens of cache needed against a 108,000-token capacity. HF returns nothing; vLLM sets part aside, finishes the rest, rebuilds the discarded work, and delivers all 64 answers for an 8% cost. Running out of memory isn't what separates these engines. How they behave when they run out is.

The biggest surprise: vLLM is the larger memory consumer for most of the sweep — 2× HF's at batch 1, dropping below only at 32. Its flat 7260 MiB is a design choice, not thriftiness: it claims a fixed budget at startup and never asks for more, then hands out slots inside that wall. Static outside, dynamic inside. HF discovers its ceiling by hitting it. Had I written this up as "vLLM is more memory-efficient," my own table would have contradicted me.

3. The number most people quote is the wrong one

Throughput above divides output words by the whole clock — including reading the prompt, up to 60% of the time here. Split the phases, and compare each against the hardware's physical limit:

Batch	HF generation	vs. limit	vLLM generation	vs. limit	vLLM/HF
1	32.3	46%	65.8	94%	2.04×
8	123.4	25%	462.6	94%	3.75×
16	146.3	17%	809.2	93%	5.53×
32	144.3	10%	1222.4	87%	8.47×
64	OOM	—	1014.1	49%	—

A 5.53× gap could mean vLLM is exceptional or HF is poor. The absolute column settles it: vLLM runs at 87–94% of what the memory bus physically permits. That also bounds my own project — nobody meaningfully beats 94% of a ceiling, so the opportunity is in the phases that ceiling doesn't cover. More useful than any speedup, and it changed what I plan to build.

4. What actually broke at batch 32

Batch	Total	Reading prompt	Share	Generating	Gen tok/s	Prompt-phase growth
1	8.2 s	0.25 s	3%	7.9 s	32.3	—
4	11.5 s	1.17 s	10%	10.4 s	98.5	2.24×
16	33.4 s	5.55 s	17%	27.9 s	146.3	2.17×
32	100.4 s	43.89 s	44%	56.6 s	144.3	7.91×

Prompt reading scales cleanly at ~2.2× per doubling until 32, where it jumps 7.91× for twice the work — while generation barely moves. Reading 32 prompts at once needs large temporary memory that hits the wall; generation produces one word per request per step and needs ~2,000× less. That names the failing phase rather than just reporting that memory ran out.

5. Where the memory actually goes

Cost per token of context — derived from the architecture as 28.0 KiB, measured three ways:

Method	Result	vs. derivation
HF live-memory slope	31.32 KiB/token	+11.9%
HF reserved-memory slope	53.33 KiB/token	+90.5%
vLLM's own capacity accounting	27.99 KiB/token	−0.03%

The gap between the first two is one prize: the naive allocator claims 1.70 KiB for every 1 KiB it stores. The other is padding — batches must be rectangular, so short requests are filled out to match the longest:


Real tokens	13,184
Slots occupied (16 × 2048)	32,768
Padding	59.8%
Ragged batch	118.0 tok/s
Same real tokens, evenly sized	224.5 tok/s
Cost of the padding	1.90×

Both batches carry identical real work; only the raggedness differs.

What I'd tell a team

Report the failure mode, not just the throughput. The decision-relevant number isn't 3.38× — it's that under identical impossible load, one engine returns nothing and the other returns everything 8% slower.

Measure against the hardware limit, not just the competitor. "5.5× faster" and "94% of what the bus allows" are different claims; only the second says whether headroom remains.

Check whether your own data contradicts the story you're about to tell. I nearly wrote that vLLM is more memory-efficient. It isn't — its advantage is a predictable budget, not a smaller one.

Say which number not to quote. The 6.06× at batch 32 pairs vLLM's best cell against HF's collapsed one. Real arithmetic, misleading result; the honest figure is 3.38× at 16.

What's next

A continuous-batching scheduler. Section 4 showed static batching doesn't degrade gradually — it scales cleanly, then falls off a cliff at one specific batch size. A scheduler that adds and removes requests from a running batch never has to guess that number in advance.

A paged memory manager in C++ behind Python bindings. Section 5 sized both prizes: 70% overhead from fragmentation, 59.8% from padding. Block table, free list, reference-counted blocks so shared prompt prefixes are reused rather than copied. A Python implementation of the same interface serves as correctness oracle and as the baseline for measuring what the C++ actually buys — including the language-boundary crossing cost, which is the only honest way to justify that boundary existing.

Then a streaming server to make it deployable, and a study of quantization — studied, not benchmarked, and I'll say so.

Limitations

Absolute throughput is capped by a GPU that throttles; both engines ran the same thermal gate, so the comparison stands, but a desktop card posts higher numbers. vLLM repeats within ±0.8% across passes while HF scatters up to 8.6% — quote the trend, not neighbouring cells. One confound remains, vLLM's fused kernels versus HF's stock attention, bounded at ~1.86× by section 1. And "batch size" isn't the same quantity on both sides: HF's is a simultaneous batch, vLLM's a request set its scheduler may run in waves — precisely the thing under study.

Everything here is reproducible from chieh006/paged-kv-engine:

benchmarks/ — the experiment suite, plus the commands to re-run it
benchmarks/results/ — raw records behind every number above, with per-cell GPU telemetry
benchmarks/thermal.py — the 70 °C cooldown gate every measurement waits on
benchmarks/hwinfo.py — the machine fingerprint, including the measured 219.8 GB/s

5.1× From Overlapping, 11.6× From Processes, 0× From Fewer Bytes

schieh710 — Sat, 01 Aug 2026 21:42:19 GMT

TL;DR. I had to read 10,000 files out of object storage, and only 6% of each file was needed. The obvious win is to ask for just that 6%. Measured, it was 2% slower than downloading every file whole. Fetching 17.8× fewer bytes bought nothing. What did work was overlapping the waiting (5.1×) and then spreading it over processes (11.6×), with byte-identical output throughout. One fact explains all of it: on a local link, bytes are nearly free and round trips are not. The post includes the limit whose removal made things 22% slower, the run that hung for 40 minutes, and the scoring code that confidently graded three of my own hypotheses wrong.

The setup

The workload is dull on purpose. Ten thousand synthetic image files sit in an object store. Each has a small header at the front and a small footer at the end. I need the fields out of those two ends, for all 10,000 files, written into a single Parquet file. The pixels in the middle — the overwhelming bulk of every file — are never read.

The numbers matter for what follows:

	medium corpus	small corpus
Files	10,000	10,000
Size each	~1.06 MiB	~125 KiB
Total	10.35 GiB	1.19 GiB
Actually needed per file	64 KiB	64 KiB

So on the medium corpus, 94% of every byte is dead weight. Reading only the two ends should be a straightforward win. That was the whole premise.

I built four readers to test it:

	How it reads	The bet
V1	Opens the file and reads it like a local file — the library decides what crosses the wire	the naive baseline
V2	Asks the store for the size, then asks for each end separately	fewer bytes = faster
V3	Same two ends, but many files in flight at once, feeding one writer	overlap the waiting
V5	Four copies of V3, one per process, each on its own slice	use more than one core

Six hypotheses were written down before anything was measured, and a scorecard grades each one PASS or FAIL directly from the raw result rows. That detail becomes the most embarrassing part of this post.

Stack: Python 3.12, s3fs + asyncio, pyarrow for Parquet, polars for the reporting, pydantic v2 for every config object. Ceph RGW in a container as the store. WSL2 dev box, 16 logical cores — relevant throughout.

Act 1: the optimization that bought nothing

Here is V1 against V2 on the medium corpus. Five runs each, medians.

	Bytes per file	Requests per file	Files/sec	Wall
V1 whole file	1,110,884	2.00	91.6	109.1 s
V2 just the ends	62,308	2.90	90.0	111.1 s

V2 moved 17.8× fewer bytes and finished two seconds later. On the small corpus it was 26% slower.

The bar chart is the optimization working perfectly. The stopwatch is the optimization not mattering. Both are true, and the arithmetic reconciles them in two lines.

What V2 saved. V1 drags 1.06 MiB per file across the wire. On this link that lands at roughly 270 MiB/s, so those bytes cost it about 3.9 ms per file.

What V2 spent. V2 pays 2.90 requests per file against V1's 2.00 — one to ask how big the file is, then one per end, and it re-reads a footer only where one exists. Each request costs about 3.5 ms here. That extra 0.90 of a request is about 3.2 ms per file.

3.9 against 3.2. The saving and the cost very nearly cancel, and what's left is noise plus V2 reading its files strictly one at a time.

This is not a refutation of ranged reads. It is a statement about the link. The plan for this work assumed a 1 Gbps network, where 1.06 MiB costs about 8.5 ms and the trade flips hard in V2's favour. My store was a container on the same machine — a 0.033 ms round trip. On that link the byte saving is worth about as much as the round trip it costs, and no more.

So the honest verdict is not "V2 loses." It's "this hypothesis cannot be tested on this machine." The slow-network runs are the obvious next step and they have still never been done. I'd rather say that than quietly bank a FAIL I didn't earn.

Act 2: what actually paid, and why

V3 reads the same two ends V2 does. It is 5.1× faster than both. Two changes, and only one of them is the obvious one.

The obvious one: V3 keeps many files in flight at once instead of waiting for each in turn. The waiting overlaps.

The one I think is more interesting: V3 also stops asking how big the file is. It looks the offsets up in a manifest it already has. That drops it from V2's 2.90 requests per file to 2.00 — exactly V1's count. So V3 gets the byte saving and the round-trip count of the thing it's beating. V2 could only ever have one of the two.

	Files/sec	Wall	vs V1
V1 whole file	91.6	109.1 s	1.00×
V2 just the ends	90.0	111.1 s	0.98×
V3 overlapped	467.0	21.4 s	5.10×
V5 four processes	1065.5	9.4 s	11.63×

The output is byte-identical across all four. One checksum per corpus, matching across every reader — and the small corpus checksum is the same one recorded the day before, on a completely different object store. Four read strategies, two stores, two campaigns, agreeing to the byte. Speed claims are worth nothing without that line, and it's the reason every run gets graded against rows rebuilt from the manifest before its timing is allowed to count.

One caveat on V5, stated plainly: four processes gave 2.28× over V3, not 4×. The store is running on the same physical cores as the workers. Past about four workers they start stealing CPU from the process that has to answer them, and everything shares a ceiling near 1,300 files/s. A remote store would probably change that number. I haven't measured it, and a sweep over process count is still on the undone list.

Act 3: three things I did not predict

The knee is at 4, and my planned axis could not have found it

V3's one real knob is how many files it keeps in flight. I swept it.

In flight	1	4	8	16	32	64	128
Files/sec	253	455	470	471	466	449	422
Slowest 1-in-100 request	4 ms	8 ms	14 ms	70 ms	104 ms	157 ms	2,235 ms

One to four nearly doubles the speed. Four to eight adds 3%. After that it gets slowly worse. Meanwhile the unluckiest requests degrade the entire way — from 4 ms to 2.2 seconds, a factor of 534.

The cause is not the store. It's my own program. The decoding and parsing run on one core, about 2.1 ms per file, and once four requests in flight keep that core busy, request number five just stands in line. The tell is that V3 hits 467 files/s on the medium corpus and 466 on the small one — 0.1% apart across a 16× change in data volume. If the store were the limit, size would have mattered.

Now the part that stings. The axis I planned to sweep was 1, 8, 32, 64, 128, 256, 512. There is no point between 1 and 8. That sweep would have run cleanly, produced a tidy chart, and reported the knee at 8 — by construction, because 8 was the first place it was allowed to look. I only found the real answer because the top of the axis blew up and forced me to re-cut it.

A measurement design can be wrong in a way that produces no error, no warning, and a plausible number. That is a considerably worse failure mode than a crash.

Removing a limit made it 22% slower

Between the readers and the Parquet writer sits a small buffer. I wanted to know whether its size matters, so I deliberately slowed the writer down — 5 ms per row, which puts a hard 50-second floor under a 10,000-row job — and then ran it with a buffer of 100 and a buffer of 10,000.

My prediction was that the size wouldn't matter. The writer sets the pace either way; a bigger buffer should just hold more rows in memory.

Buffer	Files/sec	Wall	Peak rows held	Peak memory
100	173.8	57.5 s	100 (pinned)	222 MiB
10,000	139.8	71.5 s	9,468	238 MiB

A 22% spread, with the bigger buffer the slower one.

The bottom panel shows the mechanism directly. With the big buffer the readers run flat out, pile up 9,468 rows by the 21-second mark, and finish. Only then does the writer start making real progress, alone, at 5 ms a row — about 47 seconds of pure drain stacked on top of the 21-second fill. That's the 71.5 seconds. With the small buffer, depth sits pinned at 100 the whole way: the readers are constantly forced to pause, the writer gets the core back, and the two stages interleave for 57.5 seconds.

Here's the bit worth keeping. Both stages share one core. The small buffer isn't saving memory, it's scheduling. It's the only thing making the readers yield to the writer. The big buffer lets the readers monopolize the core, which turns a pipeline into two sequential phases that don't overlap at all. Backpressure wasn't costing throughput — its absence was.

The memory half of this test was a dud, and I'll say so: the two runs are about 16 MiB apart. The rows are too small for a memory story to appear. To see that, this needs re-running with the heavier data format, which hasn't happened.

The 40-minute hang

I tried 256 files in flight. The warm-up pass finished — 40 seconds, correct output, but only 251 files/s, already half the plateau. Then the very next pass, same settings, same running program, never finished. Forty minutes later it was still going and I killed it.

The store was 3.7% busy at the time. Nothing was overloaded. But the client held 2,244 connections in a half-closed state — the store had hung up, my process never cleaned them up — against 430 working ones, out of a pool of about 520.

The diagnostic detail is that the first pass worked and the second didn't. A real capacity ceiling would have hit both equally. Something accumulated inside my process between repetitions, and that's a leak, not a limit. I re-cut the axis and moved on, so this is one observation and not a reproduced failure. It's logged as an open bug. A program that hangs forever instead of slowing down or failing cleanly is fragile at any setting, not just at 256.

The part where my own scoring code was wrong

The scorecard grades six hypotheses automatically from the raw rows. The point of that design is to stop me from grading my own homework.

The first time I ran it, it reported the knee at 1, failed a hypothesis that should have passed, and computed another one across two different corpora blended together. Three wrong verdicts in one table.

None of them were wrong about the data. They were wrong about which rows to compare. Each check was pulling in cells that differed from each other in more than the one variable under test — two corpus sizes in a single curve, the deliberately-throttled writer runs mixed into the normal sweep. Two of the six checks already grouped rows correctly. The other four didn't, and I hadn't noticed because their answers looked reasonable.

The fix was to make every check use the same grouping rule, then regenerate every number and figure. Everything in this post comes from the corrected version.

What I take from it: automating the analysis doesn't make it objective, it just moves where the mistake lives. A hand-written conclusion gets scrutinized. A generated table gets believed. I nearly published three confident, wrong verdicts in a nice monospace block.

What I'd tell another engineer

A saving is worth exactly what the bottleneck says it's worth. Cutting 17.8× of the bytes bought zero, because bytes weren't what cost. Measure what binds before you optimize what's obvious.
Count the round trips, not just the payload. V2's whole loss was 0.90 of an extra request per file. The winning reader got the same byte saving with fewer requests, by looking up what it already knew instead of asking.
Check that your measurement can even see the answer. My planned sweep would have reported the knee at 8 with a straight face, because 8 was the first place it looked. Design the axis so it can prove you wrong.
A limit can be a scheduler. Removing the buffer cap didn't just use more memory — it let one stage starve the other and cost 22% throughput. Backpressure is a coordination mechanism, not only a memory guard.
"It got slower under load" and "it hung forever" are different bugs. The first is a ceiling. The second is a leak. Don't file them together.

The headline is 11.6×. The actual deliverable is knowing which of the two optimizations produced it, being able to show that the other one produced nothing, and being able to name the exact condition — a slower network — that would flip that verdict.

Coda: the third time the same lesson has billed me

Week 1 ended on: a cost doesn't tell you where it lands — the binding constraint does. Week 2: a dependency is a cost you pay even when its benefit isn't the binding constraint.

Both of those are about costs. This one is the mirror image. A saving is worthless when it isn't the binding constraint either. Reading 17.8× less data is a real, correct, textbook optimization. It bought me exactly nothing, because on this machine moving data was never the expensive part. Same optimization, a slower network, and it becomes the best thing in the codebase.

The optimization didn't change. The constraint did.

Everything here — the campaign results file, all five figures, the generated scorecard, and the full list of what deviated from the plan and why — is in the project's engineering log on GitHub. Every number is loopback, which is precisely the regime where the most interesting hypothesis can't be tested. That axis is next. Pushback welcome.

I Replaced a Flink Cluster With One Python Process

schieh710 — Mon, 08 Jun 2026 01:13:35 GMT

TL;DR. The canonical streaming-interview feature — "clicks in the last 5 minutes" — over a 3-broker Kafka cluster. I built it on PyFlink first: a JobManager, two TaskManagers, an Apache Beam portability bridge, hand-pinned connector JARs, RocksDB, a checkpoint volume. The first smoke run died inside a Beam worker on a one-line Redis hostname typo — before a single window ever emitted. I threw the cluster away and rewrote it as one Python process with in-memory windows. It shipped. The lesson isn't "Flink bad." It's that every dependency is a liability you pay whether or not you ever collect its benefit — and at ~200 evt/s I was paying for six failure domains to get none of their payoff.

The setup

Same project as last time: a streaming feature store. This PR is the headline feature — sliding-window aggregations per user: clicks_5m, purchases_24h, twelve features across three resolutions (5 min / 1 h / 24 h), each emitting on its own slide cadence. The "route" was never in doubt; sliding windows were the chosen approach from day one. The only open question was what computes them.

The textbook answer is Apache Flink. So I started there.

Stack: Python 3.12, PyFlink DataStream API, a local Flink cluster on Docker Compose, Avro + Schema Registry, Redis as the online store. WSL2 dev box.

Act 1: the cluster that never emitted a window

The Flink design was correct. Pane-based pre-aggregation for constant-state windows, bounded-out-of-orderness watermarks (5 s skew, 30 s idleness), allowed-lateness re-firing with an emission_seq, a dual sink contract (Redis hash + Kafka topic). I'm not knocking the semantics — they're standard streaming theory and they were right.

Then I ran it. The first smoke test died like this:

  File ".../apache_beam/runners/worker/sdk_worker.py", line ...
  File ".../apache_beam/runners/worker/operations.py", line ...
  ... eight more frames of Beam + JVM plumbing ...
ConnectionError: Error connecting to redis:6379

A Redis hostname was wrong. One line. The kind of bug you fix in fifteen seconds — if you can see it. Here it surfaced ten frames deep inside a Beam SDK worker, routed through the Python↔JVM portability bridge, underneath the Flink operator that called it. No window had emitted. I couldn't yet tell whether the approach worked, because the scaffolding hadn't even finished standing up.

That's the moment that mattered. Not "Flink is slow" — I never got far enough to measure speed. The failure was opaque, and it was in the foundation, not the feature. A bug in the foundation makes you question the whole approach. A bug in the glue is a one-line patch. I was staring at the wrong kind.

Act 2: count the failure domains

I rewrote it as a plain confluent-kafka consumer that keeps the window state in ordinary Python dicts. Same windowing semantics — copied across verbatim, because the semantics were never the problem. The only thing that changed was the machine underneath them.

Here's the whole argument in one table — what has to stand up correctly before a single feature can be computed:

Moving part	PyFlink build	Plain consumer
JobManager	✅ required	—
TaskManagers (×2)	✅ required	—
JVM runtime	✅ required	—
Apache Beam portability bridge (Python↔JVM)	✅ required	—
Hand-pinned connector JARs	✅ required	—
RocksDB state backend + checkpoint volume	✅ required	—
Kafka client	(inside Flink)	`confluent-kafka` — proven in wk1
Online store client	`redis-py` in a UDF	`redis-py` — proven in wk1
Window state	RocksDB	a Python `dict`
New, unproven failure domains	6+	0

The right column is built entirely from parts that already worked end-to-end the week before. The left column introduces six independent things that had never run together in this environment — and a feature can't emit until all six are healthy at once. That's not a throughput argument. It's a probability argument: the more independent parts that must simultaneously be correct, the more first runs you lose to scaffolding you can't see into.

The honest part: the rewrite also broke. Twice.

If I stopped here it would read as "simple thing worked first try." It didn't. The plain consumer failed on its first two runs too:

A bootstrap hostname pointed at the Docker-internal broker name from a host shell — Failed to resolve 'kafka-1:9092'.
A deserializer call passed None where the library now demands a SerializationContext — TypeError on the first message decoded.

Two bugs. Same class as the one that killed the Flink build — a wrong address, a bad argument. The difference was the bill:

	PyFlink	Plain consumer
What failed	Redis hostname	Broker hostname; a `None` arg
How it surfaced	~10 frames of Beam/JVM	a 3-line Python traceback
Time to fix	(abandoned the approach)	one edit each, minutes

"It broke" is not the metric. "How much did the break cost" is. Opaque failures in a deep stack don't just take longer to fix — they make you distrust the whole design, because you can't localize the fault. Transparent failures in a shallow stack stay where they happen.

"But isn't the simple version slower?"

The reasonable objection: surely you gave up performance. For this problem, no — not on the axis that matters.

Dimension	PyFlink (promised)	Plain consumer (measured)
Feature freshness (the sliding-window output)	5 s watermark + slide cadence	identical — same 5 s + same slide
Steady-state throughput / worker	~10k evt/s/slot, via a Beam crossing	~11–14k evt/s/process, no crossing
Horizontal ceiling	12 (operator parallelism)	12 (consumer-group processes)
Restart recovery	instant (RocksDB checkpoints)	cold-start warm-up — Flink wins
State beyond RAM	spills to disk — Flink wins	heap; would OOM

Feature freshness is governed by the watermark and the slide cadence, and the consumer copies both exactly — so "the 5 minutes ending now" is equally fresh either way. Throughput is a wash: both are GIL-bound in Python and both cap at 12 partitions, except the consumer drops the Python↔JVM crossing, so per worker it's equal-or-better. (PyFlink's only road to genuinely higher throughput was switching to Java — which the Flink design itself rejected.)

Flink genuinely wins two rows — restart recovery and state-beyond-RAM. But the feeder is ~200 evt/s, the whole state is tens of MB, and restarts are rare. Neither winning row binds at this scale. You don't pay six failure domains for a benefit your workload never triggers.

And the asymmetry in that table's column headers is the real punchline: PyFlink's numbers are promised — it never emitted a window, so they were never measured. The consumer's are the ones I actually clocked.

The correction I had to make (I was half-wrong)

My first writeup of this had a tidy story: "the rewrite was easy because the hard conceptual work — panes, watermarks, lateness — was already done in the Flink design; the second doc just inherited it." A reviewer pushed back, and they were right.

That framing conflates two different questions:

Why was the rewrite cheap to produce? → it reused already-specified, already-unit-tested semantics. True.
Why did the Flink build fail? → dependencies. Full stop. Nothing to do with windowing semantics.

I had answered the first question and quietly passed it off as the answer to the second. But the semantics were never the risk — they're standard streaming theory and the sliding-window route was decided before either design existed. If you had written Flink's exact semantics on the plain consumer's dependency footprint, it would have worked first try too. The failure was the six failure domains, not the math. Crediting the rewrite's ease for the original's failure points at the wrong cause.

What I'd tell another engineer

Count failure domains before you count features. A part that must be healthy for anything to work is a liability you pay up front and a benefit you only sometimes collect.
Match the tool's weight to the problem's scale. Flink earns its machinery at 100k+ evt/s and state beyond RAM. At 200 evt/s and tens of MB, that machinery is pure downside — cost with the benefit switched off.
Opaqueness is a cost, and it's underrated. The same one-line bug was ten Beam/JVM frames in one stack and a three-line traceback in the other. Depth of stack sets the price of every future mistake.
"It also broke" isn't the comparison — "how expensive was the break" is. The shallow version failed twice and shrugged it off in minutes.
Don't confuse "why was the rewrite easy" with "why did the original fail." They have different answers. Mixing them hides the real cause.

The headline is "I replaced a Flink cluster with one Python process." The actual deliverable is knowing why that was the right call with evidence — and being able to name the exact conditions (bigger-than-RAM state, sub-second SLAs, exactly-once with large state) that would send me straight back to Flink.

Coda: the same shape as last time

Last post ended on: a cost doesn't tell you where it lands — the binding constraint does. This one is the same shape, one level up. A dependency is a cost you pay even when its benefit isn't the binding constraint. Flink's checkpointing, RocksDB spill, and JVM speed are real benefits — for a workload that triggers them. Mine didn't. So all that was left of them was the bill: six things that had to stand up before I could see whether one Redis hostname was wrong.

Account for what actually binds. Then buy exactly that, and nothing heavier.

Full design docs — the superseded Flink version kept as an artifact, the plain-consumer version that shipped, and the fix-by-fix forensics — are in the project's engineering log on GitHub. This is the narrative version. Pushback welcome — it's how the last section of this very post got written.

The GIL Fingerprint: Why My Kafka Producer Stalled at 14k events/sec

schieh710 — Sat, 16 May 2026 04:39:11 GMT

TL;DR. A Kafka load test needed to sustain 50,000+ events/sec. It plateaued at ~14k no matter what I tuned. The whole story collapses to one fact — CPython runs one process's Python on one core at a time — and that single fact explains all three puzzles: why tuning didn't help, why five independent architecture changes all failed, and why the eventual multiprocessing fix had a counter-intuitive optimal thread count (2 per process beat both 1 and 3). The fix got ~5× to ~60k. The post includes the part where I was confidently wrong.

The setup

I was building the ingestion path for a streaming feature store: synthetic events, Avro-serialized, into a 3-broker Kafka cluster via confluent-kafka's SerializingProducer, Pydantic for validation. The bar was concrete: sustain 60,000 evt/s, fail under 50,000, 10-second run, no dropped messages.

It sustained ~6,000. After tuning, ~14,500 — 3–4× short of the floor, and crucially with zero delivery failures. Nothing was breaking. It was hitting a ceiling. This is the hunt for that ceiling, and the one fact at the bottom of it.

*Stack: Python 3.12, confluent-kafka (librdkafka), pydantic v2, Avro

Schema Registry, Kafka KRaft on Docker Compose, a custom multi-threaded load runner. WSL2 dev box — relevant later.*

Act 1: the plateau, and the fix I got wrong

First move: producer tuning — linger.ms=20, lz4, acks=1 (dev only), 2 MB batches, generous queue caps. It worked exactly as designed: +18% throughput, p95/p99 latency roughly halved. And still 3.4× below the floor.

That's where a performance investigation gets interesting: the tuning did its job, but the bottleneck was somewhere else. So I kept going, one change per run (otherwise you can't attribute anything):

Change	Hypothesis	Result
librdkafka tuning	broker round-trips dominate	✅ +18%, kept
One producer per worker thread	shared serializer lock is the bottleneck	❌ flat throughput, latency 2–6× worse
Verify ≥12 partitions	broker-side serialization	✅ no-op (already optimal)
Poll once per batch, not per event	per-event poll lock contention	❌ −34%
Dedicated callback-pump thread	workers waste time polling	❌ −32% (best p50 ever, though)

Five smart, well-targeted hypotheses. Every one regressed or did nothing. That pattern is itself the clue — but first, the one I got wrong, because it's the most instructive data point in the whole investigation.

The library docstring says "Not thread-safe — one instance per producing thread." The serializer holds a schema-cache lock. Twelve threads, one lock — obviously the bottleneck. I shipped it without profiling. Throughput went flat; every latency percentile got 2–6× worse. Post-mortem: splitting one shared producer into twelve fragmented the broker-side batches ~12× (tiny frequent requests instead of one fat batched one), and the broker's per-request fixed cost swamped any lock saving. The lock was a microsecond dict lookup — 1–5% of per-event time, never the bottleneck. "Not thread-safe" is a correctness statement, not a performance one. A 5-minute py-spy run would have killed the idea before I wrote a line. I'm keeping this in because the engineer who hides their wrong turns is hiding the part where the reasoning actually happens.

Act 2: the fingerprint

Step back and look at every configuration measured:

Configuration	Sustained evt/s
Baseline	~12,250
+ librdkafka tuning	~14,500
+ per-worker producers	~14,400
+ per-batch poll	~9,600
+ dedicated pump thread	~9,800
tuning only (revert)	~14,800

Six independent architectures — different locks, topologies, polling strategies — all clustered in 9.5–14.8k. Each change moved which inner constraint was binding without ever moving the outer ceiling.

That is consistent with exactly one thing: a process-wide serialization point no in-process change can escape. In CPython there is precisely one — the Global Interpreter Lock. One thread runs Python bytecode at a time, regardless of thread or core count.

Each failure is positive evidence

The regressions aren't disappointments — each is a falsification test the GIL hypothesis survived:

Per-worker producers removed 12-way contention on two locks at once. If either bound throughput, it should have jumped. 0% change — freeing inner locks does nothing when threads queue on the GIL anyway.
Per-batch polling cut a lock's acquire frequency ~16,800×. If frequency were the cost, this recovers it. −34% — the dispatch work still happens; concentrating it just made GIL contention spikier.
The pump thread is the structurally correct fix for the symptom (a profile showed 93% of worker time in poll()). It did its job — best p50 I ever measured. But throughput fell 32%. "The architecture is right" and "throughput regressed" are only both true if the work moved to a thread competing for the same GIL. With real parallelism the pump runs on its own core and throughput climbs. It didn't.

Three changes that should have moved throughput if anything but the GIL were binding. None did.

The arithmetic that closed it

At ~14,500 evt/s, per-event Python work (Pydantic + serializer adapter

bookkeeping + poll callbacks) is ~70 µs of GIL-held time:

14,500 evt/s × 70 µs/evt ≈ 1,015 ms of GIL-held time per second
1 CPU core                = 1,000 ms of execution per second

We were saturating ~100% of one core's worth of Python — the ceiling, from first principles, matching the measured plateau. No in-process multi-threaded design beats one core of Python, because the GIL serializes bytecode across all threads. Every architecture I tried just rearranged which thread held the one lock that matters.

Act 3: the same fact, billed twice more

Here's the through-line. The GIL didn't only explain the plateau and the failed fixes — the same one-core-per-process fact dictates how to escape it, and how to tune the escape.

Escape: if the limit is one GIL per process, run more processes. Each gets its own interpreter, its own GIL, its own core. I built a separate multiprocessing harness (kept beside the threaded one — they serve different jobs): N producer processes, the proven single-process config in each, the parent aggregating results.

The tuning law — billed a third time. The projection said 4 × 14.5k ≈ ~58k. The first auto-layout, 4 processes × 3 workers, gave only 34.5k — each process slower than one running alone. 6 × 2 (same 12 total threads, redistributed) gave **60k** — same thread count, +76%. 8 × 1 ≈ 62k (tied, but 33% more processes for it). 2 × 6: a dismal 15k.

Workers-per-process has an optimum, and it's the GIL again. Treat each process's GIL as a single-server queue you want ~100% utilized with no queue forming for it:

Too few (1/proc): the lone worker spends ~half its time parked in GIL-yielding waits (rate limiter, backpressure). The GIL sits idle — half the process's Python capacity wasted. Why 8×1 ≯ 6×2.
Too many (3+/proc): more threads than the GIL can serve; the surplus block in the handoff path and you pay serialization + handoff overhead. Why 4×3 < 6×2.

The optimum is the fewest workers that keep the GIL continuously busy through the natural blocking gaps: empirically W ≈ round(1/s), where s is the fraction of a worker's wall time holding the GIL. Here s ≈ 0.5, so W = 2. Two workers tag-team the GIL — one runs while the other is parked — while librdkafka's C sender threads (no GIL) do the network in parallel. That turns "grid-search every combination" into "profile once for s, compute the layout, confirm with one run."

	Sustained evt/s	vs. 50k floor
Single process (best, tuned)	~14,500	❌ 3.4× under
4 × 3 multiprocess	~34,500	❌ under
6 × 2 multiprocess (kept)	~59–62k	✅ clears it

~5× the single-process number, latency also better (p50 16 vs 28 ms; p95 47 vs 170 ms), zero failures.

One honest caveat, because error bars are part of the result: this was a WSL2 dev box, numbers drifted as broker/OS state warmed. The load-bearing claims aren't "60,000" — they're the ~5× ratio, the mechanism, and the W ≈ round(1/s) model. Those reproduce; a laptop benchmark number doesn't.

What I'd tell another engineer

Profile before optimizing. My one regression came from shipping a plausible hypothesis without the 5-minute profile that would have killed it. "Obviously slow" is not a measurement.
% of time ≠ % wasted. 93% in poll() wasn't waste — it was useful callback work. Removing it relocated the work, not deleted it.
"Not thread-safe" ≠ "this lock is hot." One is a correctness claim, the other an empirical one. Only measurement answers the second.
Architecture changes are conservation games. Moving work between threads doesn't reduce it; it changes who waits. Real gains need less total work or escaping the serialization point.
Out of in-process options? The next move is multiprocessing or a no-GIL build — not more thread tuning. And size it with W ≈ round(1/s), not intuition.

The point of writing this up isn't the 5×. It's that the entire investigation — a plateau, five dead ends, and a surprising thread count — reduces to one fact about CPython, provable with a falsification chain and a back-of-envelope. Shipping the speedup is the deliverable. Knowing why, with evidence, is the job.

Full investigation — every run, the lock-stack diagram, the fix-by-fix forensics, the queueing-theory derivation of the worker law — is in the project's engineering log. This is the narrative version. Pushback welcome.