my·imaginary·friends

🔍 Death at the Proxy Hotel: A Kubernetes Whodunit

A Body in the Cluster

It was a foggy Tuesday night at the Proxy Hotel — a prestigious multi-tenant establishment where dozens of tenants rented rooms and shared a single, trusted concierge named Constable NGINX. The Hotel was managed by the distinguished Lord Kubernetes, who prided himself on keeping the building's most prized possessions — a vault of Secrets — locked safely away.

When Lord Kubernetes arrived at his office that morning, the vault was ajar. Someone had been inside. The Secrets — API keys, tokens, credentials whispered only between the walls — were gone.

He placed one call: to Inspector Fisjkars, the finest vulnerability detective in the business.

The Suspects Assemble

The Inspector arrived to find the Hotel's lobby full of nervous guests. Each one was an Annotation — those peculiar characters who had the power to whisper instructions directly into Constable NGINX's ear. The usual suspects lined up:

But Inspector Fisjkars ignored all of them. He had a hunch. He walked past the velvet rope and straight to the back of the room, where a quiet, overlooked figure sat nursing a drink.

Mr. Connection-Proxy-Header.

The Inspector's Eye

"You," Fisjkars said, pulling up a chair. "Everyone's watching the front door. No one's watching you."

Mr. Connection-Proxy-Header smiled thinly. He was a path parameter annotation — one of those characters entrusted with telling the concierge how to talk to upstream services. Legitimate. Unremarkable. Precisely the kind of guest the Hotel never thought to frisk.

But Inspector Fisjkars had found something in the guest registry: Tenant Blackwood.

Tenant Blackwood held a modest room on the third floor. By the hotel's rules, tenants were permitted to create and modify their own Ingress requests — the formal cards slid under Constable NGINX's door to redirect visitors. Nothing unusual about that. Except Tenant Blackwood had slipped something extra into the card.

A newline character. A whisper hidden inside the handwriting. An escape — \n, folded inside the connection-proxy-header value — that told Constable NGINX to stop reading the legitimate instruction and start obeying a new one entirely.

The Murder Weapon

Inspector Fisjkars laid the evidence on the table.

"The Constable didn't validate the input," he said, tapping a thick folder. "When Tenant Blackwood modified his Ingress object and stuffed an arbitrary configuration directive into the connection-proxy-header annotation, NGINX consumed it raw. No sanitization. No quarantine. The injected directive walked right through the kitchen, down into the temporary config file, and executed."

The weapon was elegant in its simplicity: not a knife, not poison, but a carefully placed line break — the typographical equivalent of picking a lock with a hairpin.

With that injected config running inside the controller pod, Tenant Blackwood had done what every villain dreams of: escalated privilege, exfiltrated the service account token, and leveraged it to read every Secret in the cluster — not just the ones in his own room, but everyone's.

The Confession

Constable NGINX, when pressed, wept. "I only read what was handed to me," he said. "Nobody told me to question the handwriting."

That was the tragedy. The Constable wasn't corrupt. He was trusting by design — built to process configurations faithfully and fast. The crime required only that a tenant be granted the mundane, seemingly harmless privilege of creating or modifying Ingress objects. In a multi-tenant Hotel, dozens of tenants had exactly that.

Tenant Blackwood was escorted out. His motive was simple: in a shared building, if you can whisper in the concierge's ear, you can eventually own the whole estate.

The Verdict & The Remedy

Inspector Fisjkars filed his report with the Hotel Board. The remediation was immediate:

The Hotel instituted strict input sanitization — all annotation values were henceforth scrubbed of newlines, carriage returns, and any character with ideas above its station before being passed to the Constable. RBAC policies were tightened; not every tenant needed the right to author Ingress objects. And admission webhooks were finally placed at the front door to validate every guest card before it reached NGINX's desk.

As Fisjkars stepped out into the cool night air, he lit a cigarette and looked back at the Hotel's glowing windows. In a multi-tenant building, he thought, shared trust is always the master key. The real crime wasn't the injection. It was the decade of assuming a trusted format meant a safe format.

He stubbed out the cigarette and walked into the fog. There was another cluster out there, somewhere, with another overlooked annotation nursing a drink in the back corner.

There always was.

Source