For each app that changed in a feature, deploy one new pod and three Istio objects — a VirtualService (traffic gate + cookie setter), a DestinationRule (pod selector), and optionally an EnvoyFilter (outbound header propagation for apps that don't propagate it themselves). A shared EnvoyFilter on the ingress gateway converts ?x-feat-name to a request header at the edge — request phase only, no response manipulation. The VirtualService owns Set-Cookie. Any request carrying x-feat-name: auth-refactor routes to the feature pods. Everything else falls through to stable non-prod. Go touch grass.
The problem with full-clone envs
You're running 12 microservices in non-prod. A developer needs to test a new auth flow against the real frontend and backend. The standard move: clone the whole environment, spin up all 12 services again, wire them, and tear them down when the PR merges. The cost compounds fast the moment more than one feature branch is active.
The selective deploy pattern flips this: you only run what changed. Everything else comes from the stable non-prod cluster, routed correctly via Istio's L7 traffic shaping layer. The only hard requirement is that services in the mesh run an Istio sidecar — and notably, each cluster can belong to its own mesh entirely. No shared control plane required. Your platform team will sleep better. Probably.
| Concern | Full-clone | Selective deploy (this pattern) |
|---|---|---|
| Cloud cost | Runs N services even if 2 changed | Runs only delta; routes to shared stable |
| Spin-up time | Minutes to hours per feature | Seconds — just the changed deployments |
| Config parity | Cloned envs drift quickly | Shared pods are the same nonprod pods |
| Teardown | Cleanup often delayed; costs accumulate | kubectl delete removes all Istio objects |
| Multi-mesh support | N/A | Each cluster uses its own mesh independently |
Architecture overview
The routing split happens at three layers.
- First, a dedicated EnvoyFilter on the ingress gateway operates on the request phase only — it reads
?x-feat-namefrom the URL and promotes it to a request header so downstream VirtualServices can match it. The cookie is written on the response by the VirtualService'sheaders.response.addblock. - Second, VirtualServices on each changed application match the header or cookie and route to the appropriate pod subset.
- Third, EnvoyFilters on each feature pod's sidecar propagate the header on every outbound call via a Lua script in the sidecar's outbound HTTP filter chain, keeping downstream services in the same feature lane automatically
Fig 1 — Gateway EnvoyFilter normalises the query param at the edge. Only auth service v2 and session-restapi v2 are deployed new; frontend and info-restapi are shared from stable non-prod. Each changed service gets its own VirtualService, DestinationRule, and EnvoyFilter.
How it works
The mechanism hinges on three layers working together. At the edge, a dedicated EnvoyFilter on the ingress gateway intercepts any request carrying ?x-feat-name=auth-refactor and promotes it to a request header — request phase only. The Set-Cookie on the response is written by the VirtualService's headers.response.add block, so the browser pins to the feature env after the first hit.
Inside the mesh, each participating service has a VirtualService that matches on the header or cookie and routes to the feature pod subset. On those feature pods, a per-workload EnvoyFilter injects x-feat-name onto every outbound call via a Lua script in the sidecar's outbound HTTP filter chain, keeping downstream services in the same feature lane automatically.
One important constraint runs throughout the pattern: the application itself must forward the x-feat-name header on any outbound calls it makes. The Lua EnvoyFilter handles this transparently for services in the ephemeral env, but for services that are shared from stable non-prod — like the frontend in an auth-only feature — the app code needs to propagate the header explicitly. This is covered in the gotchas section below. You've been warned. We warned you.
Fig 2 — Request path on first visit. Steps 1–2 promote the query param to a header. Step 3 is the shared-service propagation responsibility. Steps 4–6 are the Istio routing layer doing its thing.
The Istio objects
0 — Gateway EnvoyFilter: the edge normaliser
Deployed once per namespace — shared across all ephemeral envs. Its only job is to read ?x-feat-name from the query string and promote it to a request header so VirtualServices can match it. Crucially, it runs on the request phase only. The cookie write lives in the VirtualService response block — not here. This split keeps the responsibilities clean and the debugging surface small.
apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: digillect-gateway-feat-param-filter namespace: digillect-nonprod spec: workloadSelector: labels: istio: ingressgateway configPatches: - applyTo: HTTP_FILTER match: context: GATEWAY listener: filterChain: filter: name: "envoy.filters.network.http_connection_manager" subFilter: name: "envoy.filters.http.router" patch: operation: INSERT_BEFORE value: name: envoy.lua typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua inlineCode: | -- Request phase only: promote ?x-feat-name to a header. -- Set-Cookie is handled by the VirtualService response headers block. function envoy_on_request(request_handle) local path = request_handle:headers():get(":path") or "" local feat = path:match("[?&]x%-feat%-name=([^&]+)") if feat then request_handle:headers():replace("x-feat-name", feat) request_handle:logInfo("digillect: x-feat-name=" .. feat .. " promoted from queryParam") end end
The matching VirtualService then writes the cookie on the response so the browser is pinned for subsequent requests:
headers: response: set: x-feat-name: auth-refactor add: # VirtualService owns the cookie — gateway Lua only promotes the query param Set-Cookie: "x-feat-name=auth-refactor; Path=/; Domain=digillect.ca; HttpOnly" request: set: x-feat-name: auth-refactor
Centralising query-param promotion at the edge means every downstream VirtualService stays simple — header and cookie match only, no queryParams blocks. This filter is shared across all ephemeral envs in the namespace — deploy it once and forget it. Each feature's VirtualService independently handles its own Set-Cookie response header. One shared filter, infinite feature envs. That's the deal.
1 — VirtualService: the traffic gate
Sits in front of your application hostname. Because the gateway EnvoyFilter already handles query-param conversion, this only needs two match rules — header and cookie. On match it routes to the named subset and reinforces the x-feat-name response header so any service further down the chain can also read it.
apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: auth-refactor-auth-svc-vservice namespace: digillect-nonprod labels: app: digillect-auth x-feat-name: auth-refactor spec: hosts: - auth-svc.digillect.svc.cluster.local http: - name: auth-refactor match: - headers: x-feat-name: regex: .*(auth-refactor).* - headers: Cookie: regex: .*(x-feat-name=auth-refactor).* route: - destination: host: auth-svc.digillect.svc.cluster.local subset: auth-refactor port: number: 8080 headers: response: set: x-feat-name: auth-refactor request: set: x-feat-name: auth-refactor
Browsers send all cookies in a single Cookie: session=abc; x-feat-name=auth-refactor; theme=dark string. The regex is required to extract the value. Use Istio's safe_regex (RE2 engine) in production to avoid ReDoS exposure. A catastrophic backtrack in a Lua filter at 3am is a rite of passage you don't need.
2 — DestinationRule: the pod selector
Binds the subset name to concrete pod labels. Your feature Deployment must carry both a matching app label and a version label. If these drift from what the DestinationRule expects, Istio will return a 503 or silently fall through to stable — the most common production pitfall with this pattern. Check your labels twice. Deploy once.
apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: auth-refactor-auth-svc-destination-rule namespace: digillect-nonprod labels: app: digillect-auth x-feat-name: auth-refactor spec: host: auth-svc.digillect.svc.cluster.local subsets: - name: auth-refactor labels: # These must match exactly what's on your feature pod template version: auth-refactor app: digillect-auth
3 — EnvoyFilter per feature pod: outbound header propagation
Without this, when the auth service v2 pod makes an outbound call to the backend, there's no x-feat-name header on that egress request. The backend's VirtualService finds no match and routes to the stable subset — silently breaking isolation. This EnvoyFilter patches a Lua script into the SIDECAR_OUTBOUND HTTP filter chain of each feature pod. It fires on every egress response and injects the header. No application code changes required on the feature pod itself.
apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: auth-refactor-auth-svc-filter namespace: digillect-nonprod labels: app: digillect-auth x-feat-name: auth-refactor spec: # Scoped ONLY to feature pods — stable pods are untouched workloadSelector: labels: version: auth-refactor app: digillect-auth configPatches: - applyTo: HTTP_FILTER match: context: SIDECAR_OUTBOUND listener: filterChain: filter: name: "envoy.filters.network.http_connection_manager" subFilter: name: "envoy.filters.http.router" patch: operation: INSERT_BEFORE value: name: envoy.lua typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua inlineCode: | local feat_name = "auth-refactor" function envoy_on_response(response_handle) response_handle:logInfo("digillect: propagating x-feat-name=" .. feat_name) response_handle:headers():add("x-feat-name", feat_name) end
The Lua script fires when a feature pod is making a request to another service — on its outbound listener. This injects the propagation signal before the egress request leaves the pod, so downstream VirtualServices can match it. SIDECAR_INBOUND fires when the pod receives a request — too late to influence outbound routing from that pod. Wrong direction, wrong phase, wrong result.
If every service in your mesh already propagates x-feat-name in application code — the same way tracing headers are forwarded — you don't need the per-app EnvoyFilter at all. The sidecar Lua is a convenience for teams that haven't built that discipline yet, or for services you don't own and can't modify. If your HTTP client wrappers handle propagation uniformly, skip the EnvoyFilter, reduce your Istio config surface area, and rely on the app layer alone. The VirtualService and DestinationRule are still required regardless. Some assembly required.
Gotchas and precautions
Shared services must propagate the header themselves
The sidecar EnvoyFilter only runs on pods in the ephemeral env — the ones you deployed for the feature. It does not run on stable non-prod pods. This matters the moment the feature flow passes through a shared service.
Consider this scenario: only the auth service changed, so the frontend is shared from stable non-prod. A user hits the feature env via cookie. The VirtualService routes to auth service v2. But how did the request get there? The frontend pod — running in stable — made an outbound call to auth. It has no EnvoyFilter. If it doesn't explicitly forward the x-feat-name header it received from the browser, that header is silently dropped, the auth VirtualService sees no match, and the call goes to stable auth instead of v2. You tested the wrong thing. Nobody knows. Ship it.
Any service that is not in the ephemeral env but sits in the call path between the gateway and a changed service must propagate x-feat-name in its outbound calls. The Lua EnvoyFilter only covers feature pods. Shared stable pods are transparent to it. This is an application-level responsibility — the header must flow through every hop in the chain to reach the service that actually has a VirtualService match configured.
The fix is to treat x-feat-name the same way distributed tracing headers are treated: any service that participates in the mesh reads inbound context headers and forwards them on every outbound HTTP call. Build it once in a shared HTTP client wrapper, use it everywhere.
// Propagation headers — read from inbound request, forward on all outbound calls const PROPAGATE_HEADERS = [ 'x-feat-name', 'x-request-id', 'x-b3-traceid', 'x-b3-spanid', ]; function buildOutboundHeaders(inboundReq) { const headers = {}; for (const h of PROPAGATE_HEADERS) { const val = inboundReq.headers[h]; if (val) headers[h] = val; } return headers; }
TLS origination blocks the sidecar EnvoyFilter
The sidecar EnvoyFilter's Lua script runs inside the HTTP filter chain of the outbound listener. This chain only activates when the sidecar can parse the traffic as HTTP — meaning the application sends plain HTTP to the local Envoy, which then handles encryption transparently (Istio's auto-mTLS between mesh pods).
If an application originates TLS itself — calling https://backend-svc directly instead of http://backend-svc — the sidecar receives an already-encrypted byte stream. The HTTP connection manager never takes over, and the Lua filter is entirely bypassed. The header is never injected.
When a service in your ephemeral env makes an outbound call using https:// directly, the Envoy sidecar sees an opaque TLS stream. The HTTP filter chain — and therefore the Lua propagation script — is skipped. The x-feat-name header is never added to that egress call. Downstream VirtualServices see no match and fall through to stable. This failure is completely silent.
The solution is the same as the shared-service case above: explicitly forward the header in application code on every outbound call. When you cannot rely on the sidecar to propagate it (because the app owns TLS), the app must own propagation instead.
Treat x-feat-name exactly like distributed tracing context: propagate it in your HTTP client wrapper at every hop, regardless of whether you expect an EnvoyFilter to cover it. This gives you defence-in-depth — the Lua filter handles it for free on feature pods, and the app layer handles it everywhere else. Two layers. No surprises.
x-feat-name is a routing hint, not an auth mechanism. Anyone with a valid feature name can access that environment. Combine with your existing auth layer for sensitive features. Strip the header at the ingress gateway on production clusters so this pattern only operates in non-prod. Seriously — strip it on prod.
Conclusion
The pattern — a gateway-level query param normaliser, per-app VirtualServices and DestinationRules, and optional sidecar EnvoyFilters — composes into a practical ephemeral environment system that scales across clusters without a shared mesh and tears down cleanly per feature. If your services already propagate x-feat-name in application code, skip the EnvoyFilters entirely and reduce your Istio config surface area.
The two disciplines it demands are: understanding where the sidecar filter chain operates relative to TLS, and treating x-feat-name as a first-class propagation header in your HTTP client code alongside tracing headers. Build both of those habits in once, as shared infrastructure, and the operational cost of each new ephemeral env drops to deploying a pod and three YAML files. That's the deal. Not bad.
"Reliability isn't a feature you bolt on — it's a discipline you build in from the ground up. Same goes for ephemeral environments that actually work."