Cost-efficient ephemeral envs
across clusters with Istio

Deploy only the services that changed. Route feature traffic via header or cookie. Reuse the rest from your stable non-prod cluster — no duplicate stacks, no wasted compute. Your cloud bill will send a thank-you note. Your on-call rotation won't need to.

TL;DR — for the impatient engineer

For each app that changed in a feature, deploy one new pod and three Istio objects — a VirtualService (traffic gate + cookie setter), a DestinationRule (pod selector), and optionally an EnvoyFilter (outbound header propagation for apps that don't propagate it themselves). A shared EnvoyFilter on the ingress gateway converts ?x-feat-name to a request header at the edge — request phase only, no response manipulation. The VirtualService owns Set-Cookie. Any request carrying x-feat-name: auth-refactor routes to the feature pods. Everything else falls through to stable non-prod. Go touch grass.

The problem with full-clone envs

You're running 12 microservices in non-prod. A developer needs to test a new auth flow against the real frontend and backend. The standard move: clone the whole environment, spin up all 12 services again, wire them, and tear them down when the PR merges. The cost compounds fast the moment more than one feature branch is active.

The selective deploy pattern flips this: you only run what changed. Everything else comes from the stable non-prod cluster, routed correctly via Istio's L7 traffic shaping layer. The only hard requirement is that services in the mesh run an Istio sidecar — and notably, each cluster can belong to its own mesh entirely. No shared control plane required. Your platform team will sleep better. Probably.

ConcernFull-cloneSelective deploy (this pattern)
Cloud costRuns N services even if 2 changedRuns only delta; routes to shared stable
Spin-up timeMinutes to hours per featureSeconds — just the changed deployments
Config parityCloned envs drift quicklyShared pods are the same nonprod pods
TeardownCleanup often delayed; costs accumulatekubectl delete removes all Istio objects
Multi-mesh supportN/AEach cluster uses its own mesh independently

Architecture overview

The routing split happens at three layers.

browser / api client ingress gateway EnvoyFilter reads ?x-feat-name → sets header on request; VS sets cookie on response istio ingress gateway digillect-gateway · myapp.digillect.ca virtualservice — route decision match: x-feat-name header or cookie only no match header/cookie hit nonprod — stable frontend stable auth service shared session-restapi shared info-restapi shared DestinationRule subset: stable VirtualService default fallthrough ephemeral — feat: auth-refactor frontend reused — nonprod auth service v2 new — deployed session-restapi v2 new — deployed info-restapi reused — nonprod no Istio objects (propagates header) VirtualService DestinationRule EnvoyFilter VirtualService DestinationRule EnvoyFilter no Istio objects (reused stable) feature traffic (x-feat-name matched) stable nonprod reused — no new pod deployed

Fig 1 — Gateway EnvoyFilter normalises the query param at the edge. Only auth service v2 and session-restapi v2 are deployed new; frontend and info-restapi are shared from stable non-prod. Each changed service gets its own VirtualService, DestinationRule, and EnvoyFilter.

How it works

The mechanism hinges on three layers working together. At the edge, a dedicated EnvoyFilter on the ingress gateway intercepts any request carrying ?x-feat-name=auth-refactor and promotes it to a request header — request phase only. The Set-Cookie on the response is written by the VirtualService's headers.response.add block, so the browser pins to the feature env after the first hit.

Inside the mesh, each participating service has a VirtualService that matches on the header or cookie and routes to the feature pod subset. On those feature pods, a per-workload EnvoyFilter injects x-feat-name onto every outbound call via a Lua script in the sidecar's outbound HTTP filter chain, keeping downstream services in the same feature lane automatically.

One important constraint runs throughout the pattern: the application itself must forward the x-feat-name header on any outbound calls it makes. The Lua EnvoyFilter handles this transparently for services in the ephemeral env, but for services that are shared from stable non-prod — like the frontend in an auth-only feature — the app code needs to propagate the header explicitly. This is covered in the gotchas section below. You've been warned. We warned you.

request path — first visit via query param 1 — browser GET ?x-feat-name=auth-refactor 2 — gateway EnvoyFilter (request phase) reads ?x-feat-name → sets x-feat-name header on request 3 — frontend (stable — shared from nonprod) no Istio objects; app propagates x-feat-name on outbound call to auth 4 — VirtualService (auth-svc) header match → DestinationRule subset auth-refactor → auth v2 pod 5 — auth service v2 pod EnvoyFilter (Lua) injects x-feat-name on every outbound egress call calls session-restapi calls info-restapi 6a — session-restapi v2 VirtualService matches header → feature subset 6b — info-restapi (stable) no VirtualService match; serves stable response

Fig 2 — Request path on first visit. Steps 1–2 promote the query param to a header. Step 3 is the shared-service propagation responsibility. Steps 4–6 are the Istio routing layer doing its thing.

The Istio objects

0 — Gateway EnvoyFilter: the edge normaliser

Deployed once per namespace — shared across all ephemeral envs. Its only job is to read ?x-feat-name from the query string and promote it to a request header so VirtualServices can match it. Crucially, it runs on the request phase only. The cookie write lives in the VirtualService response block — not here. This split keeps the responsibilities clean and the debugging surface small.

YAMLgateway-envoyfilter.yaml
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: digillect-gateway-feat-param-filter
  namespace: digillect-nonprod
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
            subFilter:
              name: "envoy.filters.http.router"
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.lua
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
          inlineCode: |
            -- Request phase only: promote ?x-feat-name to a header.
            -- Set-Cookie is handled by the VirtualService response headers block.
            function envoy_on_request(request_handle)
              local path = request_handle:headers():get(":path") or ""
              local feat = path:match("[?&]x%-feat%-name=([^&]+)")
              if feat then
                request_handle:headers():replace("x-feat-name", feat)
                request_handle:logInfo("digillect: x-feat-name=" .. feat .. " promoted from queryParam")
              end
            end

The matching VirtualService then writes the cookie on the response so the browser is pinned for subsequent requests:

YAMLvirtualservice-auth.yaml (response headers block)
    headers:
      response:
        set:
          x-feat-name: auth-refactor
        add:
          # VirtualService owns the cookie — gateway Lua only promotes the query param
          Set-Cookie: "x-feat-name=auth-refactor; Path=/; Domain=digillect.ca; HttpOnly"
      request:
        set:
          x-feat-name: auth-refactor
Why handle this at the gateway?

Centralising query-param promotion at the edge means every downstream VirtualService stays simple — header and cookie match only, no queryParams blocks. This filter is shared across all ephemeral envs in the namespace — deploy it once and forget it. Each feature's VirtualService independently handles its own Set-Cookie response header. One shared filter, infinite feature envs. That's the deal.

1 — VirtualService: the traffic gate

Sits in front of your application hostname. Because the gateway EnvoyFilter already handles query-param conversion, this only needs two match rules — header and cookie. On match it routes to the named subset and reinforces the x-feat-name response header so any service further down the chain can also read it.

YAMLvirtualservice-auth.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: auth-refactor-auth-svc-vservice
  namespace: digillect-nonprod
  labels:
    app: digillect-auth
    x-feat-name: auth-refactor
spec:
  hosts:
  - auth-svc.digillect.svc.cluster.local
  http:
  - name: auth-refactor
    match:
    - headers:
        x-feat-name:
          regex: .*(auth-refactor).*
    - headers:
        Cookie:
          regex: .*(x-feat-name=auth-refactor).*
    route:
    - destination:
        host: auth-svc.digillect.svc.cluster.local
        subset: auth-refactor
        port:
          number: 8080
    headers:
      response:
        set:
          x-feat-name: auth-refactor
      request:
        set:
          x-feat-name: auth-refactor
Regex on Cookie header — don't skip this

Browsers send all cookies in a single Cookie: session=abc; x-feat-name=auth-refactor; theme=dark string. The regex is required to extract the value. Use Istio's safe_regex (RE2 engine) in production to avoid ReDoS exposure. A catastrophic backtrack in a Lua filter at 3am is a rite of passage you don't need.

2 — DestinationRule: the pod selector

Binds the subset name to concrete pod labels. Your feature Deployment must carry both a matching app label and a version label. If these drift from what the DestinationRule expects, Istio will return a 503 or silently fall through to stable — the most common production pitfall with this pattern. Check your labels twice. Deploy once.

YAMLdestinationrule-auth.yaml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: auth-refactor-auth-svc-destination-rule
  namespace: digillect-nonprod
  labels:
    app: digillect-auth
    x-feat-name: auth-refactor
spec:
  host: auth-svc.digillect.svc.cluster.local
  subsets:
  - name: auth-refactor
    labels:
      # These must match exactly what's on your feature pod template
      version: auth-refactor
      app: digillect-auth

3 — EnvoyFilter per feature pod: outbound header propagation

Without this, when the auth service v2 pod makes an outbound call to the backend, there's no x-feat-name header on that egress request. The backend's VirtualService finds no match and routes to the stable subset — silently breaking isolation. This EnvoyFilter patches a Lua script into the SIDECAR_OUTBOUND HTTP filter chain of each feature pod. It fires on every egress response and injects the header. No application code changes required on the feature pod itself.

YAMLenvoyfilter-auth.yaml
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: auth-refactor-auth-svc-filter
  namespace: digillect-nonprod
  labels:
    app: digillect-auth
    x-feat-name: auth-refactor
spec:
  # Scoped ONLY to feature pods — stable pods are untouched
  workloadSelector:
    labels:
      version: auth-refactor
      app: digillect-auth
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
            subFilter:
              name: "envoy.filters.http.router"
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.lua
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
          inlineCode: |
            local feat_name = "auth-refactor"

            function envoy_on_response(response_handle)
              response_handle:logInfo("digillect: propagating x-feat-name=" .. feat_name)
              response_handle:headers():add("x-feat-name", feat_name)
            end
Why SIDECAR_OUTBOUND?

The Lua script fires when a feature pod is making a request to another service — on its outbound listener. This injects the propagation signal before the egress request leaves the pod, so downstream VirtualServices can match it. SIDECAR_INBOUND fires when the pod receives a request — too late to influence outbound routing from that pod. Wrong direction, wrong phase, wrong result.

You can drop the EnvoyFilter entirely

If every service in your mesh already propagates x-feat-name in application code — the same way tracing headers are forwarded — you don't need the per-app EnvoyFilter at all. The sidecar Lua is a convenience for teams that haven't built that discipline yet, or for services you don't own and can't modify. If your HTTP client wrappers handle propagation uniformly, skip the EnvoyFilter, reduce your Istio config surface area, and rely on the app layer alone. The VirtualService and DestinationRule are still required regardless. Some assembly required.

Gotchas and precautions

Shared services must propagate the header themselves

The sidecar EnvoyFilter only runs on pods in the ephemeral env — the ones you deployed for the feature. It does not run on stable non-prod pods. This matters the moment the feature flow passes through a shared service.

Consider this scenario: only the auth service changed, so the frontend is shared from stable non-prod. A user hits the feature env via cookie. The VirtualService routes to auth service v2. But how did the request get there? The frontend pod — running in stable — made an outbound call to auth. It has no EnvoyFilter. If it doesn't explicitly forward the x-feat-name header it received from the browser, that header is silently dropped, the auth VirtualService sees no match, and the call goes to stable auth instead of v2. You tested the wrong thing. Nobody knows. Ship it.

Gotcha — shared service header blindspot

Any service that is not in the ephemeral env but sits in the call path between the gateway and a changed service must propagate x-feat-name in its outbound calls. The Lua EnvoyFilter only covers feature pods. Shared stable pods are transparent to it. This is an application-level responsibility — the header must flow through every hop in the chain to reach the service that actually has a VirtualService match configured.

The fix is to treat x-feat-name the same way distributed tracing headers are treated: any service that participates in the mesh reads inbound context headers and forwards them on every outbound HTTP call. Build it once in a shared HTTP client wrapper, use it everywhere.

Node.jshttp-client-middleware (example)
// Propagation headers — read from inbound request, forward on all outbound calls
const PROPAGATE_HEADERS = [
  'x-feat-name',
  'x-request-id',
  'x-b3-traceid',
  'x-b3-spanid',
];

function buildOutboundHeaders(inboundReq) {
  const headers = {};
  for (const h of PROPAGATE_HEADERS) {
    const val = inboundReq.headers[h];
    if (val) headers[h] = val;
  }
  return headers;
}

TLS origination blocks the sidecar EnvoyFilter

The sidecar EnvoyFilter's Lua script runs inside the HTTP filter chain of the outbound listener. This chain only activates when the sidecar can parse the traffic as HTTP — meaning the application sends plain HTTP to the local Envoy, which then handles encryption transparently (Istio's auto-mTLS between mesh pods).

If an application originates TLS itself — calling https://backend-svc directly instead of http://backend-svc — the sidecar receives an already-encrypted byte stream. The HTTP connection manager never takes over, and the Lua filter is entirely bypassed. The header is never injected.

Gotcha — TLS origination (app-originated HTTPS)

When a service in your ephemeral env makes an outbound call using https:// directly, the Envoy sidecar sees an opaque TLS stream. The HTTP filter chain — and therefore the Lua propagation script — is skipped. The x-feat-name header is never added to that egress call. Downstream VirtualServices see no match and fall through to stable. This failure is completely silent.

The solution is the same as the shared-service case above: explicitly forward the header in application code on every outbound call. When you cannot rely on the sidecar to propagate it (because the app owns TLS), the app must own propagation instead.

Design principle

Treat x-feat-name exactly like distributed tracing context: propagate it in your HTTP client wrapper at every hop, regardless of whether you expect an EnvoyFilter to cover it. This gives you defence-in-depth — the Lua filter handles it for free on feature pods, and the app layer handles it everywhere else. Two layers. No surprises.

Security note

x-feat-name is a routing hint, not an auth mechanism. Anyone with a valid feature name can access that environment. Combine with your existing auth layer for sensitive features. Strip the header at the ingress gateway on production clusters so this pattern only operates in non-prod. Seriously — strip it on prod.

Conclusion

The pattern — a gateway-level query param normaliser, per-app VirtualServices and DestinationRules, and optional sidecar EnvoyFilters — composes into a practical ephemeral environment system that scales across clusters without a shared mesh and tears down cleanly per feature. If your services already propagate x-feat-name in application code, skip the EnvoyFilters entirely and reduce your Istio config surface area.

The two disciplines it demands are: understanding where the sidecar filter chain operates relative to TLS, and treating x-feat-name as a first-class propagation header in your HTTP client code alongside tracing headers. Build both of those habits in once, as shared infrastructure, and the operational cost of each new ephemeral env drops to deploying a pod and three YAML files. That's the deal. Not bad.

"Reliability isn't a feature you bolt on — it's a discipline you build in from the ground up. Same goes for ephemeral environments that actually work."
← All Posts Work with Digillect →