Kubernetes Egress Gateway Options with Istio Ingress/Egress

Kubernetes, Networking, Service Mesh
12 min read
Kubernetes Egress Gateway Options with Istio Ingress/Egress
Page content

Istio provides a comprehensive service mesh solution with built-in ingress and egress gateway capabilities. This post explores how to leverage Istio’s gateway resources for controlling both inbound and outbound traffic in your Kubernetes cluster.

Egress Gateway Series

This series covers Kubernetes egress gateway solutions:

✓ All parts complete!

Istio Gateway Architecture

Istio uses Envoy proxy as the foundation for its gateway capabilities:

┌─────────────────────────────────────────────────────────────────┐
│                    Kubernetes Cluster                           │
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │                    Istio Service Mesh                   │   │
│  │                                                         │   │
│  │  ┌──────────────┐         ┌──────────────┐             │   │
│  │  │   Ingress    │         │    Egress    │             │   │
│  │  │   Gateway    │         │    Gateway   │             │   │
│  │  │  (Envoy)     │         │   (Envoy)    │             │   │
│  │  └──────┬───────┘         └───────┬──────┘             │   │
│  │         │                         │                     │   │
│  │         ▼                         ▼                     │   │
│  │  ┌─────────────┐           ┌─────────────┐             │   │
│  │  │   Service   │           │   External  │             │   │
│  │  │   A         │           │   APIs      │             │   │
│  │  └─────────────┘           └─────────────┘             │   │
│  │                                                         │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐                        │
│  │  Pod A  │  │  Pod B  │  │  Pod C  │  ← Sidecar Proxy       │
│  └─────────┘  └─────────┘  └─────────┘                        │
└─────────────────────────────────────────────────────────────────┘

Key Components

Component Purpose Protocol
Ingress Gateway External traffic entry point HTTP/HTTPS, TCP, TLS
Egress Gateway Controlled external access HTTP/HTTPS, TLS, TCP
Sidecar Proxy Pod-level traffic management mTLS, policies
Pilot (istiod) Configuration distribution xDS API

Benefits of Istio Gateway

Feature Benefit
Unified control plane Single configuration for ingress, egress, and mesh
mTLS support Automatic encryption between services
Traffic policies Rate limiting, circuit breaking, retries
Observability Built-in metrics, tracing, access logs
Security JWT validation, authorization policies

Prerequisites

Component Version Notes
Kubernetes 1.26+ Tested on 1.28, 1.29
Istio 1.20+ Gateway API support
kubectl Latest Configured with cluster access
istioctl 1.20+ CLI for management

Verify Istio Installation

# Check Istio version
istioctl version

# Verify Istio components
kubectl get pods -n istio-system

# Expected output:
# NAME                                    READY   STATUS
# istiod-xxxxxxxxxx-xxxxx                 1/1     Running
# istio-ingressgateway-xxxxxxxxxx-xxxxx   1/1     Running
# istio-egressgateway-xxxxxxxxxx-xxxxx    1/1     Running

# Check gateway services
kubectl get svc -n istio-system | grep gateway

Installation

Install Istio with Gateway Components

# Add Istio Helm repository
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update

# Install Istio base (CRDs)
helm install istio-base istio/base \
  -n istio-system \
  --create-namespace \
  --wait

# Install Istio with ingress and egress gateways
helm install istiod istio/istiod \
  -n istio-system \
  --set pilot.autoscaleEnabled=false \
  --set pilot.resources.requests.cpu=500m \
  --set pilot.resources.requests.memory=2Gi \
  --wait

# Install ingress gateway
helm install istio-ingressgateway istio/gateway \
  -n istio-system \
  --set service.type=LoadBalancer \
  --set autoscaleEnabled=false \
  --wait

# Install egress gateway
helm install istio-egressgateway istio/gateway \
  -n istio-system \
  --set name=istio-egressgateway \
  --set service.type=ClusterIP \
  --set autoscaleEnabled=false \
  --wait

Verify Installation

# Check all components are running
kubectl get pods -n istio-system

# Verify gateway deployment
istioctl proxy-status

# Check gateway services
kubectl get svc -n istio-system

Part 1: Istio Ingress Gateway

Architecture Overview

┌──────────────────┐
│ External Client  │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│  Load Balancer   │
└────────┬─────────┘
         │
         ▼
┌───────────────────────────┐
│ Istio Ingress Gateway     │─────── mTLS to Sidecars ───────┐
└───────────┬───────────────┘                                │
            │                                                │
            ▼                                                │
┌───────────────────────────┐                                │
│ VirtualService Routing    │                                │
└───────────┬───────────────┘                                │
            │                                                │
    ┌───────┼───────────┐                                    │
    │       │           │                                    │
    ▼       ▼           ▼                                    │
┌────────┐ ┌────────┐ ┌────────┐                             │
│Service │ │Service │ │Service │ <───────────────────────────┘
│   A    │ │   B    │ │   C    │
└────────┘ └────────┘ └────────┘

Create Gateway Resource

Create ingress-gateway.yaml:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: main-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway  # Use istio ingress controller
  servers:
    # HTTP server (redirect to HTTPS)
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "api.example.com"
        - "app.example.com"
      tls:
        httpsRedirect: true
    
    # HTTPS server
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - "api.example.com"
        - "app.example.com"
      tls:
        mode: SIMPLE
        credentialName: example-tls-secret

Create TLS Secret

# Generate or obtain certificate
kubectl create secret tls example-tls-secret \
  --cert=path/to/tls.crt \
  --key=path/to/tls.key \
  --namespace istio-system

Create VirtualService

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api-routes
  namespace: default
spec:
  hosts:
    - "api.example.com"
  gateways:
    - istio-system/main-gateway
  http:
    # Route to API service
    - match:
        - uri:
            prefix: /api/v1
      route:
        - destination:
            host: api-service
            port:
              number: 8080
    
    # Route to documentation
    - match:
        - uri:
            prefix: /docs
      route:
        - destination:
            host: docs-service
            port:
              number: 80
    
    # Default route
    - route:
        - destination:
            host: web-service
            port:
              number: 80

Apply Configuration

# Apply gateway configuration
kubectl apply -f ingress-gateway.yaml
kubectl apply -f api-routes.yaml

# Verify gateway
kubectl get gateway -n istio-system

# Verify virtual service
kubectl get virtualservice api-routes -n default

Advanced Ingress Configuration

Path-Based Routing

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: advanced-routing
  namespace: default
spec:
  hosts:
    - "example.com"
  gateways:
    - istio-system/main-gateway
  http:
    # Exact path match
    - match:
        - uri:
            exact: /health
      route:
        - destination:
            host: health-service
            port:
              number: 8080
    
    # Prefix match with rewrite
    - match:
        - uri:
            prefix: /api/v2/
      rewrite:
        uri: /api/v1/
      route:
        - destination:
            host: api-v2-service
            port:
              number: 8080
    
    # Regex match
    - match:
        - uri:
            regex: "^/users/[0-9]+/profile$"
      route:
        - destination:
            host: user-service
            port:
              number: 8080
    
    # Header-based routing (canary)
    - match:
        - headers:
            x-canary:
              exact: "true"
      route:
        - destination:
            host: api-canary
            port:
              number: 8080
    
    # Weight-based routing (90/10 split)
    - route:
        - destination:
            host: api-stable
            port:
              number: 8080
          weight: 90
        - destination:
            host: api-canary
            port:
              number: 8080
          weight: 10

Rate Limiting

apiVersion: networking.istio.io/v1beta1
kind: EnvoyFilter
metadata:
  name: rate-limit
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.local_ratelimit
          typed_config:
            "@type": type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
            value:
              stat_prefix: http_local_rate_limiter
              token_bucket:
                max_tokens: 100
                tokens_per_fill: 100
                fill_interval: 60s
              filter_enabled:
                runtime_key: local_rate_limit_enabled
                default_value:
                  numerator: 100
                  denominator: HUNDRED
              filter_enforced:
                runtime_key: local_rate_limit_enforced
                default_value:
                  numerator: 100
                  denominator: HUNDRED

Part 2: Istio Egress Gateway

Architecture Overview

┌───────────────────┐
│ Pod with Sidecar  │
└─────────┬─────────┘
          │ mTLS
          ▼
┌─────────────────────────────────────────────────────────┐
│                   Egress Gateway                        │
│                                                         │
│  ┌───────────────┐  ┌───────────────┐  ┌─────────────┐ │
│  │ServiceEntry   │  │VirtualService │  │Destination  │ │
│  │               │  │               │  │Rule         │ │
│  └───────────────┘  └───────────────┘  └─────────────┘ │
└─────────────────────────────────────────────────────────┘
          │ NAT/TLS
          ▼
┌───────────────────┐
│   External API    │
└───────────────────┘

When to Use Egress Gateway

Scenario Solution
Fixed source IP Egress gateway with static IP
External mTLS Gateway handles client certificates
Audit requirements Centralized logging of external calls
Traffic control Rate limiting, circuit breaking
Protocol translation HTTP to TLS, TCP proxy

Step 1: Enable Egress Gateway Traffic

By default, Istio allows direct egress. To force traffic through the gateway:

apiVersion: networking.istio.io/v1beta1
kind: MeshConfig
metadata:
  name: istio
  namespace: istio-system
spec:
  outboundTrafficPolicy:
    mode: REGISTRY_ONLY  # Only allow registered external services

Apply via istioctl:

istioctl install --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY

Step 2: Create ServiceEntry

Register external services that pods can access:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-api
  namespace: default
spec:
  hosts:
    - api.external-service.com
  ports:
    - number: 443
      name: https
      protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS

Step 3: Create Egress Gateway

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: istio-egress-gateway
  namespace: istio-system
spec:
  selector:
    istio: egressgateway  # Use istio egress controller
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - "api.external-service.com"
      tls:
        mode: ISTIO_MUTUAL_TLS  # mTLS from sidecar to gateway

Step 4: Create VirtualService for Egress

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: direct-external-api
  namespace: default
spec:
  hosts:
    - "api.external-service.com"
  gateways:
    - istio-system/istio-egress-gateway
    - mesh  # Apply to sidecars
  http:
    # Traffic from sidecar to egress gateway
    - match:
        - gateways:
            - mesh
          port: 443
      route:
        - destination:
            host: istio-egressgateway.istio-system.svc.cluster.local
            port:
              number: 443
    
    # Traffic from egress gateway to external service
    - match:
        - gateways:
            - istio-system/istio-egress-gateway
          port: 443
      route:
        - destination:
            host: api.external-service.com
            port:
              number: 443

Step 5: Create DestinationRule

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: egress-gateway-destination
  namespace: default
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: UPGRADE
        http1MaxPendingRequests: 100
        http2MaxRequests: 1000

Apply Egress Configuration

# Apply all egress resources
kubectl apply -f serviceentry.yaml
kubectl apply -f egress-gateway.yaml
kubectl apply -f virtualservice-egress.yaml
kubectl apply -f destinationrule.yaml

# Verify configuration
istioctl analyze

Advanced Egress Patterns

Pattern 1: Mutual TLS to External Service

When the external service requires client certificates:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: istio-egress-gateway-mtls
  namespace: istio-system
spec:
  selector:
    istio: egressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - "secure-api.example.com"
      tls:
        mode: MUTUAL
        credentialName: external-api-mtls-cert
        caCertificates: external-ca.pem

# Create secret with client certificate and key
kubectl create secret generic external-api-mtls-cert \
  --from-file=tls.crt=path/to/client.crt \
  --from-file=tls.key=path/to/client.key \
  --from-file=ca.crt=path/to/ca.crt \
  --namespace istio-system

Pattern 2: TCP Egress Gateway

For non-HTTP traffic (databases, etc.):

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-db
  namespace: default
spec:
  hosts:
    - db.external-service.com
  ports:
    - number: 5432
      name: postgres
      protocol: TCP
  location: MESH_EXTERNAL
  resolution: DNS

---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: istio-egress-gateway-tcp
  namespace: istio-system
spec:
  selector:
    istio: egressgateway
  servers:
    - port:
        number: 5432
        name: tcp-postgres
        protocol: TCP
      hosts:
        - "db.external-service.com"

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: route-external-db
  namespace: default
spec:
  hosts:
    - "db.external-service.com"
  gateways:
    - istio-system/istio-egress-gateway-tcp
    - mesh
  tcp:
    - match:
        - gateways:
            - mesh
          port: 5432
      route:
        - destination:
            host: istio-egressgateway.istio-system.svc.cluster.local
            port:
              number: 5432
    - match:
        - gateways:
            - istio-system/istio-egress-gateway-tcp
          port: 5432
      route:
        - destination:
            host: db.external-service.com
            port:
              number: 5432

Pattern 3: Egress Gateway with SNI

For TLS origination with SNI:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: sni-egress
  namespace: default
spec:
  hosts:
    - "*.googleapis.com"
  gateways:
    - istio-system/istio-egress-gateway-sni
    - mesh
  tls:
    - match:
        - gateways:
            - mesh
          port: 443
          sniHosts:
            - "*.googleapis.com"
      route:
        - destination:
            host: istio-egressgateway.istio-system.svc.cluster.local
            port:
              number: 443
    - match:
        - gateways:
            - istio-system/istio-egress-gateway-sni
          port: 443
          sniHosts:
            - "*.googleapis.com"
      route:
        - destination:
            host: www.googleapis.com
            port:
              number: 443

Security Configuration

Authorization Policy for Egress

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: restrict-egress
  namespace: default
spec:
  selector:
    matchLabels:
      istio: egressgateway
  action: ALLOW
  rules:
    - to:
        - operation:
            hosts:
              - "api.trusted-service.com"
              - "*.internal.company.com"
            ports:
              - "443"
              - "8443"

Rate Limiting for Egress

apiVersion: networking.istio.io/v1beta1
kind: EnvoyFilter
metadata:
  name: egress-rate-limit
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: egressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.local_ratelimit
          typed_config:
            "@type": type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
            value:
              stat_prefix: egress_local_rate_limit
              token_bucket:
                max_tokens: 1000
                tokens_per_fill: 1000
                fill_interval: 60s

JWT Validation at Ingress

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-auth
  namespace: default
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  jwtRules:
    - issuer: "https://auth.example.com"
      jwksUri: "https://auth.example.com/.well-known/jwks.json"
      audiences:
        - "api.example.com"

---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: require-jwt
  namespace: default
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: ALLOW
  rules:
    - from:
        - source:
            requestPrincipals: ["*"]

Monitoring and Observability

Kiali Dashboard

# Install Kiali (if not already installed)
helm install kiali-server kiali/kiali-server \
  -n istio-system \
  --set auth.strategy="anonymous" \
  --wait

# Access Kiali
istioctl dashboard kiali

                    ┌───────────────────┐
                    │  Istio Gateways   │
                    └─────────┬─────────┘
                              │
                              ▼
                    ┌───────────────────┐
                    │ Pilot/Mesh Config │
                    └─────────┬─────────┘
                              │
              ┌───────────────┼───────────────┐
              │               │               │
              ▼               ▼               ▼
     ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
     │    Kiali    │ │  Grafana    │ │   Jaeger    │
     └──────┬──────┘ └──────┬──────┘ └──────┬──────┘
            │               │               │
            ▼               ▼               ▼
     ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
     │Service Graph│ │   Metrics   │ │ Distributed │
     │             │ │  Dashboard  │ │   Tracing   │
     └─────────────┘ └─────────────┘ └─────────────┘

Custom Metrics

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: gateway-metrics
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  metrics:
    - providers:
        - name: prometheus
      overrides:
        - match:
            metric: REQUEST_COUNT
          tagOverrides:
            source_principal:
              operation: UPSERT
              value: request.auth.principal
            response_code:
              operation: UPSERT
              value: response.code

Access Logging

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: gateway-logging
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  accessLogging:
    - providers:
        - name: envoy
      filter:
        expression: response.code >= 400  # Log errors only

Troubleshooting

Issue: Gateway Not Receiving Traffic

# Check gateway status
kubectl get gateway -n istio-system

# Verify service is LoadBalancer
kubectl get svc -n istio-system istio-ingressgateway

# Check endpoint readiness
istioctl proxy-status istio-ingressgateway-xxxxxxxxxx-xxxxx.istio-system

# Test connectivity
curl -v https://<gateway-ip>/health

Issue: Egress Gateway Not Working

# Verify ServiceEntry
kubectl get serviceentry -A

# Check VirtualService routing
istioctl analyze

# Verify egress gateway pods
kubectl get pods -n istio-system -l istio=egressgateway

# Check sidecar configuration
kubectl get sidecar -A

Issue: mTLS Failures

# Check certificate status
istioctl proxy-config secret istio-ingressgateway-xxxxxxxxxx-xxxxx.istio-system

# Verify DestinationRule
kubectl get destinationrule -A

# Check PeerAuthentication
kubectl get peerauthentication -A

# Test with verbose output
curl -v --cacert ca.crt --cert client.crt --key client.key https://external-api

Common Problems and Solutions

Problem Cause Solution
503 Service Unavailable No healthy endpoints Check service and endpoints
404 Not Found VirtualService misconfiguration Verify hosts and routes
Connection timeout Egress not allowed Add ServiceEntry
mTLS handshake failed Certificate mismatch Verify secrets and DestinationRule
Rate limit exceeded Too many requests Increase token bucket or optimize calls

Performance Tuning

Gateway Resource Allocation

# Production gateway deployment
resources:
  requests:
    cpu: 1000m
    memory: 2Gi
  limits:
    cpu: 2000m
    memory: 4Gi

# High-traffic gateway
resources:
  requests:
    cpu: 2000m
    memory: 4Gi
  limits:
    cpu: 4000m
    memory: 8Gi

Connection Pool Settings

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: gateway-connection-pool
spec:
  host: istio-ingressgateway.istio-system.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1000
      http:
        h2UpgradePolicy: UPGRADE
        http1MaxPendingRequests: 1000
        http2MaxRequests: 10000
        maxRequestsPerConnection: 100
        maxRetries: 3

Envoy Proxy Tuning

apiVersion: networking.istio.io/v1beta1
kind: EnvoyFilter
metadata:
  name: gateway-performance
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
      patch:
        operation: MERGE
        value:
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            common_http_protocol_options:
              max_connection_duration: 3600s
              idle_timeout: 300s

Comparison with Cilium

Feature Istio Cilium
Architecture Envoy proxy eBPF
Performance Good (proxy-based) Excellent (kernel-level)
Service Mesh Full mesh capabilities CNI with mesh features
Learning Curve Steep Medium
Resource Usage Higher (sidecars) Lower (eBPF)
Gateway Features Rich (L7 policies) Good (L3/L4 focus)
mTLS Automatic Manual configuration
Observability Kiali, Grafana, Jaeger Hubble, Grafana

When to Choose Istio Gateway

Choose Istio when:

  • ✅ You need full service mesh capabilities
  • ✅ Advanced L7 routing and policies required
  • ✅ Automatic mTLS is important
  • ✅ Rich observability needed
  • ✅ Multi-cluster federation planned

Consider alternatives when:

  • 📋 You only need simple ingress/egress
  • 📋 Performance is critical (consider Cilium)
  • 📋 Resource constraints (sidecar overhead)
  • 📋 Simple use case (consider nginx-ingress)

Next Steps

In the next post of this series:

  • Calico Egress Gateway - Network policy-driven approach
  • Comparison with Istio and Cilium
  • Migration strategies between solutions

Conclusion

Istio Gateway provides:

Advantages:

  • ✅ Comprehensive service mesh integration
  • ✅ Advanced L7 traffic management
  • ✅ Automatic mTLS between services
  • ✅ Rich observability (Kiali, Grafana, Jaeger)
  • ✅ Security policies (JWT, authorization)
  • ✅ Multi-cluster support

Considerations:

  • 📋 Higher resource usage (sidecar per pod)
  • 📋 Steeper learning curve
  • 📋 More complex configuration
  • 📋 Performance overhead vs eBPF

For organizations needing full service mesh capabilities with advanced traffic management and security, Istio Gateway is the comprehensive choice.