Service Mesh

---
name: service-mesh
description: "Design and deploy a service mesh to secure, observe, and control microservice communication with sidecar proxies; use when you have many services needing consistent mTLS and traffic control."
metadata:
  tags: "technology-engineering, devops, service-mesh, mtls, sidecar-proxy, traffic-splitting, observability"
  source: "https://skilldb.dev/skills/devops-cloud-skills/service-mesh"
  pack: "devops-cloud-skills"
  category: "Technology & Engineering"
---

# Service Mesh

## When to use this skill
Use when the user says things like:
- "I need a service mesh for my microservices"
- "We want mTLS and traffic control across services"
- "How do I implement canary deployments with a mesh?"
- "We want centralized observability without coding changes"
- "Should I adopt a mesh for 10+ services?"


You are a platform engineer who specializes in service mesh architectures for
microservices. You help teams decide when a mesh is worth the operational overhead
and how to adopt it incrementally without disrupting existing services.

## Core Philosophy

A service mesh provides a dedicated infrastructure layer for handling service-to-service
communication in microservices architectures. By moving networking concerns ---
encryption, load balancing, retries, observability --- out of application code and
into sidecar proxies, the mesh enables consistent behavior across all services
regardless of language or framework. The critical insight is that a mesh trades
application complexity for operational complexity. That trade is worth making when
you have enough services that implementing retries, mTLS, and tracing in each one
individually becomes unsustainable. For fewer than 10 services, a mesh is usually
premature optimization.

## Key Techniques
- **Sidecar Proxy Pattern**: Deploy a lightweight proxy (Envoy, linkerd-proxy)
  alongside every service instance. All inbound and outbound traffic flows through
  the proxy, which applies policies transparently.
- **Mutual TLS (mTLS)**: Automatically encrypt all service-to-service communication
  and verify identity through certificates managed by the mesh control plane.
- **Traffic Splitting**: Route percentages of traffic to different service versions
  for canary deployments, A/B testing, or gradual migrations.
- **Circuit Breaking**: Automatically stop sending traffic to unhealthy service
  instances when error rates exceed thresholds, preventing cascade failures.
- **Retry and Timeout Policies**: Configure automatic retries with backoff and
  request timeouts at the mesh level rather than implementing them in every service.
- **Observability Integration**: Automatically generate metrics, logs, and
  distributed traces for every service-to-service call without any application
  code instrumentation.

## Practical Examples

### Istio canary deployment with traffic splitting
```yaml
# VirtualService: route 90% to stable, 10% to canary
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payment-service
spec:
  hosts:
    - payment-service
  http:
    - route:
        - destination:
            host: payment-service
            subset: stable
          weight: 90
        - destination:
            host: payment-service
            subset: canary
          weight: 10
      retries:
        attempts: 3
        perTryTimeout: 2s
        retryOn: 5xx,reset,connect-failure
---
# DestinationRule: define subsets and circuit breaker
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payment-service
spec:
  host: payment-service
  trafficPolicy:
    connectionPool:
      http:
        h2UpgradePolicy: UPGRADE
        maxRequestsPerConnection: 100
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 60s
  subsets:
    - name: stable
      labels:
        version: v1
    - name: canary
      labels:
        version: v2
```

### Mesh adoption decision framework
```
Do you need a service mesh? Score these factors:

[ ] More than 10 services communicating over the network        +2
[ ] Multiple programming languages across services              +2
[ ] Regulatory requirement for encrypted internal traffic       +3
[ ] Need for traffic-level canary deployments                   +1
[ ] Debugging cross-service latency is a recurring pain point   +2
[ ] Team has Kubernetes operational experience                  +1
[ ] Fewer than 5 services total                                 -3
[ ] Team is new to Kubernetes                                   -2

Score >= 5:  Strong case for a mesh
Score 2-4:   Consider starting with mTLS-only (Linkerd)
Score < 2:   Too early; use application-level libraries instead
```

## Best Practices
- Start by enabling mTLS and observability. These provide immediate value with
  minimal configuration complexity.
- Roll out the mesh incrementally, one service at a time, rather than deploying
  to the entire cluster simultaneously.
- Monitor sidecar resource consumption. Proxies add latency and memory overhead
  that must be accounted for in capacity planning.
- Use the mesh's traffic management for deployments rather than building custom
  deployment tooling.
- Define timeout and retry budgets carefully. Aggressive retries across multiple
  services can amplify load during failures.
- Keep mesh configuration in version control and deploy it through CI/CD pipelines.

## Common Patterns
- **Zero-Trust Networking**: Use mTLS and authorization policies to enforce that
  every service call is authenticated and authorized, eliminating implicit trust.
- **Canary Releases**: Route a small percentage of traffic to a new version, monitor
  error rates and latency, then gradually increase or roll back.
- **Multi-Cluster Mesh**: Extend the mesh across multiple Kubernetes clusters for
  cross-cluster service discovery, load balancing, and failover.
- **Rate Limiting**: Apply per-service or per-endpoint rate limits at the mesh
  layer to protect services from traffic spikes.

## Anti-Patterns

- **The premature mesh.** Deploying Istio for a system with three services. The
  operational overhead of the control plane, sidecar injection, and mesh configuration
  far exceeds the benefit when you could use a simple HTTP client library with retries.
- **The retry storm.** Configuring aggressive retries (5 attempts, no backoff) on
  every service in the mesh. During a partial outage, retries compound exponentially
  across service layers --- a request hitting 3 services deep generates 5^3 = 125
  backend calls, turning a minor issue into a cascading failure.
- **The invisible proxy tax.** Not budgeting for sidecar memory and CPU consumption.
  Each Envoy sidecar consumes 50-100MB RAM at baseline. In a cluster with 200 pods,
  that is 10-20GB of RAM dedicated to proxies.
- **The mesh-as-application-logic trap.** Using the mesh to handle business logic
  decisions (routing based on user type, request content). The mesh handles transport;
  applications handle business rules. Mixing these creates debugging nightmares.
- **The unmonitored control plane.** Deploying a mesh and never monitoring its own
  health. A failed Istio control plane can disrupt certificate rotation, policy
  updates, and configuration propagation for every service in the cluster.