Running 200+ Microservices on Kubernetes: Lessons in Cost, Scale and Reliability

Running 200+ microservices on Kubernetes exposes problems that a ten-service cluster never shows: resource requests set by guesswork, service mesh overhead multiplied across a thousand pods, and cloud bills that grow faster than traffic. This guide covers the architecture, cost controls and operational habits that keep a large EKS cluster stable.

A Kubernetes cluster that started with 12 services and grew to more than 200 does not fail dramatically. It degrades — deployments slow down, incidents take longer to diagnose, and the monthly bill climbs without anyone being able to point at a cause. The engineering problems at that scale are rarely about Kubernetes itself. They are about defaults nobody revisited.

The Cluster Architecture

The shape that held up: AWS EKS spread across three availability zones, with separate node groups for general workloads (m6i.xlarge), memory-intensive services (r6i.2xlarge) and GPU jobs (g5.xlarge). Karpenter handles provisioning, Istio provides the service mesh, ArgoCD drives deployment, and Prometheus, Grafana, Loki and Jaeger cover observability.

Steady state runs 800 to 1,200 pods, peaking above 2,000. Separating node groups by workload profile matters more than it sounds: without it, a memory-hungry service will happily schedule onto general-purpose nodes and evict things that were sized correctly.

Resource Requests and Limits: The Silent Killer

The single largest source of waste was resource requests set by intuition. Profiling with the Vertical Pod Autoscaler showed most services were using under 20% of the CPU they requested. Every one of those over-requests reserves capacity the scheduler cannot give to anything else, so the cluster scales out to satisfy reservations that no workload is actually consuming.

Right-sizing against real usage — and then enforcing it in admission policy so new services cannot reintroduce the problem — is the highest-value change available on a large cluster. It costs nothing but attention.

CI/CD: Getting to 50 Deploys a Day

GitHub Actions builds and pushes; ArgoCD reconciles. Argo Rollouts runs canary deployments with a five-minute pause at each step, and Prometheus metrics drive automatic rollback when error rates move the wrong way.

The automatic rollback is what makes frequent deployment safe. Fifty deploys a day is not achievable if each one needs a human watching a dashboard; it becomes routine once the pipeline can decide, on evidence, to undo itself.

Service Mesh: Istio's Real Tradeoff

Istio delivers mutual TLS everywhere and detailed traffic observability, and both are genuinely valuable. The cost is a sidecar per pod at 50 to 80 MB of memory each. Across a thousand pods that is 50 to 80 GB of RAM spent on proxies.

Blanket injection across every namespace is rarely the right call. Enable the mesh where mTLS and traffic policy actually matter, and leave batch jobs and internal utilities out of it.

The Namespace Strategy

Namespace-per-team with ResourceQuotas attached is the balance point between autonomy and blast radius. Teams deploy without filing tickets; a runaway workload consumes its team's quota rather than the cluster's capacity.

Distributed Systems Realities

Three conventions prevent most cascading failures:

Observability Is Not Optional

Cost Management

What We Would Do Differently

Prometheus, Grafana, Loki and Jaeger, with golden-signals dashboards per service, cost around $2,000 a month to run. That is a real line item, and it is cheap relative to a single extended outage diagnosed by guesswork.

Three changes did most of the work. Moving provisioning to Karpenter cut EC2 spend by 35%. Enabling consolidation — letting the autoscaler repack workloads onto fewer nodes — saved a further $8,000 to $10,000 a month. Auditing orphaned persistent volumes recovered storage nobody had noticed was still attached.

Kubernetes at this scale rewards restraint. Most of the wins came from removing things — fewer services, fewer sidecars, smaller resource requests — rather than adding capability.

We design, migrate and operate cloud-native platforms of this shape. Explore our engineering services or get in touch to discuss your infrastructure.