# Why We Migrated Away from AWS EKS Auto Mode Back to Standard Mode (And How We Did It) Imagine buying a brand-new, top-of-the-line autonomous electric car. The salesperson smiles and says: *"Engine tuning, battery health, tire pressure—the car handles everything automatically. You just hit the accelerator and enjoy the ride."* Sounds like a dream, right? Lekin two months into ownership, you notice a few weird things: 1. The car insists on refueling at the most expensive charging stations. 2. When you pop the hood to tweak a simple performance setting, the car sets off an alarm and locks you out. 3. Every 21 days—without fail—the car pulls into a service center and replaces its own parts, even if you are cruising at 120 km/h on a highway! **Welcome to AWS EKS Auto Mode.** AWS launched EKS Auto Mode promising "zero-touch Kubernetes" and a hands-off node management experience. For small teams or basic workloads, it feels like magic. But when you are running high-scale, multi-environment production workloads with strict FinOps goals and custom security requirements, that "magic" box can quickly turn into a straitjacket. In this post, I will break down what EKS Auto Mode actually is behind the marketing hype, the real production pain points that forced us to migrate back to **EKS Standard Mode + Karpenter**, and the exact step-by-step Terraform strategy we used to pull off a zero-downtime migration. ## Under the Hood: What EKS Auto Mode *Actually* Is AWS marketing advertises Auto Mode as an entirely new serverless-like experience for Kubernetes. But as DevOps engineers, we know there is no such thing as "no servers"—just servers managed by someone else. Behind the scenes, AWS did not invent alien technology. They simply wrapped popular cloud-native tools into a managed AWS black box: * **Autoscaling:** An AWS-managed version of open-source **Karpenter**. * **Networking:** Pre-configured **Amazon VPC CNI** with built-in Security Groups per Pod. * **Storage:** Pre-installed and auto-managed **Amazon EBS CSI Driver**. * **Node OS:** Read-only **Bottlerocket OS** (AWS's minimal, container-optimized operating system). ## The Turning Point: 4 Production Nightmares ### 1\. The Built-in NodePool Revert Loop (The "Ghost in the Machine") When you create an EKS Auto Mode cluster, AWS automatically provisions two default NodePools: `general-purpose` and `system`. Here is the catch: **These default NodePools are immutable.** #### The Incident: During a cost-optimization sprint, our FinOps team requested that we restrict worker nodes to cost-efficient instance families (like AMD `m6a`/`c6a` instances) and enforce specific Spot instance ratios. We ran `kubectl patch` to update the default NodePool constraints. Kubernetes returned `NodePool patched (updated)`. We smiled, grabbed a cup of coffee, and thought we were done. ☕ Five minutes later, chaos broke out. AWS’s background reconciliation controller detected that the default NodePool specs did not match AWS’s internal baseline. Without warning, **the controller silently reverted our changes back to the default configuration.** Karpenter went into a spin loop: 1. It launched new nodes based on our patched spec. 2. AWS background loops reverted the spec. 3. Karpenter flagged the new nodes as non-compliant and terminated them. 4. Repeat. We watched in disbelief as nodes churned and pods restarted across non-prod environments for 3 hours before we realized the background controller was fighting our changes. ### 2\. The FinOps Reality & The "Convenience Tax" Cost optimization was the primary driver for our infra overhaul. We were aiming for a **35–40% infrastructure cost reduction** across our non-production environments by leveraging: * Aggressive Karpenter consolidation rules. * Heavy usage of Spot capacity with smart fallback handling. * Workload rightsizing and AMD instance migration. With EKS Auto Mode, achieving granular FinOps control became an uphill battle: * **Premium Overhead:** You pay an additional management layer cost on managed Auto Mode compute. * **Restricted NodePool Fine-Tuning:** Because default NodePools are locked down, you cannot aggressively push Karpenter settings like `disruption.consolidationPolicy: WhenEmptyOrUnderutilized` with tight expiration timeouts. * **Limited Instance Overrides:** You lose fine-grained control over instance allocation strategies, forcing workloads onto higher-cost instance types than necessary. Instead of cutting costs by 40%, we were paying AWS extra for management overhead while locked out of optimization knobs! ### 3\. The 21-Day Forced Node Rotation To maintain security compliance, AWS EKS Auto Mode automatically enforces a **mandatory 21-day node rotation policy**. Every node in your cluster is terminated and replaced every 21 days so that it runs on the latest Bottlerocket OS image. While patch management is great, a hard 21-day lifecycle is a landmine for legacy or stateful applications: * If a team forgets to set a strict `PodDisruptionBudget` (PDB). * If a application lacks graceful `SIGTERM` signal handling. * If a database migration job is running during the forced drain window. You end up experiencing unexpected micro-outages every 3 weeks. Security updates should happen on *your* deployment pipeline schedule—not via unannounced node terminations. ### 4\. Zero OS-Level Control EKS Auto Mode uses **Bottlerocket OS**, which features a read-only root filesystem. While security-hardened, it completely blocks you from: * Adjusting custom Linux kernel parameters (`sysctl` tweaks for high-network-throughput workloads). * Installing specialized eBPF monitoring/security agents (e.g., Cilium, custom security sensors). * Mounting custom volumes or running specific host-level daemon sets requiring elevated system privileges. ## The Solution: Migrating to Standard Mode with Terraform & Open Karpenter To gain back 100% lifecycle control, eliminate management overhead, and achieve our 40% FinOps cost reduction, we decided to migrate our EKS clusters back to **Standard Mode** using module-based Terraform and self-managed Karpenter. Here is the exact battle-tested strategy we used to execute the transition with **zero downtime**. ![](https://cdn.hashnode.com/uploads/covers/69b00c50abc0d950015d60e7/3c3bca2a-08d2-422a-bea1-e0a250c73ebb.png align="center") ### Step 1: Update EKS Cluster Configuration via Terraform First, we modified our AWS EKS Terraform module to disable Auto Mode compute capabilities while keeping the EKS control plane intact. ```json # main.tf - AWS EKS Cluster Configuration module "eks" { source = "terraform-aws-modules/eks/aws" version = "~> 20.0" cluster_name = "prod-eks-cluster" cluster_version = "1.30" # Disable EKS Auto Mode Compute Configuration cluster_compute_config = { enabled = false } vpc_id = module.vpc.vpc_id subnet_ids = module.vpc.private_subnets # Enable standard OIDC provider for Karpenter controller IAM Roles enable_irsa = true tags = { Environment = "production" ManagedBy = "Terraform" } } ``` ### Step 2: Deploy Open-Source Karpenter with Fine-Tuned NodePools Next, we deployed standalone **Karpenter** using Helm via Terraform. This gave us full authority over node provisioning, Spot/On-Demand ratios, and instance families. Here is the `NodePool` manifest we applied: ```json # karpenter-nodepool.yaml apiVersion: karpenter.sh/v1beta1 kind: NodePool metadata: name: general-purpose-standard spec: template: spec: requirements: - key: karpenter.sh/capacity-type operator: In values: ["spot", "on-demand"] - key: kubernetes.io/arch operator: In values: ["amd64"] - key: karpenter.k8s.aws/instance-family operator: In values: ["c6a", "m6a", "c5", "m5"] nodeClassRef: apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass name: standard-node-class disruption: consolidationPolicy: WhenUnderutilized consolidateAfter: 30s expireAfter: 720h # 30 Days lifecycle managed on OUR terms --- apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass metadata: name: standard-node-class spec: amiFamily: AL2023 # Managed Amazon Linux 2023 role: KarpenterNodeRole-prod-eks-cluster subnetSelectorTerms: - tags: karpenter.sh/discovery: "prod-eks-cluster" securityGroupSelectorTerms: - tags: karpenter.sh/discovery: "prod-eks-cluster" tags: Environment: "production" CostCenter: "DevOps-FinOps" ``` ### Step 3: Zero-Downtime Workload Drain & Migration To shift 50+ microservices from Auto Mode nodes to our new Standard Karpenter nodes without dropping traffic, we used a progressive cordon-and-drain strategy: **Cordon Auto Mode Nodes:** Prevent new pods from scheduling onto Auto Mode nodes. ```json kubectl cordon -l eks.amazonaws.com/compute-type=auto ``` * **Deploy PodDisruptionBudgets (PDBs):** Ensure every microservice had active PDBs to guarantee minimum availability during node drains. * **Safely Drain Nodes:** Safely evict workloads. Karpenter immediately detected the pending pods and launched optimized Standard EC2 instances in parallel.Bash ```json kubectl drain -l eks.amazonaws.com/compute-type=auto \ --ignore-daemonsets \ --delete-emptydir-data \ --grace-period=60 ``` **Verify Traffic:** Using Prometheus metrics and CloudWatch Synthetics canary checks, we verified zero 5xx errors during the entire transition. ## EKS Auto Mode vs. Standard Mode: Decision Matrix Should you ever use EKS Auto Mode? **Yes—in the right context.** Use this quick decision matrix to evaluate your architecture:

Scenario / Need

🟢 Use EKS Auto Mode

🔴 Use Standard Mode (Karpenter + IaC)

Team Structure

Small team, no dedicated Platform Engineers

Dedicated DevOps / Platform Engineers

Workload Scale

Standard web apps, low-throughput APIs

Large scale, high-throughput, latency-sensitive

FinOps Strategy

Okay with standard pricing

Aggressive cost optimization (Spot, AMD, custom consolidation)

OS Customization

Bottlerocket default settings are fine

Need custom AMIs, kernel parameters (sysctl), eBPF agents

Control Preference

"Just run my containers, I don't care how"

"I need full governance over nodes, subnets, and lifecycle"

## Conclusion & Key Takeaways AWS EKS Auto Mode is an impressive engineering feat for teams that want to offload Kubernetes cluster management completely. However, **convenience always comes at the cost of control.** By migrating back to **Standard Mode with custom Karpenter NodePools**, we achieved: * **Full Node Control:** Total freedom over OS tuning, security agents, and instance types. * **35–40% Cost Reduction:** Through aggressive Spot allocation, AMD instance families, and custom Karpenter consolidation rules. * **Zero Unplanned Outages:** Elimination of forced 21-day node terminations on unready workloads. > **Rule of Thumb for Cloud Architects:** Always test managed abstraction layers against your actual production edge cases—not just happy-path benchmarks. Abstraction is amazing until you need to fix something inside the black box.