Problem#

  • Date: 2026-03-22
  • Environment: EKS Staging (goormgb-staging-eks)
  • Symptom: Elastic IPs automatically assigned to an internet-facing ALB, generating unexpected costs

Symptoms#

How It Was Discovered#

  1. Changed ArgoCD from type: LoadBalancer (NLB) to type: ClusterIP (ALB Ingress)
  2. After deleting the NLB, found 2 EIPs remaining
  3. Attempting to disassociate/release the EIPs resulted in a permission error

Error Message#

An error occurred (AuthFailure) when calling the DisassociateAddress operation:
You do not have permission to access the specified resource.

Confirmed State#

aws ec2 describe-network-interfaces \
  --filters "Name=addresses.private-ip-address,Values=10.0.18.47" \
  --query 'NetworkInterfaces[*].[Description,NetworkInterfaceId]'

# Result: ELB app/k8s-stagingalb-4f414fcf8f/...

Root Cause#

AWS Official Answer#

Automatic EIP assignment to internet-facing ALBs is expected behavior.

Key Details#

ItemDescription
Auto-assignedInternet-facing ALBs automatically get EIPs from the EC2 public IPv4 address pool
Service-managedShown as service_managed: ALB, fully managed by AWS
Cannot be modifiedManual disassociate/release is not possible (permission error is expected)
Not counted against quotaDoes not count toward the account’s EIP limit
Auto-releasedAutomatically released when the ALB is deleted

Cost Analysis#

Public IPv4 Address Pricing (effective February 2024)#

  • Per hour: $0.005 per IP
  • ALB configuration: 2 AZs = 2 IPs
  • Monthly cost: 2 × $0.005 × 24 × 30 = ~$7.20/month

Staging Environment Estimated Cost#

ALB EIP (2 IPs)     : ~$7.20/month
NAT Gateway EIP     : ~$3.60/month (separate)
----------------------------
Total Public IP cost : ~$10.80/month

Options#

Option 1: Keep Current Setup (Recommended - Staging)#

Internet → ALB (Public IP) → Pod

Pros:

  • No configuration changes
  • Simplest architecture
  • Direct access to management tools like Grafana, ArgoCD

Cons:

  • ~$7/month cost

Best for:

  • Staging/Dev environments
  • When direct internet access to management tools is needed
  • Convenience over cost

Option 2: Internal ALB + CloudFront VPC Origin#

Internet → CloudFront → (VPC Origin) → Internal ALB → Pod

Pros:

  • No ALB Public IP cost
  • CloudFront caching and security benefits
  • Easy WAF integration

Cons:

  • Separate CloudFront costs
  • Complex VPC Origin configuration
  • All domains must route through CloudFront

Configuration:

# Change Ingress annotation
alb.ingress.kubernetes.io/scheme: internal  # internet-facing → internal

# CloudFront VPC Origin (Terraform)
resource "aws_cloudfront_vpc_origin" "alb" {
  vpc_origin_endpoint_config {
    name                   = "alb-origin"
    arn                    = aws_lb.internal.arn
    http_port              = 80
    https_port             = 443
    origin_protocol_policy = "https-only"
  }
}

Best for:

  • Production environments
  • Enhanced security requirements
  • All traffic can go through CloudFront

Option 3: Management Tools via VPN Only#

api.staging.playball.one → CloudFront → Internal ALB
grafana/argocd/kiali     → VPN → Internal ALB

Pros:

  • Management tools not exposed to the internet (improved security)
  • Only the API is publicly accessible

Cons:

  • VPN setup and operation required
  • Less convenient access to management tools

Best for:

  • High-security environments
  • Organizations already running VPN infrastructure

Option 4: BYOIP + IPAM (Enterprise)#

Own IP block → AWS IPAM integration → ALB assignment

Pros:

  • Can reduce Public IPv4 costs
  • Use of fixed IP blocks
  • Easier IP allowlist management

Cons:

  • BYOIP purchase required
  • Complex setup
  • Cost-effective only at large scale

Best for:

  • Large-scale production
  • IP allowlist requirements
  • Organizations with existing IP blocks

Decision for Current Staging#

Choice: Option 1 (Keep Current Setup)#

Reasons:

  1. Staging environment - ~$7/month cost is acceptable
  2. Direct access to Grafana, ArgoCD, Kiali is needed
  3. Avoiding additional configuration complexity
  4. Project is ending (2026-04), short-term operation

Current Architecture#

                    ┌─────────────────────────────────────────┐
                    │              Internet                    │
                    └─────────────────┬───────────────────────┘
                                      │
              ┌───────────────────────┼───────────────────────┐
              │                       │                       │
              ▼                       ▼                       ▼
    ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
    │   CloudFront    │    │   Direct Access │    │   Direct Access │
    │ api.staging...  │    │ grafana.staging │    │ argocd.staging  │
    └────────┬────────┘    └────────┬────────┘    └────────┬────────┘
             │                      │                      │
             └──────────────────────┼──────────────────────┘
                                    │
                                    ▼
                    ┌───────────────────────────────┐
                    │  ALB (k8s-stagingalb-...)     │
                    │  - internet-facing            │
                    │  - Auto EIP (AWS managed)     │
                    │  - 2 AZ (ap-northeast-2a/c)   │
                    └───────────────┬───────────────┘
                                    │
              ┌─────────────────────┼─────────────────────┐
              │                     │                     │
              ▼                     ▼                     ▼
    ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
    │ Istio Gateway   │  │    Grafana      │  │    ArgoCD       │
    │ → VirtualService│  │  (monitoring)   │  │    (argocd)     │
    │ → Backend Pods  │  └─────────────────┘  └─────────────────┘
    └─────────────────┘

ALB Ingress (staging/charts/alb-ingress/values.yaml)#

ingress:
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing  # can change to internal
    alb.ingress.kubernetes.io/target-type: ip

ArgoCD Application (staging/root/values.yaml)#

gitCharts:
  alb-ingress:
    enabled: true
    path: staging/charts/alb-ingress
    namespace: istio-system
    syncWave: "5"
    syncOptions:
      - ApplyOutOfSyncOnly=true
      - PrunePropagationPolicy=foreground
      - Prune=false  # disabled due to multi-namespace deployment

References#

Troubleshooting Checklist#

When EIP errors occur#

  • Confirm EIP shows service_managed: ALB
  • Confirm ALB is internet-facing
  • EIP cannot be manually released — delete the ALB instead

Things to check when recreating an ALB#

  • Confirm External-DNS updates to the new ALB address
  • Wait for DNS propagation (1–2 minutes)
  • Confirm Target Group registration

ArgoCD alb-ingress app issues#

  • Confirm Prune=false is set (multi-namespace deployment)
  • Prevent deletion of resources outside the destination namespace