EKS Node Cluster Join Failure Checklist

Problem#

• EKS node group created but nodes are not joining the cluster
• kubectl get nodes shows no nodes
• EC2 instances are in Running state

Checklist#

1. VPC DNS Settings#

Verification:

resource "aws_vpc" "main" {
  enable_dns_hostnames = true
  enable_dns_support   = true
}

2. Subnet Tags#

The EKS controller requires specific tags to recognize subnets

Verification:

# modules/vpc/main.tf - Public Subnet
tags = {
  "kubernetes.io/role/elb"                        = "1"
  "kubernetes.io/cluster/${var.eks_cluster_name}" = "shared"
}

# modules/vpc/main.tf - Private Subnet
tags = {
  "kubernetes.io/role/internal-elb"               = "1"
  "kubernetes.io/cluster/${var.eks_cluster_name}" = "shared"
}

3. Subnet Settings#

4. Security Group Communication#

Required ports between nodes and the control plane

When using terraform-aws-modules/eks:

# Required SG rules are created automatically
attach_cluster_primary_security_group = true

5. Cluster Endpoint Settings#

Note: If private_access = false, nodes must go through NAT → public internet, which complicates routing.

6. VPC Endpoints (Required for AL2023)#

eks-auth VPC Endpoint (PrivateLink)#

Why:

• AL2023 uses nodeadm (replaces legacy bootstrap.sh)
• nodeadm uses EKS Pod Identity for authentication
• eks-auth API is PrivateLink-only (not reachable via NAT Gateway)

Symptom:

• Node bootstrap log shows nodeadm: done!
• But node never registers with the cluster

Fix:

# config.yaml
vpc_endpoints:
  - eks-auth

Other VPC Endpoints#

Conclusion: When using NAT Gateway, only eks-auth is mandatory; the rest are optional.

7. AMI Type and User Data#

AL2 vs AL2023 Differences#

User Data Conflict Check#

• Custom user_data can conflict with AL2023 bootstrap
• When using terraform-aws-modules/eks, it’s handled automatically — best not to override

Verification:

# modules/eks/main.tf - Node Group
infra = {
  ami_type = "AL2023_ARM_64_STANDARD"
  # no custom user_data or launch_template
}

8. IAM Policies#

Required policies for the node IAM Role

9. EKS Access Entries#

Authentication method for EKS 1.30+

SSO access configuration:

# config.yaml
eks:
  access_entries:
    devops_sso:
      enabled: true
      role_arn: "arn:aws:iam::ACCOUNT_ID:role/aws-reserved/sso.amazonaws.com/REGION/AWSReservedSSO_..."

Debugging Commands#

Check EC2 instance logs#

# Connect to node via SSM
aws ssm start-session --target <instance-id>

# Bootstrap logs
sudo journalctl -u nodeadm -f
sudo cat /var/log/cloud-init-output.log

# kubelet status
sudo systemctl status kubelet
sudo journalctl -u kubelet -f

Check cluster status#

# Node list
kubectl get nodes

# Cluster info
kubectl cluster-info

# Check Access Entries
aws eks list-access-entries --cluster-name <cluster-name>

Final Checklist#

• VPC DNS settings (enable_dns_hostnames, enable_dns_support)
• Subnet tags (kubernetes.io/role/elb, kubernetes.io/cluster/...)
• Public subnet map_public_ip_on_launch = true
• Security group allows ports 443 and 10250
• cluster_endpoint_private_access = true
• eks-auth VPC Endpoint created (required for AL2023)
• AMI type confirmed (AL2023_ARM_64_STANDARD)
• No custom User Data
• IAM policies include AmazonEKSWorkerNodeMinimalPolicy

Notes#

• Missing eks-auth VPC Endpoint is the primary cause for EKS 1.35 + AL2023 combinations
• NAT Gateway alone cannot reach the eks-auth API (PrivateLink only)
• terraform-aws-modules/eks v20.x handles most settings automatically