Chrome QUIC/HTTP3 Intermittent 404 Troubleshooting
Resolving intermittent 404 errors in Chrome on a home server (kubeadm) + SK Broadband environment
Home Server Infrastructure#
1.1 Overall Network Architecture#
flowchart TB
subgraph Internet["Internet"]
CF["Cloudflare Edge<br/>(Seoul PoP)"]
ISP["SK Broadband<br/>Public IP: Dynamic<br/>(39.119.192.15)"]
end
subgraph HomeNetwork["Home Network (192.168.45.0/24)"]
subgraph Router["SK Broadband Router"]
NAT["NAT/DHCP"]
FW_R["Firewall"]
PF["Port Forwarding<br/>:80 → .154:80<br/>:443 → .154:443"]
DHCP_RES["DHCP Reservation<br/>mini-gmk: .123<br/>mini-might: .154"]
end
subgraph ControlPlane["mini-gmk (Control Plane)<br/>192.168.45.123"]
UFW_C["ufw firewall<br/>22, 6443, 10250-10252<br/>2379-2380, 179"]
K8S_CP["Kubernetes Control Plane<br/>kube-apiserver<br/>etcd, scheduler<br/>controller-manager"]
ISTIOD["istiod (pilot)"]
CERTM["cert-manager"]
DDNS["ddns-cloudflare<br/>CronJob"]
ESO["external-secrets-operator"]
CALICO_CP["calico-typha<br/>calico-kube-controllers"]
end
subgraph Worker["mini-might (Worker)<br/>192.168.45.154"]
UFW_W["ufw firewall<br/>22, 80, 443<br/>10250, 30000-32767"]
INGRESS["istio-ingressgateway<br/>externalIPs: .154"]
GW["java-cloud-gateway<br/>:8085"]
AUTH["auth-guard (x2)<br/>:8080"]
APPS["order-core :8083<br/>seat :8082<br/>queue :8081"]
CALICO_W["calico-node"]
end
subgraph VPN_Access["Team Member Access"]
VPN["Tailscale/WireGuard<br/>VPN"]
SSH["SSH Tunnel<br/>:22"]
end
end
subgraph Clients["Clients"]
Chrome["Chrome<br/>(QUIC/HTTP3)"]
Safari["Safari<br/>(HTTP/2)"]
Firefox["Firefox<br/>(HTTP/2)"]
Mobile["Mobile LTE"]
TeamMember["Team Member (VPN)"]
end
Chrome -->|"QUIC/UDP:443<br/>❌ intermittent failure"| ISP
Safari -->|"HTTP/2/TCP:443"| ISP
Firefox -->|"HTTP/2/TCP:443"| ISP
Mobile -->|"HTTP/2"| ISP
Chrome -->|"QUIC/UDP:443<br/>✅ stable"| CF
CF -->|"HTTP/2/TCP:443"| ISP
ISP --> NAT
NAT --> FW_R
FW_R --> PF
PF -->|":443"| UFW_W
UFW_W --> INGRESS
INGRESS --> GW
GW --> AUTH
AUTH --> APPS
TeamMember -->|"VPN"| VPN
VPN --> SSH
SSH --> UFW_C
SSH --> UFW_W
DDNS -->|"API call"| CF
ISTIOD -.->|"xDS"| INGRESS
CERTM -.->|"TLS cert"| INGRESS
ESO -.->|"Secret sync"| DDNS
classDef problem fill:#ff6b6b,stroke:#c0392b,color:#fff
classDef solution fill:#2ecc71,stroke:#27ae60,color:#fff
classDef infrastructure fill:#3498db,stroke:#2980b9,color:#fff
class Chrome problem
class CF solution
class Router,ControlPlane,Worker infrastructure
1.2 SK Broadband Router Settings#
flowchart LR
subgraph SKRouter["SK Broadband Router Settings"]
direction TB
subgraph DHCP["DHCP Settings"]
DHCP_RANGE["DHCP range: 192.168.45.100 ~ .200"]
DHCP_RES1["Reservation 1: mini-gmk<br/>MAC: XX:XX:XX:XX:XX:XX<br/>IP: 192.168.45.123"]
DHCP_RES2["Reservation 2: mini-might<br/>MAC: YY:YY:YY:YY:YY:YY<br/>IP: 192.168.45.154"]
end
subgraph PortForward["Port Forwarding"]
PF_HTTP["External :80 → 192.168.45.154:80"]
PF_HTTPS["External :443 → 192.168.45.154:443"]
PF_VPN["External :51820 → 192.168.45.123:51820<br/>(WireGuard, optional)"]
end
subgraph Firewall["Firewall Settings"]
FW_IN["Inbound: allow 80, 443"]
FW_OUT["Outbound: allow all"]
FW_ICMP["ICMP: allow (ping)"]
end
subgraph NAT_Config["NAT Settings"]
NAT_TYPE["NAT type: Symmetric"]
NAT_UDP["UDP timeout: 30s (root cause)"]
NAT_TCP["TCP timeout: 3600s"]
end
end
style NAT_UDP fill:#ff6b6b,stroke:#c0392b,color:#fff
Router settings detail:
| Setting | Value | Note |
|---|---|---|
| Router model | SK Broadband default | ipTIME or SK OEM model |
| Internal IP | 192.168.45.1 | Gateway |
| DHCP range | 192.168.45.100 ~ .200 | Static IPs assigned outside range |
| DNS | Auto (SK DNS) | 168.126.63.1, 168.126.63.2 |
| UDP NAT timeout | 30s (estimated) | Root cause of QUIC issue |
| TCP NAT timeout | 3600s | HTTP/2 is stable |
1.3 DHCP Reservation (Fixed IPs)#
# Configure DHCP reservations in the router admin panel
# Or configure static IPs on each node
# mini-gmk (Control Plane)
# /etc/netplan/00-installer-config.yaml (Ubuntu 22.04)
network:
version: 2
ethernets:
enp2s0:
addresses:
- 192.168.45.123/24
routes:
- to: default
via: 192.168.45.1
nameservers:
addresses:
- 168.126.63.1
- 8.8.8.8
# mini-might (Worker)
network:
version: 2
ethernets:
enp3s0:
addresses:
- 192.168.45.154/24
routes:
- to: default
via: 192.168.45.1
nameservers:
addresses:
- 168.126.63.1
- 8.8.8.8
1.4 Linux Firewall (ufw) Settings#
flowchart TB
subgraph UFW_Rules["ufw Firewall Rules"]
direction LR
subgraph MiniGMK["mini-gmk (Control Plane)"]
direction TB
UFW_G1["22/tcp - SSH"]
UFW_G2["6443/tcp - K8s API Server"]
UFW_G3["2379-2380/tcp - etcd"]
UFW_G4["10250-10252/tcp - kubelet, scheduler, controller"]
UFW_G5["179/tcp - Calico BGP"]
UFW_G6["4789/udp - Calico VXLAN"]
UFW_G7["51820/udp - WireGuard VPN"]
end
subgraph MiniMight["mini-might (Worker)"]
direction TB
UFW_M1["22/tcp - SSH"]
UFW_M2["80/tcp - HTTP"]
UFW_M3["443/tcp - HTTPS"]
UFW_M4["10250/tcp - kubelet"]
UFW_M5["30000-32767/tcp - NodePort"]
UFW_M6["4789/udp - Calico VXLAN"]
end
end
style UFW_M2 fill:#2ecc71,stroke:#27ae60,color:#fff
style UFW_M3 fill:#2ecc71,stroke:#27ae60,color:#fff
mini-gmk (Control Plane) ufw settings:
# Default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing
# SSH
sudo ufw allow 22/tcp comment 'SSH'
# Kubernetes Control Plane
sudo ufw allow 6443/tcp comment 'K8s API Server'
sudo ufw allow 2379:2380/tcp comment 'etcd'
sudo ufw allow 10250/tcp comment 'kubelet'
sudo ufw allow 10251/tcp comment 'kube-scheduler'
sudo ufw allow 10252/tcp comment 'kube-controller-manager'
# Calico CNI
sudo ufw allow 179/tcp comment 'Calico BGP'
sudo ufw allow 4789/udp comment 'Calico VXLAN'
sudo ufw allow from 192.168.45.0/24 comment 'Internal network'
# VPN (optional)
sudo ufw allow 51820/udp comment 'WireGuard'
# Enable
sudo ufw enable
mini-might (Worker) ufw settings:
# Default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing
# SSH
sudo ufw allow 22/tcp comment 'SSH'
# Web Traffic (Istio IngressGateway)
sudo ufw allow 80/tcp comment 'HTTP'
sudo ufw allow 443/tcp comment 'HTTPS'
# Kubernetes Worker
sudo ufw allow 10250/tcp comment 'kubelet'
sudo ufw allow 30000:32767/tcp comment 'NodePort Services'
# Calico CNI
sudo ufw allow 4789/udp comment 'Calico VXLAN'
sudo ufw allow from 192.168.45.0/24 comment 'Internal network'
# Enable
sudo ufw enable
1.5 Team Member VPN/SSH Access#
sequenceDiagram
participant TM as Team Member
participant TS as Tailscale/WireGuard<br/>VPN Server
participant GMK as mini-gmk<br/>192.168.45.123
participant MIGHT as mini-might<br/>192.168.45.154
participant K8S as Kubernetes<br/>API Server
Note over TM,K8S: VPN connection and SSH access flow
TM->>TS: VPN connection request
TS-->>TM: VPN tunnel established<br/>(10.x.x.x range)
TM->>GMK: SSH access (via VPN tunnel)<br/>ssh admin@192.168.45.123
GMK-->>TM: SSH session established
TM->>GMK: kubectl commands
GMK->>K8S: API request (:6443)
K8S-->>GMK: API response
GMK-->>TM: Result returned
Note over TM,K8S: Using k9s, helm, argocd CLI, etc.
TM->>MIGHT: SSH access (if needed)<br/>ssh admin@192.168.45.154
MIGHT-->>TM: SSH session established
Tailscale setup (recommended):
# Install Tailscale on each node
curl -fsSL https://tailscale.com/install.sh | sh
# Login (on each node)
sudo tailscale up --ssh
# Team member from their PC
tailscale up
ssh admin@mini-gmk # using Tailscale MagicDNS
Problem Description#
2.1 Symptoms#
flowchart LR
subgraph TestResults["Test Results"]
direction TB
subgraph Success["✅ Working"]
CURL["curl (CLI)"]
SAFARI["Safari"]
FIREFOX["Firefox"]
MOBILE["Mobile LTE"]
INCOGNITO["Chrome Incognito"]
end
subgraph Fail["❌ Intermittent 404"]
CHROME["Chrome (normal)"]
CHROME_TEAM["Team members' Chrome"]
end
end
CURL -->|"HTTP/2"| OK1["200 OK"]
SAFARI -->|"HTTP/2"| OK2["200 OK"]
FIREFOX -->|"HTTP/2"| OK3["200 OK"]
MOBILE -->|"HTTP/2"| OK4["200 OK"]
INCOGNITO -->|"QUIC attempt"| RANDOM1["Random 404"]
CHROME -->|"QUIC/HTTP3"| RANDOM2["Random 404"]
CHROME_TEAM -->|"QUIC/HTTP3"| RANDOM3["Random 404"]
style CHROME fill:#ff6b6b,stroke:#c0392b,color:#fff
style CHROME_TEAM fill:#ff6b6b,stroke:#c0392b,color:#fff
style RANDOM2 fill:#ff6b6b,stroke:#c0392b,color:#fff
style RANDOM3 fill:#ff6b6b,stroke:#c0392b,color:#fff
| Test | Result | Protocol |
|---|---|---|
curl -I https://api.dev.goormgb.space/swagger-ui/index.html | 200 OK | HTTP/2 |
| Safari | OK | HTTP/2 |
| Firefox | OK | HTTP/2 |
| Mobile LTE network | OK | HTTP/2 |
| Chrome (same network) | Intermittent 404 | QUIC/HTTP3 |
| Team members’ Chrome | Same symptom | QUIC/HTTP3 |
2.2 Affected Scope#
# Affected domains (all pointing to home server)
- api.dev.goormgb.space # Backend API (Swagger UI)
- argocd.goormgb.space # GitOps
- grafana.goormgb.space # Monitoring
- kiali.goormgb.space # Service Mesh visualization
- cloudbeaver.goormgb.space # DB Admin
2.3 Detailed Symptom#
sequenceDiagram
participant User as User
participant Chrome as Chrome
participant Router as SK Router
participant GW as IngressGateway
Note over User,GW: Random 404 when pressing F5 repeatedly
User->>Chrome: F5 (refresh 1)
Chrome->>Router: QUIC/UDP:443
Router->>GW: Forward
GW-->>Router: 200 OK
Router-->>Chrome: 200 OK
Chrome-->>User: ✅ OK
User->>Chrome: F5 (refresh 2)
Chrome->>Router: QUIC/UDP:443
Note over Router: UDP NAT state unstable
Router--xChrome: Timeout/drop
Chrome->>Chrome: HTTP/2 fallback attempt
Note over Chrome: Connection state confused
Chrome-->>User: ❌ 404 Not Found
User->>Chrome: F5 (refresh 3)
Chrome->>Router: HTTP/2/TCP:443
Router->>GW: Forward
GW-->>Router: 200 OK
Router-->>Chrome: 200 OK
Chrome-->>User: ✅ OK
User->>Chrome: F5 (refresh 4)
Chrome->>Router: QUIC/UDP:443 (retry)
Router--xChrome: Fails again
Chrome-->>User: ❌ 404 Not Found
Root Cause Analysis#
3.1 How Chrome QUIC/HTTP3 Works#
flowchart TB
subgraph Chrome["Chrome Browser"]
direction TB
REQ["Request start"]
ALT_SVC["Check Alt-Svc header"]
QUIC_TRY["Attempt QUIC connection<br/>(UDP:443)"]
H2_FALLBACK["HTTP/2 Fallback<br/>(TCP:443)"]
POOL["Connection Pool"]
end
subgraph HomeRouter["SK Broadband Router"]
direction TB
NAT_UDP["UDP NAT table"]
NAT_TCP["TCP NAT table"]
UDP_TIMEOUT["UDP timeout: 30s"]
TCP_TIMEOUT["TCP timeout: 3600s"]
end
REQ --> ALT_SVC
ALT_SVC -->|"h3=':443' found"| QUIC_TRY
QUIC_TRY -->|"UDP:443"| NAT_UDP
NAT_UDP --> UDP_TIMEOUT
UDP_TIMEOUT -->|"expires after 30s"| QUIC_TRY
QUIC_TRY -->|"failure"| H2_FALLBACK
H2_FALLBACK --> NAT_TCP
NAT_TCP --> TCP_TIMEOUT
QUIC_TRY -->|"on success"| POOL
H2_FALLBACK -->|"on success"| POOL
POOL -->|"reuse"| REQ
style UDP_TIMEOUT fill:#ff6b6b,stroke:#c0392b,color:#fff
style QUIC_TRY fill:#f39c12,stroke:#d68910,color:#fff
3.2 The Core Problem: UDP NAT Timeout#
flowchart LR
subgraph Problem["Where the Problem Occurs"]
direction TB
subgraph QUIC_Conn["QUIC Connection"]
Q1["1. Chrome establishes QUIC connection"]
Q2["2. Registered in SK router UDP NAT table"]
Q3["3. No traffic for 30 seconds"]
Q4["4. Entry removed from NAT table"]
Q5["5. Next QUIC packet dropped"]
Q6["6. Chrome confused → 404"]
end
Q1 --> Q2 --> Q3 --> Q4 --> Q5 --> Q6
end
subgraph Compare["TCP vs UDP Comparison"]
direction TB
TCP["TCP (HTTP/2)<br/>Timeout: 3600s<br/>Keep-Alive supported<br/>Stable state tracking"]
UDP["UDP (QUIC)<br/>Timeout: 30s<br/>No connection state<br/>Unstable NAT traversal"]
end
style Q4 fill:#ff6b6b,stroke:#c0392b,color:#fff
style Q5 fill:#ff6b6b,stroke:#c0392b,color:#fff
style Q6 fill:#ff6b6b,stroke:#c0392b,color:#fff
style UDP fill:#ff6b6b,stroke:#c0392b,color:#fff
style TCP fill:#2ecc71,stroke:#27ae60,color:#fff
3.3 SK Broadband Environment Specifics#
flowchart TB
subgraph ISP_Issues["SK Broadband Environment Issues"]
direction TB
ISP1["1. Symmetric NAT<br/>NAT type unfavorable for QUIC"]
ISP2["2. UDP port filtering<br/>Some ISPs restrict UDP:443"]
ISP3["3. Short UDP session timeout<br/>Home router default: 30s"]
ISP4["4. Dynamic public IP<br/>Connection drops on IP change"]
end
subgraph HomeLab_Issues["Home Lab Issues"]
direction TB
HL1["1. Possible double NAT<br/>Router + modem"]
HL2["2. Port forwarding limitations<br/>Difficult to track UDP state"]
HL3["3. Firewall settings<br/>ufw UDP handling"]
HL4["4. Using externalIPs<br/>No LoadBalancer"]
end
ISP1 --> ISP2 --> ISP3 --> ISP4
HL1 --> HL2 --> HL3 --> HL4
ISP3 -.->|"combined"| PROBLEM["Unstable QUIC connection"]
HL2 -.->|"combined"| PROBLEM
style PROBLEM fill:#ff6b6b,stroke:#c0392b,color:#fff
3.4 Istio IngressGateway and QUIC#
flowchart TB
subgraph Istio["Istio IngressGateway"]
direction TB
SVC["Service: istio-ingressgateway<br/>type: ClusterIP<br/>externalIPs: [192.168.45.154]"]
ENVOY["Envoy Proxy<br/>HTTP/2, QUIC supported"]
GW_RES["Gateway Resource<br/>TLS Termination"]
VS["VirtualService<br/>Routing rules"]
end
subgraph TLS_Config["TLS Configuration"]
CERT["Let's Encrypt<br/>Wildcard certificate"]
SECRET["Secret: goormgb-tls<br/>*.goormgb.space<br/>*.dev.goormgb.space"]
end
subgraph Problem["Problem Points"]
EXT_IP["externalIPs binding<br/>also receives UDP traffic"]
UDP_FW["443/udp open in ufw?<br/>→ NAT issue even if open"]
QUIC_HANDSHAKE["QUIC handshake<br/>must complete before NAT timeout"]
end
SVC --> ENVOY --> GW_RES --> VS
CERT --> SECRET --> GW_RES
EXT_IP -.-> UDP_FW -.-> QUIC_HANDSHAKE
style QUIC_HANDSHAKE fill:#f39c12,stroke:#d68910,color:#fff
Solution#
4.1 Available Options#
flowchart TB
subgraph Solutions["Solutions"]
direction TB
subgraph Client["Client-side (not recommended)"]
C1["Disable Chrome QUIC<br/>chrome://flags/#enable-quic"]
C2["Disable DoH<br/>chrome://flags/#dns-over-https"]
end
subgraph Network["Network-side (difficult)"]
N1["Replace router<br/>Increase UDP NAT timeout"]
N2["Change ISP<br/>Not realistic"]
end
subgraph Server["Server-side (adopted)"]
S1["Cloudflare Proxy<br/>QUIC → HTTP/2 conversion"]
S2["Disable QUIC<br/>Remove Alt-Svc on server"]
end
end
C1 -->|"requires config on all team members' machines"| BAD1["❌ Not recommended"]
C2 -->|"reset on browser update"| BAD2["❌ Not recommended"]
N1 -->|"cost + complexity"| BAD3["△ Fallback"]
N2 -->|"not realistic"| BAD4["❌ Not feasible"]
S1 -->|"free + additional benefits"| GOOD1["✅ Adopted"]
S2 -->|"gives up QUIC benefits"| BAD5["△ Fallback"]
style GOOD1 fill:#2ecc71,stroke:#27ae60,color:#fff
style BAD1 fill:#e74c3c,stroke:#c0392b,color:#fff
style BAD2 fill:#e74c3c,stroke:#c0392b,color:#fff
style BAD4 fill:#e74c3c,stroke:#c0392b,color:#fff
4.2 Why Cloudflare Proxy Was Chosen#
flowchart LR
subgraph Before["Before: Direct Connection"]
B_Chrome["Chrome"]
B_ISP["SK Broadband"]
B_Router["Router<br/>UDP NAT unstable"]
B_Ingress["IngressGateway"]
B_Chrome -->|"QUIC/UDP"| B_ISP
B_ISP -->|"UDP"| B_Router
B_Router -->|"❌ intermittent drop"| B_Ingress
end
subgraph After["After: Cloudflare Proxy"]
A_Chrome["Chrome"]
A_CF["Cloudflare Edge<br/>(Seoul PoP)"]
A_ISP["SK Broadband"]
A_Router["Router<br/>TCP stable"]
A_Ingress["IngressGateway"]
A_Chrome -->|"QUIC/UDP"| A_CF
A_CF -->|"HTTP/2/TCP"| A_ISP
A_ISP -->|"TCP"| A_Router
A_Router -->|"✅ stable"| A_Ingress
end
Before -.->|"migration"| After
style B_Router fill:#ff6b6b,stroke:#c0392b,color:#fff
style A_CF fill:#2ecc71,stroke:#27ae60,color:#fff
style A_Router fill:#2ecc71,stroke:#27ae60,color:#fff
Cloudflare benefits:
| Item | Description |
|---|---|
| QUIC handling | Converts QUIC ↔ HTTP/2 at the edge |
| Connection stability | TCP connection to origin (NAT-friendly) |
| Client configuration | No changes needed |
| DDoS protection | Included by default |
| Global CDN | Leverages Seoul PoP |
| SSL certificate | Universal SSL auto-managed |
| Cost | Free plan is sufficient |
Cloudflare Migration#
5.1 Full Migration Flow#
flowchart TB
subgraph Step1["Step 1: Cloudflare Setup"]
CF_SIGNUP["Sign up for Cloudflare"]
CF_DOMAIN["Add domain<br/>goormgb.space"]
CF_RECORDS["Configure DNS records"]
CF_SSL["SSL/TLS settings<br/>Full (strict)"]
end
subgraph Step2["Step 2: Nameserver Change"]
PORKBUN["Porkbun (registrar)"]
NS_CHANGE["Change nameservers<br/>Route53 → Cloudflare"]
NS_VERIFY["Confirm propagation<br/>(up to 48 hours)"]
end
subgraph Step3["Step 3: DDNS Migration"]
DDNS_OLD["ddns-route53<br/>(delete)"]
DDNS_NEW["ddns-cloudflare<br/>(new)"]
SECRET_NEW["ExternalSecret<br/>Cloudflare API Token"]
end
subgraph Step4["Step 4: Verification"]
DNS_CHECK["Confirm DNS propagation"]
CF_HEADER["Check Cloudflare headers"]
BROWSER_TEST["Test in Chrome"]
end
CF_SIGNUP --> CF_DOMAIN --> CF_RECORDS --> CF_SSL
CF_SSL --> PORKBUN --> NS_CHANGE --> NS_VERIFY
NS_VERIFY --> DDNS_OLD --> DDNS_NEW --> SECRET_NEW
SECRET_NEW --> DNS_CHECK --> CF_HEADER --> BROWSER_TEST
style DDNS_OLD fill:#e74c3c,stroke:#c0392b,color:#fff
style DDNS_NEW fill:#2ecc71,stroke:#27ae60,color:#fff
5.2 Cloudflare DNS Record Configuration#
flowchart LR
subgraph CloudflareDNS["Cloudflare DNS Records"]
direction TB
subgraph Proxied["Proxied (orange cloud)"]
A1["A | @ | 39.119.192.15"]
A2["A | api.dev | 39.119.192.15"]
A3["A | argocd | 39.119.192.15"]
A4["A | grafana | 39.119.192.15"]
A5["A | kiali | 39.119.192.15"]
A6["A | cloudbeaver | 39.119.192.15"]
end
subgraph DNSOnly["DNS Only (grey cloud)"]
CNAME1["CNAME | dev | cname.vercel-dns.com"]
end
end
A1 --> CF_EDGE["Cloudflare Edge<br/>QUIC → HTTP/2"]
A2 --> CF_EDGE
A3 --> CF_EDGE
A4 --> CF_EDGE
A5 --> CF_EDGE
A6 --> CF_EDGE
CF_EDGE --> HOME["Home server<br/>39.119.192.15"]
CNAME1 --> VERCEL["Vercel<br/>(frontend)"]
style Proxied fill:#f39c12,stroke:#d68910,color:#fff
style DNSOnly fill:#95a5a6,stroke:#7f8c8d,color:#fff
| Type | Name | Content | Proxy | TTL | Purpose |
|---|---|---|---|---|---|
| A | @ | 39.119.192.15 | Proxied | Auto | Root domain |
| A | api.dev | 39.119.192.15 | Proxied | Auto | Backend API |
| A | argocd | 39.119.192.15 | Proxied | Auto | GitOps |
| A | grafana | 39.119.192.15 | Proxied | Auto | Monitoring |
| A | kiali | 39.119.192.15 | Proxied | Auto | Service Mesh |
| A | cloudbeaver | 39.119.192.15 | Proxied | Auto | DB Admin |
| CNAME | dev | cname.vercel-dns.com | DNS only | Auto | Frontend |
Important: The
devsubdomain is hosted on Vercel, so it must be DNS only (grey cloud)
5.3 SSL/TLS Settings#
flowchart LR
subgraph CloudflareSSL["Cloudflare SSL/TLS"]
direction TB
MODE["Encryption Mode:<br/>Full (strict)"]
EDGE_CERT["Edge Certificate:<br/>Cloudflare Universal SSL"]
ORIGIN_CERT["Origin Certificate:<br/>Let's Encrypt (existing)"]
SETTINGS["Edge Certificate Settings:"]
S1["Always Use HTTPS: On"]
S2["Minimum TLS: 1.2"]
S3["TLS 1.3: Enabled"]
S4["Automatic HTTPS Rewrites: On"]
end
CLIENT["Client"]
CF_EDGE["Cloudflare Edge"]
ORIGIN["Home server (Origin)"]
CLIENT -->|"HTTPS<br/>Cloudflare SSL"| CF_EDGE
CF_EDGE -->|"HTTPS<br/>Let's Encrypt"| ORIGIN
style MODE fill:#2ecc71,stroke:#27ae60,color:#fff
Full (strict) mode requirements:
- Origin server needs a valid SSL certificate
- We use Let’s Encrypt wildcard certificate → requirement satisfied
5.4 Nameserver Change#
sequenceDiagram
participant User as Admin
participant Porkbun as Porkbun<br/>(registrar)
participant R53 as Route53<br/>(old)
participant CF as Cloudflare<br/>(new)
Note over User,CF: Nameserver migration
User->>CF: 1. Request to add domain
CF-->>User: 2. Provide nameserver info<br/>jacob.ns.cloudflare.com<br/>rita.ns.cloudflare.com
User->>Porkbun: 3. Change nameservers
Note over Porkbun: Route53 NS → Cloudflare NS
Porkbun->>CF: 4. NS propagation begins
Note over CF: Up to 48 hours<br/>(usually 1-2 hours)
CF-->>User: 5. Domain activation complete
Note over R53: Route53 Hosted Zone<br/>no longer in use<br/>(kept as cert-manager backup)
Changing nameservers in Porkbun:
# Old (Route53)
ns-xxx.awsdns-xx.com
ns-xxx.awsdns-xx.net
ns-xxx.awsdns-xx.org
ns-xxx.awsdns-xx.co.uk
# New (Cloudflare)
jacob.ns.cloudflare.com
rita.ns.cloudflare.com
5.5 DDNS Helm Chart Migration#
flowchart TB
subgraph Before["Before: Route53 DDNS"]
direction TB
CHART_OLD["dev/charts/core/ddns-route53/"]
IMAGE_OLD["amazon/aws-cli:2.15.0"]
SECRET_OLD["route53-ddns-credentials<br/>(AWS IAM)"]
API_OLD["aws route53 change-resource-record-sets"]
end
subgraph After["After: Cloudflare DDNS"]
direction TB
CHART_NEW["dev/charts/core/ddns/"]
IMAGE_NEW["curlimages/curl:8.5.0"]
SECRET_NEW["cloudflare-ddns-credentials<br/>(ExternalSecret)"]
API_NEW["Cloudflare API<br/>PATCH /zones/{zone_id}/dns_records"]
end
CHART_OLD -->|"deleted"| REMOVED["Removed"]
CHART_NEW -->|"newly created"| ACTIVE["Active"]
style CHART_OLD fill:#e74c3c,stroke:#c0392b,color:#fff
style REMOVED fill:#e74c3c,stroke:#c0392b,color:#fff
style CHART_NEW fill:#2ecc71,stroke:#27ae60,color:#fff
style ACTIVE fill:#2ecc71,stroke:#27ae60,color:#fff
values-ddns.yaml#
# 303-goormgb-k8s-helm/dev/values/core/values-ddns.yaml
domain: "goormgb.space"
zoneID: "f9fdc81712963738bc9d5ee628849e53"
records:
- "" # goormgb.space (root)
- "api.dev" # api.dev.goormgb.space
- "argocd" # argocd.goormgb.space
- "grafana" # grafana.goormgb.space
- "kiali" # kiali.goormgb.space
- "cloudbeaver" # cloudbeaver.goormgb.space
ttl: 60 # fast IP change propagation
schedule: "*/5 * * * *" # check every 5 minutes
secret:
name: cloudflare-ddns-credentials
awsSecretKey: dev/infra/cloudflare # AWS Secrets Manager path
image:
repository: curlimages/curl
tag: "8.5.0"
CronJob Template#
# 303-goormgb-k8s-helm/dev/charts/core/ddns/templates/cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: ddns-cloudflare-updater
namespace: {{ .Release.Namespace }}
labels:
app.kubernetes.io/name: ddns-cloudflare
spec:
schedule: {{ .Values.schedule | quote }}
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
serviceAccountName: default
containers:
- name: ddns-updater
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
env:
- name: CF_API_TOKEN
valueFrom:
secretKeyRef:
name: {{ .Values.secret.name }}
key: api-token
- name: CF_ZONE_ID
value: {{ .Values.zoneID | quote }}
- name: DOMAIN
value: {{ .Values.domain | quote }}
- name: TTL
value: {{ .Values.ttl | quote }}
command:
- /bin/sh
- -c
- |
set -e
echo "=========================================="
echo "DDNS Cloudflare Updater"
echo "Started at: $(date -Iseconds)"
echo "=========================================="
# 1. Get current public IP (with fallback)
echo "[Step 1] Getting current public IP..."
CURRENT_IP=$(curl -sf --max-time 10 https://api.ipify.org || \
curl -sf --max-time 10 https://checkip.amazonaws.com || \
curl -sf --max-time 10 https://ifconfig.me)
if [ -z "$CURRENT_IP" ]; then
echo "ERROR: Failed to get public IP from all sources"
exit 1
fi
echo "Current Public IP: ${CURRENT_IP}"
echo ""
# 2. Update each record
echo "[Step 2] Processing DNS records..."
{{- range .Values.records }}
RECORD="{{ . }}"
if [ -z "$RECORD" ]; then
FULL_NAME="${DOMAIN}"
else
FULL_NAME="${RECORD}.${DOMAIN}"
fi
echo "----------------------------------------"
echo "Processing: ${FULL_NAME}"
# 2-1. Look up current record in Cloudflare
RESPONSE=$(curl -sf -X GET \
"https://api.cloudflare.com/client/v4/zones/${CF_ZONE_ID}/dns_records?type=A&name=${FULL_NAME}" \
-H "Authorization: Bearer ${CF_API_TOKEN}" \
-H "Content-Type: application/json")
RECORD_ID=$(echo "$RESPONSE" | grep -o '"id":"[^"]*"' | head -1 | cut -d'"' -f4)
CURRENT_CONTENT=$(echo "$RESPONSE" | grep -o '"content":"[^"]*"' | head -1 | cut -d'"' -f4)
PROXIED=$(echo "$RESPONSE" | grep -o '"proxied":[^,]*' | head -1 | cut -d':' -f2)
if [ -z "$RECORD_ID" ]; then
echo " WARNING: Record not found for ${FULL_NAME}, skipping..."
continue
fi
echo " Record ID: ${RECORD_ID}"
echo " Current IP in Cloudflare: ${CURRENT_CONTENT}"
echo " Proxied: ${PROXIED}"
# 2-2. Check if IP has changed
if [ "$CURRENT_CONTENT" = "$CURRENT_IP" ]; then
echo " Status: UNCHANGED (no update needed)"
continue
fi
# 2-3. Update IP
echo " Updating: ${CURRENT_CONTENT} → ${CURRENT_IP}"
UPDATE_RESULT=$(curl -sf -X PATCH \
"https://api.cloudflare.com/client/v4/zones/${CF_ZONE_ID}/dns_records/${RECORD_ID}" \
-H "Authorization: Bearer ${CF_API_TOKEN}" \
-H "Content-Type: application/json" \
--data "{\"content\":\"${CURRENT_IP}\",\"ttl\":${TTL},\"proxied\":true}")
if echo "$UPDATE_RESULT" | grep -q '"success":true'; then
echo " Status: UPDATED successfully"
else
echo " Status: FAILED"
echo " Response: $UPDATE_RESULT"
fi
{{- end }}
echo ""
echo "=========================================="
echo "DDNS update completed at $(date -Iseconds)"
echo "=========================================="
ExternalSecret Template#
# 303-goormgb-k8s-helm/dev/charts/core/ddns/templates/external-secret.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: cloudflare-ddns-credentials
namespace: {{ .Release.Namespace }}
labels:
app.kubernetes.io/name: ddns-cloudflare
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: aws-secrets-manager
target:
name: {{ .Values.secret.name }}
creationPolicy: Owner
data:
- secretKey: api-token
remoteRef:
key: {{ .Values.secret.awsSecretKey }}
property: api-token
5.6 AWS Secrets Manager Configuration#
flowchart LR
subgraph AWS["AWS Secrets Manager"]
SECRET["dev/infra/cloudflare"]
JSON["{<br/> 'api-token': 'cf_xxx...'<br/>}"]
end
subgraph K8s["Kubernetes"]
ESO["External Secrets Operator"]
K8S_SECRET["Secret:<br/>cloudflare-ddns-credentials"]
CRONJOB["CronJob:<br/>ddns-cloudflare-updater"]
end
SECRET --> JSON
ESO -->|"sync every 1h"| SECRET
ESO --> K8S_SECRET
CRONJOB -->|"env: CF_API_TOKEN"| K8S_SECRET
style SECRET fill:#f39c12,stroke:#d68910,color:#fff
# Save Cloudflare API Token in AWS Secrets Manager
aws secretsmanager create-secret \
--name dev/infra/cloudflare \
--secret-string '{"api-token":"YOUR_CLOUDFLARE_API_TOKEN"}' \
--region ap-northeast-2
# Cloudflare API Token permissions:
# - Zone > DNS > Edit
# - Zone > Zone > Read
# - Scope: goormgb.space domain only
6. Full Traffic Flow (After Migration)#
6.1 Detailed Request Flow#
sequenceDiagram
participant Chrome as Chrome<br/>(Client)
participant CF_Edge as Cloudflare Edge<br/>(Seoul PoP)
participant ISP as SK Broadband<br/>(Public IP)
participant Router as Home Router<br/>(NAT)
participant UFW as ufw Firewall<br/>(mini-might)
participant Ingress as IngressGateway<br/>(Istio)
participant GW as java-cloud-gateway<br/>(:8085)
participant Backend as Backend Service
Note over Chrome,Backend: https://api.dev.goormgb.space/swagger-ui/index.html
Chrome->>CF_Edge: 1. QUIC/HTTP3 request<br/>(UDP:443)
Note over CF_Edge: QUIC connection terminated<br/>Converted to HTTP/2
CF_Edge->>ISP: 2. HTTP/2 request<br/>(TCP:443)
Note over ISP: TCP connection<br/>(stable)
ISP->>Router: 3. Port forwarding<br/>:443 → 192.168.45.154:443
Note over Router: TCP NAT table<br/>(timeout 3600s)
Router->>UFW: 4. Pass through firewall<br/>(443/tcp allowed)
UFW->>Ingress: 5. Forward to Envoy Proxy
Note over Ingress: TLS Termination<br/>(Let's Encrypt)
Ingress->>GW: 6. VirtualService routing<br/>→ java-cloud-gateway:8085
GW->>Backend: 7. Internal routing<br/>→ swagger-ui service
Backend-->>GW: 8. Response (200 OK)
GW-->>Ingress: 9. Forward response
Ingress-->>Router: 10. HTTPS response
Router-->>ISP: 11. NAT reverse translation
ISP-->>CF_Edge: 12. HTTP/2 response
CF_Edge-->>Chrome: 13. QUIC/HTTP3 response<br/>(200 OK)
Note over Chrome,Backend: ✅ All requests handled reliably
6.2 DDNS Update Flow#
sequenceDiagram
participant Cron as CronJob<br/>(mini-gmk)
participant IPify as api.ipify.org
participant CF_API as Cloudflare API
participant CF_DNS as Cloudflare DNS
Note over Cron,CF_DNS: Runs every 5 minutes (*/5 * * * *)
Cron->>IPify: 1. GET / (get public IP)
IPify-->>Cron: 39.119.192.15
loop For each record
Cron->>CF_API: 2. GET /zones/{zone}/dns_records<br/>?name=api.dev.goormgb.space
CF_API-->>Cron: record_id, current_ip
alt IP changed
Cron->>CF_API: 3. PATCH /dns_records/{id}<br/>{"content": "39.119.192.15", "proxied": true}
CF_API-->>Cron: {"success": true}
CF_API->>CF_DNS: Update DNS record
Note over Cron: "api.dev: updated 1.2.3.4 → 39.119.192.15"
else IP unchanged
Note over Cron: "api.dev: unchanged (39.119.192.15)"
end
end
Note over Cron,CF_DNS: TTL: 60s for fast propagation
7. Verification#
7.1 DNS Propagation Check#
# Check Cloudflare nameservers
dig NS goormgb.space +short
# Expected result:
# jacob.ns.cloudflare.com.
# rita.ns.cloudflare.com.
# Check A record - returns Cloudflare Anycast IP
dig api.dev.goormgb.space +short
# Expected result (Cloudflare IP, not origin IP):
# 104.21.x.x
# 172.67.x.x
7.2 Check Cloudflare Headers#
curl -I https://api.dev.goormgb.space/swagger-ui/index.html
# Expected response:
HTTP/2 200
date: Mon, 17 Mar 2026 10:00:00 GMT
content-type: text/html
server: cloudflare
cf-ray: 8xxxxx-ICN # ICN = Seoul Edge
cf-cache-status: DYNAMIC
alt-svc: h3=":443"; ma=86400 # HTTP/3 support (CF ↔ client)
7.3 Verify DDNS CronJob#
# CronJob status
kubectl get cronjob -n core
# NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE
# ddns-cloudflare-updater */5 * * * * False 0 2m
# Recent job logs
kubectl logs -n core -l app.kubernetes.io/name=ddns-cloudflare --tail=50
# Expected output:
# ==========================================
# DDNS Cloudflare Updater
# Started at: 2026-03-17T10:05:00+09:00
# ==========================================
# [Step 1] Getting current public IP...
# Current Public IP: 39.119.192.15
#
# [Step 2] Processing DNS records...
# ----------------------------------------
# Processing: goormgb.space
# Record ID: abc123...
# Current IP in Cloudflare: 39.119.192.15
# Proxied: true
# Status: UNCHANGED (no update needed)
# ----------------------------------------
# Processing: api.dev.goormgb.space
# Record ID: def456...
# Current IP in Cloudflare: 39.119.192.15
# Proxied: true
# Status: UNCHANGED (no update needed)
# ...
# ==========================================
# DDNS update completed at 2026-03-17T10:05:03+09:00
# ==========================================
# Manual run test
kubectl create job ddns-test --from=cronjob/ddns-cloudflare-updater -n core
kubectl logs -n core job/ddns-test -f
kubectl delete job ddns-test -n core
7.4 Chrome Test#
flowchart TB
subgraph TestSteps["Chrome Test Steps"]
direction TB
STEP1["1. Clear cache<br/>chrome://net-internals/#dns<br/>→ Clear host cache"]
STEP2["2. Clear sockets<br/>chrome://net-internals/#sockets<br/>→ Close idle sockets"]
STEP3["3. Access test<br/>https://api.dev.goormgb.space/swagger-ui/index.html"]
STEP4["4. Rapid F5 test<br/>(10+ times)"]
STEP5["5. Check protocol<br/>DevTools → Network → Protocol"]
end
STEP1 --> STEP2 --> STEP3 --> STEP4 --> STEP5
STEP5 --> RESULT["Check result"]
RESULT --> H2["h2 (HTTP/2): CF↔Origin"]
RESULT --> H3["h3 (HTTP/3): Client↔CF"]
style RESULT fill:#2ecc71,stroke:#27ae60,color:#fff
Expected results:
- All requests return 200 OK
- Protocol column shows
h2orh3 cf-rayheader showsICN(Seoul)
8. Removed Resources#
flowchart LR
subgraph Deleted["Removed Resources"]
D1["ddns-route53 Helm Chart"]
D2["ddns-route53 ArgoCD App"]
D3["route53-ddns-credentials Secret"]
end
subgraph Kept["Retained Resources"]
K1["Route53 Hosted Zone<br/>(kept as cert-manager backup)"]
end
D1 -->|"rm -rf"| REMOVED["Removed"]
D2 -->|"kubectl delete app"| REMOVED
D3 -->|"replaced by ESO"| REMOVED
K1 --> BACKUP["Kept as backup"]
style Deleted fill:#e74c3c,stroke:#c0392b,color:#fff
style REMOVED fill:#e74c3c,stroke:#c0392b,color:#fff
style Kept fill:#f39c12,stroke:#d68910,color:#fff
| Resource | Status | Note |
|---|---|---|
ddns-route53 Helm Chart | Deleted | dev/charts/core/ddns-route53/ |
ddns-route53 ArgoCD App | Deleted | kubectl delete app ddns-route53 -n argocd |
route53-ddns-credentials | Deleted | Replaced by ESO |
| Route53 Hosted Zone | Retained | Backup for cert-manager DNS-01 |
9. Key Takeaways#
mindmap
root((Chrome QUIC<br/>404 Troubleshooting))
Root Cause
Chrome QUIC/HTTP3
Uses UDP:443
Activated via Alt-Svc header
SK Broadband Router
UDP NAT timeout 30s
Symmetric NAT
Home server environment
Port forwarding
externalIPs
Solution
Cloudflare Proxy
QUIC → HTTP/2 conversion
Connection terminated at edge
Only TCP to origin
DDNS migration
Route53 → Cloudflare
CronJob reimplemented
Additional benefits
DDoS protection
Global CDN
Auto SSL
Free plan
Changed files
values-ddns.yaml
cronjob.yaml
external-secret.yaml
10. References#
- Cloudflare: Understanding HTTP/3
- Chrome QUIC Protocol
- Cloudflare API: DNS Records
- Istio: Gateway Configuration
- NAT Traversal for QUIC
Appendix A: Troubleshooting Command Reference#
# === DNS checks ===
dig NS goormgb.space +short
dig api.dev.goormgb.space +short
dig api.dev.goormgb.space +trace
# === Check Cloudflare headers ===
curl -I https://api.dev.goormgb.space/swagger-ui/index.html
curl -svo /dev/null https://api.dev.goormgb.space/ 2>&1 | grep -i "cf-"
# === DDNS CronJob ===
kubectl get cronjob -n core
kubectl logs -n core -l app.kubernetes.io/name=ddns-cloudflare --tail=50
kubectl create job ddns-test --from=cronjob/ddns-cloudflare-updater -n core
# === Istio IngressGateway ===
kubectl get svc -n istio-system istio-ingressgateway -o yaml
kubectl logs -n istio-system -l app=istio-ingressgateway --tail=100
# === Gateway/VirtualService ===
kubectl get gw,vs -n istio-system
kubectl describe gw api-gateway -n istio-system
# === Certificate check ===
kubectl get certificate -n istio-system
kubectl describe certificate goormgb-tls -n istio-system
# === ufw status ===
# mini-gmk
sudo ufw status verbose
# mini-might
sudo ufw status verbose
# === Network connectivity test ===
# External access from inside
curl -I https://api.dev.goormgb.space/swagger-ui/index.html
# Direct access to service inside cluster
kubectl exec -n istio-system deploy/istio-ingressgateway -- \
curl -I http://java-cloud-gateway.dev-webs.svc.cluster.local:8085/swagger-ui/index.html
Appendix B: How to Check Router UDP NAT Timeout#
# 1. Listen for UDP on an external server (if available)
nc -lu 12345
# 2. Send UDP packet from the home server
echo "test" | nc -u <external-server-ip> 12345
# 3. Send again after 30 seconds → confirm connection is dropped
# Or check NAT type using a STUN server
# https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/
Written: 2026-03-17 Environment: kubeadm cluster + SK Broadband + Cloudflare Related repos: 302-goormgb-k8s-bootstrap, 303-goormgb-k8s-helm