CloudFront & S3: Building a Global CDN for High-Traffic Web Apps

Why CDN Architecture Matters

A poorly configured CDN is worse than no CDN. If your cache hit rate is under 80%, you're paying for CloudFront and still hammering your origin. After rearchitecting our CDN, we went from 42% to 96% cache hit rate, cut origin load by 18×, and reduced P95 global latency from 340ms to 28ms.

96%

cache hit rate

28ms

P95 global latency

18×

origin load reduction

10M+

daily requests

Multi-Origin Architecture

CloudFront Multi-Origin Architecture

Cache Policy Design

The #1 CDN mistake: using default cache behavior for everything. Different content types need radically different TTLs and cache key components.

resource "aws_cloudfront_distribution" "main" {
  # Static assets — long TTL, hash in filename
  ordered_cache_behavior {
    path_pattern    = "/assets/*"
    allowed_methods = ["GET", "HEAD"]
    # TTL: 365 days (content-hash filenames)
    cache_policy_id = aws_cloudfront_cache_policy.static.id
  }
  # API routes — no caching, pass all headers
  ordered_cache_behavior {
    path_pattern             = "/api/*"
    allowed_methods          = ["DELETE","GET","HEAD","OPTIONS","PATCH","POST","PUT"]
    cache_policy_id          = "4135ea2d-6df8-44a3-9df3-4b5a84be39ad"  # CachingDisabled
    origin_request_policy_id = "b689b0a8-53d0-40ab-baf2-68738e2966ac"  # AllViewer
    target_origin_id         = "alb-origin"
  }
}

Lambda@Edge Security Headers

// origin-response: inject security headers at the edge
export const handler = async (event) => {
  const { response } = event.Records[0].cf;
  const set = (k, v) => response.headers[k] = [{ key: k, value: v }];
  set('strict-transport-security', 'max-age=63072000; includeSubDomains; preload');
  set('x-frame-options', 'DENY');
  set('x-content-type-options', 'nosniff');
  set('referrer-policy', 'strict-origin-when-cross-origin');
  set('permissions-policy', 'camera=(), microphone=(), geolocation=()');
  return response;
};

S3 Origin Access Control

Never make your S3 bucket public. Use Origin Access Control (OAC) — the modern replacement for OAI — to ensure S3 objects are only accessible through CloudFront. OAC supports KMS-encrypted buckets, which OAI did not.

💡Enable CloudFront real-time logs → Kinesis → S3 → Athena. Query your top uncached URL paths weekly. These are your highest-leverage cache policy improvements.

Results

After this architecture: P95 global TTFB dropped from 340ms to 28ms. S3 GET request costs dropped 94%. CDN data transfer costs stayed flat despite 3× traffic growth. The platform handled a 20× Black Friday spike with zero auto-scaling events on the origin.