spicy-automation/docs/COST_OPTIMIZATION_TESTING.md

Cost Optimization for Testing Environments

Barebones setup guide for VPC → Cluster → Service with blue/green deployment and internet-facing ALB.

CloudFormation Import Pattern

All examples use the CloudFormation import pattern for minimal configuration:

• VPC Stack: Only numberOfAzs needs to be specified (exports ${vpcStackName}-NumberOfAZs)
• Cluster Stack: Uses vpcStackName to auto-import VPC ID, CIDR, subnets, and numberOfAzs. Exports ${clusterStackName}-VPCStackName for other stacks.
• Service Stack: Uses clusterName to import VPC stack name from cluster export, then imports VPC details from VPC stack
• ALB Stack: Uses clusterName to import VPC stack name from cluster export, then imports VPC details from VPC stack

Key benefit: numberOfAzs only needs to be specified once (in the VPC stack), and all other stacks automatically import it.

Cost Breakdown

Optimized Testing Setup

• VPC: 2 AZs, 2 NAT Gateways = ~$64/month (NAT) + ~$2/month (EIPs)
• Cluster: 1 x t3.small instance (Spot) = ~$5-8/month, OR Fargate-only (0 instances) = $0/month
• ALB: Individual ALB per service = ~$16/month
• Service: 1 task = minimal additional cost
• Total: ~$87-90/month (with EC2) or ~$82-84/month (Fargate-only)

Savings: ~70-80% reduction vs production setup

Barebones Blue/Green Setup

Complete Configuration

@Library(["spicy-automation@main"]) _

// 1. VPC (2 AZs, minimal cost)
spicyVPC(
    jenkinsAwsCredentialsId: "aws-credentials",
    region: "ca-central-1",
    stackName: "test-vpc",
    ownerTag: "Test",
    productTag: "test",
    componentTag: "vpc",
    numberOfAzs: 2,                          // 2 AZs instead of 4 (saves 2 NAT Gateways)
    createAdditionalPrivateSubnets: false,   // Skip NACL subnets
)

// 2. Cluster (with Fargate enabled for cheaper option)
spicyECSCluster(
    jenkinsAwsCredentialsId: "aws-credentials",
    region: "ca-central-1",
    stackName: "test-cluster",
    vpcStackName: "test-vpc",                 // Auto-imports VPC ID, CIDR, subnets, numberOfAzs
    ownerTag: "Test",
    productTag: "test",
    componentTag: "cluster",
    environment: "test",

    // Option A: EC2 with Spot (cheaper than on-demand)
    instanceType: "t3.small",                // Minimal viable instance
    minClusterSize: 1,                       // Single instance
    maxClusterSize: 1,                       // No scaling
    spotEnabled: true,                       // Use Spot for 60-70% savings
    onDemandPercentage: 0,                   // 100% Spot
    enableFargate: true,                    // Enable Fargate capacity providers

    // Option B: Fargate-only (no EC2 costs)
    // enableFargate: true,
    // minClusterSize: 0,                    // No EC2 instances
    // maxClusterSize: 0,                    // No EC2 instances

    containerInsights: false,                // Disable Container Insights (saves CloudWatch costs)
    createExternalLoadBalancer: false,       // Service creates its own ALB
    createInternalLoadBalancer: false,
)

// 3. Service with Blue/Green (pipeline auto-creates ALB stack)
spicyECSService(
    jenkinsAwsCredentialsId: "aws-credentials",
    region: "ca-central-1",
    stackName: "test-service",               // Pipeline adds -blue/-green suffixes
    serviceName: "test-api",
    clusterName: "test-cluster",             // Auto-imports VPC from cluster stack
    vpcStackName: "test-vpc",                 // Auto-imports subnets, numberOfAzs from VPC stack

    // Container (minimal resources)
    image: "nexus.kodeniks.com/docker-hosted/test-api:latest",
    containerPort: 3000,
    desiredCount: 1,                         // Single task
    cpu: 256,                                // Minimal CPU (0.25 vCPU)
    memory: 512,                             // Minimal memory (512 MiB)

    // Use Fargate Spot if cluster has no EC2 instances (Option B above)
    // capacityProviderStrategy: [
    //     [capacityProvider: "FARGATE_SPOT", weight: 1],
    // ],

    // Blue/Green Configuration
    blueGreen: true,
    hostName: "api-test.example.com",        // Auto-generates active/inactive hostnames
    // bgHostedZoneId: "Z1234567890ABCDEF", // Route53 hosted zone (optional, for Route53 DNS)

    // ALB Configuration (pipeline auto-creates ALB stack if not exists)
    useClusterAlb: false,                    // Use individual ALB (not cluster ALB)
    albScheme: "internet-facing",
    certificateArn: "arn:aws:acm:ca-central-1:123456789:certificate/xxx",
    redirectHttpToHttps: true,
    healthCheckPath: "/health",

    // Tags
    ownerTag: "Test",
    productTag: "test",
    componentTag: "api",
    environment: "test",

    // Test hooks
    blueGreenTest: { args, buildInfo ->
        sh "curl -f https://${buildInfo.inactiveHostname}/health"
    },
    smokeTest: { args, buildInfo ->
        sh "curl -f https://${buildInfo.activeHostname}/health"
    },
)

What happens:

1. VPC Stack: Creates 2 AZs with 2 NAT Gateways (~$66/mo)
2. Cluster Stack: Creates 1 Spot instance ($5-8/mo) OR 0 instances with Fargate ($0/mo)
3. ALB Stack: Pipeline automatically creates test-service-alb stack (~$16/mo)
4. Service Stack: Pipeline creates test-service-green (inactive) referencing the ALB
5. DNS: Route53 records created automatically if bgHostedZoneId provided, OR manual Cloudflare setup (see below)
6. Swap: Pipeline swaps ALB listener rules (no DNS changes)

Cost Breakdown:

Component	Cost (EC2)	Cost (Fargate)
VPC (2 AZs, 2 NAT)	$66/mo	$66/mo
Cluster (1 Spot instance)	$5-8/mo	$0/mo
Individual ALB	$16/mo	$16/mo
Service (1 task)	Minimal	$2-3/mo
Total	~$87-90/mo	~$82-84/mo

Key Simplifications:

• ✅ Uses vpcStackName and clusterName for auto-imports (no manual subnet IDs)
• ✅ Pipeline automatically creates ALB stack (test-service-alb)
• ✅ Uses hostName parameter (auto-generates active/inactive hostnames)
• ✅ Minimal resources: 1 Spot instance (or Fargate), 1 task, 256 CPU / 512 MB memory
• ✅ Route53 DNS managed automatically (if bgHostedZoneId provided)

Cloudflare DNS Setup

If your DNS is hosted with Cloudflare (not Route53), you'll need to manually configure DNS records after deployment:

1. Get ALB DNS name from ALB stack outputs (ALB stack name is {serviceStackName}-alb):

   # Get ALB DNS name from ALB stack
   aws cloudformation describe-stacks \
     --stack-name test-service-alb \
     --query 'Stacks[0].Outputs[?OutputKey==`LoadBalancerDNS`].OutputValue' \
     --output text
   

   Or find it in the AWS Console: EC2 → Load Balancers → Your ALB → DNS name

2. Create Cloudflare DNS records (set once, never change):

   Active hostname:

   • Type: CNAME
   • Name: api-test (or your subdomain)
   • Target: {alb-dns-name} (e.g., test-service-alb-123456789.ca-central-1.elb.amazonaws.com)
   • Proxy: Enabled (orange cloud) for Cloudflare protection
   • TTL: Auto

   Inactive hostname:

   • Name: inactive-api-test
   • Target: Same ALB DNS name (both point to same ALB, routing is by hostname)

   Note: These DNS records point to the ALB and never change. The ALB handles routing internally based on the Host header.

3. SSL/TLS Settings in Cloudflare:

   • Set to Full or Full (strict) mode
   • This ensures HTTPS works end-to-end (Cloudflare → ALB → Service)

How Blue/Green Works

┌──────────────────────────────────────────────────────────┐
│                    Cloudflare DNS                        │
│  ┌──────────────────┐  ┌───────────────────────────┐     │
│  │ api-test.example │  │ inactive-api-test.example │     │
│  │     .com         │  │         .com              │     │
│  └────────┬─────────┘  └──────────────┬────────────┘     │
└───────────┼───────────────────────────┼──────────────────┘
            │                           │
            │  (Both point to same ALB) │
            └──────────────┬────────────┘
                           ▼
            ┌──────────────────────────────┐
            │   AWS ALB (Individual)       │
            │  ┌─────────────┐ ┌─────────┐ │
            │  │ Host: api-  │ │ Host:   │ │
            │  │ test...     │ │ inactive│ │
            │  │ → BLUE      │ │ → GREEN │ │
            │  └─────┬───────┘ └────┬────┘ │
            └────────┼──────────────┼──────┘
                     │              │
            ┌────────▼───────┐ ┌────▼──────────┐
            │  BLUE Service  │ │ GREEN Service │
            │  (Active)      │ │ (Inactive)    │
            └────────────────┘ └───────────────┘

Deployment Flow:

1. Deploy new version to inactive (GREEN) service
2. Test via inactive-api-test.example.com in Cloudflare
3. Pipeline swaps ALB hostname routing (BLUE ↔ GREEN)
4. Active traffic now goes to new version
5. Old version stays running for rollback window

Important: The hostname does NOT change during deployments. DNS records point to the ALB (set once, never change), and the ALB routes traffic based on the Host header. The pipeline swaps ALB listener rules (not DNS), so rollback is instant (~30 seconds).

Rollback

The pipeline handles ALB routing swaps automatically. For manual rollback:

@Library(["spicy-automation@main"]) _

spicyRollback(
    jenkinsAwsCredentialsId: "aws-credentials",
    region: "ca-central-1",
    stackName: "test-service",
    serviceName: "test-api",
    // ... same config as deploy ...
    activeHostname: "api-test.example.com",
    inactiveHostname: "inactive-api-test.example.com",
    // If using Route53:
    // bgHostedZoneId: "Z1234567890ABCDEF",
)

This swaps the hostname routing in the ALB (not DNS records), so rollback is instant (~30 seconds). Your DNS records (Route53 or Cloudflare) never change.

Important Notes

1. ACM Certificate: You'll need an ACM certificate in the same region as your ALB

   • Request via AWS Console or CLI
   • Must validate domain ownership
   • Can use DNS validation (add CNAME to Cloudflare)

2. Cloudflare Proxy: Keep proxy enabled (orange cloud) for:

   • DDoS protection
   • CDN caching (if desired)
   • SSL termination at Cloudflare edge

3. Hostname Routing: The ALB routes by Host header, so both Cloudflare DNS records point to the same ALB DNS name

4. Testing: Use the inactive hostname for integration tests before swapping

5. Fargate vs EC2: For cheapest option, use Fargate-only (set minClusterSize: 0 and maxClusterSize: 0 in cluster, then use FARGATE_SPOT capacity provider in service)

Cleanup

Remember to destroy test stacks when done:

# Via Jenkins or AWS CLI
aws cloudformation delete-stack --stack-name test-service-blue
aws cloudformation delete-stack --stack-name test-service-green
aws cloudformation delete-stack --stack-name test-service-alb
aws cloudformation delete-stack --stack-name test-cluster
aws cloudformation delete-stack --stack-name test-vpc

Summary

Barebones blue/green setup:

• VPC: 2 AZs, 2 NAT Gateways
• Cluster: 1 Spot instance (or Fargate-only with 0 instances)
• Service: 1 task with blue/green deployment
• ALB: Individual internet-facing ALB (auto-created by pipeline)
• DNS: Route53 (automatic) or Cloudflare (manual)
• Total: ~$87-90/month (EC2) or ~$82-84/month (Fargate)