Files

Ryan Wilson 68684df471 Initial commit: Spicy CDK automation framework

Jenkins shared library and CDK constructs for AWS infrastructure.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2025-11-18 22:21:00 -08:00

30 KiB

Raw Blame History

ECS Service Deployment

Deploy ECS services with mixed capacity provider strategies for EC2 + Fargate burst.

Features

Blue/Green Deployments - Zero-downtime deploys with instant rollback via hostname swap
Mixed Capacity Provider Strategy - EC2 base + Fargate burst for cost optimization
Auto-scaling on CPU, Memory, and ALB request count
ALB Integration with host/path routing
Deployment Circuit Breaker with automatic rollback
ECS Exec for interactive debugging
Secrets Manager integration
CloudWatch Logs with configurable retention
Pipeline Hooks - Custom behavior at every stage

Quick Start

Minimal Jenkinsfile (EC2 Only) - Using CloudFormation Imports

Minimal props: Only clusterName required. VPC info auto-imports from cluster stack exports.

@Library(["spicy-automation@main"]) _

spicyECSService(
    jenkinsAwsCredentialsId: "aws-credentials",
    region: "ca-central-1",
    stackName: "my-api-dev",
    serviceName: "my-api",

    // Cluster info - VPC ID auto-imports from ${clusterName}-VPC
    clusterName: "my-cluster-dev",

    // Container
    image: "nexus.kodeniks.com/docker-hosted/my-api:latest",
    containerPort: 3000,

    // Tags
    ownerTag: "MyTeam",
    productTag: "my-product",
    componentTag: "api",
    environment: "dev"
)

What auto-imports:

VPC stack name from ${clusterStackName}-VPCStackName (cluster stack export)
VPC ID from ${clusterStackName}-VPC (cluster stack export)
VPC CIDR from ${vpcStackName}-VPCCIDR (VPC stack export, using imported VPC stack name)
Private subnets from ${vpcStackName}-PrivateSubnetA1ID, etc. (VPC stack exports)
Number of AZs from ${vpcStackName}-NumberOfAZs (VPC stack export)
Logs bucket from ${clusterStackName}-logs-s3-bucket (cluster stack export)

Note: The service stack imports the VPC stack name from the cluster stack export ${clusterStackName}-VPCStackName, then uses that to import all VPC details (CIDR, subnets, AZs) from the VPC stack exports.

Mixed Capacity Strategy (EC2 + Fargate Burst)

@Library(["spicy-automation@main"]) _

spicyECSService(
    jenkinsAwsCredentialsId: "aws-credentials",
    region: "ca-central-1",
    stackName: "my-api-prod",
    serviceName: "my-api",

    // Cluster info - VPC details auto-import from cluster/VPC stack exports
    clusterName: "my-cluster-prod",
    vpcStackName: "my-vpc",  // Optional: auto-imported from cluster stack if not provided

    // Container
    image: "nexus.kodeniks.com/docker-hosted/my-api:v1.2.3",
    containerPort: 3000,
    cpu: 512,
    memory: 1024,

    // Mixed capacity strategy
    capacityProviderStrategy: [
        [capacityProvider: "my-cluster-prod-ec2", base: 2, weight: 3],  // First 2 on EC2, then 75%
        [capacityProvider: "FARGATE_SPOT", weight: 1],                   // 25% burst to Fargate Spot
    ],

    // Scaling
    desiredCount: 2,
    minCapacity: 2,
    maxCapacity: 20,
    targetCpuUtilization: 70,
    targetMemoryUtilization: 80,

    // Routing - Use cluster ALB (default)
    useClusterAlb: true,  // Use cluster ALB (default)
    albScheme: "internet-facing",  // or "internal"
    hostHeader: "api.example.com",
    priority: 100,
    healthCheckPath: "/health",

    // Tags
    ownerTag: "Platform",
    productTag: "my-product",
    componentTag: "api",
    environment: "prod",

    // Approval for prod
    approvers: "admin,platform-team"
)

Parameters Reference

Required Parameters

Parameter	Description	Example
`jenkinsAwsCredentialsId`	Jenkins credential ID for AWS	`"aws-credentials"`
`region`	AWS region	`"ca-central-1"`
`stackName`	CloudFormation stack name	`"my-api-dev"`
`serviceName`	ECS service name	`"my-api"`
`clusterName`	ECS cluster name (required) - used to auto-import VPC stack name, VPC ID, and other details from cluster stack exports	`"my-cluster-dev"`
`clusterStackName`	Cluster stack name (defaults to `clusterName`) - used for CloudFormation imports	`"my-cluster-dev"`
`vpcStackName`	VPC stack name (auto-imported from `${clusterStackName}-VPCStackName`, or provide explicitly)	`"my-vpc"`
`albStackName`	ALB stack name (for individual ALB) - used to auto-import ALB ARN and listener ARNs from ALB stack	`"my-service-alb"`
`image`	Docker image URI	`"nexus.kodeniks.com/docker-hosted/my-api:latest"`
`containerPort`	Container port	`3000`
`ownerTag`	Owner tag	`"MyTeam"`
`productTag`	Product tag	`"my-product"`

Container Configuration

Parameter	Default	Description
`containerPort`	`3000`	Container port
`cpu`	`256`	CPU units (256 = 0.25 vCPU)
`memory`	`512`	Memory in MiB
`environment_vars`	-	Map of environment variables
`secrets`	-	Map of secret ARNs (env var name → ARN)

Capacity Provider Strategy

Parameter	Default	Description
`capacityProviderStrategy`	Cluster default	Array of capacity provider configs
`desiredCount`	`2`	Initial task count

Strategy Item Format:

[
    capacityProvider: "provider-name",  // EC2 provider name, "FARGATE", or "FARGATE_SPOT"
    base: 0,                            // Minimum tasks on this provider
    weight: 1                           // Distribution weight
]

Auto-Scaling

Parameter	Default	Description
`minCapacity`	-	Minimum tasks (required for scaling)
`maxCapacity`	-	Maximum tasks (required for scaling)
`targetCpuUtilization`	-	Target CPU % (0-100)
`targetMemoryUtilization`	-	Target memory % (0-100)
`targetRequestsPerTarget`	-	Target requests per task per minute

ALB Routing

Parameter	Default	Description
`useClusterAlb`	`true`	Use cluster ALB (default). Set to `false` to use individual ALB per service
`albScheme`	-	ALB scheme: `"internet-facing"` or `"internal"` (required when `useClusterAlb=true`)
`albStackName`	-	ALB stack name for individual ALB (auto-derived from `{stackName}-alb` when `useClusterAlb=false`)
`certificateArn`	-	ACM certificate ARN (required for individual ALB when `useClusterAlb=false`)
`redirectHttpToHttps`	-	Redirect HTTP to HTTPS (for individual ALB)
`hostHeader`	-	Host header for routing
`pathPatterns`	-	Comma-separated path patterns
`priority`	`100`	Listener rule priority
`stickiness`	`false`	Enable session stickiness
`stickinessDuration`	`86400`	Stickiness duration (seconds)

Health Check

Parameter	Default	Description
`healthCheckPath`	`/health`	Health check URL path

DNS (Route53) - Blue/Green Only

Parameter	Default	Description
`bgHostedZoneId`	-	Route53 hosted zone ID for blue/green DNS records (optional)
`hostName`	-	Simple hostname (e.g., "api.example.com") - auto-generates active/inactive
`activeHostname`	-	Active hostname (e.g., "api.example.com") - explicit hostname
`inactiveHostname`	-	Inactive hostname (e.g., "inactive-api.example.com") - explicit hostname

Deployment

Parameter	Default	Description
`circuitBreaker`	`true`	Enable deployment circuit breaker
`enableExecuteCommand`	`true`	Enable ECS Exec

Mixed Capacity Strategy Explained

How Task Distribution Works

capacityProviderStrategy: [
    [capacityProvider: "my-cluster-ec2", base: 2, weight: 3],
    [capacityProvider: "FARGATE_SPOT", weight: 1],
]

Distribution:

Base tasks go to their designated provider first
Additional tasks are distributed by weight ratio

Example with 10 tasks:

Tasks 1-2:  EC2 (base)
Tasks 3-8:  EC2 (weight 3 = 75%)
Tasks 9-10: FARGATE_SPOT (weight 1 = 25%)

Recommended Strategies

Development (Cost Optimized):

capacityProviderStrategy: [
    [capacityProvider: "FARGATE_SPOT", weight: 1],
]

Staging (Balanced):

capacityProviderStrategy: [
    [capacityProvider: "my-cluster-staging-ec2", base: 1, weight: 2],
    [capacityProvider: "FARGATE_SPOT", weight: 1],
]

Production (Reliability + Burst):

capacityProviderStrategy: [
    [capacityProvider: "my-cluster-prod-ec2", base: 2, weight: 3],
    [capacityProvider: "FARGATE_SPOT", weight: 1],
]

Building Docker Images

The pipeline automatically builds and pushes Docker images if a Dockerfile exists:

spicyECSService(
    // ... other params ...

    // Image will be built and pushed automatically
    serviceName: "my-api",  // Used as image name
    imageTag: env.BUILD_NUMBER,  // Or defaults to Git SHA

    // Optional: customize build
    dockerfile: "Dockerfile.prod",
    dockerContext: "./app",
    dockerBuildArgs: [
        NODE_ENV: "production",
        BUILD_DATE: new Date().format("yyyy-MM-dd")
    ],

    // Or skip build and use existing image
    buildImage: false,
    image: "nexus.kodeniks.com/docker-hosted/my-api:v1.2.3",
)

Environment Variables and Secrets

Plain Environment Variables

spicyECSService(
    // ...
    environment_vars: [
        NODE_ENV: "production",
        LOG_LEVEL: "info",
        API_URL: "https://api.example.com"
    ],
)

Secrets from Secrets Manager

spicyECSService(
    // ...
    secrets: [
        DATABASE_URL: "arn:aws:secretsmanager:ca-central-1:123456789:secret:my-db-creds",
        API_KEY: "arn:aws:secretsmanager:ca-central-1:123456789:secret:api-keys::api_key",
    ],
)

Format: "ENV_VAR_NAME": "secret-arn" or "ENV_VAR_NAME": "secret-arn::json-key"

Auto-Scaling Behavior

CPU-Based Scaling

spicyECSService(
    // ...
    minCapacity: 2,
    maxCapacity: 20,
    targetCpuUtilization: 70,
)

Scales up when average CPU > 70%
Scales down when average CPU < 70%
Cooldown: 60s scale-out, 300s scale-in

Request-Based Scaling

spicyECSService(
    // ...
    minCapacity: 2,
    maxCapacity: 20,
    targetRequestsPerTarget: 1000,  // 1000 requests/minute per task
)

Combined Scaling

spicyECSService(
    // ...
    minCapacity: 2,
    maxCapacity: 20,
    targetCpuUtilization: 70,
    targetMemoryUtilization: 80,
    targetRequestsPerTarget: 1000,
)

All policies run independently - whichever triggers first wins.

Deployment Circuit Breaker

Enabled by default. If a deployment fails:

ECS detects unhealthy tasks
After threshold failures, deployment stops
Automatic rollback to previous version
CloudWatch alarm triggered

Disable if needed:

spicyECSService(
    // ...
    circuitBreaker: false,
)

ECS Exec (Debugging)

Enabled by default. Connect to running containers:

aws ecs execute-command \
    --cluster my-cluster-dev \
    --task <task-id> \
    --container my-api \
    --interactive \
    --command "/bin/sh"

Stack Outputs

Output	Export Name	Description
`ServiceName`	`{stackName}-service-name`	ECS service name
`ServiceArn`	`{stackName}-service-arn`	ECS service ARN
`TaskDefinitionArn`	`{stackName}-task-definition-arn`	Task definition ARN
`LogGroupName`	`{stackName}-log-group`	CloudWatch log group
`TargetGroupArn`	`{stackName}-target-group-arn`	ALB target group ARN

Troubleshooting

Tasks Stuck in PENDING

Cause: No capacity available

Solutions:

Check EC2 instances have room for tasks
Verify Fargate is in capacity strategy
Check task CPU/memory fits available resources

Tasks Failing Health Checks

# Check container logs
aws logs tail /ecs/my-api --follow

# Check target group health
aws elbv2 describe-target-health --target-group-arn <arn>

Deployment Rolling Back

# Check deployment events
aws ecs describe-services --cluster my-cluster --services my-api

# Check task stopped reasons
aws ecs describe-tasks --cluster my-cluster --tasks <task-id>

ECS Exec Not Working

Verify enableExecuteCommand: true
Check task role has SSM permissions
Ensure SSM agent is running in container

Blue/Green Deployments

Blue/Green deployments provide zero-downtime releases with instant rollback capability.

How It Works

Two Services: myapp-blue and myapp-green run simultaneously
DNS Records: Both active and inactive hostnames point to the same ALB (DNS never changes)
ALB Listener Rules: Traffic routing is controlled by listener rule priorities, not DNS
Deploy to Inactive: New version deploys to inactive service with higher priority rule
Test Inactive: Run integration tests against inactive hostname (same ALB, different rule)
Swap Hostnames: Update listener rule priorities to swap active/inactive (no DNS changes)
Keep for Rollback: Old version stays running for rollback window

Important: DNS records are created once and never change. Both api.example.com and inactive-api.example.com always resolve to the same ALB. Traffic routing is controlled entirely by ALB listener rule priorities:

Active service: Lower priority (e.g., 100) for active hostname
Inactive service: Higher priority (e.g., 200) for inactive hostname
When swapping: Only the listener rule priorities are updated via CDK deployment

Architecture

                    ┌─────────────────────────────────────┐
                    │              ALB                     │
                    │  ┌─────────────┐  ┌─────────────┐   │
                    │  │ api.example │  │ inactive-api│   │
                    │  │  Priority:  │  │  Priority:  │   │
                    │  │    100      │  │    200      │   │
                    │  └──────┬──────┘  └──────┬──────┘   │
                    └─────────┼────────────────┼──────────┘
                              │                │
              ┌───────────────▼──┐      ┌──────▼───────────────┐
              │   BLUE Service   │      │   GREEN Service      │
              │   (v1 - Active)  │      │   (v2 - Inactive)    │
              │   ████████████   │      │   ████████████       │
              │   ████████████   │      │   ████████████       │
              └──────────────────┘      └──────────────────────┘

Enable Blue/Green

@Library(["spicy-automation@main"]) _

spicyECSService(
    jenkinsAwsCredentialsId: "aws-credentials",
    region: "ca-central-1",
    stackName: "my-api-prod",
    serviceName: "my-api",

    // Cluster info - VPC details auto-import from cluster/VPC stack exports
    clusterName: "my-cluster-prod",
    vpcStackName: "my-vpc",  // Optional: auto-imported from cluster stack if not provided

    // Container
    image: "nexus.kodeniks.com/docker-hosted/my-api:latest",
    containerPort: 3000,

    // Enable Blue/Green
    blueGreen: true,
    hostName: "api.example.com",  // Auto-generates active/inactive hostnames
    bgHostedZoneId: "Z1234567890",  // Route53 hosted zone (optional, for automatic DNS)

    // ALB Configuration
    useClusterAlb: false,  // Use individual ALB (required for blue/green with individual ALB)
    albScheme: "internet-facing",
    certificateArn: "arn:aws:acm:ca-central-1:123456789:certificate/xxx",
    redirectHttpToHttps: true,
    healthCheckPath: "/health",
    priority: 100,

    // Tags
    ownerTag: "Platform",
    productTag: "my-product",
    componentTag: "api",
    environment: "prod",

    // Test inactive before swap
    blueGreenTest: { args, buildInfo ->
        sh "curl -f https://${buildInfo.inactiveHostname}/health"
        sh "./run-integration-tests.sh ${buildInfo.inactiveHostname}"
    },

    // Test active after swap
    smokeTest: { args, buildInfo ->
        sh "curl -f https://${buildInfo.activeHostname}/health"
    },
)

Blue/Green Parameters

Parameter	Default	Description
`blueGreen`	`false`	Enable blue/green deployment
`hostName`	-	Simple hostname (e.g., "api.example.com") - auto-generates active/inactive
`activeHostname`	-	Active hostname (e.g., "api.example.com") - explicit hostname
`inactiveHostname`	-	Inactive hostname (e.g., "inactive-api.example.com") - explicit hostname
`bgHostedZoneId`	-	Route53 hosted zone ID for automatic DNS records (optional)
`rollbackWindowHours`	`2`	Hours to keep old version for rollback

Rollback

Instant rollback by swapping hostnames - no new deployment needed:

@Library(["spicy-automation@main"]) _

spicyRollback(
    jenkinsAwsCredentialsId: "aws-credentials",
    region: "ca-central-1",
    stackName: "my-api-prod",
    serviceName: "my-api",

    // Same params as deploy
    clusterName: "my-cluster-prod",
    vpcStackName: "my-vpc",  // Optional: auto-imported from cluster stack if not provided

    hostName: "api.example.com",  // Or use activeHostname/inactiveHostname
    bgHostedZoneId: "Z1234567890",  // If using Route53

    ownerTag: "Platform",
    productTag: "my-product",
    componentTag: "api",
    environment: "prod",
)

Rollback time: ~30 seconds (just ALB rule updates, no DNS propagation delays)

How it works:

DNS records for both active and inactive hostnames already exist and point to the same ALB
Rollback simply updates ALB listener rule priorities (e.g., swap priority 100 ↔ 200)
No DNS changes are needed, so rollback is instant

State Management

Active color is stored in SSM Parameter Store:

/spicy/{serviceName}/active-color = "blue" | "green"

Check current state:

aws ssm get-parameter --name /spicy/my-api/active-color --query 'Parameter.Value'

Testing Blue/Green DNS

To verify blue/green DNS configuration is working correctly:

Verify DNS Records:

# Both hostnames should resolve to the same ALB
dig api.example.com
dig inactive-api.example.com
# Both should return the same ALB DNS name

Verify ALB Listener Rules:

# Get ALB ARN from stack outputs
ALB_ARN=$(aws cloudformation describe-stacks --stack-name my-api-prod-alb \
  --query 'Stacks[0].Outputs[?OutputKey==`LoadBalancerArn`].OutputValue' --output text)

# Get listener ARN
LISTENER_ARN=$(aws cloudformation describe-stacks --stack-name my-api-prod-alb \
  --query 'Stacks[0].Outputs[?OutputKey==`HTTPSListenerArn`].OutputValue' --output text)

# Check listener rules
aws elbv2 describe-rules --listener-arn $LISTENER_ARN
# Active service should have priority 100 for api.example.com
# Inactive service should have priority 200 for inactive-api.example.com

Test Traffic Routing:

# Get ALB DNS name
ALB_DNS=$(aws cloudformation describe-stacks --stack-name my-api-prod-alb \
  --query 'Stacks[0].Outputs[?OutputKey==`LoadBalancerDNS`].OutputValue' --output text)

# Test active hostname (should hit active service)
curl -H "Host: api.example.com" https://$ALB_DNS/health

# Test inactive hostname (should hit inactive service)
curl -H "Host: inactive-api.example.com" https://$ALB_DNS/health

Verify After Swap:

# After swapping, priorities should be reversed
# New active should have priority 100
# Old active should have priority 200
aws elbv2 describe-rules --listener-arn $LISTENER_ARN

Important: DNS records never change - they always point to the same ALB. Only listener rule priorities change during swaps.

Pipeline Hooks

Customize pipeline behavior at every stage with Groovy closures.

Hook Execution Order

┌─────────────┐
│  Checkout   │
└──────┬──────┘
       │
       ▼
┌─────────────┐     ┌──────────────┐
│ Build Image │────▶│ onPostBuild  │  ← Unit tests, linting
└──────┬──────┘     └──────────────┘
       │
       ▼
┌─────────────┐
│ onPreDeploy │  ← Setup, integration test prep
└──────┬──────┘
       │
       ▼
┌─────────────┐
│   Deploy    │
└──────┬──────┘
       │
       ▼ (Blue/Green only)
┌───────────────┐
│ blueGreenTest │  ← Test inactive stack
└───────┬───────┘
        │
        ▼
┌─────────────┐
│ Swap (B/G)  │
└──────┬──────┘
       │
       ▼
┌──────────────┐
│ onPostDeploy │  ← Cleanup, notifications
└──────┬───────┘
       │
       ▼
┌─────────────┐
│  smokeTest  │  ← Smoke tests
└─────────────┘

Available Hooks

`buildCommand`

Custom build command (replaces default docker build):

spicyECSService(
    // ...
    buildCommand: { args, buildInfo ->
        sh "make build"
        sh "docker tag my-app:latest ${args.image}"
    },
)

`onPostBuild`

After build completes (linting, unit tests):

spicyECSService(
    // ...
    onPostBuild: { args, buildInfo ->
        sh "npm run lint"
        sh "npm run test:unit"
        junit 'coverage/junit.xml'
        publishHTML(target: [
            reportDir: 'coverage/lcov-report',
            reportFiles: 'index.html',
            reportName: 'Coverage'
        ])
    },
)

`onPreDeploy`

Before deployment (setup, test prep):

spicyECSService(
    // ...
    onPreDeploy: { args, buildInfo ->
        sh "docker-compose -f docker-compose.test.yml up -d"
        sh "./wait-for-services.sh"
    },
)

`blueGreenTest`

After inactive stack is up (integration tests):

spicyECSService(
    // ...
    blueGreen: true,
    blueGreenTest: { args, buildInfo ->
        echo "Testing inactive: ${buildInfo.inactiveHostname}"
        sh "curl -f https://${buildInfo.inactiveHostname}/health"
        sh "./run-integration-tests.sh ${buildInfo.inactiveHostname}"
    },
)

Available in buildInfo:

inactiveHostname - Inactive service hostname
activeHostname - Active service hostname
currentActive - Current active color (blue/green)
targetColor - Deployment target color

`onPostDeploy`

After deployment succeeds (cleanup, notifications):

spicyECSService(
    // ...
    onPostDeploy: { args, buildInfo ->
        sh "docker-compose -f docker-compose.test.yml down"
        slackSend(channel: '#deploys', message: "Deployed ${args.serviceName}")
    },
)

`smokeTest`

After deployment or swap (smoke tests):

spicyECSService(
    // ...
    smokeTest: { args, buildInfo ->
        sh "curl -f https://${buildInfo.activeHostname}/health"
        sh "curl -f https://${buildInfo.activeHostname}/api/status"
    },
)

Hook Parameters

All hooks receive:

Parameter	Type	Description
`args`	Map	All pipeline arguments
`buildInfo`	Map	Build context (see below)

buildInfo contents:

Key	Description
`commitSha`	Git commit SHA
`branch`	Git branch name
`image`	Built Docker image URI
`imageTag`	Docker image tag
`activeHostname`	Active service hostname
`inactiveHostname`	Inactive service hostname (B/G)
`currentActive`	Current active color (B/G)
`targetColor`	Deployment target color (B/G)
`healthCheckPath`	Health check path

Log Streaming

The pipeline automatically streams ECS logs during deployment:

spicyECSService(
    // ...
    streamLogs: true,  // Default: true
)

Logs show:

Container startup logs
Health check results
Recent ECS service events

Disable if not needed:

spicyECSService(
    // ...
    streamLogs: false,
)

Migration from Legacy

Legacy Parameter	New Parameter
`desiredCount`	`desiredCount`
`minCapacity`	`minCapacity`
`maxCapacity`	`maxCapacity`
`cpu`	`cpu`
`memory`	`memory`
`containerPort`	`containerPort`
`healthCheckUrl`	`healthCheckPath`
`albPriority`	`priority`
`albScheme: internet-facing`	`useClusterAlb: true, albScheme: "internet-facing"`
`albScheme: internal`	`useClusterAlb: true, albScheme: "internal"`
`targetGroupStickinessEnabled`	`stickiness`
`targetGroupLBCookieDurationSecs`	`stickinessDuration`
`vpcId`, `vpcCidrBlock`, etc.	`clusterName`, `vpcStackName` (auto-imports)

Key Differences

CloudFormation imports - VPC details auto-import from cluster/VPC stack exports (no explicit IDs needed)
Individual ALB support - Set useClusterAlb: false for dedicated ALB per service
No Ansible - Pure CDK deployment
Capacity providers - Native mixed EC2/Fargate support
Circuit breaker - Native deployment rollback
ECS Exec - Built-in debugging support
Blue/Green hostnames - Use hostName for simple setup or activeHostname/inactiveHostname for explicit control

30 KiB Raw Blame History