09. Deployment Guide

Setup and deployment guide for MILU2 Infra Main with OpenTofu + Terragrunt.

Prerequisites

Machine Requirements

  • Docker Desktop / OrbStack (linux/amd64 or linux/arm64)
  • Git
  • AWS SSO access with profile milu2-infra

AWS Configuration

Create AWS config file at ~/.aws/thankslab:

[profile milu2-infra]
sso_start_url = https://your-sso-url.awsapps.com/start
sso_region = ap-northeast-1
sso_account_id = 123456789012
sso_role_name = AdministratorAccess
region = ap-northeast-1
output = json

Initial Setup

Step 1: Build Dev Container

# Build the development container
make init

Container includes:

  • AWS CLI 2.33.31
  • OpenTofu 1.11.5
  • Terragrunt 0.99.4
  • crane (container tool)
  • SSM plugin
  • MariaDB client

Step 2: Start Container

# Start the container in background
make up

Step 3: AWS SSO Login

# Login to AWS SSO
make sso

Step 4: Enter Shell

# Get a shell inside the container
make bash

Downloading Environment Variables

Before planning or applying, download variables from S3:

# Download vars for test environment
sh shell/get_vars.sh test

This downloads:

  • common_vars.hcltofu/envs/common_vars.hcl
  • env_vars.hcltofu/envs/test/env_vars.hcl

Deployment Workflow

Quick Reference

# 1. Format code
make fmt

# 2. Plan changes
make plan-test

# 3. Apply changes
make apply-test

# 4. View outputs
make output-test

Full Workflow

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│   make fmt   │ ──► │ make plan-X  │ ──► │ make apply-X │
│              │     │              │     │              │
│  Format code │     │ Preview      │     │ Apply        │
└──────────────┘     │ changes      │     │ changes      │
                     └──────────────┘     └──────────────┘
                            │
                            ▼
                     ┌──────────────┐
                     │   Review     │
                     │   output     │
                     └──────────────┘

Always review plan output before applying. Check for no unexpected destroys.

First-Time Deployment

What Happens on First Apply

  1. State bucket created (S3 + lockfile)
  2. KMS keys created (Tokyo + Virginia replica)
  3. VPC + networking created
  4. Databases created (RDS, ElastiCache, MemoryDB, DocDB)
  5. DDL bootstrap runs (null_resource.db_initializer)
  6. Lambda ECR seeded (terraform_data.ecr_initial_image)
  7. GA traffic enabled (terraform_data.allow_traffic)

Bootstrap Resources

null_resource.db_initializer
├── Creates temporary SG rule on bastion
├── SSH to bastion
├── Run MySQL DDL via init.sql.tpl
└── Cleanup SG rule

terraform_data.ecr_initial_image
└── crane copy public.ecr.aws/lambda/python:3.12 → Lambda ECR

terraform_data.allow_traffic
└── aws globalaccelerator allow-custom-routing-traffic (us-west-2)

Make Commands Reference

Container Management

CommandDescription
make initBuild dev container (no cache)
make upStart container
make downStop container
make bashShell into container
make ssoAWS SSO login

Terraform Operations

CommandDescription
make fmtFormat HCL files
make docGenerate terraform-docs
make plan-<env>Plan all changes
make apply-<env>Apply all changes
make destroy-<env>Destroy all resources
make output-<env>Show outputs
make list-<env>List state resources

ECS Operations

CommandDescription
make ecs-exec-api-<env>Exec into API container
make ecs-exec-web-<env>Exec into Web container
make ecs-exec-admin-<env>Exec into Admin container
make ecs-exec-push-<env>Exec into Push container

Post-Deployment Verification

Check ECS Services

# Inside container
aws ecs describe-services \
  --cluster milu2-test-cluster \
  --services milu2-test-api-service \
  --query 'services[0].{status:status,running:runningCount,desired:desiredCount}'

Check ALB Health

# From bastion
curl -H 'X-Forwarded-App: api' http://<alb-dns>/healthcheck

Check RDS Connection

# Via bastion
mysql -h <writer-endpoint> -P 3307 -u <username> -p

View Outputs

make output-test

Outputs include:

  • vpc_id
  • alb_dns_name
  • cloudfront_*_domain
  • rds_writer_endpoint
  • elasticache_endpoint
  • bastion_public_ip
  • bastion_private_key (sensitive)

Deploying Shared Stacks

Shared stacks are environment-independent:

# Inside container, navigate to shared stack
cd tofu/envs/shared/github_provider

# Plan
terragrunt plan

# Apply
terragrunt apply

Order of deployment:

  1. github_provider (OIDC)
  2. cicd_infra (depends on #1)
  3. cicd_app_deploy (depends on #1)

Destroying Infrastructure

WARNING: This will destroy ALL resources!

# Destroy all resources
make destroy-test

Production safeguards:

  • db_deletion_protection = true
  • docdb_deletion_protection = true
  • force_destroy = false (S3 buckets)
  • force_delete = false (ECR repos)

Deployment Best Practices

Pre-Deployment Checklist

  • make sso - Refresh AWS credentials
  • sh shell/get_vars.sh <env> - Download latest vars
  • make fmt - Format code
  • make plan-<env> - Review changes
  • ☐ Verify no unexpected destroys
  • ☐ Verify resource counts

Handling Long-Running Operations

Some operations take time:

  • RDS creation: 10-15 minutes
  • CloudFront distribution: 5-10 minutes
  • Global Accelerator: 5-10 minutes

Troubleshooting Deployment

IssueSolution
SSO token expiredmake sso
State lock heldWait or check who holds lock
DDL bootstrap failsCheck bastion SG, SSH key
ECR image push failsCheck IAM permissions
GA allow traffic failsRetry (uses retry loop)