16. Secrets Management

Managing credentials and secrets for MILU2 Infrastructure.

Secrets Locations Overview

┌─────────────────────────────────────────────────────────────────┐
│                      Secrets Locations                           │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌─────────────────┐    ┌─────────────────┐    ┌─────────────┐  │
│  │   S3 Bucket     │    │ AWS Secrets     │    │  Terraform  │  │
│  │ milu2-iac-env   │    │   Manager       │    │   State     │  │
│  │                 │    │                 │    │             │  │
│  │ - env_vars.hcl  │    │ - RDS master    │    │ - SSH keys  │  │
│  │ - common_vars   │    │   password      │    │ - Sensitive │  │
│  │                 │    │                 │    │   outputs   │  │
│  └─────────────────┘    └─────────────────┘    └─────────────┘  │
│                                                                  │
│  ┌─────────────────┐    ┌─────────────────┐                     │
│  │   S3 Bucket     │    │   Parameter     │                     │
│  │ milu2-app-env   │    │    Store        │                     │
│  │                 │    │  (optional)     │                     │
│  │ - ECS env files │    │                 │                     │
│  │ - api.env       │    │ - Feature flags │                     │
│  │ - web.env       │    │ - Configs       │                     │
│  └─────────────────┘    └─────────────────┘                     │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Secret Types and Locations

Secret Type	Location	Access Method
Database credentials	env_vars.hcl (S3)	Terraform variable
RDS master password	AWS Secrets Manager	Auto-managed
ElastiCache auth token	env_vars.hcl (S3)	Terraform variable
Bastion SSH key	Terraform state	Output (sensitive)
Application secrets	S3 milu2-app-env	ECS env file

Downloading Secrets

# Download env vars (contains secrets)
sh shell/get_vars.sh test

# Verify (in container)
cat tofu/envs/test/env_vars.hcl | grep password

Accessing RDS Secret

# Get secret ARN
aws rds describe-db-clusters \
  --db-cluster-identifier milu2-test-db \
  --query 'DBClusters[0].MasterUserSecret.SecretArn'

# Retrieve secret value
aws secretsmanager get-secret-value \
  --secret-id <secret-arn> \
  --query 'SecretString'

Managing ECS Env Files

# Download current
aws s3 cp s3://milu2-app-env/ecs/api.env ./api.env

# Edit locally
vim api.env

# Upload
aws s3 cp api.env s3://milu2-app-env/ecs/api.env

# Restart service to pick up changes
aws ecs update-service \
  --cluster milu2-test-cluster \
  --service milu2-test-api-service \
  --force-new-deployment

Retrieving SSH Key

# Get key from Terraform output
make output-test | jq -r '.bastion_private_key.value' > bastion_key.pem
chmod 600 bastion_key.pem

# Connect
ssh -i bastion_key.pem ec2-user@<bastion-ip>

Security Best Practices

Do's

✅ Use Secrets Manager for database passwords
✅ Encrypt S3 buckets containing secrets
✅ Use IAM roles instead of long-lived credentials
✅ Rotate secrets regularly
✅ Audit secret access via CloudTrail

Don'ts

❌ Never commit secrets to Git
❌ Don't share secrets via chat/email
❌ Avoid storing secrets in task definitions
❌ Don't log secret values
❌ Never use default/weak passwords

Emergency Procedures

If Secret is Compromised

Identify - Determine which secret was exposed
Rotate - Change the secret immediately
Update - Update all systems using the secret
Audit - Check CloudTrail for unauthorized access
Report - Follow security incident process