Fixing Kamal's "target failed to become healthy" Error

The Silent Deployment Killer #

Deployment should be the triumphant final step in your development cycle. You’ve built your application, tested it thoroughly, and you’re ready to share it with the world. You’ve chosen Kamal as your deployment tool because it provides Docker-based simplicity without Kubernetes complexity.

Then suddenly, your deployment grinds to a halt with this message:

ERROR Failed to boot web on xxx.xxx.x.xx
INFO First web container is unhealthy, not booting any other roles
ERROR docker exit status: 1
docker stdout: Nothing written
docker stderr: Error: target failed to become healthy within configured timeout

This opaque error can derail a deploy and leave you searching through logs and configuration files. The message comes from kamal-proxy, not from Rails: the proxy never got a healthy response from your container.

Why This Error Is So Deceptively Complex #

What makes this error particularly challenging is its deceptive simplicity. It tells you something is wrong, but offers little insight into what exactly failed. Let’s unpack what’s happening beneath the surface:

1. Your container builds successfully (so your code itself is valid)
2. The container starts on your server (so your server environment is configured)
3. Kamal’s health check system attempts to verify your application is running
4. The health check fails, but the reason remains hidden

It’s like having your car stop in the middle of the road with only a generic “engine problem” warning—frustrating and uninformative.

Anatomizing the Health Check Process #

To understand the root causes, we need to understand Kamal’s health check mechanism. When deploying, Kamal:

1. Starts your container
2. Sends HTTP GET requests to /up through kamal-proxy
3. Treats a 2xx response as healthy, and keeps trying until deploy_timeout (30 seconds by default) runs out
4. Proceeds with deployment only if a check succeeds

By default the proxy requests /up once a second, with a 5-second timeout per request (Kamal proxy docs). The checked version here is Kamal 2.12.0.

This seemingly simple process can break in multiple subtle ways. In basecamp/kamal issue #1041, a Kamal 2 deploy failed with this error and the container log contains:

ERROR 2024-10-03T10:12:10.047581359Z {"time":"2024-10-03T10:12:10.047492864Z","level":"INFO","msg":"Unable to proxy request","path":"/up","error":"dial tcp 127.0.0.1:3000: connect: connection refused"}

That line is only a startup race: Puma was not listening yet. Later lines in the same issue log show /up answering 301, which is the real failure. Different causes produce different log patterns.

Five Common Failure Patterns and Their Solutions #

These are five failure patterns that recur in Kamal 2 issue reports, including #1041. Each requires a different approach to solve:

1. SSL Configuration Conflicts #

Pattern: Health check returns 301 redirects instead of 200 OK

What’s happening: Rails applications with force_ssl enabled redirect HTTP requests to HTTPS, and Kamal’s health check uses plain HTTP, so /up answers 301 instead of 200. With ssl: true, kamal-proxy does not forward X-Forwarded-Proto unless you set forward_headers: true, so Rails can’t see that the request was HTTPS. Setting config.assume_ssl = true (the Rails 7.1+ option for apps behind an SSL-terminating proxy) is the other common fix.

Solution: Add this to your config/environments/production.rb:

# Skip http-to-https redirect for the health check endpoint
config.ssl_options = { redirect: { exclude: ->(request) { request.path == "/up" } } }

This creates a targeted exception that allows health check requests while keeping SSL protection for all other routes.

2. Port Mismatch Configuration #

Pattern: Logs show “connection refused” errors

What’s happening: Kamal’s proxy tries to communicate with your application on a port different from where your application is actually listening.

Solution: Ensure port configuration consistency across three places:

• In your Dockerfile: EXPOSE 3000 (or your chosen port)
• In your deploy.yml: Add app_port: 3000 under the proxy section
• In your application startup command: Listen on 0.0.0.0:port (not just localhost)

Example deploy.yml configuration:

proxy:
  ssl: true
  host: myapp.example.com
  app_port: 3000  # Must match your application's listening port

3. Host Authorization Barriers #

Pattern: Application logs show “Blocked host” errors

What’s happening: Rails includes a host authorization system (ActionDispatch::HostAuthorization) that blocks requests with an unexpected Host header. Kamal’s health check may use an IP address or hostname that doesn’t match your configured allowed hosts.

Solution: Add an exception for the health check path in config/environments/production.rb:

# Allow health check endpoint without host verification
config.host_authorization = { exclude: ->(request) { request.path == "/up" } }

4. Resource Constraints #

Pattern: Slow SSH connection, deployment sometimes works but often fails

What’s happening: During blue-green deployment, Kamal runs two containers simultaneously. On under-provisioned servers, this can exhaust available resources.

Solution: Either upgrade your server or modify your deployment strategy:

• Increase server RAM/CPU (recommended for production)

• Add swap space as a temporary measure:

  sudo fallocate -l 1G /swapfile
  sudo chmod 600 /swapfile
  sudo mkswap /swapfile
  sudo swapon /swapfile
  

• Restart your server before deployment to clear memory

5. Application Initialization Failures #

Pattern: The container starts but your application crashes during initialization

What’s happening: Your application encounters an error during startup that isn’t visible in standard Kamal output—perhaps a missing environment variable, database connection issue, or code error only triggered in production.

Solution: Run your container interactively to see the actual error:

# SSH into your server
ssh your-server

# Find the image ID
docker image ls

# Run it with environment variables to see the real error
docker run -it --network kamal --env-file .kamal/apps/<your-app-name>/env/roles/web.env <image-id>

This will show you the actual error messages that are otherwise hidden.

A Systematic Debugging Approach #

When encountering this error, follow this systematic approach:

1. Examine the Full Logs #

kamal app logs -f

Look for patterns in the logs—are you seeing connection refused errors? SSL redirects? Application errors?

2. Verify Network Connectivity #

SSH into your server and check if basic network services are working:

# Test database connectivity
source .kamal/apps/<your-app-name>/env/roles/web.env
pg_isready -h $DB_HOST -p $DB_PORT -U $DB_USER

# Check if something else is using your port
sudo netstat -tulpn | grep 3000

3. Inspect Container Status #

# See if containers are starting and stopping
docker ps -a

# Check resource usage
docker stats

4. Run the Container Interactively #

# Get the image ID
docker image ls | grep your-app-name

# Run it with the environment variables
docker run -it --network kamal --env-file .kamal/apps/<your-app-name>/env/roles/web.env <image-id>

This interactive approach often reveals issues that aren’t visible in the logs.

5. Test the Health Check Endpoint Directly #

# On your server
curl http://localhost:3000/up

If this fails while your container is running, you’ve found the issue.

Causes Stack Up #

Fixing one cause often exposes the next. For example, an app behind ssl: true may need the SSL exception, and then also a missing environment variable that an initializer reads at boot. Fix one, redeploy, read the next error. Don’t assume the first fix is the last.

Preventive Configuration for Rails Applications #

To avoid this error in future deployments, here’s a comprehensive configuration template for Rails applications using Kamal:

1. In config/environments/production.rb:

# Allow health checks without SSL redirects
config.ssl_options = { redirect: { exclude: ->(request) { request.path == "/up" } } }

# Allow health checks without host verification
config.host_authorization = { exclude: ->(request) { request.path == "/up" } }

2. In deploy.yml:

proxy:
  ssl: true  # Set to false if using external SSL termination
  host: your-application.com
  app_port: 3000  # Match your application's port
  healthcheck:
    path: /up  # defaults: path /up, interval 1s, timeout 5s

3. In your Dockerfile:

# Ensure port is exposed
EXPOSE 3000

# Use a clear startup command
CMD ["./bin/rails", "server", "-b", "0.0.0.0", "-p", "3000"]

4. Use the built-in health endpoint (Rails 7.1+) instead of a hand-written one:

# In config/routes.rb
get "up" => "rails/health#show", as: :rails_health_check

Logs as Archaeological Artifacts #

When debugging deployment issues, think of logs as archaeological artifacts—each one tells part of a story, but you need to piece them together to understand the full narrative.

The most revealing logs for this error are often:

1. The Kamal proxy logs showing the health check attempts
2. Your application’s startup logs
3. Any database connection errors
4. Environment variable misconfigurations

Collect these logs systematically, and the pattern of failure will emerge.

Deployment as a System, Not an Event #

A key conceptual shift in resolving these errors is to view deployment not as a single event but as a system with multiple interconnected components:

• Container building
• Image distribution
• Environment configuration
• Application initialization
• Health verification
• Traffic transitioning

The “target failed to become healthy” error points to a breakdown in the health verification component, but the root cause could be in any of these areas.

By systematically testing each component and understanding their interconnections, you can identify where the system is breaking down and apply the appropriate fix.

The Path Forward #

Kamal 2 reports target failed to become healthy from kamal-proxy. When you hit it, read the app and proxy logs (kamal app logs, kamal proxy logs) and check the five causes above in order.

Remember that deployment tools are abstractions that make common cases easy but edge cases challenging. By understanding what happens beneath these abstractions, you gain the power to diagnose and resolve the issues that inevitably arise.

The next time you encounter this error, approach it methodically, test each layer of the system, and you’ll find that what initially seemed like an impenetrable wall becomes a solvable puzzle.