Skip to content

Troubleshooting and FAQ

This guide is for the local Kohakku development stack in the repo root.

Quick Triage

Start here before you chase a specific symptom:

docker compose ps
docker compose logs --tail=100 controller celery-worker celery-beat dispatcher postgres redis minio minio-init
make help

Docker Services Won't Start

Symptom

  • docker compose up -d --build fails immediately
  • a service restarts in a loop
  • Docker reports "port is already allocated" or "no space left on device"

Likely causes

  • Another process is already using one of Kohakku's ports
  • Docker is out of disk space
  • A stale container from an older run is still attached

Diagnose

docker compose ps
lsof -nP -iTCP -sTCP:LISTEN | egrep ':(8000|8080|5432|6379|9000|9001|8088)\b'
docker system df
docker compose logs --tail=100

Fix

docker compose down --remove-orphans
docker system prune -f
  • Stop or reconfigure the process that already owns the port.
  • Free Docker disk space if the daemon is full.
  • If local data is disposable, reset the stack completely:
docker compose down -v
docker compose up -d --build

docker compose down -v deletes local Postgres and MinIO data.

Database Migrations Fail

Symptom

  • make migrate fails in the controller
  • dispatcher-migrate exits non-zero during docker compose up
  • schema errors appear even though the database already exists

Likely causes

  • Postgres is not healthy yet
  • Local database volumes contain old schema state
  • Controller and Dispatcher migrations are out of sync with the current branch

Diagnose

docker compose logs postgres controller dispatcher-migrate
docker compose exec -T controller python manage.py showmigrations
docker compose run --rm dispatcher-migrate postgres://postgres:postgres@postgres:5432/kohakku-dispatcher?sslmode=disable status

Fix

make migrate
docker compose up dispatcher-migrate
  • Wait for Postgres to become healthy before retrying migrations.
  • If the schema is badly drifted and local data does not matter, rebuild from empty volumes:
docker compose down -v
docker compose up -d --build
make migrate
make seed

Remember that docker/postgres-init/init.sql only runs the first time the Postgres volume is created.

Celery Is Not Picking Up Tasks

Symptom

  • Controller actions queue work but nothing seems to execute
  • celery-worker stays quiet
  • periodic work never fires

Likely causes

  • celery-worker or celery-beat is not running
  • the broker URL is wrong
  • the worker is listening on the celery queue, but tasks are landing elsewhere

Diagnose

docker compose ps celery-worker celery-beat redis
docker compose logs --tail=100 celery-worker celery-beat controller
docker compose exec redis redis-cli -n 0 LLEN celery
docker compose exec -T controller python manage.py shell -c "from django.conf import settings; print(settings.CELERY_BROKER_URL)"

Fix

docker compose up -d redis celery-worker celery-beat
docker compose restart celery-worker celery-beat controller
  • In Docker Compose, the broker should resolve as redis://redis:6379/0.
  • The worker starts with -Q celery, so only the celery queue is consumed by default.
  • If you change broker settings, restart the controller and both Celery services.

Redis Connection Refused

Symptom

  • Controller, Celery, or Dispatcher logs show Redis connection errors
  • requests hang until a cache or broker timeout

Likely causes

  • Redis is not running
  • The hostname is wrong for the environment
  • The wrong Redis DB number is being used

Diagnose

docker compose ps redis
docker compose logs --tail=100 redis
docker compose exec redis redis-cli ping
docker compose exec -T controller python manage.py shell -c "from django.conf import settings; print(settings.REDIS_URL, settings.CELERY_BROKER_URL)"

Fix

  • Inside Docker Compose, use the hostname redis.
  • From the host machine, use localhost.
  • Keep the DB split consistent:
  • cache: redis://.../1
  • Celery broker: redis://.../0
  • Dispatcher Redis: redis://.../2
docker compose up -d redis controller celery-worker celery-beat dispatcher

Tests Pass Locally but Fail in CI

Symptom

  • Tests pass in your current shell but fail in GitHub Actions
  • CI fails to connect to Postgres or Redis
  • CI sees a different database state than your local stack

Likely causes

  • CI uses service containers on localhost, not Docker Compose service names
  • Controller CI and Dispatcher CI use different host ports
  • Your local environment has seeded data or persisted volumes that CI does not

Diagnose

sed -n '1,240p' .github/workflows/controller.yml
sed -n '1,240p' .github/workflows/dispatcher.yml

Important differences in this repo:

  • Controller CI talks to Postgres on localhost:5432 and Redis on localhost:6379.
  • Dispatcher CI talks to Postgres on localhost:5438.
  • Root Docker Compose uses postgres and redis as container hostnames inside the network.

Fix

Reproduce the CI environment instead of relying on your running compose stack:

cd controller
DJANGO_SECRET_KEY=ci-test-secret-key DJANGO_DEBUG=true DJANGO_ALLOWED_HOSTS=localhost,127.0.0.1 POSTGRES_DB=kohakku-controller POSTGRES_USER=dbadmin POSTGRES_PASSWORD=Password123 POSTGRES_HOST=localhost POSTGRES_PORT=5432 REDIS_URL=redis://localhost:6379/1 CELERY_BROKER=redis://localhost:6379/0 CELERY_BACKEND=redis://localhost:6379/0 python -m pytest --tb=short -q

cd ../dispatcher
DATABASE_URL=postgres://postgres:postgres@localhost:5438/kohakku-dispatcher?sslmode=disable go test ./... -v -count=1
  • Do not depend on pre-seeded local data in tests.
  • If your local volumes are masking problems, reset them and rerun the tests from a clean state.

MinIO Access Errors

Symptom

  • Skill upload or object storage calls fail
  • the controller reports missing objects or S3 client errors
  • MinIO is up but the bucket is missing

Likely causes

  • MinIO is not healthy yet
  • minio-init did not create the kohakku bucket
  • The access key, secret, endpoint URL, or bucket name does not match the running stack

Diagnose

docker compose ps minio minio-init controller
docker compose logs --tail=100 minio minio-init controller
curl -f http://localhost:9000/minio/health/live
docker compose exec -T controller python manage.py shell -c "from django.conf import settings; print(settings.OBJECT_STORAGE_URL, settings.OBJECT_STORAGE_BUCKET)"

Fix

docker compose up minio-init
docker compose restart controller
  • In the root stack, the expected bucket is kohakku.
  • Default dev credentials are minioadmin / minioadmin.
  • The root compose file sets USE_S3=true, so the controller expects object storage in local dev.

FAQ

What command should I run first when something looks wrong?

Run docker compose ps and then docker compose logs --tail=100 ... for the affected services.

Can I safely reset my local state?

Yes, with docker compose down -v, but it deletes local Postgres and MinIO data.

Why do some settings use redis or postgres and others use localhost?

Inside Docker Compose, services talk to each other by service name. From your host shell or CI service containers, you usually use localhost.

Why does the controller wait on MinIO in local dev?

The root compose stack enables S3-compatible storage with USE_S3=true and waits for minio-init to create the kohakku bucket.

Where should I look for CI truth?

The GitHub Actions workflow files in .github/workflows/ are the source of truth for the CI environment.