BTHLABS-49: WIP

README.md

@@ -25,7 +25,7 @@ This repository contains the _HotPocket_ project.
 **Exported services:**
 
 * The app: https://app.hotpocket.work.bthlabs.net:8000/
-* The admin: https://app.hotpocket.work.bthlabs.net:8000/
+* The admin: https://admin.hotpocket.work.bthlabs.net:8000/
 * Keycloak: https://auth.hotpocket.work.bthlabs.net:8443/
 * Postgres: postgres://postgres.hotpocket.work.bthlabs.net:5432/
 * RabbitMQ: amqp://rabbitmq.hotpocket.work.bthlabs.net:5672/
@@ -43,7 +43,120 @@ and admin using OIDC.
 
 ## Deployment
 
-TODO
+There are two deployment images - `aio` and `deployment`.
+
+### The AIO image
+
+The `aio` image is pre-configured for running small instances in a single
+container:
+
+* It defaults to SQLite database.
+* It defaults to running all background tasks in the foreground.
+* It defaults to accepting traffic with any `Host` HTTP header.
+
+The `aio` image is recommended for self-hosting with minimal use, e.g. by a
+single user.
+
+**Example:**
+
+```
+$ docker run --rm -it \
+    -v `realpath run/`:/srv/run \
+    -e HOTPOCKET_BACKEND_SECRET_KEY=thisisntright \
+    -e HOTPOCKET_BACKEND_INITIAL_ACCOUNT_USERNAME=hotpocket \
+    -e HOTPOCKET_BACKEND_INITIAL_ACCOUNT_PASSWORD=hotpocketm4st3r \
+    -p 8000:8000 \
+    docker-hosted.nexus.bthlabs.pl/hotpocket/backend:aio-v1.0.0rc1-01
+```
+
+The command above will set up and start the application. The SQLite file will
+be placed in `run/hotpocket-backend-aio.sqlite` and database migrations will
+be ran. The initial superuser account will be created with the specified
+credentials. The Web app will be reachable at `http://127.0.0.1:8000/`.
+The admin will be reachable at `http://127.0.0.1:8000/admin/`.
+
+The `DJANGO_SETTINGS_MODULE` environment variable defaults to
+`hotpocket_backend.settings.deployment.webapp`. This should be set to
+`hotpocket_backend.settings.deployment.admin` in the Admin container.
+
+**NOTE:** The command above specifies wildly insecure `SECRET_KEY` which is
+used among other things to secure the session cookie. Please *please*
+**please** don't run it like this. Not even in your homelab :).
+
+The `deployment/aio/docker-compose.yaml` file can be used as a starting
+point for AIO deployments.
+
+### The Deployment image
+
+The `deployment` image doesn't make any assumptions about the env and in turn
+will require the operator to configure database, Celery broker and result
+backend etc. The final deployment will require services for at least the Web
+app, the Celery worker and Celery Beat. Admin is optional.
+
+The `DJANGO_SETTINGS_MODULE` environment variable defaults to
+`hotpocket_backend.settings.deployment.aio`.
+
+The `deployment/fullstack/docker-compose.yaml` file can be used as a
+starting point for full-stack deployments.
+
+### Configuration environment variables
+
+HotPocket deployment images provide extensive set of environment variables
+that can be used to configure the services.
+
+| Variable                                     | Default                                                         | Description                                                                                 |
+|----------------------------------------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------|
+| `HOTPOCKET_BACKEND_ENV`                      | `deployment` or `aio`                                           | The environment name. See below.                                                            |
+| `HOTPOCKET_BACKEND_APP`                      | `webapp`                                                        | The app name. See below.                                                                    |
+| `HOTPOCKET_BACKEND_DEBUG`                    | `false`                                                         | Django `DEBUG` setting. **Do not enable in production**. Only effective in the AIO image.   |
+| `HOTPOCKET_BACKEND_ALLOWED_HOSTS`            | N/A or `*`                                                      | Django `ALLOWED_HOSTS` setting. **Required in the Deployment image.**                       |
+| `HOTPOCKET_BACKEND_SECRET_KEY`               | N/A                                                             | Django `SECRET_KEY` setting. Recommended different for the Web app and Admin. **Required**. |
+| `HOTPOCKET_BACKEND_DATABASE_ENGINE`          | `django.db.backends.postgresql` or `django.db.backends.sqlite3` | The database configuration engine.                                                          |
+| `HOTPOCKET_BACKEND_DATABASE_NAME`            | N/A or `/srv/run/hotpocket-backend-aio.sqlite`                  | The database name.                                                                          |
+| `HOTPOCKET_BACKEND_DATABASE_USER`            | N/A or N/A                                                      | The database user.                                                                          |
+| `HOTPOCKET_BACKEND_DATABASE_PASSWORD`        | N/A                                                             | The database password.                                                                      |
+| `HOTPOCKET_BACKEND_DATABASE_HOST`            | N/A                                                             | The database host.                                                                          |
+| `HOTPOCKET_BACKEND_DATABASE_PORT`            | `5432` or N/A                                                   | The database port.                                                                          |
+| `HOTPOCKET_BACKEND_MODEL_AUTH_IS_DISABLED`   | `false`                                                         | Set to `true` to disable username and password login.                                       |
+| `HOTPOCKET_BACKEND_OIDC_PAYLOAD`             | N/A                                                             | The OIDC configuration payload.                                                             |
+| `HOTPOCKET_BACKEND_CELERY_BROKER_URL`        | N/A                                                             | The Celery broker URL.                                                                      |
+| `HOTPOCKET_BACKEND_CELERY_RESULT_BACKEND`    | N/A                                                             | The Celery result backend URL.                                                              |
+| `HOTPOCKET_BACKEND_CELERY_IGNORE_RESULT`     | `false`                                                         | Set to `true` to prevent Celery from saving task results.                                   |
+| `HOTPOCKET_BACKEND_CELERY_ALWAYS_EAGER`      | `false`                                                         | Set to `true` to run Celery tasks in the foreground.                                        |
+| `HOTPOCKET_BACKEND_UPLOADS_PATH`             | `/srv/uploads` or `/srv/run/uploads`                            | The absolute path to user-uploaded files.                                                   |
+| `HOTPOCKET_BACKEND_GUNICORN_WORKERS`         | `4` or `2`                                                      | The number of Gunicorn workers to run for Web servers.                                      |
+| `HOTPOCKET_BACKEND_RUN_MIGRATIONS`           | `false` or `true`                                               | Set to `true` to run database muigrations when the container starts.                        |
+| `HOTPOCKET_BACKEND_INITIAL_ACCOUNT_USERNAME` | N/A                                                             | Username for the initial account.                                                           |
+| `HOTPOCKET_BACKEND_INITIAL_ACCOUNT_PASSWORD` | N/A                                                             | Password for the initial account.                                                           |
+
+**Env and App settings**
+
+The `HOTPOCKET_BACKEND_ENV` and `HOTPOCKET_BACKEND_APP` variables are used
+internally to resolve other settings and identify the running app.
+`HOTPOCKET_BACKEND_ENV` should only be changed if when creating heavily
+customized version of the project. `HOTPOCKET_BACKEND_APP` should generally be
+set to `admin` only in the Admin container.
+
+**OIDC login configuration**
+
+The `HOTPOCKET_BACKEND_OIDC_PAYLOAD` can be used to enable OIDC login. It must
+be a JSON string that deserializes to the following object:
+
+```
+{
+  "endpoint": "https://some.oidc.host/some-realm/",
+  "key": "client-key",
+  "secret": "client-secret",
+  "scope": ["roles"],
+  "display_name": "My OIDC server"
+}
+```
+
+The `scope` field specified additional scopes to request from IdP. It can be
+ommited and defaults to `["roles"]`. The `display_name` field specifies the
+method's name in the UI and defaults to `OIDC`.
+
+**NOTE:** Currently, only Keycloak has been tested with this login method.
 
 ## Author
 

deployment/aio/.gitignore

@@ -0,0 +1,2 @@
+run/*.sqlite
+uploads/

deployment/aio/docker-compose.yaml

@@ -0,0 +1,12 @@
+services:
+  backend:
+    image: "docker-hosted.nexus.bthlabs.pl/hotpocket/backend:aio-v1.0.0rc1-01"
+    environment:
+      HOTPOCKET_BACKEND_SECRET_KEY: "thisisntright"
+      HOTPOCKET_BACKEND_INITIAL_ACCOUNT_USERNAME: "hotpocket"
+      HOTPOCKET_BACKEND_INITIAL_ACCOUNT_PASSWORD: "hotpocketm4st3r"
+    ports:
+      - "8000:8000"
+    volumes:
+      - "./run:/srv/run"
+    restart: "unless-stopped"

deployment/fullstack/.gitignore

@@ -0,0 +1,2 @@
+run/celery-beat-schedule
+uploads/*.csv

deployment/fullstack/docker-compose.yaml

@@ -0,0 +1,77 @@
+x-backend-environment: &x-backend-environment
+  HOTPOCKET_BACKEND_DATABASE_NAME: "hotpocket_backend_staging"
+  HOTPOCKET_BACKEND_DATABASE_USER: "hotpocket"
+  HOTPOCKET_BACKEND_DATABASE_PASSWORD: "hotpocketm4st3r"
+  HOTPOCKET_BACKEND_DATABASE_HOST: "databases.bthlab.bthlabs.net"
+  HOTPOCKET_BACKEND_CELERY_BROKER_URL: "amqp://hotpocket:hotpocketm4st3r@databases.bthlab.bthlabs.net/hotpocket_backend_staging"
+  HOTPOCKET_BACKEND_CELERY_RESULT_BACKEND: "db+postgresql+psycopg://hotpocket:hotpocketm4st3r@databases.bthlab.bthlabs.net/hotpocket_backend_staging"
+
+services:
+  webapp:
+    image: "docker-hosted.nexus.bthlabs.pl/hotpocket/backend:deployment-v1.0.0rc1-01"
+    environment:
+      <<: *x-backend-environment
+      HOTPOCKET_BACKEND_ALLOWED_HOSTS: "app.staging.hotpocket.bthlab.bthlabs.net"
+      HOTPOCKET_BACKEND_SECRET_KEY: "thisisntright"
+    ports:
+      - "8000:8000"
+    volumes:
+      - "./run:/srv/run"
+      - "./uploads:/srv/uploads"
+    restart: "unless-stopped"
+
+  admin:
+    image: "docker-hosted.nexus.bthlabs.pl/hotpocket/backend:deployment-v1.0.0rc1-01"
+    environment:
+      <<: *x-backend-environment
+      HOTPOCKET_BACKEND_APP: "admin"
+      HOTPOCKET_BACKEND_ALLOWED_HOSTS: "app.staging.hotpocket.bthlab.bthlabs.net"
+      HOTPOCKET_BACKEND_SECRET_KEY: "thisisntright"
+    ports:
+      - "8001:8000"
+    volumes:
+      - "./run:/srv/run"
+      - "./uploads:/srv/uploads"
+    restart: "unless-stopped"
+
+  celery-worker:
+    image: "docker-hosted.nexus.bthlabs.pl/hotpocket/backend:deployment-v1.0.0rc1-01"
+    command:
+      - "/srv/venv/bin/celery"
+      - "-A"
+      - "hotpocket_backend.celery:app"
+      - "worker"
+      - "-l"
+      - "INFO"
+      - "-Q"
+      - "celery,webapp"
+      - "-c"
+      - "2"
+    environment:
+      <<: *x-backend-environment
+      HOTPOCKET_BACKEND_ALLOWED_HOSTS: "app.staging.hotpocket.bthlab.bthlabs.net"
+      HOTPOCKET_BACKEND_SECRET_KEY: "thisisntright"
+    volumes:
+      - "./run:/srv/run"
+      - "./uploads:/srv/uploads"
+    restart: "unless-stopped"
+
+  celery-beat:
+    image: "docker-hosted.nexus.bthlabs.pl/hotpocket/backend:deployment-v1.0.0rc1-01"
+    command:
+      - "/srv/venv/bin/celery"
+      - "-A"
+      - "hotpocket_backend.celery:app"
+      - "beat"
+      - "-l"
+      - "INFO"
+      - "-s"
+      - "/srv/run/celery-beat-schedule"
+    environment:
+      <<: *x-backend-environment
+      HOTPOCKET_BACKEND_ALLOWED_HOSTS: "app.staging.hotpocket.bthlab.bthlabs.net"
+      HOTPOCKET_BACKEND_SECRET_KEY: "thisisntright"
+    volumes:
+      - "./run:/srv/run"
+      - "./uploads:/srv/uploads"
+    restart: "unless-stopped"

services/backend/Dockerfile

@@ -72,6 +72,10 @@ ARG APP_USER_UID
 ARG APP_USER_GID
 ARG IMAGE_ID
 
+ENV DJANGO_SETTINGS_MODULE=hotpocket_backend.settings.deployment.webapp
+ENV HOTPOCKET_BACKEND_ENV=deployment
+ENV HOTPOCKET_BACKEND_APP=webapp
+
 VOLUME ["/srv/run", "/srv/uploads"]
 
 FROM deployment-base AS aio
@@ -84,7 +88,12 @@ ENV DJANGO_SETTINGS_MODULE=hotpocket_backend.settings.aio
 ENV HOTPOCKET_BACKEND_ENV=aio
 ENV HOTPOCKET_BACKEND_APP=webapp
 ENV HOTPOCKET_BACKEND_DEBUG=false
-ENV HOTPOCKET_BACKEND_DATABASE_PAYLOAD={"engine":"django.db.backends.sqlite3","name":"/srv/run/hotpocket-backend-aio.sqlite"}
+ENV HOTPOCKET_BACKEND_DATABASE_ENGINE=django.db.backends.sqlite3
+ENV HOTPOCKET_BACKEND_DATABASE_NAME=/srv/run/hotpocket-backend-aio.sqlite
+ENV HOTPOCKET_BACKEND_DATABASE_USER=
+ENV HOTPOCKET_BACKEND_DATABASE_PASSWORD=
+ENV HOTPOCKET_BACKEND_DATABASE_HOST=
+ENV HOTPOCKET_BACKEND_DATABASE_PORT=
 ENV HOTPOCKET_BACKEND_CELERY_IGNORE_RESULT=true
 ENV HOTPOCKET_BACKEND_CELERY_ALWAYS_EAGER=true
 ENV HOTPOCKET_BACKEND_GUNICORN_WORKERS=2

services/backend/docker-compose-aio.yaml

@@ -1,5 +1,5 @@
 services:
-  backend-aio-webapp:
+  webapp:
     build:
       context: ".."
       dockerfile: "backend/Dockerfile"

services/backend/hotpocket_backend/secrets/deployment/common.py

@@ -17,10 +17,10 @@ class DeploymentDatabaseSecrets(DatabaseSecrets):
     payload: str = LiteralField.new(
         json.dumps({
             'engine': os.getenv('HOTPOCKET_BACKEND_DATABASE_ENGINE', 'django.db.backends.postgresql'),
-            'name': os.getenv('HOTPOCKET_BACKEND_DATABASE_NAME', 'hotpocket'),
+            'name': os.getenv('HOTPOCKET_BACKEND_DATABASE_NAME', ''),
-            'user': os.getenv('HOTPOCKET_BACKEND_DATABASE_USER', 'hotpocket'),
+            'user': os.getenv('HOTPOCKET_BACKEND_DATABASE_USER', ''),
-            'password': os.getenv('HOTPOCKET_BACKEND_DATABASE_PASSWORD', 'hotpocketm4st3r'),
+            'password': os.getenv('HOTPOCKET_BACKEND_DATABASE_PASSWORD', ''),
-            'host': os.getenv('HOTPOCKET_BACKEND_DATABASE_HOST', 'postgres.hotpocket.work.bthlabs.net'),
+            'host': os.getenv('HOTPOCKET_BACKEND_DATABASE_HOST', ''),
             'port': os.getenv('HOTPOCKET_BACKEND_DATABASE_PORT', '5432'),
         }),
     )