22 — Multi-Tenancy (v1)

Tier 3.2 — DONE

Forail Platform supports soft multi-tenancy: a single Forail install can host many customer tenants, each with its own quotas, branding, and an admin team, without spinning up separate databases or separate Celery workers. Tenancy is layered on top of the existing Organization + RBAC primitives — no Postgres row-level security, no per-tenant schemas.

Everything is additive and gated by a single kill switch (TENANCY_ENABLED). With the switch off, all tenant hooks short-circuit and Forail behaves exactly as before.

Scope (v1)

In:

• Organization promoted to "Tenant" via 11 additive fields (no new Org model).
• Per-tenant quotas enforced at job launch: concurrent jobs, daily launches, host count, storage MB.
• Tenant branding (logo, colors, custom domain) served by a public /api/v2/branding/ endpoint so the UI can skin itself before login.
• Tenant provisioning REST API (POST /api/v2/tenants/) that atomically creates an Organization, admin user, default team, and TenantUsage row.
• Quota audit log (TenantQuotaEvent) — one row per allow/block decision.
• Cross-tenant access audit (TenantIsolationEvent) — audit only in v1, no blocking.
• Celery beat reconciliation task that refreshes host count / storage usage.

Out (deferred to v2):

• Postgres row-level security policies
• Strict-mode enforcement (blocking cross-tenant queries)
• Per-tenant Celery queues
• Per-tenant API rate limiting (token bucket)
• Custom-domain TLS provisioning
• Billing / metering hooks

Architecture

  POST /api/v2/job_templates/{id}/launch/
                │
                ▼
  ┌─────────────────────────────────────────┐
  │ forail.launch  (OTel span, Tier 3.6)      │
  │                                          │
  │   check_tenant_quota(job, request)  ◄── NEW  (Tier 3.2)
  │        │ allowed → inc counters         │
  │        │ blocked → 429 + QuotaEvent     │
  │        ▼                                │
  │   evaluate_launch(...)  (Policy-as-Code)│
  │        ▼                                │
  │   run_scanners_for_launch(...)          │
  │        ▼                                │
  │   job.signal_start()                    │
  └─────────────────────────────────────────┘

The quota gate is the first gate inside the launch span so that blocked tenants never consume policy/scanning work. A post-finish signal decrements concurrent_jobs_count so the counter stays honest.

Models — forail/main/models/tenancy.py

Organization (extended in place)

11 additive fields on the existing Organization model. All default to False / null / empty string so existing orgs are untouched.

null on any quota means unlimited.

TenantUsage (new)

One row per tenant Org. The launch hook reads/writes concurrent_jobs_count / launches_today_count; the beat task refreshes hosts_count / storage_mb_used.

TenantQuotaEvent (new)

One row per quota decision. Mirrors PolicyDecision shape.

TenantIsolationEvent (new)

One row per cross-tenant read observed when tenant_isolation_strict=True.

Pure helpers — forail/main/tenancy/helpers.py

Exported for standalone test; zero Django imports.

def check_quota_value(current: int, limit: int | None) -> bool: ...
def is_window_expired(window_start: datetime, now: datetime) -> bool: ...
def reset_daily_window(now: datetime) -> tuple[datetime, int]: ...
def format_quota_message(kind: str, current: int, limit: int | None) -> str: ...
def normalize_branding_host(host: str) -> str: ...  # lowercase + strip port
def validate_hex_color(value: str) -> bool: ...

All four quota kinds use check_quota_value so the semantics are identical (None → unlimited; current >= limit → blocked).

Tenancy package layout — forail/main/tenancy/

| Module | Role | | ----------------- | ------------------------------------------------------------------------- | ----- | | __init__.py | Re-exports the public surface | | helpers.py | Pure functions (standalone-testable) | | quota.py | QuotaResult, check_tenant_quota(job, request), on_job_finished(job) | | provisioning.py | provision_tenant(payload) — atomic Org + User + Team + Usage | | branding.py | get_branding_for_host(host) -> dict | None | | usage.py | recalculate_tenant_usage(org) — drift reconciliation | | isolation.py | TenantIsolationMiddleware — cross-tenant audit (v1 log only) |

Launch hook order

In forail/api/views/job_templates.py, workflows.py, and ad_hoc_commands.py, the launch sequence inside the forail.launch OTel span is:

check_tenant_quota   (NEW — Tier 3.2)   — 429 on block
evaluate_launch      (Tier 2.2)         — 403 on block
run_scanners_for_launch (Tier 3.4)      — 403 on block
job.signal_start()

The quota gate sets quota_blocked=<kind> on the current span for easy trace filtering.

Job-finished signal

forail.main.signals.handlers dispatches on_job_finished(job) on each unified_job_finished signal, which atomically decrements concurrent_jobs_count for the tenant. The Celery reconciliation task also corrects drift (concurrent_jobs_count = actual running UnifiedJob count) on every tick.

Settings — forail/main/conf.py

All four live under the System category and show up in /api/v2/settings/system/.

Security posture (v2 hardening)

The v2 isolation stack is designed to fail closed and is off by default so single-tenant installs are unaffected. A multi-tenant deployment enables at least TENANCY_ENABLED + TENANCY_RLS_ENABLED, plus TENANCY_STRICT_ISOLATION_ENABLED for hard cross-tenant blocking.

• RLS gate fails closed. If the per-request Postgres tenant scope cannot be installed, the request is aborted (HTTP 500) rather than run with global row visibility (an unset scope makes RLS treat all rows as visible).
• Strict gate sees cross-tenant objects. The 403 gate resolves the target resource's organization with the caller's RLS scope removed — otherwise a genuinely cross-tenant object is invisible to the lookup and could never be blocked. When a covered resource's org can't be determined, it defaults to deny.
• RLS coverage. Every table with its own organization_id has a policy (including main_eventlog); child tables of main_unifiedjobtemplate / main_unifiedjob (projects, workflow templates, schedules, nodes, job events) are covered indirectly through the parent via the multi-table-inheritance join. Policies cast the tenant GUC as NULLIF(current_setting(...), '')::int so the empty "no scope" sentinel can't raise.
• Rate limiter logs Redis outages loudly and can be made to fail closed with TENANCY_RATE_LIMIT_FAIL_CLOSED (default open, favouring availability).
• Provisioning refuses to silently reuse an existing username (which would discard the supplied password and cross-link accounts) unless attach_existing_admin is set.

REST API

All /api/v2/tenants/* endpoints require is_superuser. /api/v2/branding/ is public (no auth classes).

Provisioning payload

{
  "name": "Acme Corp",
  "admin_username": "acme-admin",
  "admin_email": "admin@acme.example",
  "admin_password": "...",
  "quota": {
    "max_concurrent_jobs": 10,
    "max_daily_launches": 500,
    "max_hosts": 200,
    "max_storage_mb": 5000
  },
  "branding": {
    "logo_url": "https://...",
    "primary_color": "#5B47E0",
    "custom_domain": "acme.forail.example"
  }
}

Idempotent on name: a POST with an existing tenant name returns the existing tenant.

Branding response

{
  "tenant_id": 42,
  "name": "Acme Corp",
  "logo_url": "https://...",
  "primary_color": "#5B47E0",
  "secondary_color": "#3B2799"
}

Standalone tests — tests_standalone/test_tenancy.py

Pure-helper tests (no Django setup). Covers:

• check_quota_value under/at/over limit, None unlimited
• is_window_expired under/over 24h boundary
• reset_daily_window returns top-of-UTC-day
• format_quota_message human-readable strings for each kind
• QuotaResult aggregation (any block → overall block)
• normalize_branding_host lowercase + strip port
• validate_hex_color valid/invalid patterns
• Provisioning payload validation (required fields, password strength, email)

Run:

python -m unittest tests_standalone.test_tenancy -v

Verification (E2E smoke)

With the stack running:

1. Enable tenancy — Settings → System → TENANCY_ENABLED = true.
2. POST /api/v2/tenants/ provisioning a fake "Acme" tenant with custom_domain=acme.localhost, max_concurrent_jobs=1.
3. Sidebar → Tenancy → Tenants → Acme appears with usage bars.
4. As acme-admin, launch two jobs back-to-back. Second returns 429 with reasons=[concurrent_jobs:...]; a TenantQuotaEvent row appears in the audit log.
5. curl -sk 'https://localhost/api/v2/branding/?host=acme.localhost' → returns Acme's branding JSON without an auth header.
6. Load https://localhost with Host: acme.localhost → UI skins with Acme colors and logo before login.
7. POST /api/v2/tenants/{id}/recalculate/ → usage values refresh.
8. DELETE /api/v2/tenants/{id}/?confirm=true → tenant + all its resources removed.

Files

• forail/main/models/tenancy.py — models
• forail/main/models/organization.py — extended in place
• forail/main/migrations/0204_multi_tenancy.py — schema migration
• forail/main/tenancy/ — helpers + quota + provisioning + branding + usage + isolation
• forail/main/tasks/tenancy.py — Celery beat task
• forail/main/conf.py — settings registration
• forail/api/serializers/tenancy.py
• forail/api/views/tenancy.py
• forail/api/urls/tenancy.py
• tests_standalone/test_tenancy.py