rename dmc12 inventory to cloudlorean and vault admin_password

- replace inventory/dmc12.yml with cloudlorean.yml (+ group_vars) and update all README/AGENTS call examples and the inventory table - move admin_password hash into encrypted vault.yml; remove plaintext copies from roles/bootstrap/defaults and group_vars/external.yml - bootstrap.yml loads vault.yml via vars_files (same pattern as monitoring) - monitoring: PSK config/registration updates, docker role defaults, os-updates logging/reboot refinements, ansible.cfg/execute.sh tweaks - add .project-wiki knowledge base and .github agent instructions
2026-09-09 14:14:18 +02:00
parent a31751af1d
commit 30f3e9a4c6
96 changed files with 3602 additions and 138 deletions
@@ -0,0 +1,34 @@
+# Technical Documentation
+
+This section records implemented or observed code behavior and technical structure.
+
+## When To Read
+
+- Work with architecture, modules, APIs, data, integrations, tests, deployment, or security.
+
+## Key Files
+
+- [Architecture](./architecture.md)
+- [Codebase map](./codebase-map.md)
+- [Testing](./testing.md)
+- [Deployment](./deployment.md)
+- [Security](./security.md)
+
+## Modules
+
+- [Bootstrap role](./modules/MOD-001-bootstrap-role.md)
+- [Docker role](./modules/MOD-002-docker-role.md)
+- [Monitoring role (Zabbix Agent 2)](./modules/MOD-003-monitoring-role.md)
+- [OS updates role](./modules/MOD-004-os-updates-role.md)
+- [Healthcheck role](./modules/MOD-005-healthcheck-role.md)
+- [Hawser role](./modules/MOD-006-hawser-role.md)
+- [Manage SSH keys role](./modules/MOD-007-manage-ssh-keys-role.md)
+
+## Integrations
+
+- [Zabbix](./integrations/zabbix.md)
+
+## Placeholders Or Stale Areas
+
+- None. `technical/api/` and `technical/data/` remain empty (no public APIs
+  or data schemas in an Ansible repository).
@@ -0,0 +1,102 @@
+---
+id: TECH-ARCHITECTURE
+type: technical
+status: active
+title: Architecture
+created: '2026-09-09'
+updated: '2026-09-09'
+tags:
+- technical
+- architecture
+- ansible
+related:
+- TECH-CODEBASE-MAP
+- TECH-DEPLOYMENT
+- TECH-INTEGRATIONS-ZABBIX
+- SCAN-20260909
+source_paths:
+- ansible.cfg
+- execute.sh
+- bootstrap.yml
+- playbooks/
+- roles/
+confidence: confirmed
+---
+
+# Architecture
+
+## Overview
+
+A push-based Ansible controller-in-container setup: an interactive Docker
+container (built from `debian:trixie-slim`, `ansible-core 2.20.x`) runs the
+playbooks from the mounted repository and connects over SSH to Debian hosts
+(`user: admin`, `become: true`). No agent is pushed except Zabbix Agent 2.
+Unverified claims are marked `confidence: inferred`.
+
+## Layered View
+
+```
+Docker runtime (execute.sh)
+  └─ ansible-playbook -i inventory/<site>.yml <playbook>
+       └─ SSH (admin, pipelining, no host-key checking) → Debian hosts
+            ├─ roles/bootstrap   ── composes ──► roles/docker
+            │                                 └─► roles/monitoring
+            ├─ roles/docker         (Docker Engine from download.docker.com)
+            ├─ roles/monitoring     (zabbix-agent2 → repo.zabbix.com, PSK,
+            │                        Zabbix API registration via httpapi)
+            ├─ roles/os-updates     (apt full upgrade, reboot, controller logs)
+            ├─ roles/healthcheck    (container quality gate, controller logs)
+            ├─ roles/hawser         (docker_compose_v2, pull: always)
+            └─ roles/manage-ssh-keys (authorized_keys allow/deny lists)
+```
+
+## Role Composition
+
+- `bootstrap` is the only composing role: it imports the `docker` role and
+  the `monitoring` role unless `skip_docker` / `skip_monitoring` are set.
+- `os-updates` and `healthcheck` are paired at playbook level
+  (`playbooks/os-updates-deb.yml`); healthcheck reuses os-updates logging
+  defaults (`healthcheck_log_file` derives from `os_update_log_dir`).
+
+## Inventory Model
+
+- Three sites: `cloudlorean` (LAN 192.168.212.0/24, groups `debian` +
+  `proxmox`), `home` (10.13.37.0/24, groups `debian` + `proxmox`),
+  `external` (IPv6-only, group `debian` only).
+- `proxmox` group membership drives reboot deferral in `os-updates`
+  (`'proxmox' in group_names`); `virtualization_type != lxc` skips kernel
+  and reboot logic for LXC containers.
+- Per-site `group_vars/<site>.yml` override Zabbix connectivity (server,
+  API URL, proxy), host groups/templates; `group_vars/debian.yml` holds
+  os-update defaults including the hardcoded
+  `os_update_version_codename: "trixie"`.
+- Secrets: `vault.yml` (encrypted, tracked) provides
+  `monitoring_zabbix_api_password` (and possibly more; vault not decrypted -
+  `confidence: unknown`).
+
+## Controller-Side State
+
+- `.zabbix-psk/<hostname>.psk` - per-host PSKs generated once on the
+  controller and reused across runs (keeps agent and Zabbix registration in
+  sync).
+- `logs/<inventory>/<hostname>/update.log` - append-only update + quality
+  gate log written via `delegate_to: localhost` / `blockinfile`.
+
+## Execution Flow (os-updates-deb.yml)
+
+1. Debian guard (fail non-Debian).
+2. `os-updates`: optional mirror rewrite → preflight logging (upgradable
+   list, start time) → `apt upgrade: full` → kernel-comparison reboot
+   decision (async reboot + `wait_for_connection`, deferred on `proxmox`
+   group unless `os_update_reboot_on_proxmox`) → postflight logging.
+3. `healthcheck`: docker presence check → container comparison →
+   `quality_gate` log section → optional failure
+   (`healthcheck_fail_on_unhealthy`).
+
+## Dependencies
+
+- Collections: `ansible.posix`, `community.general`, `community.docker`,
+  `community.zabbix` (pinned in `docker/requirements.yml`).
+- External services: repo.zabbix.com, download.docker.com, Debian mirrors,
+  Zabbix API (`mon.cloudlorean.de`). See
+  [Integrations: Zabbix](./integrations/zabbix.md).
@@ -0,0 +1,90 @@
+---
+id: TECH-CODEBASE-MAP
+type: technical
+status: active
+title: Codebase Map
+created: '2026-09-09'
+updated: '2026-09-09'
+tags:
+- technical
+- codebase
+related:
+- TECH-ARCHITECTURE
+- SCAN-20260909
+source_paths: []
+confidence: confirmed
+---
+
+# Codebase Map
+
+Ansible repository, no application source code. Structure as of 2026-09-09.
+
+## Root Files
+
+| Path | Purpose |
+| --- | --- |
+| `ansible.cfg` | Global Ansible config: no host key checking, `roles_path=./roles`, `forks=20`, smart gathering, SSH pipelining |
+| `bootstrap.yml` | Top-level playbook, binds the `bootstrap` role with Debian guard |
+| `execute.sh` | Starts the interactive Docker runtime container (`ansible:deb13`) |
+| `vault.yml` | Ansible-Vault encrypted secrets (tracked, encrypted) |
+| `setenv.sh` | Proxmox credentials (git-ignored, not in repo) |
+| `.vault_pass`, `.zabbix-psk/`, `logs/` | Vault password, per-host PSKs, update logs (git-ignored) |
+
+## Directories
+
+```
+docker/                  Dockerfile (debian:trixie-slim + ansible-core 2.20.x),
+                         requirements.yml (pinned collections), ansible-functs.sh
+                         (aliases ap/ag/av)
+inventory/               cloudlorean.yml, home.yml, external.yml + group_vars/
+playbooks/               docker.yml, monitoring.yml, os-updates-deb.yml,
+                         upgrade-hawser.yml, hardening/manage-ssh-keys.yml
+roles/bootstrap/         tasks/ (8 subtasks), templates/ (4), handlers/, README.md
+roles/docker/            tasks/ (2), templates/sources.list.j2, handlers/, README.md
+roles/hawser/            tasks/ (2), templates/docker-compose.yml.j2, handlers/, README.md
+roles/healthcheck/       tasks/main.yml, defaults/, README.md (no handlers/templates)
+roles/manage-ssh-keys/   tasks/ (4), handlers/, defaults/, README.md
+roles/monitoring/        tasks/ (8), templates/ (2: zabbix_agent2.conf.j2, docker.conf.j2),
+                         handlers/, README.md
+roles/os-updates/        tasks/ (6), templates/ (2: sources.list.j2, sources.list-deb822.j2),
+                         handlers/, README.md
+```
+
+## Entry Points
+
+| Playbook | Role(s) | Notes |
+| --- | --- | --- |
+| `bootstrap.yml` | bootstrap (+ docker, monitoring) | Full initial setup |
+| `playbooks/docker.yml` | docker | Docker Engine install |
+| `playbooks/monitoring.yml` | monitoring | Loads `../vault.yml` |
+| `playbooks/os-updates-deb.yml` | os-updates, healthcheck | Update + quality gate |
+| `playbooks/upgrade-hawser.yml` | hawser | Compose stack update |
+| `playbooks/hardening/manage-ssh-keys.yml` | manage-ssh-keys | No Debian guard, no become |
+
+## Conventions (Observed)
+
+- Roles split logic into `tasks/<name>.yml` subtasks; `tasks/main.yml` only
+  imports/includes (`bootstrap`, `os-updates`) or delegates to a single
+  subtask (`docker`, `hawser`).
+- Newer tasks use FQCN; older tasks (apt, file, blockinfile, set_fact,
+  include_role) still use short names.
+- Per-role README documents variables, templates, handlers, tags.
+- Tags observed: `motd`, `ssh`, `bashrc`, `sysctl` (bootstrap),
+  `monitoring` (monitoring role).
+
+## Tests
+
+- No automated test suite. Verification is manual: run in the Docker
+  container, optional `ansible-lint`, `--check` dry runs. See
+  [Testing](./testing.md).
+
+## Inventories
+
+| Site | File | Groups | Hosts |
+| --- | --- | --- | --- |
+| cloudlorean | `inventory/cloudlorean.yml` | debian (13), proxmox (2: dmc12, pbs) | 192.168.212.0/24 |
+| home | `inventory/home.yml` | debian (6), proxmox (1: tardis) | 10.13.37.0/24 |
+| external | `inventory/external.yml` | debian (4) | IPv6-only |
+
+Per-site `group_vars/<site>.yml` override Zabbix servers, API URL, proxy,
+host groups and templates. See [Architecture](./architecture.md).
@@ -0,0 +1,59 @@
+---
+id: TECH-DEPLOYMENT
+type: technical
+status: active
+title: Deployment (Runtime Environment)
+created: '2026-09-09'
+updated: '2026-09-09'
+tags:
+- technical
+- deployment
+- docker
+related:
+- TECH-ARCHITECTURE
+source_paths:
+- execute.sh
+- docker/Dockerfile
+- docker/requirements.yml
+- docker/ansible-functs.sh
+- ansible.cfg
+confidence: confirmed
+---
+
+# Deployment (Runtime Environment)
+
+## Docker Image
+
+- `docker/Dockerfile`: `debian:trixie-slim`, `TZ=Europe/Berlin`,
+  `ANSIBLE_VERSION=2.20.6` → `pip3 install ansible-core~=2.20.6`.
+- Utilities: nano, git, wget, unzip, curl, python3-pip, tzdata, openssh-client,
+  ping, vim, less.
+- Bakes SSH client policy: `StrictHostKeyChecking no`,
+  `UserKnownHostsFile /dev/null` (`/etc/ssh/ssh_config.d/99-no-hostkey-check.conf`).
+- Installs pinned collections from `docker/requirements.yml`:
+  `ansible.posix` 2.2.0, `community.general` 13.0.1, `community.docker`
+  5.2.0, `community.zabbix` 4.2.0.
+- Copies `ansible-functs.sh` into `/root/.bashrc.d/` providing aliases
+  `ap` (ansible-playbook), `ag` (ansible-galaxy), `av` (ansible-vault).
+
+## Launch Flow (execute.sh)
+
+1. Pick next free container name `ansible-<n>`.
+2. Ensure IPv6-enabled bridge network `ansible-dualstack` (fails fast with
+   a clear message if the network exists without IPv6).
+3. Run the container detached: repo mounted at `/ansible` (workdir), SSH
+   keys from the `ansible_ssh_keys` volume into `/root/.ssh`, env
+   `ANSIBLE_HOST_KEY_CHECKING=False`, `ANSIBLE_CONFIG=./ansible.cfg`.
+4. Write a per-container SSH client config, print name/IPv4/IPv6/OS, then
+   `docker attach` for interactive use.
+
+## Controller-Side Artifacts
+
+- `.zabbix-psk/` and `logs/` are written by delegation from the container
+  into the mounted repo (git-ignored).
+
+## Operational Notes
+
+- Changing collections requires updating `docker/requirements.yml` and an
+  image rebuild (AGENTS.md).
+- Default image tag: `ansible:deb13`.
@@ -0,0 +1,63 @@
+---
+id: TECH-INTEGRATIONS-ZABBIX
+type: technical
+status: active
+title: Integrations - Zabbix
+created: 2026-09-09
+updated: 2026-09-09
+tags:
+- technical
+- integration
+- zabbix
+related:
+- MOD-003
+source_paths:
+- roles/monitoring/
+- inventory/group_vars/
+confidence: confirmed
+---
+
+# Integrations - Zabbix
+
+## Agent Side (outbound/inbound to monitored hosts)
+
+- Package source: `repo.zabbix.com` (release package for Zabbix 7.0 on
+  Debian, downloaded to `/tmp` and installed).
+- Agent: `zabbix-agent2`, listen port 10050, config
+  `/etc/zabbix/zabbix_agent2.conf`.
+- TLS: PSK only (`tls_connect=psk`, `tls_accept=psk`), identity per host,
+  PSK file `/etc/zabbix/zabbix_agent2.psk`.
+- Docker monitoring: plugin config `/etc/zabbix/zabbix_agent2.d/plugins.d/
+  docker.conf` pointing at `unix:///var/run/docker.sock`, deployed when the
+  socket exists.
+- Passive/active server lists come from site group_vars
+  (`monitoring_zabbix_passive_servers`).
+
+## Controller Side (Zabbix API registration)
+
+- API endpoint per site: `https://mon.cloudlorean.de/api_jsonrpc.php`
+  (group_vars); credentials `monitoring_zabbix_api_user: Admin` +
+  `monitoring_zabbix_api_password` from `vault.yml`.
+- Module: `community.zabbix.zabbix_host` over `httpapi`; connection
+  parameters derived from the API URL with `urlsplit` (scheme, port, path).
+- Host object: name = `monitoring_zabbix_host_name` (default
+  `inventory_hostname`), agent interface IP = `ansible_host`, host groups
+  and templates per site, `inventory_mode: manual`, TLS PSK identity/value
+  from the deployed PSK, `status: enabled`, `state: present` (idempotent
+  register-or-update).
+- Optional proxy: when `monitoring_zabbix_proxy` is set, `monitored_by`
+  switches to the proxy (`gatekeeper.cloudlorean.de` for cloudlorean,
+  `gatekeeper.meyer-kempen.de` for home).
+
+## Site Overrides (observed)
+
+| Site | Passive servers | Proxy | Host groups |
+| --- | --- | --- | --- |
+| cloudlorean | 192.168.212.1 | gatekeeper.cloudlorean.de (192.168.212.1) | Linux servers, Cloudlorean |
+| home | 10.13.37.1 | gatekeeper.meyer-kempen.de (10.13.37.1) | Linux servers, Meyer-Kempen |
+| external | 2a01:4f8:1c1e:62dd::1 (see OQ-004) | none | Linux servers, Cloudlorean |
+
+## Other External Services
+
+- `download.docker.com` (docker role apt repository), Debian mirrors
+  (`deb.debian.org`, `security.debian.org` default; site-overridable).
@@ -0,0 +1,65 @@
+---
+id: MOD-001
+type: module
+status: active
+title: Bootstrap Role
+created: 2026-09-09
+updated: 2026-09-09
+tags: [module, bootstrap, ansible]
+related: [MOD-002, MOD-003, REQ-001]
+source_paths:
+- roles/bootstrap/
+confidence: confirmed
+---
+
+# Bootstrap Role
+
+## Responsibility
+
+Full initial setup of a Debian host: packages, admin user, SSH, MOTD,
+keyboard layout, bashrc, sysctl; then composes Docker and monitoring.
+
+## Source Paths
+
+- `roles/bootstrap/tasks/main.yml` - orchestration only
+- Subtasks: `install-basicpackages.yml`, `create-admin-user.yml`,
+  `set-motd.yml`, `set-keyboardlayout.yml`, `install-openssh.yml`,
+  `configure-ssh.yml`, `add-ssh-keys.yml`, `setup-bashrc.yml`,
+  `configure-sysctl.yml`
+- Templates: `authorized_keys.j2`, `keyboard.j2`, `sshd.conf.j2`,
+  `sudoers-admin.j2`
+
+## Public Interfaces
+
+- Playbook: `bootstrap.yml` (top level, Debian guard, `become`, `user: admin`,
+  `vars_files: vault.yml`)
+- Skip flags: `skip_docker`, `skip_monitoring` (control role composition)
+- Tags: `motd`, `ssh`, `bashrc`, `sysctl`
+
+## Important Flows
+
+1. Basic packages → admin user (+sudo via `sudoers-admin.j2`, authorized
+   keys via `authorized_keys.j2`) → MOTD → keyboard layout.
+2. OpenSSH install, hardening config (`sshd.conf.j2`), authorized keys.
+3. bashrc, sysctl.
+4. `import_role: docker` (when `not skip_docker`), then
+   `import_role: monitoring` (when `not skip_monitoring`).
+
+## Dependencies
+
+- Composed roles: [docker](./MOD-002-docker-role.md),
+  [monitoring](./MOD-003-monitoring-role.md).
+- `admin_authorized_keys` (public keys, plaintext OK) from
+  `defaults/main.yml`; `admin_password` (SHA-512 hash) lives encrypted in
+  `vault.yml`, loaded via `vars_files` in `bootstrap.yml` (resolved
+  ALERT-20260909-001).
+
+## Tests
+
+- None automated; manual runs via `./execute.sh`.
+
+## Notes For Agents
+
+- `sshd` service name differs per host (`ssh_service_name`, set for `gitea`
+  and `ipam` in `inventory/cloudlorean.yml`).
+- Keep `tasks/main.yml` as pure import list when extending.
@@ -0,0 +1,52 @@
+---
+id: MOD-002
+type: module
+status: active
+title: Docker Role
+created: 2026-09-09
+updated: 2026-09-09
+tags: [module, docker, ansible]
+related: [MOD-001, MOD-006, REQ-002]
+source_paths:
+- roles/docker/
+confidence: confirmed
+---
+
+# Docker Role
+
+## Responsibility
+
+Install Docker Engine from the official Docker apt repository.
+
+## Source Paths
+
+- `roles/docker/tasks/main.yml` → `install-docker.yml`
+- Template: `roles/docker/templates/sources.list.j2`
+- Handlers: `roles/docker/handlers/main.yml`
+
+## Public Interfaces
+
+- Playbook: `playbooks/docker.yml`
+- Defaults: `docker_mirror` (download.docker.com/linux/debian),
+  `os_version_codename` (derived from `ansible_facts['distribution_release']`),
+  `docker_packages` (docker-ce, docker-ce-cli, containerd.io, buildx and
+  compose plugins).
+
+## Important Flows
+
+1. Set up the official Docker apt repository (key + `sources.list.j2`).
+2. Install `docker_packages` (handlers restart the service when needed).
+
+## Dependencies
+
+- Used standalone (`playbooks/docker.yml`) and composed by
+  [bootstrap](./MOD-001-bootstrap-role.md). Prerequisite for
+  [hawser](./MOD-006-hawser-role.md).
+
+## Tests
+
+- None automated.
+
+## Notes For Agents
+
+- Codename comes from host facts; no LXC-specific handling observed.
@@ -0,0 +1,77 @@
+---
+id: MOD-003
+type: module
+status: active
+title: Monitoring Role (Zabbix Agent 2)
+created: 2026-09-09
+updated: 2026-09-09
+tags: [module, monitoring, zabbix, ansible]
+related: [MOD-001, TECH-INTEGRATIONS-ZABBIX, REQ-003, NFR-002]
+source_paths:
+- roles/monitoring/
+confidence: confirmed
+---
+
+# Monitoring Role (Zabbix Agent 2)
+
+## Responsibility
+
+Install Zabbix Agent 2, configure TLS-PSK and the Docker plugin, and
+register/update the host on the Zabbix server via the API.
+
+## Source Paths
+
+- `roles/monitoring/tasks/main.yml` → `validate.yml`,
+  `install-repository.yml`, `install-agent.yml`, `configure-docker.yml`,
+  `configure-psk.yml`, `configure-agent.yml`, `register-host.yml`
+- Templates: `zabbix_agent2.conf.j2`, `docker.conf.j2`
+
+## Public Interfaces
+
+- Playbook: `playbooks/monitoring.yml` (loads `../vault.yml`)
+- Tag: `monitoring`
+- Key variables (`defaults/main.yml`, overridden per site in
+  `inventory/group_vars/<site>.yml`): `monitoring_zabbix_version` (7.0),
+  `monitoring_zabbix_passive_servers`, `monitoring_zabbix_active_servers`,
+  `monitoring_zabbix_api_url`, `monitoring_zabbix_api_user`,
+  `monitoring_zabbix_api_password` (vault), `monitoring_zabbix_register_host`,
+  `monitoring_zabbix_proxy` / `monitoring_zabbix_proxy_address`,
+  `monitoring_zabbix_host_groups`, `monitoring_zabbix_templates`,
+  `monitoring_zabbix_docker_templates`, PSK settings
+  (`monitoring_zabbix_psk_bits: 256`,
+  `monitoring_zabbix_psk_store: {{ role_path }}/../../.zabbix-psk`).
+
+## Important Flows
+
+1. Validate inputs → add repo.zabbix.com repo → install `zabbix-agent2`.
+2. Docker plugin config when `/var/run/docker.sock` exists.
+3. PSK: generate once on the controller (`openssl rand -hex 32`, umask 077,
+   `creates:` guard), persist under `.zabbix-psk/<hostname>.psk` (0700),
+   validate hex format/length, deploy to `/etc/zabbix/zabbix_agent2.psk`
+   (0600), notify restart handler.
+4. Agent config: TLS connect/accept = PSK, identity =
+   `monitoring_zabbix_host_name`, passive/active servers, listen port 10050.
+5. Registration: `community.zabbix.zabbix_host` over httpapi (connection
+   params derived from `monitoring_zabbix_api_url` via `urlsplit`), agent
+   interface from `ansible_host`, template union adds
+   `monitoring_zabbix_docker_templates` when the Docker socket is present,
+   `monitored_by` switches to proxy when `monitoring_zabbix_proxy` is set.
+
+## Dependencies
+
+- `community.zabbix` collection; vault variable
+  `monitoring_zabbix_api_password`.
+- Composed by [bootstrap](./MOD-001-bootstrap-role.md) unless
+  `skip_monitoring`.
+
+## Tests
+
+- None automated.
+
+## Notes For Agents
+
+- PSK store is controller-side state; never commit it (git-ignored). When
+  renaming a host, migrate/regenerate the PSK, otherwise registration
+  drifts (documented in AGENTS.md).
+- Template name mismatch between role default (`Docker by Zabbix agent2`)
+  and group_vars (`Docker by Zabbix agent 2`) - see open question OQ-005.
@@ -0,0 +1,73 @@
+---
+id: MOD-004
+type: module
+status: active
+title: OS Updates Role
+created: 2026-09-09
+updated: 2026-09-09
+tags: [module, os-updates, ansible]
+related: [MOD-005, NFR-003, NFR-006, REQ-004]
+source_paths:
+- roles/os-updates/
+confidence: confirmed
+---
+
+# OS Updates Role
+
+## Responsibility
+
+Upgrade Debian packages with optional mirror rewrite, kernel-based reboot
+handling, and controller-side update logging.
+
+## Source Paths
+
+- `roles/os-updates/tasks/main.yml` → `update_mirrors.yml` (optional),
+  `upgrade_packages.yml` → `logging_preflight.yml`, `reboot.yml`,
+  `logging_postflight.yml`
+- Templates: `sources.list.j2`, `sources.list-deb822.j2`
+- Handlers: apt cleanup, apt autoremove
+
+## Public Interfaces
+
+- Playbook: `playbooks/os-updates-deb.yml` (chains
+  [healthcheck](./MOD-005-healthcheck-role.md))
+- Key variables: `os_update_auto_upgrade`, `os_also_update_mirror` (false),
+  `os_update_mirrors` (main + security mirror),
+  `os_update_version_codename` (role default: host facts;
+  `group_vars/debian.yml` hardcodes `trixie`),
+  `os_update_debian_codenames` (trixie, bookworm, bullseye),
+  `os_update_reboot_on_proxmox` (false),
+  `os_update_logging_enabled` (true), `os_update_log_dir` (/ansible/logs),
+  `os_update_log_file` (logs/<inventory>/<hostname>/update.log).
+
+## Important Flows
+
+1. Mirror rewrite (only when `os_also_update_mirror`) with
+   `ignore_errors: true`; only whitelisted Debian codenames are rewritten in
+   `sources.list.d`.
+2. Preflight logging: refresh cache, record start epoch/ISO, parse
+   `apt list --upgradable` into names/versions/count.
+3. `apt upgrade: full` (notifies cleanup/autoremove handlers).
+4. Reboot decision: compare running kernel (`uname -r`) vs newest installed
+   `linux-image-*`; skipped on LXC (`virtualization_type != lxc`). On
+   `proxmox` group hosts the reboot is deferred unless
+   `os_update_reboot_on_proxmox`. Otherwise: async `/sbin/reboot`,
+   `wait_for_connection` (10s delay, 600s timeout), downtime and kernel
+   after reboot recorded.
+5. Postflight logging: duration, upgrade result, reboot details, kernel
+   before/after appended to the update log (`delegate_to: localhost`).
+
+## Dependencies
+
+- None (ansible.builtin only). Paired with healthcheck at playbook level.
+
+## Tests
+
+- None automated.
+
+## Notes For Agents
+
+- Logging facts are shared with the healthcheck role via the same log file
+  and start timestamp conventions.
+- Old-style module names (apt, file, blockinfile) remain here; use FQCN for
+  new tasks.
@@ -0,0 +1,59 @@
+---
+id: MOD-005
+type: module
+status: active
+title: Healthcheck Role
+created: 2026-09-09
+updated: 2026-09-09
+tags: [module, healthcheck, docker, ansible]
+related: [MOD-004, REQ-004, NFR-003]
+source_paths:
+- roles/healthcheck/
+confidence: confirmed
+---
+
+# Healthcheck Role
+
+## Responsibility
+
+Post-update quality gate: verify that every Docker container on the host is
+running and record the result in the update log.
+
+## Source Paths
+
+- `roles/healthcheck/tasks/main.yml` (single file, no subtasks)
+- `roles/healthcheck/defaults/main.yml`
+
+## Public Interfaces
+
+- Playbook: `playbooks/os-updates-deb.yml` (runs after os-updates)
+- Key variables: `healthcheck_fail_on_unhealthy` (default false),
+  `healthcheck_logging_enabled` (derives from `os_update_logging_enabled`),
+  `healthcheck_log_file` (reuses os-updates log path conventions).
+
+## Important Flows
+
+1. Detect Docker binary (`which docker`, tolerant).
+2. If Docker present: list all container names and running container names
+   (`docker ps -a` / `docker ps --filter status=running`), compute
+   non-running set.
+3. If Docker absent: neutral result (0 containers, passed).
+4. Append a `quality_gate:` section (docker_installed, total/running counts,
+   non-running list, healthcheck_passed) to
+   `logs/<inventory>/<hostname>/update.log` via `blockinfile` delegated to
+   localhost.
+5. Fail the play when `healthcheck_fail_on_unhealthy` is true and any
+   container is not running.
+
+## Dependencies
+
+- Shares log file conventions with [os-updates](./MOD-004-os-updates-role.md).
+
+## Tests
+
+- None automated.
+
+## Notes For Agents
+
+- The gate does not fail by default; set `healthcheck_fail_on_unhealthy`
+  to make failures visible in CI-like runs.
@@ -0,0 +1,57 @@
+---
+id: MOD-006
+type: module
+status: active
+title: Hawser Role
+created: 2026-09-09
+updated: 2026-09-09
+tags: [module, hawser, docker, ansible]
+related: [MOD-002, REQ-005]
+source_paths:
+- roles/hawser/
+confidence: confirmed
+---
+
+# Hawser Role
+
+## Responsibility
+
+Update the Hawser Docker Compose stack on a host that already runs Docker.
+
+## Source Paths
+
+- `roles/hawser/tasks/main.yml` → `update-stack.yml`
+- Template: `roles/hawser/templates/docker-compose.yml.j2`
+- Handlers: `roles/hawser/handlers/main.yml`
+
+## Public Interfaces
+
+- Playbook: `playbooks/upgrade-hawser.yml`
+- Key variables: `hawser_project_name` (hawser),
+  `hawser_compose_dir` (/opt/hawser), `hawser_port` (2376),
+  `hawser_stacks_volume` (hawser_stacks),
+  `hawser_allow_insecure_no_auth` (true - documented as requiring an
+  otherwise secured network).
+
+## Important Flows
+
+1. Gather Docker host info (`community.docker.docker_host_info`,
+   `ignore_errors: true`).
+2. Require existing compose file at `<hawser_compose_dir>/docker-compose.yml`;
+   re-render it from the template.
+3. `community.docker.docker_compose_v2` with `state: present`,
+   `pull: always` to pull latest images and recreate the stack.
+
+## Dependencies
+
+- Requires Docker on the target (role skips gracefully when Docker is not
+  reachable). Collection: `community.docker`.
+
+## Tests
+
+- None automated.
+
+## Notes For Agents
+
+- `hawser_allow_insecure_no_auth: true` allows token-less standard mode on
+  non-loopback addresses; treat as a deliberate, documented trade-off.
@@ -0,0 +1,58 @@
+---
+id: MOD-007
+type: module
+status: active
+title: Manage SSH Keys Role
+created: 2026-09-09
+updated: 2026-09-09
+tags: [module, ssh, hardening, ansible]
+related: [REQ-006, ALERT-20260909-001]
+source_paths:
+- roles/manage-ssh-keys/
+confidence: confirmed
+---
+
+# Manage SSH Keys Role
+
+## Responsibility
+
+Harden SSH access by enforcing an allow list (`good_keys`) and removing
+deny-listed keys (`bad_keys`) from the target user's authorized keys.
+
+## Source Paths
+
+- `roles/manage-ssh-keys/tasks/main.yml` → `validate-authorized-keys.yml`,
+  `add-goodkeys.yml`, `remove-badkeys.yml`
+- Handlers: `roles/manage-ssh-keys/handlers/main.yml`
+
+## Public Interfaces
+
+- Playbook: `playbooks/hardening/manage-ssh-keys.yml`
+  (binds the role directly; **no Debian guard, no `become`**)
+- Key variables: `ssh_user` (admin), `good_keys` (allow list),
+  `bad_keys` (deny list; supports env-provided JSON overrides via commented
+  vars in the playbook).
+- Conditional execution: `good_keys.defined and good_keys | length > 0`
+  (same for `bad_keys`).
+
+## Important Flows
+
+1. Validate/ensure `~/.ssh` of the logged-in user (0700).
+2. Add `good_keys` to authorized keys.
+3. Remove `bad_keys` from authorized keys.
+
+## Dependencies
+
+- None (ansible.builtin).
+
+## Tests
+
+- None automated.
+
+## Notes For Agents
+
+- The playbook deviates from the documented convention (Debian guard,
+  `become: true`); treat as an intentional gap to confirm before adding
+  more hardening playbooks.
+- Keys in `defaults/main.yml` are public keys (not secrets); the same allow
+  list also exists in `roles/bootstrap/defaults/main.yml`.
@@ -0,0 +1,69 @@
+---
+id: TECH-SECURITY
+type: technical
+status: active
+title: Security
+created: '2026-09-09'
+updated: '2026-09-09'
+tags:
+- technical
+- security
+related:
+- ALERT-20260909-001
+- ALERT-20260909-002
+- NFR-002
+- MOD-003
+- MOD-007
+source_paths:
+- vault.yml
+- roles/monitoring/tasks/configure-psk.yml
+- roles/bootstrap/defaults/main.yml
+- ansible.cfg
+confidence: confirmed
+---
+
+# Security
+
+## Secrets Handling
+
+- `vault.yml` (tracked, encrypted) holds `admin_password` (SHA-512 hash),
+  `monitoring_zabbix_api_password`, and Proxmox credentials
+  (`PROXMOX_USER_NAME`, `PROXMOX_USER_PASS`); it is loaded via playbook-level
+  `vars_files` in `bootstrap.yml` and `playbooks/monitoring.yml`.
+- Git-ignored (verified): `.vault_pass`, `setenv.sh` (Proxmox credentials),
+  `.zabbix-psk/` (per-host PSKs), `logs/`.
+- Resolved 2026-09-09: the `admin_password` SHA-512 hash that was committed
+  in plaintext in `roles/bootstrap/defaults/main.yml` and
+  `inventory/group_vars/external.yml` was moved to encrypted `vault.yml`
+  (see [ALERT-20260909-001](../alerts/ALERT-20260909-001.md)).
+
+## SSH Posture
+
+- Controller/transport: `host_key_checking = False`,
+  `StrictHostKeyChecking=no`, `UserKnownHostsFile=/dev/null` (ansible.cfg,
+  execute.sh, Dockerfile). Deliberate for first-time provisioning; README
+  explicitly warns to restrict this in production.
+- Host hardening: `manage-ssh-keys` role enforces allow/deny key lists;
+  bootstrap deploys hardened `sshd.conf.j2` and per-host authorized keys.
+- Per-host exceptions: `ssh_service_name: ssh` on `gitea`/`ipam`.
+
+## Zabbix Agent TLS
+
+- PSK-only encryption (TLS connect + accept), 256-bit PSKs generated with
+  `openssl rand -hex 32` on the controller, persisted with umask 077 in a
+  0700 directory, deployed 0600 to the agent host. Identity = host name.
+
+## Risky Defaults (documented trade-offs)
+
+- `hawser_allow_insecure_no_auth: true` - Hawser standard mode without token
+  on non-loopback addresses; comment in defaults demands other network
+  protections.
+- `execute.sh` disables host key checking inside the runtime container.
+- Mirror rewrite runs with `ignore_errors: true` in os-updates.
+
+## Git Hygiene
+
+- Resolved 2026-09-09: the previously uncommitted change set (including the
+  `dmc12` → `cloudlorean` inventory rename and stale documentation) was
+  committed as one coherent change; see
+  [ALERT-20260909-002](../alerts/ALERT-20260909-002.md).
@@ -0,0 +1,41 @@
+---
+id: TECH-TESTING
+type: technical
+status: active
+title: Testing
+created: '2026-09-09'
+updated: 2026-09-09
+tags:
+- technical
+- testing
+related:
+- MOD-005
+source_paths: []
+confidence: confirmed
+---
+
+# Testing
+
+## Automated Tests
+
+- None found. The repository has no test suite, no CI configuration, and no
+  lint config (`confidence: confirmed` for absence in the working tree).
+
+## Verification Approach (documented in AGENTS.md / README)
+
+1. Run changes in the Docker container: `./execute.sh` (image
+   `ansible:deb13`, repo mounted at `/ansible`).
+2. Optional lint: `ansible-lint` (recommended by AGENTS.md, not installed in
+   the image by default).
+3. Dry runs with `ansible-playbook --check` where sensible.
+4. Role-level documentation duty: update the role README with every
+   variable/task/template change.
+
+## Built-In Quality Gates
+
+- Playbook-level Debian guard (`os_family` check) on all playbooks except
+  `playbooks/hardening/manage-ssh-keys.yml`.
+- `healthcheck` role: post-update container quality gate, optionally failing
+  the play (`healthcheck_fail_on_unhealthy`, default false).
+- `monitoring` role: PSK format/length assertion, role input validation task
+  (`validate.yml`).