A multi-environment OpenTofu infrastructure project for managing Incus containers across multiple hosts, providing a complete observability and monitoring stack.
This project manages two Incus environments with separate Terraform state:
iapetus- Control plane / aggregation (IncusOS standalone host)cluster01- Production workloads (3-node IncusOS cluster)
iapetus (Control Plane):
- Grafana - Central dashboards and visualization
- Prometheus - Federated metrics (pulls from cluster01)
- Loki - Aggregated log storage
- step-ca - Internal PKI for TLS certificates
- Cloudflared - Cloudflare Tunnel for Zero Trust access
- CoreDNS - Split-horizon DNS with cross-environment forwarding
- HAProxy - TCP/HTTP load balancer (optional)
- Dex - OIDC identity provider (optional)
- OpenFGA - Fine-grained authorization (optional)
- Atlantis - GitOps controller (optional)
Host metrics come from the native Incus metrics API (incus-metrics module), not a separate node-exporter container.
cluster01 (Production):
- Prometheus - Local metrics scraping
- Alloy - Ships logs to iapetus Loki
- Alertmanager - Alert routing and notifications
- Mosquitto - MQTT broker for IoT messaging
- CoreDNS - Split-horizon DNS
- PostgreSQL - Database backend for Forgejo and Prefect (optional,
enable_postgresql) - Forgejo - Self-hosted Git forge (optional,
enable_forgejo) - Forgejo Runner - Forgejo Actions CI runner (optional,
enable_forgejo_runner) - Prefect - Workflow orchestration server (optional,
enable_prefect) - Ceph - Distributed storage cluster (optional,
enable_ceph) - OVN networking - Software-defined networking with load balancers (optional,
network_backend = "ovn")
Host metrics likewise come from the native Incus metrics API - each cluster member exposes its own metrics endpoint, no separate node-exporter containers.
All services run in Incus system containers (Debian Trixie + cloud-init) with persistent storage, network isolation, and automatic configuration management.
atlas/
├── modules/ # Shared Terraform modules (used by all environments)
│ ├── alertmanager/ # Alert routing and notifications
│ ├── alloy/ # Log collection and shipping (replaces Promtail)
│ ├── atlantis/ # GitOps controller
│ ├── base-infrastructure/ # Networks and base profiles
│ ├── ceph/ # Distributed storage cluster
│ ├── cloudflared/ # Cloudflare Tunnel client
│ ├── coredns/ # Split-horizon DNS
│ ├── dex/ # OIDC identity provider
│ ├── forgejo/ # Self-hosted Git forge
│ ├── forgejo-runner/ # Forgejo Actions CI runner
│ ├── grafana/ # Visualization platform
│ ├── haproxy/ # TCP/HTTP load balancer
│ ├── incus-loki/ # Native Incus → Loki logging
│ ├── incus-metrics/ # Incus metrics certificates
│ ├── incus-network-zone/ # Incus DNS zone configuration
│ ├── incus-vm/ # Virtual machine support
│ ├── loki/ # Log aggregation
│ ├── mosquitto/ # MQTT broker
│ ├── network-acl/ # Incus network ACLs
│ ├── node-exporter/ # Host metrics (unused - see incus-metrics)
│ ├── openfga/ # Fine-grained authorization
│ ├── ovn-central/ # OVN databases
│ ├── ovn-config/ # Incus OVN settings
│ ├── ovn-load-balancer/ # OVN load balancers
│ ├── postgresql/ # Database backend
│ ├── prefect/ # Workflow orchestration server
│ ├── prometheus/ # Metrics collection
│ └── step-ca/ # Internal CA
│
├── environments/
│ ├── iapetus/ # Control plane environment
│ │ ├── main.tf # Module instantiations
│ │ ├── variables.tf
│ │ ├── outputs.tf
│ │ ├── locals.tf
│ │ ├── templates/ # Configuration templates
│ │ ├── terraform.tfvars # Secrets (gitignored)
│ │ └── bootstrap/ # Remote state setup
│ │
│ └── cluster01/ # Production cluster environment
│ ├── main.tf
│ ├── variables.tf
│ ├── outputs.tf
│ ├── locals.tf
│ └── terraform.tfvars # Secrets (gitignored)
│
├── docker/ # Custom Docker images (Atlantis only)
│ └── atlantis/
│
├── .github/workflows/ # CI/CD workflows
│ ├── ci.yml # Validates both environments
│ └── release.yml # Build and publish images
│
├── Makefile # ENV=iapetus make plan (default: iapetus)
├── CLAUDE.md # Detailed architecture documentation
├── CONTRIBUTING.md # Contribution guidelines
├── BACKUP.md # Backup and disaster recovery
├── GITOPS.md # GitOps workflow with Atlantis
├── SECURITY.md # Security documentation
├── TROUBLESHOOTING.md # Troubleshooting guide
└── README.md # This file
- Incus installed and running
- OpenTofu >= 1.6 (or Terraform >= 1.6)
- Cloudflare API token (for Cloudflare Tunnel access)
- GitHub account (images published to ghcr.io)
-
Clone the repository:
git clone https://github.com/accuser-dev/atlas.git cd atlas -
Bootstrap remote state (first time only):
make bootstrap
-
Create terraform.tfvars:
cd environments/iapetus cp terraform.tfvars.example terraform.tfvars # Edit with your values
-
Deploy iapetus:
make init make plan make apply
-
View outputs:
cd environments/iapetus tofu output
-
Configure Incus remote (from iapetus or management host):
incus remote add cluster01 https://<cluster-ip>:8443 export INCUS_REMOTE=cluster01
-
Bootstrap cluster state:
ENV=cluster01 make bootstrap
-
Create terraform.tfvars:
cd environments/cluster01 cp terraform.tfvars.example terraform.tfvars # Edit with cluster-specific values including loki_push_url
-
Deploy cluster01:
ENV=cluster01 make init ENV=cluster01 make plan ENV=cluster01 make apply
Every deployed service except Atlantis uses Debian Trixie (images:debian/trixie/cloud) system containers with cloud-init and systemd - no custom images needed. Most modules follow a hybrid Terraform + Ansible pattern: cloud-init only bootstraps the container (Python3, so Ansible can connect), and an Ansible role in ansible/roles/ installs and configures the service:
- Grafana, Prometheus, Loki, Alertmanager
- step-ca, Alloy, Mosquitto, CoreDNS, Dex, OpenFGA
- PostgreSQL, Forgejo, Forgejo Runner
A few services still configure themselves entirely via cloud-init at first boot, with no Ansible role:
- Cloudflared, HAProxy, Ceph (all daemon types)
Prefect's cloud-init also targets the Ansible pattern, but the ansible/roles/prefect role and playbook haven't landed yet - see ansible/playbooks/site.yml for the services currently covered.
See .claude/architecture.md for details on the hybrid provisioning pattern.
Only Atlantis uses a custom Docker image:
- Atlantis:
ghcr.io/accuser-dev/atlas/atlantis:latest
Built on every push to main and published to GitHub Container Registry.
Build Atlantis image locally for testing:
make build-atlantisAll commands support the ENV variable to target specific environments:
# iapetus (default)
make init
make plan
make apply
make destroy
# cluster01 environment
ENV=cluster01 make init
ENV=cluster01 make plan
ENV=cluster01 make apply
ENV=cluster01 make destroy# Deploy iapetus
make deploy
# Deploy cluster01
ENV=cluster01 make deployEach environment supports multiple network modes:
Bridge Networks (default):
- production (10.10.0.0/24) - For external services (Mosquitto, CoreDNS)
- management (10.20.0.0/24) - For internal services (monitoring stack, PKI)
- gitops (10.30.0.0/24) - For GitOps automation (optional, iapetus only)
OVN Networks (optional):
- Provides native load balancers with LAN-routable VIPs
- Supports external access without proxy devices
- Configured via
network_backend = "ovn"in terraform.tfvars
Configure IP addresses in environments/*/terraform.tfvars.
The cluster01 environment connects to iapetus for:
- Log shipping: Alloy → iapetus Loki (
loki_push_urlvariable) - Prometheus federation: iapetus pulls metrics from cluster01
- TLS certificates: step-ca on iapetus issues certs for cluster services
- DNS forwarding: Cross-environment DNS resolution via CoreDNS
See CLAUDE.md for detailed instructions on adding new services.
- Multi-Environment - Separate state for iapetus (control plane) and cluster01 (production)
- Declarative Infrastructure - Everything defined in OpenTofu
- Modular Design - Shared modules across environments
- System Containers - Debian Trixie with cloud-init/systemd (no custom images for most services)
- Persistent Storage - Data survives container restarts with optional automated snapshots
- Zero Trust Access - Cloudflare Tunnel for external access
- Network Isolation - Separate networks for different service types
- Native Incus Integration - Container metrics and logging via Incus API
- OVN Support - Optional software-defined networking with load balancers
┌─────────────────────────────────────────────────────────────────────┐
│ iapetus (IncusOS) │
│ Control Plane / Aggregation │
├─────────────────────────────────────────────────────────────────────┤
│ - Atlantis (GitOps) → manages iapetus + cluster via remote Incus │
│ - Grafana (central dashboards) │
│ - Prometheus (federated, pulls from cluster01) │
│ - Loki (aggregated logs via Alloy on cluster01) │
│ - Cloudflared (tunnel ingress) │
│ - step-ca (central CA for all environments) │
│ - CoreDNS, HAProxy, Dex, OpenFGA (optional) │
└──────────────────────────────┬──────────────────────────────────────┘
│ incus remote + prometheus federation
▼
┌─────────────────────────────────────────────────────────────────────┐
│ cluster01 (IncusOS 3-node cluster) │
│ Production Workloads │
├─────────────────────────────────────────────────────────────────────┤
│ - Prometheus (local scraping, federated by iapetus) │
│ - Alloy → ships logs to iapetus Loki │
│ - Mosquitto, CoreDNS, Alertmanager │
│ - PostgreSQL, Forgejo, Forgejo Runner, Prefect (optional) │
│ - Ceph, OVN networking (optional) │
└─────────────────────────────────────────────────────────────────────┘
iapetus Services (control plane):
- Grafana:
https://grafana.yourdomain.com(via Cloudflare Tunnel) - Prometheus:
http://prometheus01.incus:9090 - Loki:
http://loki01.incus:3100 - step-ca:
https://step-ca01.incus:9000
cluster01 Services (production):
- Prometheus:
http://prometheus01.incus:9090 - Alertmanager:
http://alertmanager01.incus:9093 - Mosquitto: Host ports 1883 (MQTT), 8883 (MQTTS)
- CoreDNS: Host port 53 (DNS)
- PostgreSQL:
postgresql01.incus:5432(optional) - Forgejo:
http://forgejo01.incus:3000(optional) - Prefect:
http://prefect01.incus:4200(optional)
iapetus:
grafana01-data(10GB) - Dashboards and settingsloki01-data(50GB) - Log storageprometheus01-data(100GB) - Metrics storagestep-ca01-data(1GB) - CA keys and databaseatlantis01-data(10GB) - GitOps state (optional)dex01-data(1GB) - OIDC state (optional)openfga01-data(1GB) - Authorization store (optional)
cluster01:
prometheus01-data(100GB) - Metrics storagealertmanager01-data(1GB) - Silences and statemosquitto01-data(5GB) - MQTT retained messagespostgresql01-data(20GB) - Database storage (optional,enable_postgresql)forgejo01-data(50GB) - Git repositories and forge data (optional,enable_forgejo)forgejo-runner01-data(20GB) - CI job workspace (optional,enable_forgejo_runner)prefect01-data(5GB) - Workflow orchestration state (optional,enable_prefect)
All volumes support optional automated snapshot scheduling. See BACKUP.md for details.
The project uses separate workflows for validation and releases:
Triggers: Pull requests and pushes to feature branches
What it does:
- OpenTofu format and validation for both environments
- Atlantis Docker image build (without publish)
- Security scanning
Triggers: Push to main branch
What it does:
- Builds Atlantis Docker image
- Publishes to GitHub Container Registry
- Tags with
latestand commit SHA
All targets support ENV=iapetus (default) or ENV=cluster01:
make help # Show all available commands
# Bootstrap (first time setup per environment)
make bootstrap # Set up remote state storage
# OpenTofu operations
make init # Initialize OpenTofu with remote backend
make plan # Plan infrastructure changes
make apply # Apply infrastructure changes
make destroy # Destroy infrastructure
make deploy # Apply OpenTofu
make format # Format OpenTofu files
# Docker operations (Atlantis only)
make build-atlantis # Build Atlantis image locally
# Cleanup
make clean # Clean all build artifacts
make clean-docker # Clean Docker cache
make clean-tofu # Clean OpenTofu cache
make clean-images # Remove Atlas images from Incus cache
# Backup operations
make backup-snapshot # Create snapshots of all storage volumes
make backup-export # Export all volumes to tarballs
make backup-list # List all volume snapshots-
modules/- Shared Terraform modules- Used by both environments
- Each service has its own directory
-
environments/- Environment-specific configurationiapetus/- Control planecluster01/- Production workloads- Each has
main.tf,variables.tf,terraform.tfvars(gitignored)
-
docker/- Custom Docker images (Atlantis only)
Check container logs:
incus info <container-name>
incus console <container-name>incus exec <container-name> -- systemctl status <service-name>View current state:
cd environments/iapetus # or environments/cluster01
tofu showTest internal DNS:
incus exec grafana01 -- ping prometheus01.incusEnsure the cluster is accessible from iapetus:
# On iapetus or management host
incus remote list
incus list cluster01:See TROUBLESHOOTING.md for comprehensive troubleshooting guidance.
See CONTRIBUTING.md for detailed contribution guidelines.
Quick start:
- Create a feature branch from
main - Make changes and test with
make plan - Format code:
make format - Submit pull request to
main - GitHub Actions will validate and build images
[Your license here]
For detailed architecture, design patterns, and development guidance, see:
- CLAUDE.md - Complete architecture documentation
- CONTRIBUTING.md - Contribution guidelines and GitHub Flow
- BACKUP.md - Backup and disaster recovery procedures
- GITOPS.md - GitOps workflow with Atlantis
- SECURITY.md - Security architecture and controls
- TROUBLESHOOTING.md - Common issues and solutions
- modules/*/README.md - OpenTofu module documentation