discover · compare · collections · requirements · contribute · feed · watching · stats · help with your tokens
fenio/homelab · source
Snapshot 0 · 2026-11-04T12:47:27Z · Shell · MIT
Content fingerprint
e596085d30081d4a17ba56248a3b1b2fb1447119abcb28357e4a8cc328251dd4GitHub description at capture
k8s homelab powered by Talos/omni and flux
README at capture
# Talos-based homelab powered by Flux
<details open>
<summary><h2 style="display: inline-block; margin: 0;">Overview</h2></summary>
This is home to my personal Kubernetes lab cluster running on [Talos Linux](https://www.talos.dev/) and managed by [Sidero Omni](https://omni.siderolabs.com/). [Flux](https://github.com/fluxcd/flux2) watches this Git repository and makes the changes to my cluster based on the manifests in the [cluster](./cluster/) directory. [Renovate](https://github.com/renovatebot/renovate) also watches this Git repository and creates pull requests when it finds updates to Docker images, Helm charts, and other dependencies.
</details>
<details>
<summary><h2 style="display: inline-block; margin: 0;">Hardware</h2></summary>
| Device | Count | OS Disk Size | Data Disk Size | Ram | Operating System | Purpose |
| ----------------------------- | ----- | ------------ | -------------- | ---- | ---------------- | ----------------- |
| Mikrotik RB4011iGS+5HacQ2HnD | 1 | 512MB | | 1GB | RouterOS 7.13 | Router |
| Dell Wyse 5070 | 3 | 16GB | 128GB | 12GB | Talos v1.13.3 | Control plane |
| Odroid H3+ | 1 | 64GB | 8x480GB SSD | 32GB | TrueNAS Scale | NAS / storage |
</details>
<details>
<summary><h2 style="display: inline-block; margin: 0;">Cluster architecture</h2></summary>
- **OS**: Talos Linux v1.13.3 (managed via Sidero Omni)
- **Kubernetes**: v1.36.1
- **CNI**: Cilium v1.19.2 (kube-proxy replacement, Wireguard encryption, BGP control plane)
- **GitOps**: Flux v2.7.3 (installed via flux-operator)
- **Secrets**: SOPS with AGE encryption
- **Storage**: tns-csi (NFS, iSCSI, NVMe-oF) backed by TrueNAS
- **Monitoring**: VictoriaMetrics, Grafana, Coroot, Hubble
### Network layout
| Subnet | Purpose |
| ------------- | --------------- |
| 10.10.20.97 | Gateway |
| 10.10.20.100 | TrueNAS |
| 10.10.20.101 | node1 (enp1s0) |
| 10.10.20.102 | node2 (enp1s0) |
| 10.10.20.103 | node3 (enp3s0) |
| 10.20.0.0/16 | Pod CIDR |
</details>
<details>
<summary><h2 style="display: inline-block; margin: 0;">Components</h2></summary>
### Networking (`net`)
| Component | Description |
| --------- | ----------- |
| [Cilium](https://cilium.io/) | eBPF-based CNI with kube-proxy replacement, Wireguard encryption, and Hubble observability |
| [ingress-nginx](https://kubernetes.github.io/ingress-nginx/) | Ingress controllers for public and private traffic |
| [cert-manager](https://cert-manager.io/) | Automatic TLS certificates via Let's Encrypt |
| [external-dns](https://github.com/kubernetes-sigs/external-dns) | Automatic DNS record management via Cloudflare |
### Storage (`disk`)
| Component | Description |
| --------- | ----------- |
| [tns-csi](https://github.com/fenio/tns-csi) | TrueNAS CSI driver providing NFS, iSCSI, and NVMe-oF storage classes |
| [snapshot-controller](https://github.com/kubernetes-csi/external-snapshotter) | Volume snapshot support |
### Database (`db`)
| Component | Description |
| --------- | ----------- |
| [CloudNative-PG](https://cloudnative-pg.io/) | PostgreSQL operator and cluster |
### Observability (`o11y`)
| Component | Description |
| --------- | ----------- |
| [VictoriaMetrics](https://victoriametrics.com/) | Metrics storage, vmagent, vmalert, and alertmanager |
| [VictoriaLogs](https://docs.victoriametrics.com/victorialogs/) | Log aggregation with Vector collection |
| [Grafana](https://grafana.com/) | Dashboards and visualization |
| [Coroot](https://coroot.com/) | Application performance monitoring with ClickHouse backend |
| [Goldpinger](https://github.com/bloomberg/goldpinger) | Cluster network connectivity monitoring |
| [smartctl-exporter](https://github.com/prometheus-community/smartctl_exporter) | Disk S.M.A.R.T. health metrics |
| [metrics-server](https://github.com/kubernetes-sigs/metrics-server) | Kubernetes resource metrics |
| [Flux Web UI](https://github.com/fluxcd/webui) | Web dashboard for Flux CD |
### Media (`media`)
| Component | Description |
| --------- | ----------- |
| [Kyoo](https://github.com/zoriya/kyoo) | Media server with transcoding and metadata management |
| [qBittorrent](https://www.qbittorrent.org/) | BitTorrent client with VueTorrent WebUI |
| [DMS](https://github.com/anacrolix/dms) | DLNA/UPnP media server |
### Self-hosted (`selfhosted`)
| Component | Description |
| --------- | ----------- |
| [Ollama](https://ollama.com/) | Local LLM runtime with ROCm GPU support |
| [Open WebUI](https://github.com/open-webui/open-webui) | ChatGPT-like interface for Ollama |
| [Perplexica](https://github.com/ItzCrazyKns/Perplexica) | AI-powered search engine |
| [SearXNG](https://github.com/searxng/searxng) | Privacy-focused meta search engine |
### System (`kube-system`, `default`)
| Component | Description |
| --------- | ----------- |
| [AMD GPU device plugin](https://github.com/ROCm/k8s-device-plugin) | ROCm GPU support for AI workloads |
| [Reloader](https://github.com/stakater/Reloader) | Auto-restart pods on ConfigMap/Secret changes |
| [Replicator](https://github.com/mittwald/kubernetes-replicator) | Cross-namespace secret and configmap replication |
| [Homepage](https://gethomepage.dev/) | Personal dashboard |
</details>
## Installing the cluster from scratch
### Prerequisites
A running [Sidero Omni](https://omni.siderolabs.com/) instance is required. Omni manages the Talos lifecycle and provides the Kubernetes API proxy.
Install the following tools on your workstation:
- [omnictl](https://omni.siderolabs.com/docs/reference/cli/) - Sidero Omni CLI
- [helmfile](https://github.com/helmfile/helmfile) - declarative Helm chart management
- [helm](https://helm.sh/) - Kubernetes package manager
- [kubectl](https://kubernetes.io/docs/tasks/tools/) - Kubernetes CLI
- [cilium](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli) - Cilium CLI (for status checks)
- [sops](https://github.com/getsops/sops) - secrets encryption
- [age](https://github.com/FiloSottile/age) - encryption tool used by SOPS
- [task](https://taskfile.dev/) - task runner (optional, for automation)
### Step 1: Prepare Talos ISO
Create an installation media preset in Omni with the required extensions and kernel args, then download an ISO from it:
```sh
omnictl media preset create homelab-iso \
--arch amd64 \
--secureboot \
--extensions amdgpu,amd-ucode,i915,intel-ice-firmware,intel-ucode,iscsi-tools,realtek-firmware \
--extra-kernel-args=-lockdown \
--extra-kernel-args=lockdown=integrity \
--extra-kernel-args=mitigations=off
omnictl media download homelab-iso --output /tmp/talos.iso
```
The three `--extra-kernel-args` flags are intentional — each kernel arg must be passed as a separate flag (the flag is a `stringArray`, not a comma-split list). The leading `-` on `-lockdown` removes Talos's default `lockdown=confidentiality` before adding `lockdown=integrity`.
Boot all three nodes from this ISO. They will register with Omni automatically.
### Step 2: Apply cluster template via Omni
The cluster is defined in [`talos/homelab.yaml`](./talos/homelab.yaml). This template configures:
- 3-node control plane (no dedicated workers, scheduling on control planes enabled)
- No built-in CNI (Cilium installed separately)
- kube-proxy disabled (Cilium replaces it)
- Static IPs for each node
```sh
omnictl cluster template sync -f talos/homelab.yaml
```
Wait for all nodes to join Omni. Nodes will remain `NotReady` until Cilium is installed in step 4.
### Step 3: Get kubeconfig
```sh
omnictl kubeconfig -c homelab
```
Verify the nodes are visible (they will be `NotReady` without CNI):
```sh
kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 NotReady control-plane 1m v1.36.1
node2 NotReady control-plane 1m v1.36.1
node3 NotReady control-plane 1m v1.36.1
```
### Step 4: Install base components with helmfile
[`helmfile`](./talos/helmfile.yaml) installs components in order:
1. **prometheus-operator-crds** - CRDs needed by ServiceMonitors
2. **cilium** - CNI with kube-proxy replacement, Wireguard, Hubble
3. **flux-operator** - Flux lifecycle manager
4. **flux-instance** - Flux deployment pointing to this Git repository
```sh
helmfile --file talos/helmfile.yaml apply --skip-diff-on-install --suppress-diff
```
This will take a few minutes. After completion, nodes should become `Ready`:
```sh
kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 Ready control-plane 5m v1.36.1
node2 Ready control-plane 5m v1.36.1
node3 Ready control-plane 5m v1.36.1
```
Verify Cilium is healthy:
```sh
❯ cilium status -n net
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: OK
\__/ ClusterMesh: disabled
DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3
DaemonSet cilium-envoy Desired: 3, Ready: 3/3, Available: 3/3
Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 3
cilium-envoy Running: 3
cilium-operator Running: 1
clustermesh-apiserver
hubble-relay Running: 1
hubble-ui Running: 1
Cluster Pods: 9/9 managed by Cilium
Helm chart version: 1.19.2
Image versions cilium quay.io/cilium/cilium:v1.19.2@sha256:7bc7e0be845cae0a70241e622cd03c3b169001c9383dd84329c59ca86a8b1341: 3
cilium-envoy quay.io/cilium/cilium-envoy:v1.35.9-1773656288-7b052e66eb2cfc5ac130ce0a5be66202a10d83be@sha256:60031f39669542b21aedf05a3317d14e8d3ea48255790af039b315a1c9637361: 3
cilium-operator quay.io/cilium/operator-generic:v1.19.2@sha256:e363f4f634c2a66a36e01618734ea17e7b541b949b9a5632f9c180ab16de23f0: 1
hubble-relay quay.io/cilium/hubble-relay:v1.19.2@sha256:9987c73bad48c987fd065185535fd15a6717cbe8a8caf7fc7ef0413532cf490e: 1
hubble-ui quay.io/cilium/hubble-ui-backend:v0.13.3@sha256:db1454e45dc39ca41fbf7cad31eec95d99e5b9949c39daaad0fa81ef29d56953: 1
hubble-ui quay.io/cilium/hubble-ui:v0.13.3@sha256:661d5de7050182d495c6497ff0b007a7a1e379648e60830dd68c4d78ae21761d: 1
```
### Step 5: Bootstrap Flux secrets
Flux is now running but needs secrets to decrypt SOPS-encrypted values and access the Git repository.
**Make sure your AGE key file is available:**
```sh
export SOPS_AGE_KEY_FILE=~/AGE/sops-key.txt
```
**Apply the SOPS decryption key:**
```sh
kubectl -n flux-system create secret generic sops-age --from-file=age.agekey=$SOPS_AGE_KEY_FILE
```
**Apply cluster secrets and settings:**
```sh
sops --decrypt cluster/flux/vars/cluster-secrets.sops.yaml | kubectl apply -f -
kubectl apply -f cluster/flux/vars/cluster-settings.yaml
```
### Step 6: Wait for Flux to reconcile
Flux will now start reconciling all resources from this repository. Monitor progress:
```sh
flux get kustomization
```
All kustomizations should eventually show `Ready: True`. The dependency chain ensures components are installed in the correct order (tns-csi -> databases -> applications).
### Automated alternative
If you have [task](https://taskfile.dev/) installed, steps 1-4 can be run with a single command from the repo root:
```sh
task cluster:bootstrap
```
This runs: download-iso -> template-sync -> kubeconfig -> helmfile-apply
Step 5 (Flux secrets) can also be automated:
```sh
export SOPS_AGE_KEY_FILE=~/AGE/sops-key.txt
task flux:bootstrap
```
## AGE / SOPS secrets
Generate a new AGE key (only needed once):
```sh
age-keygen -o sops-key.txt
```
Encrypt/decrypt the cluster secrets file:
```sh
# Decrypt (to edit)
sops cluster/flux/vars/cluster-secrets.sops.yaml
# The .sops.yaml file at the repo root defines encryption rules
```