A StatefulSet's volumeClaimTemplates are immutable, and the API server
injects creationTimestamp, volumeMode and a status block that are not in
the manifest. ArgoCD diffed those and reported OutOfSync permanently,
since no sync could ever resolve them.
infisical-postgres and infisical-valkey are the only StatefulSets here
using volumeClaimTemplates, which is why this app alone was affected.
gitea-runner-pvc referenced volumeName gitea-runner-data, but neither that
PV nor a Longhorn volume of that name exists, so it sat Pending since May.
ArgoCD treats a Pending PVC as Progressing and rolled that up to the whole
app, which is why gitea never reported Healthy.
Nothing mounted it. The runner uses the PVC the gitea/actions chart
generates from its own volumeClaimTemplate.
The kubelet probes the pod IP, which Django rejects with a 400 because
ALLOWED_HOSTS is netbox.jsme.be. Setting the Host header keeps
ALLOWED_HOSTS tight instead of widening it to '*'.
Five components under applications/netbox: the web pod, an rqworker,
a daily housekeeping CronJob, postgres 18 and two valkey instances.
The task queue runs appendonly on its own PVC so queued jobs survive a
restart, while the cache instance is disposable.
Worker and cronjob override args rather than command, which replaces
CMD while keeping tini as the entrypoint, so only the web pod runs
migrations. Media, reports and scripts share one RWX PVC via subPaths
because both the web pod and the worker mount them.
Exposed on the internal gateway only.
The console will not work from outside without this. WebsocketController hands
the browser wss://<node fqdn>:443/api/servers/<uuid>/ws built from
Node::getConnectionAddress(), so the browser talks to wings directly and never
through the panel. Same for file upload and download, which FileUploadController
builds from the same address. Everything else (login, server list, file
browsing, power actions, sending console commands) already worked publicly,
because those go through the panel API.
Adds public listeners for both nodes reusing the existing wings-mc-tls and
wings-ark-tls certificates, and a gateway-public parentRef on each route.
Extends the wings ClientTrafficPolicy to the two new listeners. Without it the
public listeners would negotiate h2 and long-lived consoles would drop: wings
only speaks HTTP/1.1, and the idle timeout needs to be 3600s rather than the
default.
CrowdSec ext_authz already covers every listener on gateway-public, so both
nodes are behind the WAF and community blocklist from the moment they are live.
Wings itself is JWT-gated and returns 401 unauthenticated.
Adds a pelican listener to gateway-public and a gateway-public parentRef to
pelican-route. Reuses the existing pelican-tls certificate, which both
listeners reference.
Wings deliberately stays internal-only. That is a real functional limit, not
an oversight: the browser talks to wings directly for the console websocket
and for file upload/download, using the address from Node::getConnectionAddress.
From outside the LAN those calls will fail. Panel login, server list, file
browsing and editing, and power actions all go through the panel API and keep
working, because the panel reaches wings server-side over the internal gateway.
CrowdSec ext_authz already covers every listener on gateway-public, so the
panel is behind the WAF and community blocklist from the moment it is live.
Prerequisites completed first: TRUSTED_PROXIES=10.42.0.0/16 and
APP_2FA_REQUIRED=1 set in Infisical and verified live in the panel config.
pelican.jsme.be has no public DNS record yet, so this makes the panel reachable
through 10.8.11.105 but not from the internet. The Azure DNS CNAME in the
OpenTofu repo is the actual cutover.
My previous commit staged the whole kustomization file and swept in the
in-flight netbox lines, which I had said I would keep out. Those reference
infra/envoy-gateway/netbox/, which is untracked, so ArgoCD could not render
the app at all:
kustomize build failed: accumulating resources from
'netbox/netbox-certificate.yaml': no such file or directory
That left envoy-gateway-config unable to load target state, so the orphaned
listener cleanup never applied. The running gateways were unaffected.
Commenting the two entries keeps the work visible in place and makes it a
two line uncomment once infra/envoy-gateway/netbox/ is committed.
Ten Gateway listeners had zero attached routes. Eight of them are in this repo
and are removed here by commenting them out of the kustomization, which prunes
the listener and its cert-manager Certificate.
Deprecated, workloads already live in deprecated/:
vaultwarden (vault.jsme.be), affine (affine.jsme.be)
Superseded by minio-aistor:
resume-minio (resume-minio.jsme.be), resume-minioweb (resume-minioweb.jsme.be)
Written but never deployed, no namespace in the cluster:
n8n (n8n.jsme.be), wikijs (wiki.jsme.be), ollama (ollama.jsme.be),
openwebui (forge.jsme.be)
wiki.jsme.be was one of these on the PUBLIC gateway: a listener with a valid
certificate, no backend, and an allowedRoutes selector that would attach the
first HTTPRoute appearing in a matching namespace. Deploying Wiki.js later
expecting it to be internal would have published it.
Commented rather than deleted so redeploying an app is a two line uncomment.
The certificate and the gateway patch must be uncommented together.
Verified with kubectl kustomize: 45 listeners render, down from 53, with all
eight orphan hostnames gone and every live service still present.
Wires crowdsec/gateway-public-securitypolicy.yaml into the kustomization now
that LAPI, AppSec and the bouncer are all healthy and the envoy bouncer is
registered with LAPI.
Covers every listener on gateway-public rather than just one app, since EG
v1.3.2 rejects targetRefs.sectionName on SecurityPolicy. gateway-internal is a
separate Gateway and is untouched, so LAN access is unaffected. failOpen is
true, so a bouncer outage lets traffic through instead of denying it.
Chart defaults set requests == limits == 500m for both. The node is at 95% of
allocatable CPU in requests while actually using about 10%, so lapi sat Pending
with Insufficient cpu. Requests dropped to 50m/100m with the limits left
generous, since appsec runs inline on every public request.
Detection is CAPI community blocklists plus AppSec inline WAF, not Envoy
access-log parsing. Enforcement is gRPC ext_authz from Envoy Gateway.
Changes from the draft that was held:
Agent disabled rather than given an empty acquisition list. Chart 0.24.0
refuses to render the DaemonSet with acquisition: [] ("No acquisition or
additionalAcquisition configured"), and with no log tailing the agent has
nothing to do. The WAF collections moved to appsec.env, where they belong,
and base-http-scenarios was dropped since it only feeds log parsing. AppSec
registers with LAPI on its own, so it works without the agent.
Bouncer values updated for chart 0.8.0 (the app already pinned 0.8.0 while the
values were written against 0.6.3). Added waf.failOpen: true, whose chart
default is false and would deny every request if AppSec were unreachable.
SecurityPolicy now targets the whole gateway-public rather than the it-tools
listener. EG v1.3.2 rejects targetRefs.sectionName on SecurityPolicy, and
targeting the HTTPRoute instead would gate LAN traffic too since public routes
also parent gateway-internal. Blanket coverage of the public gateway is what we
want anyway, and failOpen keeps a bouncer outage from taking public apps down.
envoyproxy-public.yaml is kept as documentation but not applied. EG v1.3.2
already defaults envoyService.externalTrafficPolicy to Local, verified live on
all three gateway LB services, so the real client IP already reaches Envoy.
Not wired into the envoy-gateway kustomization yet. That lands once the
bouncer is up and healthy.
The ghcr.io/pelican-dev/panel package is no longer anonymously pullable
(token endpoint returns DENIED for every tag, package page 404s). Upstream
renamed the org to "pelican" in beta38 and compose.yml now points at
ghcr.io/pelican/panel. Verified a real pull of v1.0.0-beta38 from the new
location succeeds.
Pinning the tag also replaces :latest, which combined with IfNotPresent meant
the running version depended on whatever the node happened to have cached.
beta36 was clean against all six published advisories, but four of them landed
in the last five months, so a deliberate patch path matters here.
beta37/38 carry no breaking changes or manual migration steps. beta37 fixes
Passkeys origin validation, which is wanted before enrolling 2FA.
Bumps every outdated image and chart except databases, which are
deliberately left on their current versions.
Applications:
authentik 2026.5.2 -> 2026.8.0 (server and worker)
immich v2.7.5 -> v3.1.0
gitea 1.25 -> 1.27.2
gotify 2.9.1 -> 3.0.0
uptime-kuma 2.2.1 -> 2.5.3
zipline 4.5.3 -> 4.7.0
outline 1.8.1 -> 1.9.2
reactive-resume v5.0 -> v5.2.8
netbootxyz nbxyz18 -> nbxyz24
bentopdf v2.8.2 -> v2.8.7
jellyfin 10.11.9 -> 10.11.11
gitea runner init busybox 1.37.0 -> 1.38.0
Infra:
kube-vip v0.9.1 -> v1.2.3
victoria-metrics-k8s-stack 0.77.0 -> 0.91.2
intel-device-plugins v0.35.0 -> v0.36.0
crowdsec-envoy-bouncer 0.6.3 -> 0.8.0
Immich v3 drops pgvecto.rs support. Verified the live database already
runs vchord 0.4.3 and pgvector 0.8.1 with no pgvecto.rs extension, both
inside the ranges v3 accepts, so no database change is required.
The victoria-metrics chart renamed defaultRules.create to
defaultRules.enabled at both the top level and per group. Migrated those
keys so the etcd, kubeScheduler, kubernetesSystemControllerManager and
kubernetesSystemScheduler exclusions keep applying. Without the rename
those groups revert to enabled and alert on control-plane components
that k3s runs embedded.
That chart also moved default rules and dashboards to a runtime sync job
instead of templating them, so ArgoCD will prune the VMRules and
dashboard ConfigMaps it currently owns and the job will recreate them.
kube-vip is not managed by ArgoCD. The manifest change is inert until
applied by hand.
Both ran as BestEffort QoS, making them first in line for eviction under node memory pressure. Requests also give the scheduler real numbers to place them with.
Longhorn auto-creates this class when missing and it is not backed by the longhorn-storageclass ConfigMap, so it had no source of truth. Set to Retain.
Flat CR narrowed to recursive: false; it still serves the SMB mount credentials and the Actions runner token, which the runner Helm chart reads via existingSecret.
Folds the pelican-config ConfigMap into Infisical so the container has a single envFrom. Pins imagePullPolicy to IfNotPresent because ghcr.io/pelican-dev/panel no longer allows anonymous pulls.
All 31 env entries, secrets and literals alike, now live in Infisical under /outline/outline and /outline/postgres. Manifest rationale carried across as Infisical secret comments.
Both ran as BestEffort QoS, making them first in line for eviction
under node memory pressure. Requests also give the scheduler real
numbers to place them with.
Jeffrey