Troubleshooting

This page covers troubleshooting Pangolin Kubernetes deployments across Helm, Kustomize, Argo CD, and Flux workflows. Start with the core checks, then use the section that matches the symptom.

Core diagnostics

Set the namespace and release name used by your installation:

export PANGOLIN_NAMESPACE=pangolin
export PANGOLIN_RELEASE=pangolin

Helm diagnostics

Check the release:

helm status "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE"
helm history "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE"
helm get values "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE" --all

Render the chart locally with your values file:

helm repo update fossorial

helm template "$PANGOLIN_RELEASE" fossorial/pangolin \
  --namespace "$PANGOLIN_NAMESPACE" \
  --values values-pangolin.yaml

Preview an upgrade:

helm upgrade "$PANGOLIN_RELEASE" fossorial/pangolin \
  --namespace "$PANGOLIN_NAMESPACE" \
  --values values-pangolin.yaml \
  --dry-run

helm lint charts/pangolin is only useful when you are working inside the Helm chart repository. For normal installs, use helm template and helm upgrade --dry-run.

Kubernetes diagnostics

Check workloads and events:

kubectl get pods --namespace "$PANGOLIN_NAMESPACE"
kubectl get deploy,statefulset,job,cronjob --namespace "$PANGOLIN_NAMESPACE"
kubectl get events --namespace "$PANGOLIN_NAMESPACE" --sort-by=.lastTimestamp

Inspect a pod:

kubectl describe pod <pod-name> --namespace "$PANGOLIN_NAMESPACE"
kubectl logs <pod-name> --namespace "$PANGOLIN_NAMESPACE" --all-containers --tail=200

Check services, PVCs, and policies:

kubectl get svc,pvc,secret,configmap --namespace "$PANGOLIN_NAMESPACE"
kubectl get networkpolicy --namespace "$PANGOLIN_NAMESPACE"

Traefik diagnostics

In controller mode, the chart uses Traefik CRDs such as IngressRoute. Check whether Traefik CRDs are installed:

kubectl get crd | grep traefik

Check rendered or applied Traefik resources:

kubectl get ingressroute --namespace "$PANGOLIN_NAMESPACE"
kubectl describe ingressroute <name> --namespace "$PANGOLIN_NAMESPACE"

Depending on your Traefik setup, also check:

kubectl get middleware,tlsoption,traefikservice --namespace "$PANGOLIN_NAMESPACE"

kubectl get ingress is only useful if your selected deployment mode renders standard Kubernetes Ingress resources. In controller mode, use IngressRoute.

Database diagnostics

If you use CloudNativePG, first check that the CRD exists:

kubectl get crd | grep postgresql.cnpg.io

Then check CNPG resources:

kubectl get cluster --namespace "$PANGOLIN_NAMESPACE"
kubectl describe cluster <cluster-name> --namespace "$PANGOLIN_NAMESPACE"
kubectl get pods --namespace "$PANGOLIN_NAMESPACE" | grep -E 'pangolin-db|postgres'
kubectl get secret --namespace "$PANGOLIN_NAMESPACE" | grep -E 'pangolin-db|postgres'

If you use external PostgreSQL, verify the connection Secret:

kubectl get secret <connection-secret-name> --namespace "$PANGOLIN_NAMESPACE"
kubectl describe secret <connection-secret-name> --namespace "$PANGOLIN_NAMESPACE"

Do not decode and paste database credentials into logs, screenshots, or issue reports.

Common issues and solutions

Gerbil fails during the first install

Symptoms

Gerbil pod crashes during a fresh install.
Logs mention missing setup data, missing exit node, or tunnel configuration not being ready.
Pangolin itself is not initialized yet.

CauseOn first install, Gerbil may start before Pangolin has completed the initial setup. The chart supports gerbil.startupMode for this case.ResolutionUse delayed startup for the first install:

gerbil:
  startupMode: delayed

Install or upgrade with the values file:

helm upgrade --install "$PANGOLIN_RELEASE" fossorial/pangolin \
  --namespace "$PANGOLIN_NAMESPACE" \
  --values values-pangolin.yaml

After Pangolin setup is complete, switch Gerbil to normal startup:

helm upgrade "$PANGOLIN_RELEASE" fossorial/pangolin \
  --namespace "$PANGOLIN_NAMESPACE" \
  --reuse-values \
  --set gerbil.startupMode=normal

Check Gerbil resources:

kubectl get pods,svc,pvc --namespace "$PANGOLIN_NAMESPACE" \
  -l app.kubernetes.io/name=gerbil

Gerbil is blocked by Pod Security Admission

Symptoms

Gerbil pod does not start.
Events mention Pod Security Admission.
Events mention forbidden capabilities.
Logs or events mention NET_ADMIN.

CauseGerbil requires the NET_ADMIN Linux capability for WireGuard interface management. A namespace using a restricted Pod Security profile can block this.ResolutionCheck namespace labels:

kubectl get namespace "$PANGOLIN_NAMESPACE" --show-labels

For a namespace running Gerbil, use a policy profile that allows the required capability. Example:

kubectl label namespace "$PANGOLIN_NAMESPACE" \
  pod-security.kubernetes.io/enforce=privileged \
  pod-security.kubernetes.io/warn=baseline \
  pod-security.kubernetes.io/audit=restricted \
  --overwrite

Then restart the affected pods:

kubectl rollout restart deploy --namespace "$PANGOLIN_NAMESPACE"

Do not use a restricted Pod Security profile for Gerbil unless you have validated the selected chart mode and security context. Removing NET_ADMIN breaks WireGuard management.

Dashboard is not reachable through IngressRoute

Symptoms

The dashboard URL does not load.
Browser shows timeout, bad gateway, 404, or TLS error.
API path /api/v1 fails while the dashboard path works, or the reverse.

Common causes

DNS points to the wrong load balancer or ingress endpoint.
Traefik CRDs are missing.
Traefik controller is not watching the namespace or selector labels.
IngressRoute host does not match the dashboard URL.
API route was changed and no longer matches PathPrefix(/api/v1).
TLS resolver or TLS Secret is misconfigured.

ChecksCheck DNS:

nslookup pangolin.example.com

Check Traefik CRDs:

kubectl get crd | grep traefik

Check IngressRoute resources:

kubectl get ingressroute --namespace "$PANGOLIN_NAMESPACE"
kubectl describe ingressroute <name> --namespace "$PANGOLIN_NAMESPACE"

Check the rendered values:

helm get values "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE" --all | grep -A30 ingressRoute

Check Traefik logs. Adjust the namespace and label selector to your Traefik installation:

kubectl logs --namespace traefik -l app.kubernetes.io/name=traefik --tail=100

Temporary local check for the dashboard/UI service port:

kubectl port-forward --namespace "$PANGOLIN_NAMESPACE" svc/pangolin 8080:3002

Then open:

http://localhost:8080

The dashboard/UI port is 3002. The API/external port is 3000. Port-forward 3002 when checking the dashboard locally.

TLS certResolver and secretName conflict

Symptoms

IngressRoute is created but TLS does not work.
Traefik logs mention TLS configuration problems.
Certificate is not issued or the TLS Secret is not found.

CauseThe dashboard IngressRoute TLS configuration should use either a Traefik certificate resolver or an existing TLS Secret.ResolutionUse Traefik ACME certificate resolver:

pangolin:
  ingressRoute:
    dashboard:
      tls:
        enabled: true
        certResolver: letsencrypt
        secretName: ""

Or use an existing TLS Secret:

pangolin:
  ingressRoute:
    dashboard:
      tls:
        enabled: true
        certResolver: ""
        secretName: pangolin-dashboard-tls

Verify the Secret if using secretName:

kubectl get secret pangolin-dashboard-tls --namespace "$PANGOLIN_NAMESPACE"

certResolver is a Traefik ACME resolver setting. It is not a cert-manager issuer reference.

Newt cannot reach Gerbil WireGuard ports

Symptoms

Newt shows repeated connection or tunnel errors.
Tunnel traffic does not pass.
WireGuard UDP ports are unreachable from the Newt location.

Common causes

pangolin.config.gerbil.base_endpoint points to the wrong host.
Gerbil Service is not exposed as expected.
External firewall blocks UDP traffic.
NetworkPolicy blocks the required traffic.
pangolin.config.gerbil.start_port and gerbil.ports.wg1 are not aligned.
pangolin.config.gerbil.clients_start_port and gerbil.ports.wg2 are not aligned.

ChecksCheck Gerbil Service:

kubectl get svc --namespace "$PANGOLIN_NAMESPACE" \
  -l app.kubernetes.io/name=gerbil

kubectl describe svc <gerbil-service-name> --namespace "$PANGOLIN_NAMESPACE"

Check Gerbil values:

helm get values "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE" --all | grep -A30 gerbil

Check NetworkPolicies:

kubectl get networkpolicy --namespace "$PANGOLIN_NAMESPACE"
kubectl describe networkpolicy --namespace "$PANGOLIN_NAMESPACE"

Verify external firewall rules for the configured UDP ports.

Gerbil is behind reverse proxy or UDP gateway and tunnels fail

Symptoms

Newt peers do not establish stable handshakes.
Tunnel traffic drops even though Gerbil pods are healthy.
Logs show connection resets or malformed upstream traffic.

CauseProxy protocol handling is inconsistent between the upstream hop and Gerbil.

If Gerbil is exposed through a reverse proxy or UDP gateway, keep proxy protocol settings aligned end-to-end. Do not enable proxy protocol on the upstream hop unless Gerbil is configured to accept it.

ChecksCheck endpoint and port alignment:

helm get values "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE" --all | grep -A40 gerbil

Check Gerbil logs:

kubectl logs --namespace "$PANGOLIN_NAMESPACE" \
  -l app.kubernetes.io/name=gerbil \
  --tail=200

Check Service exposure:

kubectl get svc --namespace "$PANGOLIN_NAMESPACE" \
  -l app.kubernetes.io/name=gerbil -o wide

External database mode missing or invalid Secret

Symptoms

Pangolin pod crashes.
Logs mention database connection errors.
Events mention missing Secret or missing Secret key.

Causedatabase.mode=external needs a valid database connection Secret unless the chart is configured to generate one from values.ResolutionCreate a connection Secret:

kubectl create secret generic pangolin-db-connection \
  --namespace "$PANGOLIN_NAMESPACE" \
  --from-literal=connectionString='postgresql://pangolin:password@postgres.example.com:5432/pangolin?sslmode=require'

Reference it in values:

database:
  mode: external
  connection:
    existingSecretName: pangolin-db-connection
    existingSecretKey: connectionString

Check the Secret:

kubectl describe secret pangolin-db-connection --namespace "$PANGOLIN_NAMESPACE"

Do not put database passwords directly in values files for production. Use an existing Secret or your normal secret-management workflow.

CloudNativePG cluster does not provision

Symptoms

CNPG Cluster resource is missing.
CNPG pods do not start.
Pangolin cannot connect to the generated CNPG database.
Secret such as pangolin-db-app is missing.

Common causes

CloudNativePG CRDs/operator are not installed.
cnpg-cluster.enabled is false when you expected the chart to create a cluster.
cnpg-operator.enabled is false and no operator exists.
database.cloudnativepg.cluster.name does not match the CNPG cluster name.
StorageClass or PVC provisioning fails.

ChecksCheck CRDs:

kubectl get crd | grep postgresql.cnpg.io

Check CNPG operator pods:

kubectl get pods --all-namespaces | grep -i cnpg

Check CNPG Cluster:

kubectl get cluster --namespace "$PANGOLIN_NAMESPACE"
kubectl describe cluster pangolin-db --namespace "$PANGOLIN_NAMESPACE"

Check PVCs and Secrets:

kubectl get pvc --namespace "$PANGOLIN_NAMESPACE"
kubectl get secret --namespace "$PANGOLIN_NAMESPACE" | grep pangolin-db

Expected naming when using the default example:

database:
  cloudnativepg:
    cluster:
      name: pangolin-db

cnpg-cluster:
  enabled: true
  fullnameOverride: pangolin-db

NetworkPolicy blocks DNS, database, controller, or tunnel traffic

Symptoms

DNS lookups fail.
Pangolin cannot connect to the database.
Controller cannot reach the Kubernetes API.
Gerbil or Newt traffic does not work.
External services such as SMTP, OIDC, or webhooks time out.

CauseThe chart can render NetworkPolicies. If your CNI enforces them, missing egress or ingress rules can break required paths.Checks

kubectl get networkpolicy --namespace "$PANGOLIN_NAMESPACE"
kubectl describe networkpolicy --namespace "$PANGOLIN_NAMESPACE"

Check whether DNS is allowed:

networkPolicy:
  dns:
    enabled: true

Check database egress:

networkPolicy:
  database:
    enabled: true
    port: 5432

Check controller API access:

networkPolicy:
  controller:
    egress:
      enabled: true
      kubernetesApi:
        enabled: true
        port: 443

For external integrations, add scoped egress rules for the required services instead of allowing broad egress.For a temporary isolation test, disable NetworkPolicy and re-apply:

networkPolicy:
  enabled: false

If this fixes the issue, re-enable policies and add the missing rules.

Pangolin pod is CrashLoopBackOff or Pending

Symptoms

Pangolin pod restarts repeatedly.
Pod stays Pending.
Readiness never becomes true.

ChecksFind the pod:

kubectl get pods --namespace "$PANGOLIN_NAMESPACE" \
  -l app.kubernetes.io/name=pangolin

Inspect it:

kubectl describe pod <pod-name> --namespace "$PANGOLIN_NAMESPACE"
kubectl logs <pod-name> --namespace "$PANGOLIN_NAMESPACE" --tail=200
kubectl logs <pod-name> --namespace "$PANGOLIN_NAMESPACE" --previous --tail=200

Check PVCs:

kubectl get pvc --namespace "$PANGOLIN_NAMESPACE"
kubectl describe pvc <pvc-name> --namespace "$PANGOLIN_NAMESPACE"

Common causes:

Status	Common causes
`CrashLoopBackOff`	Database connection issue, missing Secret, invalid config, startup dependency not ready
`Pending`	PVC not bound, insufficient resources, node selector/affinity mismatch, Pod Security policy rejection
`ImagePullBackOff`	Wrong image override, registry access issue, missing imagePullSecret

Do not assume tools such as psql, curl, or dig are available inside the Pangolin container. Use logs, Events, or a temporary debug pod when needed.

Run a temporary debug pod for network tests:

kubectl run net-debug \
  --namespace "$PANGOLIN_NAMESPACE" \
  --rm -it \
  --image=curlimages/curl:latest \
  --restart=Never \
  -- sh

Traefik CRDs or resources are missing

Symptoms

Helm template or install succeeds, but Traefik resources are not reconciled.
kubectl get ingressroute fails with unknown resource type.
Argo CD or Flux reports missing kind IngressRoute.

CauseController mode expects Traefik CRDs and a Traefik controller. They must be installed separately or through the bundled dependency when enabled.Checks

kubectl get crd | grep traefik
kubectl get pods --all-namespaces | grep -i traefik

If you want the chart to install the bundled Traefik controller, enable it:

deployment:
  type: controller
  installTraefikController: true

If Traefik is already installed elsewhere, keep it disabled and make sure the controller watches the namespace and labels used by the Pangolin IngressRoute.

Helm upgrade fails or rendered output is unexpected

Symptoms

helm upgrade fails.
Rendered resources changed unexpectedly.
Existing resources conflict with chart-managed resources.
GitOps reports immutable field changes or ownership conflicts.

ChecksRender before upgrading:

helm template "$PANGOLIN_RELEASE" fossorial/pangolin \
  --namespace "$PANGOLIN_NAMESPACE" \
  --values values-pangolin.yaml > rendered.yaml

Run a server-side dry run:

kubectl apply -f rendered.yaml --dry-run=server

Compare the current live release:

helm get manifest "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE" > live-release.yaml
diff -u live-release.yaml rendered.yaml

Check ownership conflicts:

kubectl get all --namespace "$PANGOLIN_NAMESPACE" -o yaml | grep -E "meta.helm.sh|app.kubernetes.io/managed-by"

Avoid --force unless you understand which resources will be recreated.

helm upgrade --force can delete and recreate resources. That can interrupt traffic and may affect persistent workloads depending on the resource type.

Kustomize patches do not apply

Symptoms

Kustomize build succeeds but changes are missing.
Patch target does not match any resource.
Patch breaks after chart upgrade.

ChecksList generated resource names:

kustomize build base | grep -E "^(kind:|  name:)"

Validate the overlay:

kustomize build overlays/prod

Run a server-side dry run:

kustomize build overlays/prod | kubectl apply -f - --dry-run=server

Preview live changes:

kustomize build overlays/prod | kubectl diff -f -

Use modern Kustomize patches syntax:

patches:
  - path: patches/pangolin-resources.patch.yaml
    target:
      group: apps
      version: v1
      kind: Deployment
      name: pangolin

For Helm-rendered bases, do not assume resource names. Check the rendered manifests after each chart upgrade.

GitOps deployment does not sync

Symptoms

Argo CD Application is OutOfSync or Degraded.
Flux HelmRelease or Kustomization is not Ready.
Resources are missing or constantly reverted.

Argo CD checks

kubectl describe application pangolin --namespace argocd
kubectl logs --namespace argocd deployment/argocd-application-controller --tail=100
argocd app diff pangolin
argocd app sync pangolin

Flux checks

flux get sources all --all-namespaces
flux get helmreleases --all-namespaces
flux get kustomizations --all-namespaces
flux logs --all-namespaces --follow

Reconcile manually:

flux reconcile helmrelease pangolin --namespace "$PANGOLIN_NAMESPACE"
flux reconcile kustomization pangolin --namespace flux-system

Common causes:

chart repository or OCI source not reachable
wrong chart version
missing CRDs
invalid values
rendered resource ownership conflict
Secret not available in the expected namespace

Routing issues to the right repository

Use the repository that matches the failing area:

Area	Repository
Chart templates, values, examples, rendered manifests	`fosrl/helm-charts`
Pangolin runtime, API, UI, auth, application behavior	`fosrl/pangolin`
Newt client behavior or connectivity	`fosrl/newt`
Documentation	`fosrl/docs-v2`

Before opening an issue, collect

Collect this information before opening an issue:

chart version
Pangolin app version
Kubernetes version
Helm version
deployment method: Helm, Kustomize, Argo CD, or Flux
sanitized values file
pod logs
namespace events
Traefik logs, if routing is involved
rendered manifests from helm template or kustomize build
Helm release status or GitOps sync status
reproduction steps

Collect basic diagnostics:

kubectl version
helm version

helm status "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE"
helm get values "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE" --all > pangolin-values.yaml
helm get manifest "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE" > pangolin-manifest.yaml

kubectl get pods --namespace "$PANGOLIN_NAMESPACE" -o wide > pangolin-pods.txt
kubectl get events --namespace "$PANGOLIN_NAMESPACE" --sort-by=.lastTimestamp > pangolin-events.txt

Before sharing diagnostics, remove:

database passwords
SERVER_SECRET
API keys
OAuth/OIDC client secrets
TLS private keys
internal hostnames, if sensitive

Useful command reference

# General cluster info
kubectl cluster-info
kubectl version

# Namespace overview
kubectl get all --namespace "$PANGOLIN_NAMESPACE"
kubectl get pvc,secret,configmap --namespace "$PANGOLIN_NAMESPACE"
kubectl get events --namespace "$PANGOLIN_NAMESPACE" --sort-by=.lastTimestamp

# Logs
kubectl logs --namespace "$PANGOLIN_NAMESPACE" \
  -l app.kubernetes.io/name=pangolin \
  --tail=200

kubectl logs --namespace "$PANGOLIN_NAMESPACE" \
  -l app.kubernetes.io/name=gerbil \
  --tail=200

# Dashboard local test
kubectl port-forward --namespace "$PANGOLIN_NAMESPACE" svc/pangolin 8080:3002

# Traefik resources
kubectl get ingressroute --namespace "$PANGOLIN_NAMESPACE"

# Resource usage
kubectl top pod --namespace "$PANGOLIN_NAMESPACE"
kubectl top node

Next steps

Pangolin Configuration

Review Pangolin chart options.

Helm Quick-Start

Install Pangolin with Helm.

Kustomize Quick-Start

Install Pangolin with rendered manifests and Kustomize overlays.

GitOps Overview

Deploy Pangolin with Argo CD or Flux.

About

Manage Pangolin

Self-host Pangolin

Development

Core diagnostics

Helm diagnostics

Kubernetes diagnostics

Traefik diagnostics

Database diagnostics

Common issues and solutions

Routing issues to the right repository

Before opening an issue, collect

Useful command reference

Next steps

Pangolin Configuration

Helm Quick-Start

Kustomize Quick-Start

GitOps Overview

​Core diagnostics

​Helm diagnostics

​Kubernetes diagnostics

​Traefik diagnostics

​Database diagnostics

​Common issues and solutions

​Routing issues to the right repository

​Before opening an issue, collect

​Useful command reference

​Next steps

Pangolin Configuration

Helm Quick-Start

Kustomize Quick-Start

GitOps Overview

Core diagnostics

Helm diagnostics

Kubernetes diagnostics

Traefik diagnostics

Database diagnostics

Common issues and solutions

Routing issues to the right repository

Before opening an issue, collect

Useful command reference

Next steps