Troubleshooting

Symptoms

• Gerbil pod crashes during a fresh install.
• Logs mention missing setup data, missing exit node, or tunnel configuration not being ready.
• Pangolin itself is not initialized yet.

Cause

On first install, Gerbil may start before Pangolin has completed the initial setup. The chart supports gerbil.startupMode for this case.

Resolution

Use delayed startup for the first install:

gerbil:
  startupMode: delayed

Install or upgrade with the values file:

helm upgrade --install "$PANGOLIN_RELEASE" fossorial/pangolin \
  --namespace "$PANGOLIN_NAMESPACE" \
  --values values-pangolin.yaml

After Pangolin setup is complete, switch Gerbil to normal startup:

helm upgrade "$PANGOLIN_RELEASE" fossorial/pangolin \
  --namespace "$PANGOLIN_NAMESPACE" \
  --reuse-values \
  --set gerbil.startupMode=normal

Check Gerbil resources:

kubectl get pods,svc,pvc --namespace "$PANGOLIN_NAMESPACE" \
  -l app.kubernetes.io/name=gerbil

Symptoms

• Gerbil pod does not start.
• Events mention Pod Security Admission.
• Events mention forbidden capabilities.
• Logs or events mention NET_ADMIN.

Cause

Gerbil requires the NET_ADMIN Linux capability for WireGuard interface management. A namespace using a restricted Pod Security profile can block this.

Resolution

Check namespace labels:

kubectl get namespace "$PANGOLIN_NAMESPACE" --show-labels

For a namespace running Gerbil, use a policy profile that allows the required capability. Example:

kubectl label namespace "$PANGOLIN_NAMESPACE" \
  pod-security.kubernetes.io/enforce=privileged \
  pod-security.kubernetes.io/warn=baseline \
  pod-security.kubernetes.io/audit=restricted \
  --overwrite

Then restart the affected pods:

kubectl rollout restart deploy --namespace "$PANGOLIN_NAMESPACE"

Symptoms

• The dashboard URL does not load.
• Browser shows timeout, bad gateway, 404, or TLS error.
• API path /api/v1 fails while the dashboard path works, or the reverse.

Common causes

• DNS points to the wrong load balancer or ingress endpoint.
• Traefik CRDs are missing.
• Traefik controller is not watching the namespace or selector labels.
• IngressRoute host does not match the dashboard URL.
• API route was changed and no longer matches PathPrefix(/api/v1).
• TLS resolver or TLS Secret is misconfigured.

Checks

Check DNS:

nslookup pangolin.example.com

Check Traefik CRDs:

kubectl get crd | grep traefik

Check IngressRoute resources:

kubectl get ingressroute --namespace "$PANGOLIN_NAMESPACE"
kubectl describe ingressroute <name> --namespace "$PANGOLIN_NAMESPACE"

Check the rendered values:

helm get values "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE" --all | grep -A30 ingressRoute

Check Traefik logs. Adjust the namespace and label selector to your Traefik installation:

kubectl logs --namespace traefik -l app.kubernetes.io/name=traefik --tail=100

Temporary local check for the dashboard/UI service port:

kubectl port-forward --namespace "$PANGOLIN_NAMESPACE" svc/pangolin 8080:3002

Then open:

http://localhost:8080

Symptoms

• IngressRoute is created but TLS does not work.
• Traefik logs mention TLS configuration problems.
• Certificate is not issued or the TLS Secret is not found.

Cause

The dashboard IngressRoute TLS configuration should use either a Traefik certificate resolver or an existing TLS Secret.

Resolution

Use Traefik ACME certificate resolver:

pangolin:
  ingressRoute:
    dashboard:
      tls:
        enabled: true
        certResolver: letsencrypt
        secretName: ""

Or use an existing TLS Secret:

pangolin:
  ingressRoute:
    dashboard:
      tls:
        enabled: true
        certResolver: ""
        secretName: pangolin-dashboard-tls

Verify the Secret if using secretName:

kubectl get secret pangolin-dashboard-tls --namespace "$PANGOLIN_NAMESPACE"

Symptoms

• Newt shows repeated connection or tunnel errors.
• Tunnel traffic does not pass.
• WireGuard UDP ports are unreachable from the Newt location.

Common causes

• pangolin.config.gerbil.base_endpoint points to the wrong host.
• Gerbil Service is not exposed as expected.
• External firewall blocks UDP traffic.
• NetworkPolicy blocks the required traffic.
• pangolin.config.gerbil.start_port and gerbil.ports.wg1 are not aligned.
• pangolin.config.gerbil.clients_start_port and gerbil.ports.wg2 are not aligned.

Checks

Check Gerbil Service:

kubectl get svc --namespace "$PANGOLIN_NAMESPACE" \
  -l app.kubernetes.io/name=gerbil

kubectl describe svc <gerbil-service-name> --namespace "$PANGOLIN_NAMESPACE"

Check Gerbil values:

helm get values "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE" --all | grep -A30 gerbil

Check NetworkPolicies:

kubectl get networkpolicy --namespace "$PANGOLIN_NAMESPACE"
kubectl describe networkpolicy --namespace "$PANGOLIN_NAMESPACE"

Verify external firewall rules for the configured UDP ports.

Symptoms

• Newt peers do not establish stable handshakes.
• Tunnel traffic drops even though Gerbil pods are healthy.
• Logs show connection resets or malformed upstream traffic.

Cause

Proxy protocol handling is inconsistent between the upstream hop and Gerbil.

Checks

Check endpoint and port alignment:

helm get values "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE" --all | grep -A40 gerbil

Check Gerbil logs:

kubectl logs --namespace "$PANGOLIN_NAMESPACE" \
  -l app.kubernetes.io/name=gerbil \
  --tail=200

Check Service exposure:

kubectl get svc --namespace "$PANGOLIN_NAMESPACE" \
  -l app.kubernetes.io/name=gerbil -o wide

Symptoms

• Pangolin pod crashes.
• Logs mention database connection errors.
• Events mention missing Secret or missing Secret key.

Cause

database.mode=external needs a valid database connection Secret unless the chart is configured to generate one from values.

Resolution

Create a connection Secret:

kubectl create secret generic pangolin-db-connection \
  --namespace "$PANGOLIN_NAMESPACE" \
  --from-literal=connectionString='postgresql://pangolin:password@postgres.example.com:5432/pangolin?sslmode=require'

Reference it in values:

database:
  mode: external
  connection:
    existingSecretName: pangolin-db-connection
    existingSecretKey: connectionString

Check the Secret:

kubectl describe secret pangolin-db-connection --namespace "$PANGOLIN_NAMESPACE"

Symptoms

• CNPG Cluster resource is missing.
• CNPG pods do not start.
• Pangolin cannot connect to the generated CNPG database.
• Secret such as pangolin-db-app is missing.

Common causes

• CloudNativePG CRDs/operator are not installed.
• cnpg-cluster.enabled is false when you expected the chart to create a cluster.
• cnpg-operator.enabled is false and no operator exists.
• database.cloudnativepg.cluster.name does not match the CNPG cluster name.
• StorageClass or PVC provisioning fails.

Checks

Check CRDs:

kubectl get crd | grep postgresql.cnpg.io

Check CNPG operator pods:

kubectl get pods --all-namespaces | grep -i cnpg

Check CNPG Cluster:

kubectl get cluster --namespace "$PANGOLIN_NAMESPACE"
kubectl describe cluster pangolin-db --namespace "$PANGOLIN_NAMESPACE"

Check PVCs and Secrets:

kubectl get pvc --namespace "$PANGOLIN_NAMESPACE"
kubectl get secret --namespace "$PANGOLIN_NAMESPACE" | grep pangolin-db

Expected naming when using the default example:

database:
  cloudnativepg:
    cluster:
      name: pangolin-db

cnpg-cluster:
  enabled: true
  fullnameOverride: pangolin-db

Symptoms

• DNS lookups fail.
• Pangolin cannot connect to the database.
• Controller cannot reach the Kubernetes API.
• Gerbil or Newt traffic does not work.
• External services such as SMTP, OIDC, or webhooks time out.

Cause

The chart can render NetworkPolicies. If your CNI enforces them, missing egress or ingress rules can break required paths.

Checks

kubectl get networkpolicy --namespace "$PANGOLIN_NAMESPACE"
kubectl describe networkpolicy --namespace "$PANGOLIN_NAMESPACE"

Check whether DNS is allowed:

networkPolicy:
  dns:
    enabled: true

Check database egress:

networkPolicy:
  database:
    enabled: true
    port: 5432

Check controller API access:

networkPolicy:
  controller:
    egress:
      enabled: true
      kubernetesApi:
        enabled: true
        port: 443

For external integrations, add scoped egress rules for the required services instead of allowing broad egress.

For a temporary isolation test, disable NetworkPolicy and re-apply:

networkPolicy:
  enabled: false

If this fixes the issue, re-enable policies and add the missing rules.

Symptoms

• Pangolin pod restarts repeatedly.
• Pod stays Pending.
• Readiness never becomes true.

Checks

Find the pod:

kubectl get pods --namespace "$PANGOLIN_NAMESPACE" \
  -l app.kubernetes.io/name=pangolin

Inspect it:

kubectl describe pod <pod-name> --namespace "$PANGOLIN_NAMESPACE"
kubectl logs <pod-name> --namespace "$PANGOLIN_NAMESPACE" --tail=200
kubectl logs <pod-name> --namespace "$PANGOLIN_NAMESPACE" --previous --tail=200

Check PVCs:

kubectl get pvc --namespace "$PANGOLIN_NAMESPACE"
kubectl describe pvc <pvc-name> --namespace "$PANGOLIN_NAMESPACE"

Common causes:

Run a temporary debug pod for network tests:

kubectl run net-debug \
  --namespace "$PANGOLIN_NAMESPACE" \
  --rm -it \
  --image=curlimages/curl:latest \
  --restart=Never \
  -- sh

Symptoms

• Helm template or install succeeds, but Traefik resources are not reconciled.
• kubectl get ingressroute fails with unknown resource type.
• Argo CD or Flux reports missing kind IngressRoute.

Cause

Controller mode expects Traefik CRDs and a Traefik controller. They must be installed separately or through the bundled dependency when enabled.

Checks

kubectl get crd | grep traefik
kubectl get pods --all-namespaces | grep -i traefik

If you want the chart to install the bundled Traefik controller, enable it:

deployment:
  type: controller
  installTraefikController: true

If Traefik is already installed elsewhere, keep it disabled and make sure the controller watches the namespace and labels used by the Pangolin IngressRoute.

Symptoms

• helm upgrade fails.
• Rendered resources changed unexpectedly.
• Existing resources conflict with chart-managed resources.
• GitOps reports immutable field changes or ownership conflicts.

Checks

Render before upgrading:

helm template "$PANGOLIN_RELEASE" fossorial/pangolin \
  --namespace "$PANGOLIN_NAMESPACE" \
  --values values-pangolin.yaml > rendered.yaml

Run a server-side dry run:

kubectl apply -f rendered.yaml --dry-run=server

Compare the current live release:

helm get manifest "$PANGOLIN_RELEASE" --namespace "$PANGOLIN_NAMESPACE" > live-release.yaml
diff -u live-release.yaml rendered.yaml

Check ownership conflicts:

kubectl get all --namespace "$PANGOLIN_NAMESPACE" -o yaml | grep -E "meta.helm.sh|app.kubernetes.io/managed-by"

Avoid --force unless you understand which resources will be recreated.

Symptoms

• Kustomize build succeeds but changes are missing.
• Patch target does not match any resource.
• Patch breaks after chart upgrade.

Checks

List generated resource names:

kustomize build base | grep -E "^(kind:|  name:)"

Validate the overlay:

kustomize build overlays/prod

Run a server-side dry run:

kustomize build overlays/prod | kubectl apply -f - --dry-run=server

Preview live changes:

kustomize build overlays/prod | kubectl diff -f -

Use modern Kustomize patches syntax:

patches:
  - path: patches/pangolin-resources.patch.yaml
    target:
      group: apps
      version: v1
      kind: Deployment
      name: pangolin

Symptoms

• Argo CD Application is OutOfSync or Degraded.
• Flux HelmRelease or Kustomization is not Ready.
• Resources are missing or constantly reverted.

Argo CD checks

kubectl describe application pangolin --namespace argocd
kubectl logs --namespace argocd deployment/argocd-application-controller --tail=100
argocd app diff pangolin
argocd app sync pangolin

Flux checks

flux get sources all --all-namespaces
flux get helmreleases --all-namespaces
flux get kustomizations --all-namespaces
flux logs --all-namespaces --follow

Reconcile manually:

flux reconcile helmrelease pangolin --namespace "$PANGOLIN_NAMESPACE"
flux reconcile kustomization pangolin --namespace flux-system

Common causes:

• chart repository or OCI source not reachable
• wrong chart version
• missing CRDs
• invalid values
• rendered resource ownership conflict
• Secret not available in the expected namespace