> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pangolin.net/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Diagnose and resolve common Newt Kubernetes deployment issues.

Use this guide to troubleshoot Newt Kubernetes deployments installed with Helm, Kustomize, Argo CD, or Flux.

Start with the basic checks, then move to the section that matches the symptom.

## Quick checks

Set the namespace and release name used by your installation:

```bash theme={"theme":"gruvbox-light-hard"}
export NEWT_NAMESPACE=pangolin
export NEWT_RELEASE=newt
```

Check the Helm release:

```bash theme={"theme":"gruvbox-light-hard"}
helm status "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
helm history "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
```

Check Newt pods:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl get pods --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt
```

Check recent events:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl get events --namespace "$NEWT_NAMESPACE" \
  --sort-by=.lastTimestamp
```

Check logs:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl logs --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt \
  --tail=100
```

Check the applied Helm values:

```bash theme={"theme":"gruvbox-light-hard"}
helm get values "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
```

<Note>
  Do not assume the pod or Deployment name. Chart-generated names can change with the Helm release name, instance name, `nameOverride`, or `fullnameOverride`.
</Note>

## Get the generated resource names

List Newt resources:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl get deploy,sts,svc,secret,cm --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt
```

List pods with labels:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl get pods --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt \
  --show-labels
```

Store the first Newt pod name:

```bash theme={"theme":"gruvbox-light-hard"}
export NEWT_POD="$(kubectl get pod --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt \
  -o jsonpath='{.items[0].metadata.name}')"
```

Then use:

```bash theme={"theme":"gruvbox-light-hard"}
echo "$NEWT_POD"
```

## Pod fails to start

### Symptoms

```text theme={"theme":"gruvbox-light-hard"}
STATUS             RESTARTS
CrashLoopBackOff   5
Error              3
CreateContainerConfigError
ImagePullBackOff
```

### Check pod details

```bash theme={"theme":"gruvbox-light-hard"}
kubectl describe pod "$NEWT_POD" --namespace "$NEWT_NAMESPACE"
```

Check logs:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl logs "$NEWT_POD" --namespace "$NEWT_NAMESPACE" --tail=100
```

If the container restarts quickly, check the previous logs:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl logs "$NEWT_POD" --namespace "$NEWT_NAMESPACE" --previous --tail=100
```

### Common causes

| Symptom                             | Likely cause                                           | Check                                                   |
| ----------------------------------- | ------------------------------------------------------ | ------------------------------------------------------- |
| `Secret "..." not found`            | Secret name does not match `auth.existingSecretName`   | `kubectl get secret -n "$NEWT_NAMESPACE"`               |
| Missing env var or empty credential | Secret exists but key names do not match `auth.keys.*` | `kubectl describe secret <secret> -n "$NEWT_NAMESPACE"` |
| Authentication failure              | Wrong `NEWT_ID`, `NEWT_SECRET`, or provisioning key    | Check credentials in Pangolin                           |
| Endpoint connection errors          | `PANGOLIN_ENDPOINT` is wrong or unreachable            | Test DNS and HTTPS from the pod                         |
| Image pull failure                  | Registry or image settings are wrong                   | `kubectl describe pod`                                  |

## Secret issues

### Verify the Secret exists

```bash theme={"theme":"gruvbox-light-hard"}
kubectl get secret newt-auth --namespace "$NEWT_NAMESPACE"
```

### Check Secret keys

```bash theme={"theme":"gruvbox-light-hard"}
kubectl describe secret newt-auth --namespace "$NEWT_NAMESPACE"
```

The default keys are:

```text theme={"theme":"gruvbox-light-hard"}
PANGOLIN_ENDPOINT
NEWT_ID
NEWT_SECRET
```

If your Secret uses different key names, map them in values:

```yaml theme={"theme":"gruvbox-light-hard"}
newtInstances:
  - name: main-tunnel
    enabled: true
    auth:
      existingSecretName: newt-auth
      keys:
        endpointKey: PANGOLIN_ENDPOINT
        idKey: NEWT_ID
        secretKey: NEWT_SECRET
```

<Warning>
  Do not paste decoded secrets into issue reports, logs, screenshots, or public repositories.
</Warning>

### Check which Secret the pod uses

```bash theme={"theme":"gruvbox-light-hard"}
kubectl get pod "$NEWT_POD" --namespace "$NEWT_NAMESPACE" \
  -o jsonpath='{range .spec.containers[*].envFrom[*]}{.secretRef.name}{"\n"}{end}'
```

Also inspect explicit Secret references:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl get pod "$NEWT_POD" --namespace "$NEWT_NAMESPACE" -o yaml | grep -A5 -B2 secretKeyRef
```

## Newt cannot reach Pangolin

### Test DNS from the Newt pod

```bash theme={"theme":"gruvbox-light-hard"}
kubectl exec "$NEWT_POD" --namespace "$NEWT_NAMESPACE" -- \
  nslookup pangolin.example.com
```

### Test HTTPS from the Newt pod

```bash theme={"theme":"gruvbox-light-hard"}
kubectl exec "$NEWT_POD" --namespace "$NEWT_NAMESPACE" -- \
  wget -S -O- https://pangolin.example.com 2>&1 | head -40
```

Depending on the image, `curl`, `wget`, `nc`, or `nslookup` may not be available. If needed, run a temporary debug pod in the same namespace:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl run net-debug \
  --namespace "$NEWT_NAMESPACE" \
  --rm -it \
  --image=curlimages/curl:latest \
  --restart=Never \
  -- sh
```

Then test:

```bash theme={"theme":"gruvbox-light-hard"}
curl -vk https://pangolin.example.com
```

### Common causes

| Problem                          | What to check                                             |
| -------------------------------- | --------------------------------------------------------- |
| DNS fails                        | CoreDNS, NetworkPolicy egress to DNS, wrong hostname      |
| HTTPS fails                      | ingress, TLS certificate, firewall, proxy, wrong endpoint |
| TLS verification fails           | certificate chain, hostname mismatch, private CA          |
| Works locally but not in cluster | egress policies, proxy settings, DNS split-horizon        |

## Newt pod is running but site is offline

Check logs:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl logs "$NEWT_POD" --namespace "$NEWT_NAMESPACE" --tail=200
```

Check the site in the Pangolin dashboard.

Verify:

* the site credentials belong to the same site
* the site was not deleted or regenerated in Pangolin
* `PANGOLIN_ENDPOINT` points to the correct Pangolin URL
* the cluster can resolve and reach the Pangolin endpoint
* outbound HTTPS is allowed from the Newt namespace
* the Secret is in the same namespace as the Newt workload

If you use provisioning, also verify:

* `provisioningKey` is valid
* `newtName` is set as expected
* `configPersistence.enabled=true`
* the configured `CONFIG_FILE` path is writable

## Provisioning issues

Provisioning requires writable config persistence.

### Symptoms

* Newt starts but does not keep generated credentials after restart.
* Newt provisions repeatedly.
* Logs mention config file or write errors.
* Pod restarts cause the site to appear as a new or unconfigured instance.

### Check values

```bash theme={"theme":"gruvbox-light-hard"}
helm get values "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
```

Provisioning example:

```yaml theme={"theme":"gruvbox-light-hard"}
newtInstances:
  - name: main-tunnel
    enabled: true
    auth:
      pangolinEndpoint: https://pangolin.example.com
      provisioningKey: "<provisioning-key>"
      newtName: "my-site"
    configPersistence:
      enabled: true
      type: emptyDir
      mountPath: /var/lib/newt
      fileName: config.json
```

For durable state, use an existing PVC:

```yaml theme={"theme":"gruvbox-light-hard"}
newtInstances:
  - name: main-tunnel
    enabled: true
    auth:
      pangolinEndpoint: https://pangolin.example.com
      provisioningKey: "<provisioning-key>"
      newtName: "my-site"
    configPersistence:
      enabled: true
      type: persistentVolumeClaim
      existingClaim: my-newt-config
      mountPath: /var/lib/newt
      fileName: config.json
```

<Note>
  `emptyDir` is recreated when the pod is recreated. Use a PVC if the generated configuration must survive pod replacement.
</Note>

## Service not created or not reachable

### Important behavior

`acceptClients` does not create a Service.

A Service is created through:

```yaml theme={"theme":"gruvbox-light-hard"}
newtInstances:
  - name: main-tunnel
    service:
      enabled: true
```

The chart also has `service.enabledWhenAcceptClients`, but runtime client behavior and Service rendering should still be verified in the rendered manifests.

### Check Services

```bash theme={"theme":"gruvbox-light-hard"}
kubectl get svc --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt
```

Describe the Service:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl describe svc <service-name> --namespace "$NEWT_NAMESPACE"
```

### LoadBalancer stuck in pending

```text theme={"theme":"gruvbox-light-hard"}
EXTERNAL-IP   <pending>
```

Common causes:

* the cluster has no cloud load balancer integration
* bare-metal cluster without MetalLB or equivalent
* cloud provider quota or permission issue
* invalid `loadBalancerClass`
* invalid `loadBalancerSourceRanges`

For bare-metal clusters, use MetalLB or another load balancer implementation, or use `NodePort` if appropriate.

## Metrics scraping does not work

Metrics are disabled by default.

Enable metrics:

```yaml theme={"theme":"gruvbox-light-hard"}
global:
  metrics:
    enabled: true
```

The chart default admin address is:

```yaml theme={"theme":"gruvbox-light-hard"}
global:
  metrics:
    adminAddr: ":2112"
```

This listens on all interfaces and allows in-cluster scraping. Do not set it to `127.0.0.1:2112` if Prometheus scrapes from another pod.

### Metrics Service

Enable the metrics Service:

```yaml theme={"theme":"gruvbox-light-hard"}
global:
  metrics:
    enabled: true
    service:
      enabled: true
      port: 2112
```

### ServiceMonitor

If you use Prometheus Operator:

```yaml theme={"theme":"gruvbox-light-hard"}
global:
  metrics:
    enabled: true
    service:
      enabled: true
    serviceMonitor:
      enabled: true
```

Check resources:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl get svc,podmonitor,servicemonitor,prometheusrule \
  --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt
```

<Note>
  The chart has separate metrics values for container port, admin address, and metrics Service port. Check the rendered manifest when changing these values.
</Note>

## NetworkPolicy blocks traffic

If NetworkPolicy is enabled, check that the policy allows required egress.

Newt usually needs egress to:

* DNS
* Pangolin endpoint over HTTPS
* any tunnel or connectivity endpoints used by your deployment

Check policies:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl get networkpolicy --namespace "$NEWT_NAMESPACE"
kubectl describe networkpolicy --namespace "$NEWT_NAMESPACE"
```

If DNS is blocked, enable or add DNS egress rules.

Example:

```yaml theme={"theme":"gruvbox-light-hard"}
global:
  networkPolicy:
    enabled: true
    components:
      dns:
        enabled: true
```

If HTTPS egress is blocked, add an appropriate custom egress rule for your environment.

## Multiple Newt instances conflict

### Symptoms

* Multiple pods run, but only one site connects.
* Both instances use the same credentials.
* A site appears to flap between instances.
* Logs show authentication or registration conflicts.

### Check values

```bash theme={"theme":"gruvbox-light-hard"}
helm get values "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
```

Each instance should use its own credentials or provisioning identity:

```yaml theme={"theme":"gruvbox-light-hard"}
newtInstances:
  - name: site-a
    enabled: true
    auth:
      existingSecretName: newt-auth-site-a

  - name: site-b
    enabled: true
    auth:
      existingSecretName: newt-auth-site-b
```

Create separate Secrets:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl create secret generic newt-auth-site-a \
  --namespace "$NEWT_NAMESPACE" \
  --from-literal=PANGOLIN_ENDPOINT=https://pangolin.example.com \
  --from-literal=NEWT_ID=<site-a-newt-id> \
  --from-literal=NEWT_SECRET=<site-a-newt-secret>

kubectl create secret generic newt-auth-site-b \
  --namespace "$NEWT_NAMESPACE" \
  --from-literal=PANGOLIN_ENDPOINT=https://pangolin.example.com \
  --from-literal=NEWT_ID=<site-b-newt-id> \
  --from-literal=NEWT_SECRET=<site-b-newt-secret>
```

## RBAC or service account issues

Chart `1.4.0` disables RBAC creation by default.

Check service account and RBAC:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl get serviceaccount,role,rolebinding \
  --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt
```

If your configuration requires Kubernetes API access, enable RBAC:

```yaml theme={"theme":"gruvbox-light-hard"}
rbac:
  create: true
  clusterRole: false
```

For most Newt deployments, RBAC is not required.

## High CPU or memory usage

Check resource usage:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl top pod --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt
```

Check current resource settings:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl get pod "$NEWT_POD" --namespace "$NEWT_NAMESPACE" \
  -o jsonpath='{.spec.containers[0].resources}'
```

Tune resources in values:

```yaml theme={"theme":"gruvbox-light-hard"}
newtInstances:
  - name: main-tunnel
    resources:
      requests:
        cpu: 200m
        memory: 256Mi
      limits:
        cpu: 1000m
        memory: 512Mi
```

Then upgrade:

```bash theme={"theme":"gruvbox-light-hard"}
helm upgrade "$NEWT_RELEASE" fossorial/newt \
  --namespace "$NEWT_NAMESPACE" \
  --values values-newt.yaml
```

Common causes of high usage:

* high tunnel traffic
* too low resource limits
* repeated reconnect loops
* excessive debug logging
* MTU or network path issues

## MTU issues

### Symptoms

* Connections establish but large transfers fail.
* Some websites or services work, others hang.
* Logs show repeated reconnects.
* Throughput is much lower than expected.

Newt defaults to MTU `1280`.

Try another MTU only after confirming basic connectivity:

```yaml theme={"theme":"gruvbox-light-hard"}
newtInstances:
  - name: main-tunnel
    mtu: 1280
```

Upgrade after changing values:

```bash theme={"theme":"gruvbox-light-hard"}
helm upgrade "$NEWT_RELEASE" fossorial/newt \
  --namespace "$NEWT_NAMESPACE" \
  --values values-newt.yaml
```

## Helm debugging

Preview an upgrade:

```bash theme={"theme":"gruvbox-light-hard"}
helm upgrade "$NEWT_RELEASE" fossorial/newt \
  --namespace "$NEWT_NAMESPACE" \
  --values values-newt.yaml \
  --dry-run
```

Render the chart locally:

```bash theme={"theme":"gruvbox-light-hard"}
helm template "$NEWT_RELEASE" fossorial/newt \
  --namespace "$NEWT_NAMESPACE" \
  --values values-newt.yaml
```

Show rendered manifests from the live release:

```bash theme={"theme":"gruvbox-light-hard"}
helm get manifest "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
```

Show values from the live release:

```bash theme={"theme":"gruvbox-light-hard"}
helm get values "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
```

Rollback:

```bash theme={"theme":"gruvbox-light-hard"}
helm rollback "$NEWT_RELEASE" <revision> --namespace "$NEWT_NAMESPACE"
```

## Kustomize debugging

Validate the overlay:

```bash theme={"theme":"gruvbox-light-hard"}
kustomize build overlays/site-a
```

Run a server-side dry run:

```bash theme={"theme":"gruvbox-light-hard"}
kustomize build overlays/site-a | kubectl apply -f - --dry-run=server
```

Preview live changes:

```bash theme={"theme":"gruvbox-light-hard"}
kustomize build overlays/site-a | kubectl diff -f -
```

If a patch does not apply, inspect generated resource names:

```bash theme={"theme":"gruvbox-light-hard"}
kustomize build base | grep -E "^(kind:|  name:)"
```

## Collect diagnostics

Collect logs and resource information:

```bash theme={"theme":"gruvbox-light-hard"}
kubectl logs --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt \
  --tail=200 > newt-logs.txt

kubectl get pods --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt \
  -o yaml > newt-pods.yaml

kubectl get events --namespace "$NEWT_NAMESPACE" \
  --sort-by=.lastTimestamp > newt-events.txt

helm get values "$NEWT_RELEASE" \
  --namespace "$NEWT_NAMESPACE" > newt-helm-values.yaml

helm get manifest "$NEWT_RELEASE" \
  --namespace "$NEWT_NAMESPACE" > newt-helm-manifest.yaml
```

If using Kustomize:

```bash theme={"theme":"gruvbox-light-hard"}
kustomize build overlays/site-a > newt-kustomize-output.yaml
```

Before sharing diagnostics, remove:

* Newt credentials
* provisioning keys
* TLS private keys
* tokens
* passwords
* internal hostnames if sensitive

## Next steps

<CardGroup cols={2}>
  <Card title="Configuration" href="/manage/sites/kubernetes/configuration" icon="sliders">
    Review Newt chart options.
  </Card>

  <Card title="Helm Install" href="/manage/sites/kubernetes/helm" icon="box">
    Install Newt with Helm.
  </Card>

  <Card title="Kustomize Install" href="/manage/sites/kubernetes/kustomize" icon="layer-group">
    Install Newt with rendered manifests and Kustomize overlays.
  </Card>

  <Card title="GitOps" href="/self-host/manual/kubernetes/gitops/overview" icon="code-branch">
    Deploy Newt with Argo CD or Flux.
  </Card>
</CardGroup>