# Troubleshooting (/manage/sites/kubernetes/troubleshooting)

> Diagnose and resolve common Newt Kubernetes deployment issues.



Use this guide to troubleshoot Newt Kubernetes deployments installed with Helm, Kustomize, Argo CD, or Flux.

Start with the basic checks, then move to the section that matches the symptom.

## Quick checks [#quick-checks]

Set the namespace and release name used by your installation:

```bash
export NEWT_NAMESPACE=pangolin
export NEWT_RELEASE=newt
```

Check the Helm release:

```bash
helm status "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
helm history "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
```

Check Newt pods:

```bash
kubectl get pods --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt
```

Check recent events:

```bash
kubectl get events --namespace "$NEWT_NAMESPACE" \
  --sort-by=.lastTimestamp
```

Check logs:

```bash
kubectl logs --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt \
  --tail=100
```

Check the applied Helm values:

```bash
helm get values "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
```

<Note>
  Do not assume the pod or Deployment name. Chart-generated names can change with the Helm release name, instance name, `nameOverride`, or `fullnameOverride`.
</Note>

## Get the generated resource names [#get-the-generated-resource-names]

List Newt resources:

```bash
kubectl get deploy,sts,svc,secret,cm --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt
```

List pods with labels:

```bash
kubectl get pods --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt \
  --show-labels
```

Store the first Newt pod name:

```bash
export NEWT_POD="$(kubectl get pod --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt \
  -o jsonpath='{.items[0].metadata.name}')"
```

Then use:

```bash
echo "$NEWT_POD"
```

## Pod fails to start [#pod-fails-to-start]

### Symptoms [#symptoms]

```text
STATUS             RESTARTS
CrashLoopBackOff   5
Error              3
CreateContainerConfigError
ImagePullBackOff
```

### Check pod details [#check-pod-details]

```bash
kubectl describe pod "$NEWT_POD" --namespace "$NEWT_NAMESPACE"
```

Check logs:

```bash
kubectl logs "$NEWT_POD" --namespace "$NEWT_NAMESPACE" --tail=100
```

If the container restarts quickly, check the previous logs:

```bash
kubectl logs "$NEWT_POD" --namespace "$NEWT_NAMESPACE" --previous --tail=100
```

### Common causes [#common-causes]

| Symptom                             | Likely cause                                           | Check                                                   |
| ----------------------------------- | ------------------------------------------------------ | ------------------------------------------------------- |
| `Secret "..." not found`            | Secret name does not match `auth.existingSecretName`   | `kubectl get secret -n "$NEWT_NAMESPACE"`               |
| Missing env var or empty credential | Secret exists but key names do not match `auth.keys.*` | `kubectl describe secret <secret> -n "$NEWT_NAMESPACE"` |
| Authentication failure              | Wrong `NEWT_ID`, `NEWT_SECRET`, or provisioning key    | Check credentials in Pangolin                           |
| Endpoint connection errors          | `PANGOLIN_ENDPOINT` is wrong or unreachable            | Test DNS and HTTPS from the pod                         |
| Image pull failure                  | Registry or image settings are wrong                   | `kubectl describe pod`                                  |

## Secret issues [#secret-issues]

### Verify the Secret exists [#verify-the-secret-exists]

```bash
kubectl get secret newt-auth --namespace "$NEWT_NAMESPACE"
```

### Check Secret keys [#check-secret-keys]

```bash
kubectl describe secret newt-auth --namespace "$NEWT_NAMESPACE"
```

The default keys are:

```text
PANGOLIN_ENDPOINT
NEWT_ID
NEWT_SECRET
```

If your Secret uses different key names, map them in values:

```yaml
newtInstances:
  - name: main-tunnel
    enabled: true
    auth:
      existingSecretName: newt-auth
      keys:
        endpointKey: PANGOLIN_ENDPOINT
        idKey: NEWT_ID
        secretKey: NEWT_SECRET
```

<Warning>
  Do not paste decoded secrets into issue reports, logs, screenshots, or public repositories.
</Warning>

### Check which Secret the pod uses [#check-which-secret-the-pod-uses]

```bash
kubectl get pod "$NEWT_POD" --namespace "$NEWT_NAMESPACE" \
  -o jsonpath='{range .spec.containers[*].envFrom[*]}{.secretRef.name}{"\n"}{end}'
```

Also inspect explicit Secret references:

```bash
kubectl get pod "$NEWT_POD" --namespace "$NEWT_NAMESPACE" -o yaml | grep -A5 -B2 secretKeyRef
```

## Newt cannot reach Pangolin [#newt-cannot-reach-pangolin]

### Test DNS from the Newt pod [#test-dns-from-the-newt-pod]

```bash
kubectl exec "$NEWT_POD" --namespace "$NEWT_NAMESPACE" -- \
  nslookup pangolin.example.com
```

### Test HTTPS from the Newt pod [#test-https-from-the-newt-pod]

```bash
kubectl exec "$NEWT_POD" --namespace "$NEWT_NAMESPACE" -- \
  wget -S -O- https://pangolin.example.com 2>&1 | head -40
```

Depending on the image, `curl`, `wget`, `nc`, or `nslookup` may not be available. If needed, run a temporary debug pod in the same namespace:

```bash
kubectl run net-debug \
  --namespace "$NEWT_NAMESPACE" \
  --rm -it \
  --image=curlimages/curl:latest \
  --restart=Never \
  -- sh
```

Then test:

```bash
curl -vk https://pangolin.example.com
```

### Common causes [#common-causes-1]

| Problem                          | What to check                                             |
| -------------------------------- | --------------------------------------------------------- |
| DNS fails                        | CoreDNS, NetworkPolicy egress to DNS, wrong hostname      |
| HTTPS fails                      | ingress, TLS certificate, firewall, proxy, wrong endpoint |
| TLS verification fails           | certificate chain, hostname mismatch, private CA          |
| Works locally but not in cluster | egress policies, proxy settings, DNS split-horizon        |

## Newt pod is running but site is offline [#newt-pod-is-running-but-site-is-offline]

Check logs:

```bash
kubectl logs "$NEWT_POD" --namespace "$NEWT_NAMESPACE" --tail=200
```

Check the site in the Pangolin dashboard.

Verify:

* the site credentials belong to the same site
* the site was not deleted or regenerated in Pangolin
* `PANGOLIN_ENDPOINT` points to the correct Pangolin URL
* the cluster can resolve and reach the Pangolin endpoint
* outbound HTTPS is allowed from the Newt namespace
* the Secret is in the same namespace as the Newt workload

If you use provisioning, also verify:

* `provisioningKey` is valid
* `newtName` is set as expected
* `configPersistence.enabled=true`
* the configured `CONFIG_FILE` path is writable

## Provisioning issues [#provisioning-issues]

Provisioning requires writable config persistence.

### Symptoms [#symptoms-1]

* Newt starts but does not keep generated credentials after restart.
* Newt provisions repeatedly.
* Logs mention config file or write errors.
* Pod restarts cause the site to appear as a new or unconfigured instance.

### Check values [#check-values]

```bash
helm get values "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
```

Provisioning example:

```yaml
newtInstances:
  - name: main-tunnel
    enabled: true
    auth:
      pangolinEndpoint: https://pangolin.example.com
      provisioningKey: "<provisioning-key>"
      newtName: "my-site"
    configPersistence:
      enabled: true
      type: emptyDir
      mountPath: /var/lib/newt
      fileName: config.json
```

For durable state, use an existing PVC:

```yaml
newtInstances:
  - name: main-tunnel
    enabled: true
    auth:
      pangolinEndpoint: https://pangolin.example.com
      provisioningKey: "<provisioning-key>"
      newtName: "my-site"
    configPersistence:
      enabled: true
      type: persistentVolumeClaim
      existingClaim: my-newt-config
      mountPath: /var/lib/newt
      fileName: config.json
```

<Note>
  `emptyDir` is recreated when the pod is recreated. Use a PVC if the generated configuration must survive pod replacement.
</Note>

## Service not created or not reachable [#service-not-created-or-not-reachable]

### Important behavior [#important-behavior]

`acceptClients` does not create a Service.

A Service is created through:

```yaml
newtInstances:
  - name: main-tunnel
    service:
      enabled: true
```

The chart also has `service.enabledWhenAcceptClients`, but runtime client behavior and Service rendering should still be verified in the rendered manifests.

### Check Services [#check-services]

```bash
kubectl get svc --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt
```

Describe the Service:

```bash
kubectl describe svc <service-name> --namespace "$NEWT_NAMESPACE"
```

### LoadBalancer stuck in pending [#loadbalancer-stuck-in-pending]

```text
EXTERNAL-IP   <pending>
```

Common causes:

* the cluster has no cloud load balancer integration
* bare-metal cluster without MetalLB or equivalent
* cloud provider quota or permission issue
* invalid `loadBalancerClass`
* invalid `loadBalancerSourceRanges`

For bare-metal clusters, use MetalLB or another load balancer implementation, or use `NodePort` if appropriate.

## Metrics scraping does not work [#metrics-scraping-does-not-work]

Metrics are disabled by default.

Enable metrics:

```yaml
global:
  metrics:
    enabled: true
```

The chart default admin address is:

```yaml
global:
  metrics:
    adminAddr: ":2112"
```

This listens on all interfaces and allows in-cluster scraping. Do not set it to `127.0.0.1:2112` if Prometheus scrapes from another pod.

### Metrics Service [#metrics-service]

Enable the metrics Service:

```yaml
global:
  metrics:
    enabled: true
    service:
      enabled: true
      port: 2112
```

### ServiceMonitor [#servicemonitor]

If you use Prometheus Operator:

```yaml
global:
  metrics:
    enabled: true
    service:
      enabled: true
    serviceMonitor:
      enabled: true
```

Check resources:

```bash
kubectl get svc,podmonitor,servicemonitor,prometheusrule \
  --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt
```

<Note>
  The chart has separate metrics values for container port, admin address, and metrics Service port. Check the rendered manifest when changing these values.
</Note>

## NetworkPolicy blocks traffic [#networkpolicy-blocks-traffic]

If NetworkPolicy is enabled, check that the policy allows required egress.

Newt usually needs egress to:

* DNS
* Pangolin endpoint over HTTPS
* any tunnel or connectivity endpoints used by your deployment

Check policies:

```bash
kubectl get networkpolicy --namespace "$NEWT_NAMESPACE"
kubectl describe networkpolicy --namespace "$NEWT_NAMESPACE"
```

If DNS is blocked, enable or add DNS egress rules.

Example:

```yaml
global:
  networkPolicy:
    enabled: true
    components:
      dns:
        enabled: true
```

If HTTPS egress is blocked, add an appropriate custom egress rule for your environment.

## Multiple Newt instances conflict [#multiple-newt-instances-conflict]

### Symptoms [#symptoms-2]

* Multiple pods run, but only one site connects.
* Both instances use the same credentials.
* A site appears to flap between instances.
* Logs show authentication or registration conflicts.

### Check values [#check-values-1]

```bash
helm get values "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
```

Each instance should use its own credentials or provisioning identity:

```yaml
newtInstances:
  - name: site-a
    enabled: true
    auth:
      existingSecretName: newt-auth-site-a

  - name: site-b
    enabled: true
    auth:
      existingSecretName: newt-auth-site-b
```

Create separate Secrets:

```bash
kubectl create secret generic newt-auth-site-a \
  --namespace "$NEWT_NAMESPACE" \
  --from-literal=PANGOLIN_ENDPOINT=https://pangolin.example.com \
  --from-literal=NEWT_ID=<site-a-newt-id> \
  --from-literal=NEWT_SECRET=<site-a-newt-secret>

kubectl create secret generic newt-auth-site-b \
  --namespace "$NEWT_NAMESPACE" \
  --from-literal=PANGOLIN_ENDPOINT=https://pangolin.example.com \
  --from-literal=NEWT_ID=<site-b-newt-id> \
  --from-literal=NEWT_SECRET=<site-b-newt-secret>
```

## RBAC or service account issues [#rbac-or-service-account-issues]

Chart `1.4.0` disables RBAC creation by default.

Check service account and RBAC:

```bash
kubectl get serviceaccount,role,rolebinding \
  --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt
```

If your configuration requires Kubernetes API access, enable RBAC:

```yaml
rbac:
  create: true
  clusterRole: false
```

For most Newt deployments, RBAC is not required.

## High CPU or memory usage [#high-cpu-or-memory-usage]

Check resource usage:

```bash
kubectl top pod --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt
```

Check current resource settings:

```bash
kubectl get pod "$NEWT_POD" --namespace "$NEWT_NAMESPACE" \
  -o jsonpath='{.spec.containers[0].resources}'
```

Tune resources in values:

```yaml
newtInstances:
  - name: main-tunnel
    resources:
      requests:
        cpu: 200m
        memory: 256Mi
      limits:
        cpu: 1000m
        memory: 512Mi
```

Then upgrade:

```bash
helm upgrade "$NEWT_RELEASE" fossorial/newt \
  --namespace "$NEWT_NAMESPACE" \
  --values values-newt.yaml
```

Common causes of high usage:

* high tunnel traffic
* too low resource limits
* repeated reconnect loops
* excessive debug logging
* MTU or network path issues

## MTU issues [#mtu-issues]

### Symptoms [#symptoms-3]

* Connections establish but large transfers fail.
* Some websites or services work, others hang.
* Logs show repeated reconnects.
* Throughput is much lower than expected.

Newt defaults to MTU `1280`.

Try another MTU only after confirming basic connectivity:

```yaml
newtInstances:
  - name: main-tunnel
    mtu: 1280
```

Upgrade after changing values:

```bash
helm upgrade "$NEWT_RELEASE" fossorial/newt \
  --namespace "$NEWT_NAMESPACE" \
  --values values-newt.yaml
```

## Helm debugging [#helm-debugging]

Preview an upgrade:

```bash
helm upgrade "$NEWT_RELEASE" fossorial/newt \
  --namespace "$NEWT_NAMESPACE" \
  --values values-newt.yaml \
  --dry-run
```

Render the chart locally:

```bash
helm template "$NEWT_RELEASE" fossorial/newt \
  --namespace "$NEWT_NAMESPACE" \
  --values values-newt.yaml
```

Show rendered manifests from the live release:

```bash
helm get manifest "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
```

Show values from the live release:

```bash
helm get values "$NEWT_RELEASE" --namespace "$NEWT_NAMESPACE"
```

Rollback:

```bash
helm rollback "$NEWT_RELEASE" <revision> --namespace "$NEWT_NAMESPACE"
```

## Kustomize debugging [#kustomize-debugging]

Validate the overlay:

```bash
kustomize build overlays/site-a
```

Run a server-side dry run:

```bash
kustomize build overlays/site-a | kubectl apply -f - --dry-run=server
```

Preview live changes:

```bash
kustomize build overlays/site-a | kubectl diff -f -
```

If a patch does not apply, inspect generated resource names:

```bash
kustomize build base | grep -E "^(kind:|  name:)"
```

## Collect diagnostics [#collect-diagnostics]

Collect logs and resource information:

```bash
kubectl logs --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt \
  --tail=200 > newt-logs.txt

kubectl get pods --namespace "$NEWT_NAMESPACE" \
  -l app.kubernetes.io/name=newt \
  -o yaml > newt-pods.yaml

kubectl get events --namespace "$NEWT_NAMESPACE" \
  --sort-by=.lastTimestamp > newt-events.txt

helm get values "$NEWT_RELEASE" \
  --namespace "$NEWT_NAMESPACE" > newt-helm-values.yaml

helm get manifest "$NEWT_RELEASE" \
  --namespace "$NEWT_NAMESPACE" > newt-helm-manifest.yaml
```

If using Kustomize:

```bash
kustomize build overlays/site-a > newt-kustomize-output.yaml
```

Before sharing diagnostics, remove:

* Newt credentials
* provisioning keys
* TLS private keys
* tokens
* passwords
* internal hostnames if sensitive

## Next steps [#next-steps]

<CardGroup cols="2">
  <Card title="Configuration" href="/manage/sites/kubernetes/configuration" icon="sliders">
    Review Newt chart options.
  </Card>

  <Card title="Helm Install" href="/manage/sites/kubernetes/helm" icon="box">
    Install Newt with Helm.
  </Card>

  <Card title="Kustomize Install" href="/manage/sites/kubernetes/kustomize" icon="layer-group">
    Install Newt with rendered manifests and Kustomize overlays.
  </Card>

  <Card title="GitOps" href="/self-host/manual/kubernetes/gitops/overview" icon="code-branch">
    Deploy Newt with Argo CD or Flux.
  </Card>
</CardGroup>