> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pangolin.net/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics and Observability

> Understand metrics, traces, logs, and profiling support across Pangolin components

<div id="pangolin-toc-cta" className="pangolin-toc-cta-source">
  <Card title="Try free on Pangolin Cloud" icon="cloud" href="https://app.pangolin.net/auth/signup" arrow="true" cta="Sign up free">
    Fastest way to get started with Pangolin using the hosted control plane. No credit card required.
  </Card>
</div>

Pangolin exposes observability signals across multiple components, but not every component provides the same telemetry surface. This page explains which metrics, traces, logs, and profiling endpoints are available today, how they are exposed, and which collection patterns are recommended for production deployments.

**Newt** provides the broadest native observability support with Prometheus metrics, OTLP metrics, OTLP traces, health checks, and optional `pprof`. **Gerbil** supports metrics through either a Prometheus backend or an OTLP backend. The **Pangolin Kubernetes Controller** exposes a Prometheus-compatible scrape endpoint and includes additional OTel-backed metric instruments on that endpoint.

<Note>
  This page focuses on Pangolin-native observability. For a community walkthrough that collects Traefik metrics with Prometheus and Grafana, see the [community metrics guide](/self-host/community-guides/metrics).

  All currently documented component metrics are listed in the [Full Metric Reference](#full-metric-reference).
</Note>

<CardGroup cols={2}>
  <Card title="Capability Matrix" icon="chart-line" href="#observability-capability-matrix">
    Compare metrics, traces, logs, and profiling support across Pangolin components.
  </Card>

  <Card title="Newt" icon="route" href="#newt">
    Native Prometheus metrics, OTLP metrics, OTLP traces, health checks, and optional pprof.
  </Card>

  <Card title="Gerbil" icon="network-wired" href="#gerbil">
    Metrics-only observability using either a Prometheus or OTLP backend.
  </Card>

  <Card title="Kubernetes Controller" icon="cubes" href="#pangolin-kubernetes-controller">
    Prometheus scraping, health probes, ServiceMonitor examples, and optional pprof.
  </Card>
</CardGroup>

## Observability Capability Matrix

| Component                  | Metrics            | Traces | Logs   | Profiling      |
| -------------------------- | ------------------ | ------ | ------ | -------------- |
| `newt`                     | Prometheus, OTLP   | OTLP   | stdout | pprof optional |
| `gerbil`                   | Prometheus or OTLP | —      | stdout | —              |
| `pangolin-kube-controller` | Prometheus scrape  | —      | stdout | pprof optional |

## Supported Signals

| Signal                    | What it means                                                                    |
| ------------------------- | -------------------------------------------------------------------------------- |
| Prometheus scrape metrics | Pull-based metrics served over HTTP on `/metrics`                                |
| OTLP metrics              | Push-based OpenTelemetry metrics sent to an OTel Collector or compatible backend |
| OTLP traces               | Distributed traces sent to an OTel Collector or trace backend                    |
| Application logs          | stdout, file logs, audit logs, or platform logs                                  |
| Profiling                 | Debug endpoints such as `pprof`                                                  |

## Collection Patterns

Use one or more of the following patterns depending on your deployment model. Prometheus scrape mode is the simplest option for local or Kubernetes monitoring. OTLP is useful when you already operate an OpenTelemetry Collector or want to forward telemetry to a managed backend such as Grafana Cloud, Mimir, or Tempo.

<Tabs>
  <Tab title="Prometheus scrape mode">
    ```yaml title="prometheus.yml (fragment)" theme={"theme":"gruvbox-light-hard"}
    scrape_configs:
      - job_name: newt
        static_configs:
          - targets: ["newt:2112"]

      - job_name: gerbil
        metrics_path: /metrics
        static_configs:
          - targets: ["gerbil:3003"]

      - job_name: pangolin-kube-controller
        static_configs:
          - targets: ["pangolin-kube-controller:9090"]
    ```
  </Tab>

  <Tab title="OTel Collector">
    ```yaml title="otel-collector.yaml" theme={"theme":"gruvbox-light-hard"}
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      prometheus:
        config:
          scrape_configs:
            - job_name: pangolin-kube-controller
              static_configs:
                - targets: ["pangolin-kube-controller:9090"]

    processors:
      batch: {}

    exporters:
      prometheusremotewrite:
        endpoint: https://mimir.example.com/api/v1/push
      otlp/tempo:
        endpoint: tempo:4317
        tls:
          insecure: true

    service:
      pipelines:
        metrics:
          receivers: [otlp, prometheus]
          processors: [batch]
          exporters: [prometheusremotewrite]
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp/tempo]
    ```

    <Tip>
      Today the traces pipeline is relevant only for `newt`. `pangolin-kube-controller` exposes additional OTel instruments, but they are exported on the same Prometheus scrape endpoint rather than pushed through OTLP.
    </Tip>
  </Tab>

  <Tab title="Kubernetes ServiceMonitor">
    ```yaml title="servicemonitor.yaml" theme={"theme":"gruvbox-light-hard"}
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: pangolin-kube-controller
    spec:
      selector:
        matchLabels:
          app: pangolin-kube-controller
      endpoints:
        - port: http-metrics
          path: /metrics
          interval: 30s
    ```

    ```yaml title="helm-values.yaml" theme={"theme":"gruvbox-light-hard"}
    global:
      metrics:
        enabled: true
        service:
          enabled: true
        serviceMonitor:
          enabled: true

    controller:
      monitoring:
        serviceMonitor:
          enabled: true
    ```

    <Note>
      Helm values for Newt are documented on the [Kubernetes Newt configuration page](/self-host/manual/kubernetes/newt/configuration). Controller and Gerbil ServiceMonitor values are documented on the [Kubernetes Pangolin configuration page](/self-host/manual/kubernetes/pangolin/configuration).
    </Note>
  </Tab>

  <Tab title="Logs to Loki">
    ```yaml title="promtail.yaml" theme={"theme":"gruvbox-light-hard"}
    server:
      http_listen_port: 9080

    positions:
      filename: /tmp/positions.yaml

    clients:
      - url: http://loki:3100/loki/api/v1/push

    scrape_configs:
      - job_name: pangolin
        static_configs:
          - targets: [localhost]
            labels:
              job: pangolin
              __path__: /var/log/containers/*pangolin*.log
    ```

    Use Grafana Alloy, Promtail, Fluent Bit, or your preferred SIEM forwarder to collect container and application logs.
  </Tab>
</Tabs>

## Newt

Newt metrics are not typically enabled in default deployments. Turn them on explicitly and expose the admin address only where you intend to scrape or profile. Available metrics are listed in [Newt metrics](#newt-metrics).

<AccordionGroup>
  <Accordion title="Configuration">
    <Tabs>
      <Tab title="Environment Variables">
        ```text theme={"theme":"gruvbox-light-hard"}
        NEWT_METRICS_PROMETHEUS_ENABLED=true
        NEWT_METRICS_OTLP_ENABLED=true
        NEWT_ADMIN_ADDR=:2112
        OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
        OTEL_EXPORTER_OTLP_INSECURE=true
        OTEL_METRIC_EXPORT_INTERVAL=15s
        NEWT_PPROF_ENABLED=false
        ```
      </Tab>

      <Tab title="CLI Args">
        ```text theme={"theme":"gruvbox-light-hard"}
        newt \
          --metrics=true \
          --otlp=true \
          --metrics-admin-addr=:2112 \
          --pprof=false \
          --endpoint=https://pangolin.example.com \
          --id=saz281jfa8z37zg \
          --secret=your-secret
        ```
      </Tab>

      <Tab title="Helm Values">
        ```yaml theme={"theme":"gruvbox-light-hard"}
        global:
          metrics:
            enabled: true
            adminAddr: ":2112"
            service:
              enabled: true
            serviceMonitor:
              enabled: true
            pprofEnabled: false
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="Signals and endpoints">
    * Prometheus scrape endpoint: `/metrics` on `NEWT_ADMIN_ADDR`
    * Health endpoint: `/healthz` on the same admin server
    * OTLP metrics: enabled with `NEWT_METRICS_OTLP_ENABLED=true` or `--otlp=true`
    * OTLP traces: initialized when OTLP export is enabled
    * Profiling: `/debug/pprof/*` when `NEWT_PPROF_ENABLED=true`

    <Note>
      The Newt OTLP switch enables the OTLP telemetry pipeline. The current environment variable name contains `METRICS`, but the implementation also initializes OTLP tracing when OTLP is enabled.
    </Note>

    <Warning>
      Newt's admin server defaults to a loopback bind address. In containers or Kubernetes, set `NEWT_ADMIN_ADDR=:2112` or another non-loopback address only when you intentionally want Prometheus, pprof, or health checks to reach it.
    </Warning>
  </Accordion>

  <Accordion title="Examples">
    <Tabs>
      <Tab title="Direct Prometheus">
        ```yaml title="prometheus.yml (fragment)" theme={"theme":"gruvbox-light-hard"}
        scrape_configs:
          - job_name: newt
            static_configs:
              - targets: ["newt:2112"]
        ```
      </Tab>

      <Tab title="OTLP metrics and traces">
        ```yaml title="otel-collector.yaml" theme={"theme":"gruvbox-light-hard"}
        receivers:
          otlp:
            protocols:
              grpc:
              http:

        processors:
          batch: {}

        exporters:
          prometheusremotewrite:
            endpoint: https://mimir.example.com/api/v1/push
          otlp/tempo:
            endpoint: tempo:4317
            tls:
              insecure: true

        service:
          pipelines:
            metrics:
              receivers: [otlp]
              processors: [batch]
              exporters: [prometheusremotewrite]
            traces:
              receivers: [otlp]
              processors: [batch]
              exporters: [otlp/tempo]
        ```
      </Tab>
    </Tabs>
  </Accordion>
</AccordionGroup>

## Gerbil

Gerbil supports **metrics only**. Choose either a native Prometheus backend or an OTLP metrics backend at runtime. Those backends are mutually exclusive. Available metrics are listed in [Gerbil metrics](#gerbil-metrics).

<AccordionGroup>
  <Accordion title="Configuration">
    <Tabs>
      <Tab title="Environment Variables">
        ```text theme={"theme":"gruvbox-light-hard"}
        METRICS_ENABLED=true
        METRICS_BACKEND=prometheus
        METRICS_PATH=/metrics
        LISTEN=:3003

        # OTel mode
        OTEL_METRICS_PROTOCOL=grpc
        OTEL_METRICS_ENDPOINT=otel-collector:4317
        OTEL_METRICS_INSECURE=true
        OTEL_METRICS_EXPORT_INTERVAL=60s
        OTEL_METRICS_TIMEOUT=10s
        ```
      </Tab>

      <Tab title="CLI Args">
        ```text theme={"theme":"gruvbox-light-hard"}
        ./gerbil \
          --metrics-enabled \
          --metrics-backend=prometheus \
          --metrics-path=/metrics \
          --config=/etc/gerbil/config.json

        ./gerbil \
          --metrics-enabled \
          --metrics-backend=otel \
          --otel-metrics-protocol=grpc \
          --otel-metrics-endpoint=otel-collector:4317 \
          --otel-metrics-insecure \
          --otel-metrics-export-interval=10s \
          --otel-metrics-timeout=10s \
          --config=/etc/gerbil/config.json
        ```
      </Tab>
    </Tabs>
  </Accordion>

  <Accordion title="Signals and endpoints">
    * Prometheus metrics endpoint: `METRICS_PATH`, default `/metrics`
    * Metrics are served on Gerbil's configured HTTP listen address
    * The Docker Compose metrics example commonly scrapes `gerbil:3003`
    * Health endpoint: `/healthz`
    * OTLP metrics: enabled when `METRICS_BACKEND=otel`
    * Traces: not supported yet
    * Profiling: not supported yet
  </Accordion>

  <Accordion title="Examples">
    <Tabs>
      <Tab title="Prometheus scrape">
        ```yaml title="prometheus.yml (fragment)" theme={"theme":"gruvbox-light-hard"}
        scrape_configs:
          - job_name: gerbil
            metrics_path: /metrics
            static_configs:
              - targets: ["gerbil:3003"]
        ```

        <Note>
          Replace `gerbil:3003` with the actual Gerbil HTTP listen address in your deployment.
        </Note>
      </Tab>

      <Tab title="OTLP metrics">
        ```yaml title="otel-collector.yaml" theme={"theme":"gruvbox-light-hard"}
        receivers:
          otlp:
            protocols:
              grpc:
              http:

        processors:
          batch: {}

        exporters:
          prometheusremotewrite:
            endpoint: https://mimir.example.com/api/v1/push

        service:
          pipelines:
            metrics:
              receivers: [otlp]
              processors: [batch]
              exporters: [prometheusremotewrite]
        ```
      </Tab>
    </Tabs>
  </Accordion>
</AccordionGroup>

## Pangolin Kubernetes Controller

The controller exposes a Prometheus-compatible `/metrics` endpoint and standard health probes. It also registers additional OpenTelemetry metric instruments, but those instruments are exported on the same scrape endpoint rather than pushed through OTLP. Available metrics are listed in [Controller metrics](#pangolin-kubernetes-controller-metrics).

<Note>
  `pprof` is available only when explicitly enabled via `ENABLE_PPROF=true`.
</Note>

<AccordionGroup>
  <Accordion title="Configuration">
    <Tabs>
      <Tab title="Environment Variables">
        ```text theme={"theme":"gruvbox-light-hard"}
        METRICS_ADDR=:9090
        DISABLE_LIVEZ=false
        ENABLE_PPROF=false
        ```
      </Tab>

      <Tab title="Helm Values">
        ```yaml theme={"theme":"gruvbox-light-hard"}
        controller:
          service:
            enabled: true
            port: 9090
            portName: metrics
          monitoring:
            serviceMonitor:
              enabled: true
            podMonitor:
              enabled: false
            prometheusRule:
              enabled: false
        ```
      </Tab>
    </Tabs>

    <Warning>
      Expose the controller metrics endpoint only inside trusted networks or through Kubernetes-native monitoring resources. If TLS or auth is required, terminate it with a Service mesh, Ingress, sidecar, or platform-specific monitoring gateway unless native TLS support is verified.
    </Warning>
  </Accordion>

  <Accordion title="Endpoints">
    * `/metrics` on `METRICS_ADDR`
    * `/healthz` and `/readyz` for readiness
    * `/livez` and `/health/live` for liveness unless `DISABLE_LIVEZ=true`
    * `/debug/pprof/*` when `ENABLE_PPROF=true`
  </Accordion>

  <Accordion title="Representative metrics and Kubernetes examples">
    ```yaml title="service-and-servicemonitor.yaml" theme={"theme":"gruvbox-light-hard"}
    apiVersion: v1
    kind: Service
    metadata:
      name: pangolin-kube-controller
      labels:
        app: pangolin-kube-controller
    spec:
      selector:
        app: pangolin-kube-controller
      ports:
        - name: http-metrics
          port: 9090
          targetPort: 9090
    ---
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: pangolin-kube-controller
    spec:
      selector:
        matchLabels:
          app: pangolin-kube-controller
      endpoints:
        - port: http-metrics
          path: /metrics
          interval: 30s
    ```
  </Accordion>
</AccordionGroup>

## Alerting Examples

<Tabs>
  <Tab title="Newt">
    ```promql theme={"theme":"gruvbox-light-hard"}
    increase(newt_connection_errors_total[5m]) > 10
    ```

    ```promql theme={"theme":"gruvbox-light-hard"}
    histogram_quantile(
      0.95,
      sum(rate(newt_tunnel_latency_seconds_bucket[5m])) by (le)
    ) > 1
    ```

    ```promql theme={"theme":"gruvbox-light-hard"}
    increase(newt_tunnel_reconnects_total[10m]) > 20
    ```
  </Tab>

  <Tab title="Gerbil">
    ```promql theme={"theme":"gruvbox-light-hard"}
    max_over_time(gerbil_wg_interface_up[5m]) by (ifname) < 1
    ```

    ```promql theme={"theme":"gruvbox-light-hard"}
    sum(gerbil_wg_peers_total) > 0
    and
    sum(gerbil_wg_peer_connected) == 0
    ```
  </Tab>

  <Tab title="Controller">
    ```promql theme={"theme":"gruvbox-light-hard"}
    min_over_time(pangolin_kube_controller_ready[5m]) == 0
    ```

    ```promql theme={"theme":"gruvbox-light-hard"}
    increase(pangolin_kube_controller_reconcile_errors_total[10m]) > 5
    ```
  </Tab>
</Tabs>

<CardGroup cols={2}>
  <Card title="Community Metrics Guide" icon="chart-simple" href="/self-host/community-guides/metrics">
    Traefik and metrics collection with Prometheus and Grafana.
  </Card>

  <Card title="Newt Kubernetes Monitoring" icon="cubes" href="/self-host/manual/kubernetes/newt/configuration">
    Verified Newt chart values for metrics, Services, and ServiceMonitor resources.
  </Card>

  <Card title="Controller Monitoring Values" icon="cubes" href="/self-host/manual/kubernetes/pangolin/configuration">
    Verified chart values for controller Services, ServiceMonitor, PodMonitor, and PrometheusRule resources.
  </Card>
</CardGroup>

## Versions

| Component                      | Signal                    | Since version    |
| ------------------------------ | ------------------------- | ---------------- |
| Newt                           | Prometheus scrape metrics | `v1.6.0`         |
| Newt                           | OTLP metrics              | `v1.6.0`         |
| Newt                           | OTLP traces               | `v1.6.0`         |
| Newt                           | pprof                     | `v1.10.4`        |
| Gerbil                         | Prometheus scrape metrics | `v1.4.0`         |
| Gerbil                         | OTLP metrics              | `v1.4.0`         |
| Pangolin Kubernetes Controller | Prometheus scrape metrics | `v0.1.0-alpha.1` |

## Full Metric Reference

The full reference below is grouped by component.

* Newt: Prometheus metrics, OTLP metrics, OTLP traces
* Gerbil: Prometheus or OTLP metrics
* Pangolin Kubernetes Controller: Prometheus-native metrics and additional OTel-backed scrape metrics

<Info>
  Metric names, labels, and defaults can change between component releases.
</Info>

### Newt metrics

<Tabs>
  <Tab title="OpenTelemetry (OTel)">
    <ResponseField name="newt">
      OpenTelemetry metric instruments exposed by Newt. Expand each section to see individual metrics with labels, units, emission points, and examples.

      <Expandable title="Site & Build">
        <ResponseField name="newt_site_registrations_total" type="Counter">
          Counts Pangolin registration attempts keyed by result.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result` (`success`|`failure`), `site_id`\
            **Emission path:** `telemetry.IncSiteRegistration`\
            **Example:** `newt_site_registrations_total{result="success",site_id="abc"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_site_online" type="ObservableGauge">
          0/1 heartbeat for the active site.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `site_id`\
            **Emission path:** `state.TelemetryView` (callback)\
            **Example:** `newt_site_online{site_id="self"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_site_last_heartbeat_seconds" type="ObservableGauge">
          Seconds since last Pangolin heartbeat.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `site_id`\
            **Emission path:** `TouchHeartbeat` (callback)\
            **Example:** `newt_site_last_heartbeat_seconds{site_id="self"} 3.2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_build_info" type="ObservableGauge">
          Constant 1 with build metadata labels.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `version`, `commit`\
            **Emission path:** Build info registration\
            **Example:** `newt_build_info{version="1.2.3",commit="abc123"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_restart_count_total" type="Counter">
          Process boot indicator (increments once per process start).

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** —\
            **Emission path:** `RegisterBuildInfo`\
            **Example:** `newt_restart_count_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_cert_rotation_total" type="Counter">
          Certificate rotation events keyed by result.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result`\
            **Emission path:** `IncCertRotation`\
            **Example:** `newt_cert_rotation_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_config_reloads_total" type="Counter">
          Config reload attempts keyed by result.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result`\
            **Emission path:** `telemetry.IncConfigReload`\
            **Example:** `newt_config_reloads_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_config_apply_seconds" type="Histogram (s)">
          Duration per config-apply phase keyed by `phase` and `result`.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `phase`, `result`\
            **Emission path:** `telemetry.ObserveConfigApply`\
            **Example:** `newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Tunnel">
        <ResponseField name="newt_tunnel_sessions" type="ObservableGauge">
          Active sessions per tunnel (or collapsed).

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `site_id`, `tunnel_id`\
            **Emission path:** `RegisterStateView`\
            **Example:** `newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_tunnel_bytes_total" type="Counter (bytes)">
          Traffic per tunnel, direction, and protocol.

          <Expandable title="Details">
            **Unit:** bytes\
            **Labels:** `tunnel_id`, `direction` (`ingress`|`egress`), `protocol` (`tcp`|`udp`)\
            **Emission path:** Proxy manager\
            **Example:** `newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_tunnel_latency_seconds" type="Histogram (s)">
          RTT samples per tunnel/transport.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `tunnel_id`, `transport`\
            **Emission path:** Health checks\
            **Example:** `newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_tunnel_reconnects_total" type="Counter">
          Reconnect attempts keyed by initiator & reason.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `tunnel_id`, `initiator` (`client`|`server`), `reason`\
            **Emission path:** `telemetry.IncReconnect`\
            **Example:** `newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Connection & Auth">
        <ResponseField name="newt_connection_attempts_total" type="Counter">
          Auth/WebSocket connection attempts keyed by transport & result.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `transport`, `result`\
            **Emission path:** `telemetry.IncConnAttempt`\
            **Example:** `newt_connection_attempts_total{transport="websocket",result="failure"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_connection_errors_total" type="Counter">
          Connection errors keyed by transport and type.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `transport`, `error_type`\
            **Emission path:** `telemetry.IncConnError`\
            **Example:** `newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="WebSocket">
        <ResponseField name="newt_websocket_connect_latency_seconds" type="Histogram (s)">
          Dial latency for Pangolin WebSocket.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `result`, `transport`\
            **Emission path:** `ObserveWSConnectLatency`\
            **Example:** `newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_disconnects_total" type="Counter">
          WebSocket disconnects keyed by reason.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `reason`, `tunnel_id`\
            **Emission path:** `IncWSDisconnect`\
            **Example:** `newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_keepalive_failures_total" type="Counter">
          Ping/Pong failures observed by keepalive.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `reason` (e.g., `ping_write`, `pong_timeout`)\
            **Emission path:** `telemetry.IncWSKeepaliveFailure(ctx, "ping_write")`\
            **Example:** `newt_websocket_keepalive_failures_total{reason="ping_write"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_session_duration_seconds" type="Histogram (s)">
          Duration of established WS sessions keyed by result.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `result` (`success`|`error`)\
            **Emission path:** `telemetry.ObserveWSSessionDuration(ctx, time.Since(start).Seconds(), "error")`\
            **Example:** `newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_connected" type="ObservableGauge">
          Current WS connection state (0/1).

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** —\
            **Emission path:** `telemetry.SetWSConnectionState(true|false)`\
            **Example:** `newt_websocket_connected 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_reconnects_total" type="Counter">
          WebSocket reconnect attempts keyed by reason.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `reason`\
            **Emission path:** `telemetry.IncWSReconnect(ctx, "ping_write")`\
            **Example:** `newt_websocket_reconnects_total{reason="ping_write"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_messages_total" type="Counter">
          In/out WS messages keyed by direction & type.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `direction` (`in`|`out`), `msg_type` (`ping`|`pong`|`text`|...)\
            **Emission path:** `IncWSMessage`\
            **Example:** `newt_websocket_messages_total{direction="out",msg_type="ping"} 4`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Proxy">
        <ResponseField name="newt_proxy_active_connections" type="ObservableGauge">
          Active TCP/UDP proxy connections per tunnel/protocol.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `protocol`, `tunnel_id`\
            **Emission path:** Proxy callback\
            **Example:** `newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_buffer_bytes" type="ObservableGauge (bytes)">
          Proxy buffer pool size.

          <Expandable title="Details">
            **Unit:** bytes\
            **Labels:** `protocol`, `tunnel_id`\
            **Emission path:** Proxy callback\
            **Example:** `newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_async_backlog_bytes" type="ObservableGauge (bytes)">
          Unflushed async byte backlog.

          <Expandable title="Details">
            **Unit:** bytes\
            **Labels:** `protocol`, `tunnel_id`\
            **Emission path:** Proxy callback\
            **Example:** `newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_drops_total" type="Counter">
          Proxy write drops keyed by protocol/tunnel.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `protocol`, `tunnel_id`\
            **Emission path:** `IncProxyDrops`\
            **Example:** `newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_accept_total" type="Counter">
          Proxy accept events keyed by result/reason.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `tunnel_id`, `protocol`, `result`, `reason`\
            **Emission path:** `telemetry.IncProxyAccept(ctx, tunnelID, "tcp", "failure", "timeout")`\
            **Example:** `newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_connections_total" type="Counter">
          Lifecycle events (opened/closed) per connection.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `tunnel_id`, `protocol`, `event` (`opened`|`closed`)\
            **Emission path:** `telemetry.IncProxyConnectionEvent(ctx, tunnelID, "tcp", telemetry.ProxyConnectionOpened)`\
            **Example:** `newt_proxy_connections_total{protocol="tcp",event="opened"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_connection_duration_seconds" type="Histogram (s)">
          Duration of completed proxy connections.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `tunnel_id`, `protocol`, `result`\
            **Emission path:** `telemetry.ObserveProxyConnectionDuration(ctx, tunnelID, "tcp", "success", seconds)`\
            **Example:** `newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3`
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Tab>

  <Tab title="Prometheus">
    <ResponseField name="newt">
      Prometheus-style series for the same Newt metrics. Names, labels, and examples mirror the OTel tab.

      <Expandable title="Site & Build">
        <ResponseField name="newt_site_registrations_total" type="counter">
          Counts Pangolin registration attempts keyed by result.

          <Expandable title="Details">
            **Labels:** `result`, `site_id` • **Unit:** 1 • **Path:** `telemetry.IncSiteRegistration`\
            **Example:** `newt_site_registrations_total{result="success",site_id="abc"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_site_online" type="gauge">
          0/1 heartbeat for the active site.

          <Expandable title="Details">
            **Labels:** `site_id` • **Unit:** 1 • **Path:** `state.TelemetryView`\
            **Example:** `newt_site_online{site_id="self"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_site_last_heartbeat_seconds" type="gauge">
          Seconds since last Pangolin heartbeat.

          <Expandable title="Details">
            **Labels:** `site_id` • **Unit:** seconds • **Path:** `TouchHeartbeat`\
            **Example:** `newt_site_last_heartbeat_seconds{site_id="self"} 3.2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_build_info" type="gauge">
          Constant 1 with build metadata labels.

          <Expandable title="Details">
            **Labels:** `version`, `commit` • **Unit:** 1 • **Path:** Build info registration\
            **Example:** `newt_build_info{version="1.2.3",commit="abc123"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_restart_count_total" type="counter">
          Process boot indicator (increments once).

          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `RegisterBuildInfo`\
            **Example:** `newt_restart_count_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_cert_rotation_total" type="counter">
          Certificate rotation events keyed by result.

          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `IncCertRotation`\
            **Example:** `newt_cert_rotation_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_config_reloads_total" type="counter">
          Config reload attempts keyed by result.

          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `telemetry.IncConfigReload`\
            **Example:** `newt_config_reloads_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_config_apply_seconds" type="histogram">
          Duration per config-apply phase & result.

          <Expandable title="Details">
            **Labels:** `phase`, `result` • **Unit:** seconds • **Path:** `telemetry.ObserveConfigApply`\
            **Example:** `newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Tunnel">
        <ResponseField name="newt_tunnel_sessions" type="gauge">
          Active sessions per tunnel (or collapsed).

          <Expandable title="Details">
            **Labels:** `site_id`, `tunnel_id` • **Unit:** 1 • **Path:** `RegisterStateView`\
            **Example:** `newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_tunnel_bytes_total" type="counter">
          Traffic per tunnel/direction/protocol.

          <Expandable title="Details">
            **Labels:** `tunnel_id`, `direction`, `protocol` • **Unit:** bytes • **Path:** Proxy manager\
            **Example:** `newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_tunnel_latency_seconds" type="histogram">
          RTT samples per tunnel/transport.

          <Expandable title="Details">
            **Labels:** `tunnel_id`, `transport` • **Unit:** seconds • **Path:** Health checks\
            **Example:** `newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_tunnel_reconnects_total" type="counter">
          Reconnect attempts by initiator & reason.

          <Expandable title="Details">
            **Labels:** `tunnel_id`, `initiator`, `reason` • **Unit:** 1 • **Path:** `telemetry.IncReconnect`\
            **Example:** `newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Connection & Auth">
        <ResponseField name="newt_connection_attempts_total" type="counter">
          Auth/WebSocket attempts by transport & result.

          <Expandable title="Details">
            **Labels:** `transport`, `result` • **Unit:** 1 • **Path:** `telemetry.IncConnAttempt`\
            **Example:** `newt_connection_attempts_total{transport="websocket",result="failure"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_connection_errors_total" type="counter">
          Connection errors by transport and type.

          <Expandable title="Details">
            **Labels:** `transport`, `error_type` • **Unit:** 1 • **Path:** `telemetry.IncConnError`\
            **Example:** `newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="WebSocket">
        <ResponseField name="newt_websocket_connect_latency_seconds" type="histogram">
          Dial latency for Pangolin WebSocket.

          <Expandable title="Details">
            **Labels:** `result`, `transport` • **Unit:** seconds • **Path:** `ObserveWSConnectLatency`\
            **Example:** `newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_disconnects_total" type="counter">
          WS disconnects by reason.

          <Expandable title="Details">
            **Labels:** `reason`, `tunnel_id` • **Unit:** 1 • **Path:** `IncWSDisconnect`\
            **Example:** `newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_keepalive_failures_total" type="counter">
          Keepalive Ping/Pong failures.

          <Expandable title="Details">
            **Labels:** `reason` • **Unit:** 1 • **Path:** `telemetry.IncWSKeepaliveFailure(ctx, "ping_write")`\
            **Example:** `newt_websocket_keepalive_failures_total{reason="ping_write"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_session_duration_seconds" type="histogram">
          Duration of established WebSocket sessions by result.

          <Expandable title="Details">
            **Labels:** `result` • **Unit:** seconds • **Path:** `telemetry.ObserveWSSessionDuration(...)`\
            **Example:** `newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_connected" type="gauge">
          Current WS connection status (0/1).

          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `telemetry.SetWSConnectionState(true|false)`\
            **Example:** `newt_websocket_connected 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_reconnects_total" type="counter">
          Reconnect attempts by reason.

          <Expandable title="Details">
            **Labels:** `reason` • **Unit:** 1 • **Path:** `telemetry.IncWSReconnect(ctx, "ping_write")`\
            **Example:** `newt_websocket_reconnects_total{reason="ping_write"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_websocket_messages_total" type="counter">
          In/out WS messages by direction & type.

          <Expandable title="Details">
            **Labels:** `direction`, `msg_type` • **Unit:** 1 • **Path:** `IncWSMessage`\
            **Example:** `newt_websocket_messages_total{direction="out",msg_type="ping"} 4`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Proxy">
        <ResponseField name="newt_proxy_active_connections" type="gauge">
          Active TCP/UDP proxy connections per tunnel/protocol.

          <Expandable title="Details">
            **Labels:** `protocol`, `tunnel_id` • **Unit:** 1 • **Path:** Proxy callback\
            **Example:** `newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_buffer_bytes" type="gauge">
          Proxy buffer pool size.

          <Expandable title="Details">
            **Labels:** `protocol`, `tunnel_id` • **Unit:** bytes • **Path:** Proxy callback\
            **Example:** `newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_async_backlog_bytes" type="gauge">
          Unflushed async byte backlog.

          <Expandable title="Details">
            **Labels:** `protocol`, `tunnel_id` • **Unit:** bytes • **Path:** Proxy callback\
            **Example:** `newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_drops_total" type="counter">
          Proxy write drops per protocol/tunnel.

          <Expandable title="Details">
            **Labels:** `protocol`, `tunnel_id` • **Unit:** 1 • **Path:** `IncProxyDrops`\
            **Example:** `newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_accept_total" type="counter">
          Proxy accept events by result/reason.

          <Expandable title="Details">
            **Labels:** `tunnel_id`, `protocol`, `result`, `reason` • **Unit:** 1 • **Path:** `telemetry.IncProxyAccept(...)`\
            **Example:** `newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_connections_total" type="counter">
          Connection lifecycle events (opened/closed).

          <Expandable title="Details">
            **Labels:** `tunnel_id`, `protocol`, `event` • **Unit:** 1 • **Path:** `telemetry.IncProxyConnectionEvent(...)`\
            **Example:** `newt_proxy_connections_total{protocol="tcp",event="opened"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="newt_proxy_connection_duration_seconds" type="histogram">
          Duration of completed proxy connections.

          <Expandable title="Details">
            **Labels:** `tunnel_id`, `protocol`, `result` • **Unit:** seconds • **Path:** `telemetry.ObserveProxyConnectionDuration(...)`\
            **Example:** `newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3`
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Tab>
</Tabs>

### Gerbil metrics

<Tabs>
  <Tab title="OpenTelemetry (OTel)">
    <ResponseField name="gerbil">
      OpenTelemetry metric instruments exposed by Gerbil. Gerbil supports exactly one metrics backend at runtime: `prometheus`, `otel`, or `none`.

      <Note>
        In `otel` mode, Gerbil pushes metrics to an OTLP collector. The `/metrics` endpoint is not exposed in this mode.
      </Note>

      <Expandable title="WireGuard">
        <ResponseField name="gerbil_wg_interface_up" type="Int64Gauge">
          Operational state of a WireGuard interface.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`, `instance`\
            **Emission path:** `metrics.RecordInterfaceUp`\
            **Example:** `gerbil_wg_interface_up{ifname="wg0",instance="gerbil-1"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_peers_total" type="UpDownCounter">
          Number of configured peers per interface.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`\
            **Emission path:** `metrics.RecordPeersTotal`\
            **Example:** `gerbil_wg_peers_total{ifname="wg0"} 5`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_peer_connected" type="Int64Gauge">
          Current peer connection state.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`, `peer`\
            **Emission path:** `metrics.RecordPeerConnected`\
            **Example:** `gerbil_wg_peer_connected{ifname="wg0",peer="abc"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_allowed_ips_count" type="UpDownCounter">
          Number of allowed IPs configured per peer.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`, `peer`\
            **Emission path:** `metrics.RecordAllowedIPsCount`\
            **Example:** `gerbil_allowed_ips_count{ifname="wg0",peer="abc"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_key_rotation_total" type="Counter">
          Key rotation events.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`, `reason`\
            **Emission path:** `metrics.RecordKeyRotation`\
            **Example:** `gerbil_key_rotation_total{ifname="wg0",reason="scheduled"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_handshakes_total" type="Counter">
          WireGuard handshake attempts keyed by result.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`, `peer`, `result`\
            **Emission path:** `metrics.RecordHandshake`\
            **Example:** `gerbil_wg_handshakes_total{ifname="wg0",peer="abc",result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_handshake_latency_seconds" type="Histogram (s)">
          Distribution of WireGuard handshake latencies.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `ifname`, `peer`\
            **Emission path:** `metrics.RecordHandshakeLatency`\
            **Example:** `gerbil_wg_handshake_latency_seconds_bucket{ifname="wg0",peer="abc",le="0.1"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_peer_rtt_seconds" type="Histogram (s)">
          Observed peer round-trip time.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `ifname`, `peer`\
            **Emission path:** `metrics.RecordPeerRTT`\
            **Example:** `gerbil_wg_peer_rtt_seconds_bucket{ifname="wg0",peer="abc",le="0.05"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_bytes_received_total" type="Counter (bytes)">
          Bytes received from a WireGuard peer.

          <Expandable title="Details">
            **Unit:** bytes\
            **Labels:** `ifname`, `peer`\
            **Emission path:** `metrics.RecordBytesReceived`\
            **Example:** `gerbil_wg_bytes_received_total{ifname="wg0",peer="abc"} 8192`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_bytes_transmitted_total" type="Counter (bytes)">
          Bytes transmitted to a WireGuard peer.

          <Expandable title="Details">
            **Unit:** bytes\
            **Labels:** `ifname`, `peer`\
            **Emission path:** `metrics.RecordBytesTransmitted`\
            **Example:** `gerbil_wg_bytes_transmitted_total{ifname="wg0",peer="abc"} 16384`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Relay">
        <ResponseField name="gerbil_active_sessions" type="UpDownCounter">
          Number of active UDP relay sessions.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`\
            **Emission path:** `metrics.RecordActiveSession` / `metrics.RecordSession`\
            **Example:** `gerbil_active_sessions{ifname="wg0"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_udp_packets_total" type="Counter">
          UDP packets processed by relay workers.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`, `type`, `direction`\
            **Emission path:** `metrics.RecordUDPPacket`\
            **Example:** `gerbil_udp_packets_total{ifname="wg0",type="data",direction="rx"} 42`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_udp_packet_size_bytes" type="Histogram (bytes)">
          Size distribution of packets forwarded through the relay.

          <Expandable title="Details">
            **Unit:** bytes\
            **Labels:** `ifname`, `type`\
            **Emission path:** `metrics.RecordUDPPacketSize`\
            **Example:** `gerbil_udp_packet_size_bytes_bucket{ifname="wg0",type="data",le="1024"} 7`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_hole_punch_events_total" type="Counter">
          Hole punch messages processed by result.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`, `result`\
            **Emission path:** `metrics.RecordHolePunchEvent`\
            **Example:** `gerbil_hole_punch_events_total{ifname="wg0",result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_mapping_active" type="UpDownCounter">
          Active proxy mappings.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`\
            **Emission path:** `metrics.RecordProxyMapping`\
            **Example:** `gerbil_proxy_mapping_active{ifname="wg0"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_session_rebuilt_total" type="Counter">
          Sessions rebuilt from communication patterns.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`\
            **Emission path:** `metrics.RecordSessionRebuilt`\
            **Example:** `gerbil_session_rebuilt_total{ifname="wg0"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_comm_pattern_active" type="UpDownCounter">
          Active communication patterns.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`\
            **Emission path:** `metrics.RecordCommPattern`\
            **Example:** `gerbil_comm_pattern_active{ifname="wg0"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_cleanup_removed_total" type="Counter">
          Items removed by cleanup routines.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`, `component`\
            **Emission path:** `metrics.RecordProxyCleanupRemoved`\
            **Example:** `gerbil_proxy_cleanup_removed_total{ifname="wg0",component="sessions"} 5`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_connection_errors_total" type="Counter">
          Proxy connection errors.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`, `error_type`\
            **Emission path:** `metrics.RecordProxyConnectionError`\
            **Example:** `gerbil_proxy_connection_errors_total{ifname="wg0",error_type="timeout"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_initial_mappings" type="Int64Gauge">
          Initial proxy mappings loaded.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`\
            **Emission path:** `metrics.RecordProxyInitialMappings`\
            **Example:** `gerbil_proxy_initial_mappings{ifname="wg0"} 8`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_mapping_updates_total" type="Counter">
          Proxy mapping updates.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`\
            **Emission path:** `metrics.RecordProxyMappingUpdate`\
            **Example:** `gerbil_proxy_mapping_updates_total{ifname="wg0"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_idle_cleanup_duration_seconds" type="Histogram (s)">
          Duration of idle cleanup cycles.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `ifname`, `component`\
            **Emission path:** `metrics.RecordProxyIdleCleanupDuration`\
            **Example:** `gerbil_proxy_idle_cleanup_duration_seconds_bucket{ifname="wg0",component="sessions",le="0.1"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="SNI Proxy">
        <ResponseField name="gerbil_active_proxy_connections" type="UpDownCounter">
          Active SNI proxy connections.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** —\
            **Emission path:** `metrics.RecordActiveProxyConnection`\
            **Example:** `gerbil_active_proxy_connections 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_route_lookups_total" type="Counter">
          Route lookups keyed by result.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result`\
            **Emission path:** `metrics.RecordProxyRouteLookup`\
            **Example:** `gerbil_proxy_route_lookups_total{result="hit"} 6`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_tls_handshake_seconds" type="Histogram (s)">
          TLS handshake duration for the SNI proxy.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** —\
            **Emission path:** `metrics.RecordProxyTLSHandshake`\
            **Example:** `gerbil_proxy_tls_handshake_seconds_bucket{le="0.1"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_bytes_transmitted_total" type="Counter (bytes)">
          Bytes sent or received by the SNI proxy.

          <Expandable title="Details">
            **Unit:** bytes\
            **Labels:** `direction`\
            **Emission path:** `metrics.RecordProxyBytesTransmitted`\
            **Example:** `gerbil_proxy_bytes_transmitted_total{direction="egress"} 16384`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_connections_total" type="Counter">
          Connections processed by the SNI proxy.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result`\
            **Emission path:** `metrics.RecordSNIConnection`\
            **Example:** `gerbil_sni_connections_total{result="success"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_connection_duration_seconds" type="Histogram (s)">
          Lifetime distribution of proxied TLS connections.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** —\
            **Emission path:** `metrics.RecordSNIConnectionDuration`\
            **Example:** `gerbil_sni_connection_duration_seconds_bucket{le="10"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_active_connections" type="UpDownCounter">
          Active SNI tunnels.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** —\
            **Emission path:** `metrics.RecordSNIActiveConnection`\
            **Example:** `gerbil_sni_active_connections 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_route_cache_hits_total" type="Counter">
          SNI route cache hits and misses.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result`\
            **Emission path:** `metrics.RecordSNIRouteCacheHit`\
            **Example:** `gerbil_sni_route_cache_hits_total{result="hit"} 10`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_route_api_requests_total" type="Counter">
          SNI route API requests.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result`\
            **Emission path:** `metrics.RecordSNIRouteAPIRequest`\
            **Example:** `gerbil_sni_route_api_requests_total{result="success"} 5`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_route_api_latency_seconds" type="Histogram (s)">
          Route API call latency.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** —\
            **Emission path:** `metrics.RecordSNIRouteAPILatency`\
            **Example:** `gerbil_sni_route_api_latency_seconds_bucket{le="0.25"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_local_override_total" type="Counter">
          Routes using local overrides.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `hit`\
            **Emission path:** `metrics.RecordSNILocalOverride`\
            **Example:** `gerbil_sni_local_override_total{hit="true"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_trusted_proxy_events_total" type="Counter">
          PROXY protocol events.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `event`\
            **Emission path:** `metrics.RecordSNITrustedProxyEvent`\
            **Example:** `gerbil_sni_trusted_proxy_events_total{event="parsed"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_proxy_protocol_parse_errors_total" type="Counter">
          PROXY protocol parse failures.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** —\
            **Emission path:** `metrics.RecordSNIProxyProtocolParseError`\
            **Example:** `gerbil_sni_proxy_protocol_parse_errors_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_data_bytes_total" type="Counter (bytes)">
          Bytes proxied through SNI tunnels.

          <Expandable title="Details">
            **Unit:** bytes\
            **Labels:** `direction`\
            **Emission path:** `metrics.RecordSNIDataBytes`\
            **Example:** `gerbil_sni_data_bytes_total{direction="ingress"} 4096`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_tunnel_terminations_total" type="Counter">
          SNI tunnel terminations keyed by reason.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `reason`\
            **Emission path:** `metrics.RecordSNITunnelTermination`\
            **Example:** `gerbil_sni_tunnel_terminations_total{reason="client_close"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="HTTP API & Peer Management">
        <ResponseField name="gerbil_http_requests_total" type="Counter">
          HTTP requests to the management API.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `endpoint`, `method`, `status_code`\
            **Emission path:** `metrics.RecordHTTPRequest`\
            **Example:** `gerbil_http_requests_total{endpoint="/peer",method="POST",status_code="200"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_http_request_duration_seconds" type="Histogram (s)">
          HTTP request handling time.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `endpoint`, `method`\
            **Emission path:** `metrics.RecordHTTPRequestDuration`\
            **Example:** `gerbil_http_request_duration_seconds_bucket{endpoint="/peer",method="POST",le="0.1"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_peer_operations_total" type="Counter">
          Peer lifecycle operations.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `operation`, `result`\
            **Emission path:** `metrics.RecordPeerOperation`\
            **Example:** `gerbil_peer_operations_total{operation="add",result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_mapping_update_requests_total" type="Counter">
          Proxy mapping update API calls.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result`\
            **Emission path:** `metrics.RecordProxyMappingUpdateRequest`\
            **Example:** `gerbil_proxy_mapping_update_requests_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_destinations_update_requests_total" type="Counter">
          Destination update API calls.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result`\
            **Emission path:** `metrics.RecordDestinationsUpdateRequest`\
            **Example:** `gerbil_destinations_update_requests_total{result="success"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Remote Config & Reporting">
        <ResponseField name="gerbil_remote_config_fetches_total" type="Counter">
          Remote configuration fetch attempts.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result`\
            **Emission path:** `metrics.RecordRemoteConfigFetch`\
            **Example:** `gerbil_remote_config_fetches_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_bandwidth_reports_total" type="Counter">
          Bandwidth report transmissions.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result`\
            **Emission path:** `metrics.RecordBandwidthReport`\
            **Example:** `gerbil_bandwidth_reports_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_peer_bandwidth_bytes_total" type="Counter (bytes)">
          Bytes per peer tracked by bandwidth calculation.

          <Expandable title="Details">
            **Unit:** bytes\
            **Labels:** `peer`, `direction`\
            **Emission path:** `metrics.RecordPeerBandwidthBytes`\
            **Example:** `gerbil_peer_bandwidth_bytes_total{peer="abc",direction="rx"} 8192`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="System & Operations">
        <ResponseField name="gerbil_netlink_events_total" type="Counter">
          Netlink events processed.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `event_type`\
            **Emission path:** `metrics.RecordNetlinkEvent`\
            **Example:** `gerbil_netlink_events_total{event_type="link_up"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_netlink_errors_total" type="Counter">
          Netlink or kernel errors.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `component`, `error_type`\
            **Emission path:** `metrics.RecordNetlinkError`\
            **Example:** `gerbil_netlink_errors_total{component="wg",error_type="permission"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sync_duration_seconds" type="Histogram (s)">
          Duration of reconciliation or sync loops.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `component`\
            **Emission path:** `metrics.RecordSyncDuration`\
            **Example:** `gerbil_sync_duration_seconds_bucket{component="remote_config",le="0.5"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_workqueue_depth" type="UpDownCounter">
          Current length of internal work queues.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `queue`\
            **Emission path:** `metrics.RecordWorkqueueDepth`\
            **Example:** `gerbil_workqueue_depth{queue="relay"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_kernel_module_loads_total" type="Counter">
          Kernel module load attempts.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result`\
            **Emission path:** `metrics.RecordKernelModuleLoad`\
            **Example:** `gerbil_kernel_module_loads_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_firewall_rules_applied_total" type="Counter">
          Firewall rules applied.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result`, `chain`\
            **Emission path:** `metrics.RecordFirewallRuleApplied`\
            **Example:** `gerbil_firewall_rules_applied_total{result="success",chain="FORWARD"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_config_reloads_total" type="Counter">
          Configuration reloads.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `result`\
            **Emission path:** `metrics.RecordConfigReload`\
            **Example:** `gerbil_config_reloads_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_restart_total" type="Counter">
          Process restart count.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** —\
            **Emission path:** `metrics.RecordRestart`\
            **Example:** `gerbil_restart_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_auth_failures_total" type="Counter">
          Authentication or peer validation failures.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `peer`, `reason`\
            **Emission path:** `metrics.RecordAuthFailure`\
            **Example:** `gerbil_auth_failures_total{peer="abc",reason="invalid_key"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_acl_denied_total" type="Counter">
          Access-control denied events.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `ifname`, `peer`, `policy`\
            **Emission path:** `metrics.RecordACLDenied`\
            **Example:** `gerbil_acl_denied_total{ifname="wg0",peer="abc",policy="deny"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_certificate_expiry_days" type="Float64Gauge">
          Days until certificate expiry.

          <Expandable title="Details">
            **Unit:** days\
            **Labels:** `cert_name`, `ifname`\
            **Emission path:** `metrics.RecordCertificateExpiry`\
            **Example:** `gerbil_certificate_expiry_days{cert_name="server",ifname="wg0"} 42`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_memory_spike_total" type="Counter">
          Memory spikes detected by severity.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `severity`\
            **Emission path:** `metrics.RecordMemorySpike`\
            **Example:** `gerbil_memory_spike_total{severity="warning"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_heap_profiles_written_total" type="Counter">
          Heap profile files generated.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** —\
            **Emission path:** `metrics.RecordHeapProfileWritten`\
            **Example:** `gerbil_heap_profiles_written_total 1`
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Tab>

  <Tab title="Prometheus">
    <ResponseField name="gerbil">
      Prometheus-style series for the same Gerbil metrics. Gerbil exposes these only when `METRICS_BACKEND=prometheus`.

      <Note>
        In Prometheus mode, Gerbil registers a native Prometheus client and exposes the configured metrics path, default `/metrics`.
        In OTel mode, `/metrics` is not exposed.
      </Note>

      <Expandable title="WireGuard">
        <ResponseField name="gerbil_wg_interface_up" type="gauge">
          Operational state of a WireGuard interface.

          <Expandable title="Details">
            **Labels:** `ifname`, `instance` • **Unit:** 1 • **Path:** `metrics.RecordInterfaceUp`\
            **Example:** `gerbil_wg_interface_up{ifname="wg0",instance="gerbil-1"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_peers_total" type="gauge">
          Number of configured peers per interface.

          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordPeersTotal`\
            **Example:** `gerbil_wg_peers_total{ifname="wg0"} 5`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_peer_connected" type="gauge">
          Current peer connection state.

          <Expandable title="Details">
            **Labels:** `ifname`, `peer` • **Unit:** 1 • **Path:** `metrics.RecordPeerConnected`\
            **Example:** `gerbil_wg_peer_connected{ifname="wg0",peer="abc"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_allowed_ips_count" type="gauge">
          Number of allowed IPs configured per peer.

          <Expandable title="Details">
            **Labels:** `ifname`, `peer` • **Unit:** 1 • **Path:** `metrics.RecordAllowedIPsCount`\
            **Example:** `gerbil_allowed_ips_count{ifname="wg0",peer="abc"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_key_rotation_total" type="counter">
          Key rotation events.

          <Expandable title="Details">
            **Labels:** `ifname`, `reason` • **Unit:** 1 • **Path:** `metrics.RecordKeyRotation`\
            **Example:** `gerbil_key_rotation_total{ifname="wg0",reason="scheduled"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_handshakes_total" type="counter">
          WireGuard handshake attempts keyed by result.

          <Expandable title="Details">
            **Labels:** `ifname`, `peer`, `result` • **Unit:** 1 • **Path:** `metrics.RecordHandshake`\
            **Example:** `gerbil_wg_handshakes_total{ifname="wg0",peer="abc",result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_handshake_latency_seconds" type="histogram">
          Distribution of WireGuard handshake latencies.

          <Expandable title="Details">
            **Labels:** `ifname`, `peer` • **Unit:** seconds • **Path:** `metrics.RecordHandshakeLatency`\
            **Example:** `gerbil_wg_handshake_latency_seconds_bucket{ifname="wg0",peer="abc",le="0.1"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_peer_rtt_seconds" type="histogram">
          Observed peer round-trip time.

          <Expandable title="Details">
            **Labels:** `ifname`, `peer` • **Unit:** seconds • **Path:** `metrics.RecordPeerRTT`\
            **Example:** `gerbil_wg_peer_rtt_seconds_bucket{ifname="wg0",peer="abc",le="0.05"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_bytes_received_total" type="counter">
          Bytes received from a WireGuard peer.

          <Expandable title="Details">
            **Labels:** `ifname`, `peer` • **Unit:** bytes • **Path:** `metrics.RecordBytesReceived`\
            **Example:** `gerbil_wg_bytes_received_total{ifname="wg0",peer="abc"} 8192`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_wg_bytes_transmitted_total" type="counter">
          Bytes transmitted to a WireGuard peer.

          <Expandable title="Details">
            **Labels:** `ifname`, `peer` • **Unit:** bytes • **Path:** `metrics.RecordBytesTransmitted`\
            **Example:** `gerbil_wg_bytes_transmitted_total{ifname="wg0",peer="abc"} 16384`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Relay">
        <ResponseField name="gerbil_active_sessions" type="gauge">
          Active UDP relay sessions.

          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordActiveSession` / `metrics.RecordSession`\
            **Example:** `gerbil_active_sessions{ifname="wg0"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_udp_packets_total" type="counter">
          UDP packets processed by relay workers.

          <Expandable title="Details">
            **Labels:** `ifname`, `type`, `direction` • **Unit:** 1 • **Path:** `metrics.RecordUDPPacket`\
            **Example:** `gerbil_udp_packets_total{ifname="wg0",type="data",direction="rx"} 42`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_udp_packet_size_bytes" type="histogram">
          Size distribution of packets forwarded through the relay.

          <Expandable title="Details">
            **Labels:** `ifname`, `type` • **Unit:** bytes • **Path:** `metrics.RecordUDPPacketSize`\
            **Example:** `gerbil_udp_packet_size_bytes_bucket{ifname="wg0",type="data",le="1024"} 7`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_hole_punch_events_total" type="counter">
          Hole punch messages processed by result.

          <Expandable title="Details">
            **Labels:** `ifname`, `result` • **Unit:** 1 • **Path:** `metrics.RecordHolePunchEvent`\
            **Example:** `gerbil_hole_punch_events_total{ifname="wg0",result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_mapping_active" type="gauge">
          Active proxy mappings.

          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordProxyMapping`\
            **Example:** `gerbil_proxy_mapping_active{ifname="wg0"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_session_rebuilt_total" type="counter">
          Sessions rebuilt from communication patterns.

          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordSessionRebuilt`\
            **Example:** `gerbil_session_rebuilt_total{ifname="wg0"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_comm_pattern_active" type="gauge">
          Active communication patterns.

          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordCommPattern`\
            **Example:** `gerbil_comm_pattern_active{ifname="wg0"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_cleanup_removed_total" type="counter">
          Items removed by cleanup routines.

          <Expandable title="Details">
            **Labels:** `ifname`, `component` • **Unit:** 1 • **Path:** `metrics.RecordProxyCleanupRemoved`\
            **Example:** `gerbil_proxy_cleanup_removed_total{ifname="wg0",component="sessions"} 5`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_connection_errors_total" type="counter">
          Proxy connection errors.

          <Expandable title="Details">
            **Labels:** `ifname`, `error_type` • **Unit:** 1 • **Path:** `metrics.RecordProxyConnectionError`\
            **Example:** `gerbil_proxy_connection_errors_total{ifname="wg0",error_type="timeout"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_initial_mappings" type="gauge">
          Initial proxy mappings loaded.

          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordProxyInitialMappings`\
            **Example:** `gerbil_proxy_initial_mappings{ifname="wg0"} 8`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_mapping_updates_total" type="counter">
          Proxy mapping updates.

          <Expandable title="Details">
            **Labels:** `ifname` • **Unit:** 1 • **Path:** `metrics.RecordProxyMappingUpdate`\
            **Example:** `gerbil_proxy_mapping_updates_total{ifname="wg0"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_idle_cleanup_duration_seconds" type="histogram">
          Duration of idle cleanup cycles.

          <Expandable title="Details">
            **Labels:** `ifname`, `component` • **Unit:** seconds • **Path:** `metrics.RecordProxyIdleCleanupDuration`\
            **Example:** `gerbil_proxy_idle_cleanup_duration_seconds_bucket{ifname="wg0",component="sessions",le="0.1"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="SNI Proxy">
        <ResponseField name="gerbil_active_proxy_connections" type="gauge">
          Active SNI proxy connections.

          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordActiveProxyConnection`\
            **Example:** `gerbil_active_proxy_connections 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_route_lookups_total" type="counter">
          Route lookups keyed by result.

          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordProxyRouteLookup`\
            **Example:** `gerbil_proxy_route_lookups_total{result="hit"} 6`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_tls_handshake_seconds" type="histogram">
          TLS handshake duration for the SNI proxy.

          <Expandable title="Details">
            **Labels:** — • **Unit:** seconds • **Path:** `metrics.RecordProxyTLSHandshake`\
            **Example:** `gerbil_proxy_tls_handshake_seconds_bucket{le="0.1"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_bytes_transmitted_total" type="counter">
          Bytes sent or received by the SNI proxy.

          <Expandable title="Details">
            **Labels:** `direction` • **Unit:** bytes • **Path:** `metrics.RecordProxyBytesTransmitted`\
            **Example:** `gerbil_proxy_bytes_transmitted_total{direction="egress"} 16384`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_connections_total" type="counter">
          Connections processed by the SNI proxy.

          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordSNIConnection`\
            **Example:** `gerbil_sni_connections_total{result="success"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_connection_duration_seconds" type="histogram">
          Lifetime distribution of proxied TLS connections.

          <Expandable title="Details">
            **Labels:** — • **Unit:** seconds • **Path:** `metrics.RecordSNIConnectionDuration`\
            **Example:** `gerbil_sni_connection_duration_seconds_bucket{le="10"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_active_connections" type="gauge">
          Active SNI tunnels.

          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordSNIActiveConnection`\
            **Example:** `gerbil_sni_active_connections 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_route_cache_hits_total" type="counter">
          SNI route cache hits and misses.

          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordSNIRouteCacheHit`\
            **Example:** `gerbil_sni_route_cache_hits_total{result="hit"} 10`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_route_api_requests_total" type="counter">
          SNI route API requests.

          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordSNIRouteAPIRequest`\
            **Example:** `gerbil_sni_route_api_requests_total{result="success"} 5`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_route_api_latency_seconds" type="histogram">
          Route API call latency.

          <Expandable title="Details">
            **Labels:** — • **Unit:** seconds • **Path:** `metrics.RecordSNIRouteAPILatency`\
            **Example:** `gerbil_sni_route_api_latency_seconds_bucket{le="0.25"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_local_override_total" type="counter">
          Routes using local overrides.

          <Expandable title="Details">
            **Labels:** `hit` • **Unit:** 1 • **Path:** `metrics.RecordSNILocalOverride`\
            **Example:** `gerbil_sni_local_override_total{hit="true"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_trusted_proxy_events_total" type="counter">
          PROXY protocol events.

          <Expandable title="Details">
            **Labels:** `event` • **Unit:** 1 • **Path:** `metrics.RecordSNITrustedProxyEvent`\
            **Example:** `gerbil_sni_trusted_proxy_events_total{event="parsed"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_proxy_protocol_parse_errors_total" type="counter">
          PROXY protocol parse failures.

          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordSNIProxyProtocolParseError`\
            **Example:** `gerbil_sni_proxy_protocol_parse_errors_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_data_bytes_total" type="counter">
          Bytes proxied through SNI tunnels.

          <Expandable title="Details">
            **Labels:** `direction` • **Unit:** bytes • **Path:** `metrics.RecordSNIDataBytes`\
            **Example:** `gerbil_sni_data_bytes_total{direction="ingress"} 4096`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sni_tunnel_terminations_total" type="counter">
          SNI tunnel terminations keyed by reason.

          <Expandable title="Details">
            **Labels:** `reason` • **Unit:** 1 • **Path:** `metrics.RecordSNITunnelTermination`\
            **Example:** `gerbil_sni_tunnel_terminations_total{reason="client_close"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="HTTP API & Peer Management">
        <ResponseField name="gerbil_http_requests_total" type="counter">
          HTTP requests to the management API.

          <Expandable title="Details">
            **Labels:** `endpoint`, `method`, `status_code` • **Unit:** 1 • **Path:** `metrics.RecordHTTPRequest`\
            **Example:** `gerbil_http_requests_total{endpoint="/peer",method="POST",status_code="200"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_http_request_duration_seconds" type="histogram">
          HTTP request handling time.

          <Expandable title="Details">
            **Labels:** `endpoint`, `method` • **Unit:** seconds • **Path:** `metrics.RecordHTTPRequestDuration`\
            **Example:** `gerbil_http_request_duration_seconds_bucket{endpoint="/peer",method="POST",le="0.1"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_peer_operations_total" type="counter">
          Peer lifecycle operations.

          <Expandable title="Details">
            **Labels:** `operation`, `result` • **Unit:** 1 • **Path:** `metrics.RecordPeerOperation`\
            **Example:** `gerbil_peer_operations_total{operation="add",result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_proxy_mapping_update_requests_total" type="counter">
          Proxy mapping update API calls.

          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordProxyMappingUpdateRequest`\
            **Example:** `gerbil_proxy_mapping_update_requests_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_destinations_update_requests_total" type="counter">
          Destination update API calls.

          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordDestinationsUpdateRequest`\
            **Example:** `gerbil_destinations_update_requests_total{result="success"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Remote Config & Reporting">
        <ResponseField name="gerbil_remote_config_fetches_total" type="counter">
          Remote configuration fetch attempts.

          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordRemoteConfigFetch`\
            **Example:** `gerbil_remote_config_fetches_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_bandwidth_reports_total" type="counter">
          Bandwidth report transmissions.

          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordBandwidthReport`\
            **Example:** `gerbil_bandwidth_reports_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_peer_bandwidth_bytes_total" type="counter">
          Bytes per peer tracked by bandwidth calculation.

          <Expandable title="Details">
            **Labels:** `peer`, `direction` • **Unit:** bytes • **Path:** `metrics.RecordPeerBandwidthBytes`\
            **Example:** `gerbil_peer_bandwidth_bytes_total{peer="abc",direction="rx"} 8192`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="System & Operations">
        <ResponseField name="gerbil_netlink_events_total" type="counter">
          Netlink events processed.

          <Expandable title="Details">
            **Labels:** `event_type` • **Unit:** 1 • **Path:** `metrics.RecordNetlinkEvent`\
            **Example:** `gerbil_netlink_events_total{event_type="link_up"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_netlink_errors_total" type="counter">
          Netlink or kernel errors.

          <Expandable title="Details">
            **Labels:** `component`, `error_type` • **Unit:** 1 • **Path:** `metrics.RecordNetlinkError`\
            **Example:** `gerbil_netlink_errors_total{component="wg",error_type="permission"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_sync_duration_seconds" type="histogram">
          Duration of reconciliation or sync loops.

          <Expandable title="Details">
            **Labels:** `component` • **Unit:** seconds • **Path:** `metrics.RecordSyncDuration`\
            **Example:** `gerbil_sync_duration_seconds_bucket{component="remote_config",le="0.5"} 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_workqueue_depth" type="gauge">
          Current length of internal work queues.

          <Expandable title="Details">
            **Labels:** `queue` • **Unit:** 1 • **Path:** `metrics.RecordWorkqueueDepth`\
            **Example:** `gerbil_workqueue_depth{queue="relay"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_kernel_module_loads_total" type="counter">
          Kernel module load attempts.

          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordKernelModuleLoad`\
            **Example:** `gerbil_kernel_module_loads_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_firewall_rules_applied_total" type="counter">
          Firewall rules applied.

          <Expandable title="Details">
            **Labels:** `result`, `chain` • **Unit:** 1 • **Path:** `metrics.RecordFirewallRuleApplied`\
            **Example:** `gerbil_firewall_rules_applied_total{result="success",chain="FORWARD"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_config_reloads_total" type="counter">
          Configuration reloads.

          <Expandable title="Details">
            **Labels:** `result` • **Unit:** 1 • **Path:** `metrics.RecordConfigReload`\
            **Example:** `gerbil_config_reloads_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_restart_total" type="counter">
          Process restart count.

          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordRestart`\
            **Example:** `gerbil_restart_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_auth_failures_total" type="counter">
          Authentication or peer validation failures.

          <Expandable title="Details">
            **Labels:** `peer`, `reason` • **Unit:** 1 • **Path:** `metrics.RecordAuthFailure`\
            **Example:** `gerbil_auth_failures_total{peer="abc",reason="invalid_key"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_acl_denied_total" type="counter">
          Access-control denied events.

          <Expandable title="Details">
            **Labels:** `ifname`, `peer`, `policy` • **Unit:** 1 • **Path:** `metrics.RecordACLDenied`\
            **Example:** `gerbil_acl_denied_total{ifname="wg0",peer="abc",policy="deny"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_certificate_expiry_days" type="gauge">
          Days until certificate expiry.

          <Expandable title="Details">
            **Labels:** `cert_name`, `ifname` • **Unit:** days • **Path:** `metrics.RecordCertificateExpiry`\
            **Example:** `gerbil_certificate_expiry_days{cert_name="server",ifname="wg0"} 42`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_memory_spike_total" type="counter">
          Memory spikes detected by severity.

          <Expandable title="Details">
            **Labels:** `severity` • **Unit:** 1 • **Path:** `metrics.RecordMemorySpike`\
            **Example:** `gerbil_memory_spike_total{severity="warning"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="gerbil_heap_profiles_written_total" type="counter">
          Heap profile files generated.

          <Expandable title="Details">
            **Labels:** — • **Unit:** 1 • **Path:** `metrics.RecordHeapProfileWritten`\
            **Example:** `gerbil_heap_profiles_written_total 1`
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Tab>
</Tabs>

### Pangolin Kubernetes Controller metrics

<Tabs>
  <Tab title="OpenTelemetry (OTel)">
    <ResponseField name="pangolin-kube-controller">
      Additional OpenTelemetry metric instruments exposed by the Pangolin Kubernetes Controller.

      <Note>
        The controller exposes Prometheus-native metrics and additional OTel-backed metrics on the same `/metrics` endpoint.
        The OTel-backed series use the `pangolin_controller_*` prefix.
      </Note>

      <Expandable title="Reconcile">
        <ResponseField name="pangolin_controller_reconcile_phase_duration_seconds" type="Histogram (s)">
          Duration of each reconcile phase.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `phase`, `result`\
            **Label values:**\
            `phase`: `middlewares` | `routers` | `serversTransports` | `services` | `tcp` | `udp`\
            `result`: `success` | `error`\
            **Emission path:** OTel reconcile phase instrumentation\
            **Example:** `pangolin_controller_reconcile_phase_duration_seconds_bucket{phase="routers",result="success",le="0.5"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_controller_active_reconcile_routines" type="UpDownCounter">
          Number of active reconcile routines by phase.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `phase`\
            **Label values:** `middlewares` | `routers` | `serversTransports` | `services` | `tcp` | `udp`\
            **Emission path:** Parallel reconcile instrumentation\
            **Example:** `pangolin_controller_active_reconcile_routines{phase="routers"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_controller_loop_iterations_total" type="Counter">
          Controller loop iterations by outcome.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `outcome`\
            **Label values:** `success` | `nochange` | `error`\
            **Emission path:** Controller loop instrumentation\
            **Example:** `pangolin_controller_loop_iterations_total{outcome="success"} 10`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Fetch & Config">
        <ResponseField name="pangolin_controller_fetch_duration_seconds" type="Histogram (s)">
          Duration of remote fetch cycle HTTP requests.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `status_code`, `status_class`\
            **Label values:**\
            `status_code`: `200` | `304` | `401` | `403` | `404` | `5xx`\
            `status_class`: `2xx` | `3xx` | `4xx` | `5xx`\
            **Emission path:** Remote config fetch instrumentation\
            **Example:** `pangolin_controller_fetch_duration_seconds_bucket{status_code="200",status_class="2xx",le="0.25"} 4`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_controller_config_parse_duration_seconds" type="Histogram (s)">
          Duration of configuration parsing.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `section`\
            **Label values:** `full`\
            **Emission path:** Config parse instrumentation\
            **Example:** `pangolin_controller_config_parse_duration_seconds_bucket{section="full",le="0.1"} 2`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Kubernetes API">
        <ResponseField name="pangolin_controller_k8s_request_duration_seconds" type="Histogram (s)">
          Duration of Kubernetes API requests.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `verb`, `resource_kind`, `result`, `forced`\
            **Label values:**\
            `verb`: `get` | `create` | `patch` | `update` | `delete` | `list`\
            `resource_kind`: `IngressRoute` | `Middleware` | `TraefikService` | `ServersTransport` | `ServersTransportTCP` | `Service` | `EndpointSlice`\
            `result`: `success` | `error` | `conflict`\
            `forced`: `true` | `false`\
            **Emission path:** Kubernetes API request instrumentation\
            **Example:** `pangolin_controller_k8s_request_duration_seconds_bucket{verb="patch",resource_kind="IngressRoute",result="success",forced="false",le="0.25"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_controller_k8s_requests_total" type="Counter">
          Total Kubernetes API requests.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `verb`, `resource_kind`, `result`, `forced`\
            **Emission path:** Kubernetes API request instrumentation\
            **Example:** `pangolin_controller_k8s_requests_total{verb="patch",resource_kind="IngressRoute",result="success",forced="false"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_controller_retries_total" type="Counter">
          Retry attempts in the SSA apply loop.

          <Expandable title="Details">
            **Unit:** 1\
            **Labels:** `reason`, `operation`, `resource_kind`\
            **Label values:**\
            `reason`: `conflict` | `transient` | `timeout`\
            `operation`: `get` | `create` | `patch` | `delete` | `apply`\
            **Emission path:** SSA apply retry instrumentation\
            **Example:** `pangolin_controller_retries_total{reason="conflict",operation="patch",resource_kind="IngressRoute"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Garbage Collection">
        <ResponseField name="pangolin_controller_gc_run_duration_seconds" type="Histogram (s)">
          Duration of garbage collection runs.

          <Expandable title="Details">
            **Unit:** seconds\
            **Labels:** `result`\
            **Label values:** `success` | `fail` | `dryrun`\
            **Emission path:** GC instrumentation\
            **Example:** `pangolin_controller_gc_run_duration_seconds_bucket{result="success",le="0.5"} 1`
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Tab>

  <Tab title="Prometheus">
    <ResponseField name="pangolin-kube-controller">
      Prometheus-native metrics exposed by the Pangolin Kubernetes Controller. These use the `pangolin_kube_controller_*` prefix.

      <Note>
        The metrics endpoint is exposed at `GET /metrics` on `METRICS_ADDR`, default `:9090`.
        The same HTTP server also exposes `/healthz`, `/readyz`, and optionally `/debug/pprof/` when pprof is enabled.
      </Note>

      <Expandable title="Reconcile">
        <ResponseField name="pangolin_kube_controller_reconcile_seconds" type="histogram">
          Duration of a full successful reconcile loop.

          <Expandable title="Details">
            **Labels:** —\
            **Unit:** seconds\
            **Path:** `Collector.ReconcileDuration`\
            **Example:** `pangolin_kube_controller_reconcile_seconds_bucket{le="0.5"} 3`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_reconcile_errors_total" type="counter">
          Total errors during reconcile steps.

          <Expandable title="Details">
            **Labels:** —\
            **Unit:** 1\
            **Path:** `Collector.ReconcileErrors`\
            **Example:** `pangolin_kube_controller_reconcile_errors_total 2`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_consecutive_errors" type="gauge">
          Number of consecutive reconcile or fetch errors.

          <Expandable title="Details">
            **Labels:** —\
            **Unit:** 1\
            **Path:** `Collector.ConsecutiveErrors`\
            **Example:** `pangolin_kube_controller_consecutive_errors 0`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_last_fetch_success_timestamp_seconds" type="gauge">
          Unix timestamp of the last successful fetch.

          <Expandable title="Details">
            **Labels:** —\
            **Unit:** seconds since Unix epoch\
            **Path:** `Collector.LastFetchSuccess`\
            **Example:** `pangolin_kube_controller_last_fetch_success_timestamp_seconds 1767225600`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_ready" type="gauge">
          Controller readiness state.

          <Expandable title="Details">
            **Labels:** —\
            **Unit:** 1\
            **Value:** `1` = ready, `0` = not ready\
            **Path:** `Collector.Ready`\
            **Example:** `pangolin_kube_controller_ready 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Objects">
        <ResponseField name="pangolin_kube_controller_objects_applied_total" type="counter">
          Applied objects by kind and action.

          <Expandable title="Details">
            **Labels:** `kind`, `action`\
            **Label values:**\
            `kind`: `IngressRoute` | `Middleware` | `TraefikService` | other managed Traefik CRDs\
            `action`: `create` | `patch`\
            **Unit:** 1\
            **Path:** `Collector.AppliedObjects`\
            **Example:** `pangolin_kube_controller_objects_applied_total{kind="IngressRoute",action="patch"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_objects_deleted_total" type="counter">
          Deleted objects by kind.

          <Expandable title="Details">
            **Labels:** `kind`\
            **Unit:** 1\
            **Path:** `Collector.DeletedObjects`\
            **Example:** `pangolin_kube_controller_objects_deleted_total{kind="Middleware"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_desired_objects_count" type="gauge">
          Desired objects count by kind from the last processed config.

          <Expandable title="Details">
            **Labels:** `kind`\
            **Unit:** 1\
            **Path:** `Collector.DesiredObjects`\
            **Example:** `pangolin_kube_controller_desired_objects_count{kind="IngressRoute"} 12`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Garbage Collection">
        <ResponseField name="pangolin_kube_controller_gc_deleted_total" type="counter">
          Garbage-collected objects by kind and reason.

          <Expandable title="Details">
            **Labels:** `kind`, `reason`\
            **Label values:**\
            `reason`: `immediate` | `grace`\
            **Unit:** 1\
            **Path:** `Collector.GCDeletedTotal`\
            **Example:** `pangolin_kube_controller_gc_deleted_total{kind="IngressRoute",reason="grace"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_gc_runs_total" type="counter">
          Garbage collection runs by result.

          <Expandable title="Details">
            **Labels:** `result`\
            **Label values:** `start` | `success` | `fail` | `dryrun`\
            **Unit:** 1\
            **Path:** `Collector.GCRunsTotal`\
            **Example:** `pangolin_kube_controller_gc_runs_total{result="success"} 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_grace_queue_dropped_total" type="counter">
          Grace deletion queue items dropped due to queue overflow.

          <Expandable title="Details">
            **Labels:** —\
            **Unit:** 1\
            **Path:** `Collector.GraceQueueDropped`\
            **Example:** `pangolin_kube_controller_grace_queue_dropped_total 0`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_grace_queue_depth" type="gauge">
          Current depth of the grace deletion queue.

          <Expandable title="Details">
            **Labels:** —\
            **Unit:** 1\
            **Path:** `Collector.GraceQueueDepth`\
            **Example:** `pangolin_kube_controller_grace_queue_depth 3`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Instance Label">
        <ResponseField name="pangolin_kube_controller_instance_label_detect_success_total" type="counter">
          Successful Traefik instance label resolutions.

          <Expandable title="Details">
            **Labels:** —\
            **Unit:** 1\
            **Path:** `Collector.InstanceLabelDetectSuccess`\
            **Example:** `pangolin_kube_controller_instance_label_detect_success_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_instance_label_detect_failure_total" type="counter">
          Failed Traefik instance label resolutions or verifications.

          <Expandable title="Details">
            **Labels:** —\
            **Unit:** 1\
            **Path:** `Collector.InstanceLabelDetectFailures`\
            **Example:** `pangolin_kube_controller_instance_label_detect_failure_total 1`
          </Expandable>
        </ResponseField>

        <ResponseField name="pangolin_kube_controller_instance_label_last_check_timestamp_seconds" type="gauge">
          Unix timestamp of the last instance label verification.

          <Expandable title="Details">
            **Labels:** —\
            **Unit:** seconds since Unix epoch\
            **Path:** `Collector.InstanceLabelLastCheck`\
            **Example:** `pangolin_kube_controller_instance_label_last_check_timestamp_seconds 1767225600`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Leader Election">
        <ResponseField name="pangolin_kube_controller_leader_state" type="gauge">
          Current leader election state of the controller.

          <Expandable title="Details">
            **Labels:** —\
            **Unit:** 1\
            **Value:** `1` = leader, `0` = follower, `-1` = leader election disabled\
            **Path:** `Collector.LeaderState`\
            **Example:** `pangolin_kube_controller_leader_state 1`
          </Expandable>
        </ResponseField>
      </Expandable>

      <Expandable title="Runtime">
        <ResponseField name="go_*" type="runtime metrics">
          Go runtime metrics registered by the Prometheus Go collector.

          <Expandable title="Details">
            **Labels:** varies\
            **Unit:** varies\
            **Path:** `collectors.NewGoCollector()`\
            **Example:** `go_goroutines 14`
          </Expandable>
        </ResponseField>

        <ResponseField name="process_*" type="process metrics">
          Process metrics registered by the Prometheus process collector.

          <Expandable title="Details">
            **Labels:** varies\
            **Unit:** varies\
            **Path:** `collectors.NewProcessCollector()`\
            **Example:** `process_resident_memory_bytes 52428800`
          </Expandable>
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Tab>
</Tabs>

***

## References

* <Link href="https://opentelemetry.io/docs/">OpenTelemetry Documentation</Link>
* <Link href="https://prometheus.io/docs/introduction/overview/">Prometheus Documentation</Link>

<Tip>
  Have improvements or a missing metric? Open an issue or PR referencing this page.
</Tip>