Metrics and Observability

Pangolin exposes observability signals across multiple components, but not every component provides the same telemetry surface. This page explains which metrics, traces, logs, and profiling endpoints are available today, how they are exposed, and which collection patterns are recommended for production deployments. Newt provides the broadest native observability support with Prometheus metrics, OTLP metrics, OTLP traces, health checks, and optional pprof. Gerbil supports metrics through either a Prometheus backend or an OTLP backend. The Pangolin Kubernetes Controller exposes a Prometheus-compatible scrape endpoint and includes additional OTel-backed metric instruments on that endpoint.

This page focuses on Pangolin-native observability. For a community walkthrough that collects Traefik metrics with Prometheus and Grafana, see the community metrics guide.All currently documented component metrics are listed in the Full Metric Reference.

Capability Matrix

Compare metrics, traces, logs, and profiling support across Pangolin components.

Newt

Native Prometheus metrics, OTLP metrics, OTLP traces, health checks, and optional pprof.

Gerbil

Metrics-only observability using either a Prometheus or OTLP backend.

Kubernetes Controller

Prometheus scraping, health probes, ServiceMonitor examples, and optional pprof.

Observability Capability Matrix

Component	Metrics	Traces	Logs	Profiling
`newt`	Prometheus, OTLP	OTLP	stdout	pprof optional
`gerbil`	Prometheus or OTLP	—	stdout	—
`pangolin-kube-controller`	Prometheus scrape	—	stdout	pprof optional

Supported Signals

Signal	What it means
Prometheus scrape metrics	Pull-based metrics served over HTTP on `/metrics`
OTLP metrics	Push-based OpenTelemetry metrics sent to an OTel Collector or compatible backend
OTLP traces	Distributed traces sent to an OTel Collector or trace backend
Application logs	stdout, file logs, audit logs, or platform logs
Profiling	Debug endpoints such as `pprof`

Collection Patterns

Use one or more of the following patterns depending on your deployment model. Prometheus scrape mode is the simplest option for local or Kubernetes monitoring. OTLP is useful when you already operate an OpenTelemetry Collector or want to forward telemetry to a managed backend such as Grafana Cloud, Mimir, or Tempo.

Prometheus scrape mode
OTel Collector
Kubernetes ServiceMonitor
Logs to Loki

prometheus.yml (fragment)

scrape_configs:
  - job_name: newt
    static_configs:
      - targets: ["newt:2112"]

  - job_name: gerbil
    metrics_path: /metrics
    static_configs:
      - targets: ["gerbil:3003"]

  - job_name: pangolin-kube-controller
    static_configs:
      - targets: ["pangolin-kube-controller:9090"]

otel-collector.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:
  prometheus:
    config:
      scrape_configs:
        - job_name: pangolin-kube-controller
          static_configs:
            - targets: ["pangolin-kube-controller:9090"]

processors:
  batch: {}

exporters:
  prometheusremotewrite:
    endpoint: https://mimir.example.com/api/v1/push
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Today the traces pipeline is relevant only for newt. pangolin-kube-controller exposes additional OTel instruments, but they are exported on the same Prometheus scrape endpoint rather than pushed through OTLP.

servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pangolin-kube-controller
spec:
  selector:
    matchLabels:
      app: pangolin-kube-controller
  endpoints:
    - port: http-metrics
      path: /metrics
      interval: 30s

helm-values.yaml

global:
  metrics:
    enabled: true
    service:
      enabled: true
    serviceMonitor:
      enabled: true

controller:
  monitoring:
    serviceMonitor:
      enabled: true

Helm values for Newt are documented on the Kubernetes Newt configuration page. Controller and Gerbil ServiceMonitor values are documented on the Kubernetes Pangolin configuration page.

promtail.yaml

server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: pangolin
    static_configs:
      - targets: [localhost]
        labels:
          job: pangolin
          __path__: /var/log/containers/*pangolin*.log

Use Grafana Alloy, Promtail, Fluent Bit, or your preferred SIEM forwarder to collect container and application logs.

Newt

Newt metrics are not typically enabled in default deployments. Turn them on explicitly and expose the admin address only where you intend to scrape or profile. Available metrics are listed in Newt metrics.

Configuration

Environment Variables
CLI Args
Helm Values

NEWT_METRICS_PROMETHEUS_ENABLED=true
NEWT_METRICS_OTLP_ENABLED=true
NEWT_ADMIN_ADDR=:2112
OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
OTEL_EXPORTER_OTLP_INSECURE=true
OTEL_METRIC_EXPORT_INTERVAL=15s
NEWT_PPROF_ENABLED=false

newt \
  --metrics=true \
  --otlp=true \
  --metrics-admin-addr=:2112 \
  --pprof=false \
  --endpoint=https://pangolin.example.com \
  --id=saz281jfa8z37zg \
  --secret=your-secret

global:
  metrics:
    enabled: true
    adminAddr: ":2112"
    service:
      enabled: true
    serviceMonitor:
      enabled: true
    pprofEnabled: false

Signals and endpoints

Prometheus scrape endpoint: /metrics on NEWT_ADMIN_ADDR
Health endpoint: /healthz on the same admin server
OTLP metrics: enabled with NEWT_METRICS_OTLP_ENABLED=true or --otlp=true
OTLP traces: initialized when OTLP export is enabled
Profiling: /debug/pprof/* when NEWT_PPROF_ENABLED=true

The Newt OTLP switch enables the OTLP telemetry pipeline. The current environment variable name contains METRICS, but the implementation also initializes OTLP tracing when OTLP is enabled.

Newt’s admin server defaults to a loopback bind address. In containers or Kubernetes, set NEWT_ADMIN_ADDR=:2112 or another non-loopback address only when you intentionally want Prometheus, pprof, or health checks to reach it.

Examples

Direct Prometheus
OTLP metrics and traces

prometheus.yml (fragment)

scrape_configs:
  - job_name: newt
    static_configs:
      - targets: ["newt:2112"]

otel-collector.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch: {}

exporters:
  prometheusremotewrite:
    endpoint: https://mimir.example.com/api/v1/push
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Gerbil

Gerbil supports metrics only. Choose either a native Prometheus backend or an OTLP metrics backend at runtime. Those backends are mutually exclusive. Available metrics are listed in Gerbil metrics.

Configuration

Environment Variables
CLI Args

METRICS_ENABLED=true
METRICS_BACKEND=prometheus
METRICS_PATH=/metrics
LISTEN=:3003

# OTel mode
OTEL_METRICS_PROTOCOL=grpc
OTEL_METRICS_ENDPOINT=otel-collector:4317
OTEL_METRICS_INSECURE=true
OTEL_METRICS_EXPORT_INTERVAL=60s
OTEL_METRICS_TIMEOUT=10s

./gerbil \
  --metrics-enabled \
  --metrics-backend=prometheus \
  --metrics-path=/metrics \
  --config=/etc/gerbil/config.json

./gerbil \
  --metrics-enabled \
  --metrics-backend=otel \
  --otel-metrics-protocol=grpc \
  --otel-metrics-endpoint=otel-collector:4317 \
  --otel-metrics-insecure \
  --otel-metrics-export-interval=10s \
  --otel-metrics-timeout=10s \
  --config=/etc/gerbil/config.json

Signals and endpoints

Prometheus metrics endpoint: METRICS_PATH, default /metrics
Metrics are served on Gerbil’s configured HTTP listen address
The Docker Compose metrics example commonly scrapes gerbil:3003
Health endpoint: /healthz
OTLP metrics: enabled when METRICS_BACKEND=otel
Traces: not supported yet
Profiling: not supported yet

Examples

Prometheus scrape
OTLP metrics

prometheus.yml (fragment)

scrape_configs:
  - job_name: gerbil
    metrics_path: /metrics
    static_configs:
      - targets: ["gerbil:3003"]

Replace gerbil:3003 with the actual Gerbil HTTP listen address in your deployment.

otel-collector.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch: {}

exporters:
  prometheusremotewrite:
    endpoint: https://mimir.example.com/api/v1/push

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite]

Pangolin Kubernetes Controller

The controller exposes a Prometheus-compatible /metrics endpoint and standard health probes. It also registers additional OpenTelemetry metric instruments, but those instruments are exported on the same scrape endpoint rather than pushed through OTLP. Available metrics are listed in Controller metrics.

pprof is available only when explicitly enabled via ENABLE_PPROF=true.

Configuration

Environment Variables
Helm Values

METRICS_ADDR=:9090
DISABLE_LIVEZ=false
ENABLE_PPROF=false

controller:
  service:
    enabled: true
    port: 9090
    portName: metrics
  monitoring:
    serviceMonitor:
      enabled: true
    podMonitor:
      enabled: false
    prometheusRule:
      enabled: false

Expose the controller metrics endpoint only inside trusted networks or through Kubernetes-native monitoring resources. If TLS or auth is required, terminate it with a Service mesh, Ingress, sidecar, or platform-specific monitoring gateway unless native TLS support is verified.

Endpoints

/metrics on METRICS_ADDR
/healthz and /readyz for readiness
/livez and /health/live for liveness unless DISABLE_LIVEZ=true
/debug/pprof/* when ENABLE_PPROF=true

Representative metrics and Kubernetes examples

service-and-servicemonitor.yaml

apiVersion: v1
kind: Service
metadata:
  name: pangolin-kube-controller
  labels:
    app: pangolin-kube-controller
spec:
  selector:
    app: pangolin-kube-controller
  ports:
    - name: http-metrics
      port: 9090
      targetPort: 9090
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pangolin-kube-controller
spec:
  selector:
    matchLabels:
      app: pangolin-kube-controller
  endpoints:
    - port: http-metrics
      path: /metrics
      interval: 30s

Alerting Examples

Newt
Gerbil
Controller

increase(newt_connection_errors_total[5m]) > 10

histogram_quantile(
  0.95,
  sum(rate(newt_tunnel_latency_seconds_bucket[5m])) by (le)
) > 1

increase(newt_tunnel_reconnects_total[10m]) > 20

max_over_time(gerbil_wg_interface_up[5m]) by (ifname) < 1

sum(gerbil_wg_peers_total) > 0
and
sum(gerbil_wg_peer_connected) == 0

min_over_time(pangolin_kube_controller_ready[5m]) == 0

increase(pangolin_kube_controller_reconcile_errors_total[10m]) > 5

Community Metrics Guide

Traefik and metrics collection with Prometheus and Grafana.

Newt Kubernetes Monitoring

Verified Newt chart values for metrics, Services, and ServiceMonitor resources.

Controller Monitoring Values

Verified chart values for controller Services, ServiceMonitor, PodMonitor, and PrometheusRule resources.

Versions

Component	Signal	Since version
Newt	Prometheus scrape metrics	`v1.6.0`
Newt	OTLP metrics	`v1.6.0`
Newt	OTLP traces	`v1.6.0`
Newt	pprof	`v1.10.4`
Gerbil	Prometheus scrape metrics	`v1.4.0`
Gerbil	OTLP metrics	`v1.4.0`
Pangolin Kubernetes Controller	Prometheus scrape metrics	`v0.1.0-alpha.1`

Full Metric Reference

The full reference below is grouped by component.

Newt: Prometheus metrics, OTLP metrics, OTLP traces
Gerbil: Prometheus or OTLP metrics
Pangolin Kubernetes Controller: Prometheus-native metrics and additional OTel-backed scrape metrics

Metric names, labels, and defaults can change between component releases.

Newt metrics

OpenTelemetry (OTel)
Prometheus

OpenTelemetry metric instruments exposed by Newt. Expand each section to see individual metrics with labels, units, emission points, and examples.

Show Site & Build

Counter

Counts Pangolin registration attempts keyed by result.

Show Details

Unit: 1
Labels: result (success|failure), site_id
Emission path: telemetry.IncSiteRegistration
Example: newt_site_registrations_total{result="success",site_id="abc"} 1

ObservableGauge

0/1 heartbeat for the active site.

Show Details

Unit: 1
Labels: site_id
Emission path: state.TelemetryView (callback)
Example: newt_site_online{site_id="self"} 1

ObservableGauge

Seconds since last Pangolin heartbeat.

Show Details

Unit: seconds
Labels: site_id
Emission path: TouchHeartbeat (callback)
Example: newt_site_last_heartbeat_seconds{site_id="self"} 3.2

ObservableGauge

Constant 1 with build metadata labels.

Show Details

Unit: 1
Labels: version, commit
Emission path: Build info registration
Example: newt_build_info{version="1.2.3",commit="abc123"} 1

Counter

Process boot indicator (increments once per process start).

Show Details

Unit: 1
Labels: —
Emission path: RegisterBuildInfo
Example: newt_restart_count_total 1

Counter

Certificate rotation events keyed by result.

Show Details

Unit: 1
Labels: result
Emission path: IncCertRotation
Example: newt_cert_rotation_total{result="success"} 1

Counter

Config reload attempts keyed by result.

Show Details

Unit: 1
Labels: result
Emission path: telemetry.IncConfigReload
Example: newt_config_reloads_total{result="success"} 1

Histogram (s)

Duration per config-apply phase keyed by phase and result.

Show Details

Unit: seconds
Labels: phase, result
Emission path: telemetry.ObserveConfigApply
Example: newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3

Show Tunnel

ObservableGauge

Active sessions per tunnel (or collapsed).

Show Details

Unit: 1
Labels: site_id, tunnel_id
Emission path: RegisterStateView
Example: newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2

Counter (bytes)

Traffic per tunnel, direction, and protocol.

Show Details

Unit: bytes
Labels: tunnel_id, direction (ingress|egress), protocol (tcp|udp)
Emission path: Proxy manager
Example: newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192

Histogram (s)

RTT samples per tunnel/transport.

Show Details

Unit: seconds
Labels: tunnel_id, transport
Emission path: Health checks
Example: newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4

Counter

Reconnect attempts keyed by initiator & reason.

Show Details

Unit: 1
Labels: tunnel_id, initiator (client|server), reason
Emission path: telemetry.IncReconnect
Example: newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3

Show Connection & Auth

Counter

Auth/WebSocket connection attempts keyed by transport & result.

Show Details

Unit: 1
Labels: transport, result
Emission path: telemetry.IncConnAttempt
Example: newt_connection_attempts_total{transport="websocket",result="failure"} 2

Counter

Connection errors keyed by transport and type.

Show Details

Unit: 1
Labels: transport, error_type
Emission path: telemetry.IncConnError
Example: newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1

Show WebSocket

Histogram (s)

Dial latency for Pangolin WebSocket.

Show Details

Unit: seconds
Labels: result, transport
Emission path: ObserveWSConnectLatency
Example: newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1

Counter

WebSocket disconnects keyed by reason.

Show Details

Unit: 1
Labels: reason, tunnel_id
Emission path: IncWSDisconnect
Example: newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2

Counter

Ping/Pong failures observed by keepalive.

Show Details

Unit: 1
Labels: reason (e.g., ping_write, pong_timeout)
Emission path: telemetry.IncWSKeepaliveFailure(ctx, "ping_write")
Example: newt_websocket_keepalive_failures_total{reason="ping_write"} 1

Histogram (s)

Duration of established WS sessions keyed by result.

Show Details

Unit: seconds
Labels: result (success|error)
Emission path: telemetry.ObserveWSSessionDuration(ctx, time.Since(start).Seconds(), "error")
Example: newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3

ObservableGauge

Current WS connection state (0/1).

Show Details

Unit: 1
Labels: —
Emission path: telemetry.SetWSConnectionState(true|false)
Example: newt_websocket_connected 1

Counter

WebSocket reconnect attempts keyed by reason.

Show Details

Unit: 1
Labels: reason
Emission path: telemetry.IncWSReconnect(ctx, "ping_write")
Example: newt_websocket_reconnects_total{reason="ping_write"} 1

Counter

In/out WS messages keyed by direction & type.

Show Details

Unit: 1
Labels: direction (in|out), msg_type (ping|pong|text|…)
Emission path: IncWSMessage
Example: newt_websocket_messages_total{direction="out",msg_type="ping"} 4

Show Proxy

ObservableGauge

Active TCP/UDP proxy connections per tunnel/protocol.

Show Details

Unit: 1
Labels: protocol, tunnel_id
Emission path: Proxy callback
Example: newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3

ObservableGauge (bytes)

Proxy buffer pool size.

Show Details

Unit: bytes
Labels: protocol, tunnel_id
Emission path: Proxy callback
Example: newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240

ObservableGauge (bytes)

Unflushed async byte backlog.

Show Details

Unit: bytes
Labels: protocol, tunnel_id
Emission path: Proxy callback
Example: newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096

Counter

Proxy write drops keyed by protocol/tunnel.

Show Details

Unit: 1
Labels: protocol, tunnel_id
Emission path: IncProxyDrops
Example: newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2

Counter

Proxy accept events keyed by result/reason.

Show Details

Unit: 1
Labels: tunnel_id, protocol, result, reason
Emission path: telemetry.IncProxyAccept(ctx, tunnelID, "tcp", "failure", "timeout")
Example: newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1

Counter

Lifecycle events (opened/closed) per connection.

Show Details

Unit: 1
Labels: tunnel_id, protocol, event (opened|closed)
Emission path: telemetry.IncProxyConnectionEvent(ctx, tunnelID, "tcp", telemetry.ProxyConnectionOpened)
Example: newt_proxy_connections_total{protocol="tcp",event="opened"} 1

Histogram (s)

Duration of completed proxy connections.

Show Details

Unit: seconds
Labels: tunnel_id, protocol, result
Emission path: telemetry.ObserveProxyConnectionDuration(ctx, tunnelID, "tcp", "success", seconds)
Example: newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3

Prometheus-style series for the same Newt metrics. Names, labels, and examples mirror the OTel tab.

Show Site & Build

counter

Counts Pangolin registration attempts keyed by result.

Show Details

Labels: result, site_id • Unit: 1 • Path: telemetry.IncSiteRegistration
Example: newt_site_registrations_total{result="success",site_id="abc"} 1

gauge

0/1 heartbeat for the active site.

Show Details

Labels: site_id • Unit: 1 • Path: state.TelemetryView
Example: newt_site_online{site_id="self"} 1

gauge

Seconds since last Pangolin heartbeat.

Show Details

Labels: site_id • Unit: seconds • Path: TouchHeartbeat
Example: newt_site_last_heartbeat_seconds{site_id="self"} 3.2

gauge

Constant 1 with build metadata labels.

Show Details

Labels: version, commit • Unit: 1 • Path: Build info registration
Example: newt_build_info{version="1.2.3",commit="abc123"} 1

counter

Process boot indicator (increments once).

Show Details

Labels: — • Unit: 1 • Path: RegisterBuildInfo
Example: newt_restart_count_total 1

counter

Certificate rotation events keyed by result.

Show Details

Labels: result • Unit: 1 • Path: IncCertRotation
Example: newt_cert_rotation_total{result="success"} 1

counter

Config reload attempts keyed by result.

Show Details

Labels: result • Unit: 1 • Path: telemetry.IncConfigReload
Example: newt_config_reloads_total{result="success"} 1

histogram

Duration per config-apply phase & result.

Show Details

Labels: phase, result • Unit: seconds • Path: telemetry.ObserveConfigApply
Example: newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3

Show Tunnel

gauge

Active sessions per tunnel (or collapsed).

Show Details

Labels: site_id, tunnel_id • Unit: 1 • Path: RegisterStateView
Example: newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2

counter

Traffic per tunnel/direction/protocol.

Show Details

Labels: tunnel_id, direction, protocol • Unit: bytes • Path: Proxy manager
Example: newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192

histogram

RTT samples per tunnel/transport.

Show Details

Labels: tunnel_id, transport • Unit: seconds • Path: Health checks
Example: newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4

counter

Reconnect attempts by initiator & reason.

Show Details

Labels: tunnel_id, initiator, reason • Unit: 1 • Path: telemetry.IncReconnect
Example: newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3

Show Connection & Auth

counter

Auth/WebSocket attempts by transport & result.

Show Details

Labels: transport, result • Unit: 1 • Path: telemetry.IncConnAttempt
Example: newt_connection_attempts_total{transport="websocket",result="failure"} 2

counter

Connection errors by transport and type.

Show Details

Labels: transport, error_type • Unit: 1 • Path: telemetry.IncConnError
Example: newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1

Show WebSocket

histogram

Dial latency for Pangolin WebSocket.

Show Details

Labels: result, transport • Unit: seconds • Path: ObserveWSConnectLatency
Example: newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1

counter

WS disconnects by reason.

Show Details

Labels: reason, tunnel_id • Unit: 1 • Path: IncWSDisconnect
Example: newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2

counter

Keepalive Ping/Pong failures.

Show Details

Labels: reason • Unit: 1 • Path: telemetry.IncWSKeepaliveFailure(ctx, "ping_write")
Example: newt_websocket_keepalive_failures_total{reason="ping_write"} 1

histogram

Duration of established WebSocket sessions by result.

Show Details

Labels: result • Unit: seconds • Path: telemetry.ObserveWSSessionDuration(...)
Example: newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3

gauge

Current WS connection status (0/1).

Show Details

Labels: — • Unit: 1 • Path: telemetry.SetWSConnectionState(true|false)
Example: newt_websocket_connected 1

counter

Reconnect attempts by reason.

Show Details

Labels: reason • Unit: 1 • Path: telemetry.IncWSReconnect(ctx, "ping_write")
Example: newt_websocket_reconnects_total{reason="ping_write"} 1

counter

In/out WS messages by direction & type.

Show Details

Labels: direction, msg_type • Unit: 1 • Path: IncWSMessage
Example: newt_websocket_messages_total{direction="out",msg_type="ping"} 4

Show Proxy

gauge

Active TCP/UDP proxy connections per tunnel/protocol.

Show Details

Labels: protocol, tunnel_id • Unit: 1 • Path: Proxy callback
Example: newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3

gauge

Proxy buffer pool size.

Show Details

Labels: protocol, tunnel_id • Unit: bytes • Path: Proxy callback
Example: newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240

gauge

Unflushed async byte backlog.

Show Details

Labels: protocol, tunnel_id • Unit: bytes • Path: Proxy callback
Example: newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096

counter

Proxy write drops per protocol/tunnel.

Show Details

Labels: protocol, tunnel_id • Unit: 1 • Path: IncProxyDrops
Example: newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2

counter

Proxy accept events by result/reason.

Show Details

Labels: tunnel_id, protocol, result, reason • Unit: 1 • Path: telemetry.IncProxyAccept(...)
Example: newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1

counter

Connection lifecycle events (opened/closed).

Show Details

Labels: tunnel_id, protocol, event • Unit: 1 • Path: telemetry.IncProxyConnectionEvent(...)
Example: newt_proxy_connections_total{protocol="tcp",event="opened"} 1

histogram

Duration of completed proxy connections.

Show Details

Labels: tunnel_id, protocol, result • Unit: seconds • Path: telemetry.ObserveProxyConnectionDuration(...)
Example: newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3

Gerbil metrics

OpenTelemetry (OTel)
Prometheus

OpenTelemetry metric instruments exposed by Gerbil. Gerbil supports exactly one metrics backend at runtime: prometheus, otel, or none.

In otel mode, Gerbil pushes metrics to an OTLP collector. The /metrics endpoint is not exposed in this mode.

Show WireGuard

Int64Gauge

Operational state of a WireGuard interface.

Show Details

Unit: 1
Labels: ifname, instance
Emission path: metrics.RecordInterfaceUp
Example: gerbil_wg_interface_up{ifname="wg0",instance="gerbil-1"} 1

UpDownCounter

Number of configured peers per interface.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordPeersTotal
Example: gerbil_wg_peers_total{ifname="wg0"} 5

Int64Gauge

Current peer connection state.

Show Details

Unit: 1
Labels: ifname, peer
Emission path: metrics.RecordPeerConnected
Example: gerbil_wg_peer_connected{ifname="wg0",peer="abc"} 1

UpDownCounter

Number of allowed IPs configured per peer.

Show Details

Unit: 1
Labels: ifname, peer
Emission path: metrics.RecordAllowedIPsCount
Example: gerbil_allowed_ips_count{ifname="wg0",peer="abc"} 2

Counter

Key rotation events.

Show Details

Unit: 1
Labels: ifname, reason
Emission path: metrics.RecordKeyRotation
Example: gerbil_key_rotation_total{ifname="wg0",reason="scheduled"} 1

Counter

WireGuard handshake attempts keyed by result.

Show Details

Unit: 1
Labels: ifname, peer, result
Emission path: metrics.RecordHandshake
Example: gerbil_wg_handshakes_total{ifname="wg0",peer="abc",result="success"} 1

Histogram (s)

Distribution of WireGuard handshake latencies.

Show Details

Unit: seconds
Labels: ifname, peer
Emission path: metrics.RecordHandshakeLatency
Example: gerbil_wg_handshake_latency_seconds_bucket{ifname="wg0",peer="abc",le="0.1"} 3

Histogram (s)

Observed peer round-trip time.

Show Details

Unit: seconds
Labels: ifname, peer
Emission path: metrics.RecordPeerRTT
Example: gerbil_wg_peer_rtt_seconds_bucket{ifname="wg0",peer="abc",le="0.05"} 4

Counter (bytes)

Bytes received from a WireGuard peer.

Show Details

Unit: bytes
Labels: ifname, peer
Emission path: metrics.RecordBytesReceived
Example: gerbil_wg_bytes_received_total{ifname="wg0",peer="abc"} 8192

Counter (bytes)

Bytes transmitted to a WireGuard peer.

Show Details

Unit: bytes
Labels: ifname, peer
Emission path: metrics.RecordBytesTransmitted
Example: gerbil_wg_bytes_transmitted_total{ifname="wg0",peer="abc"} 16384

Show Relay

UpDownCounter

Number of active UDP relay sessions.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordActiveSession / metrics.RecordSession
Example: gerbil_active_sessions{ifname="wg0"} 3

Counter

UDP packets processed by relay workers.

Show Details

Unit: 1
Labels: ifname, type, direction
Emission path: metrics.RecordUDPPacket
Example: gerbil_udp_packets_total{ifname="wg0",type="data",direction="rx"} 42

Histogram (bytes)

Size distribution of packets forwarded through the relay.

Show Details

Unit: bytes
Labels: ifname, type
Emission path: metrics.RecordUDPPacketSize
Example: gerbil_udp_packet_size_bytes_bucket{ifname="wg0",type="data",le="1024"} 7

Counter

Hole punch messages processed by result.

Show Details

Unit: 1
Labels: ifname, result
Emission path: metrics.RecordHolePunchEvent
Example: gerbil_hole_punch_events_total{ifname="wg0",result="success"} 1

UpDownCounter

Active proxy mappings.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordProxyMapping
Example: gerbil_proxy_mapping_active{ifname="wg0"} 4

Counter

Sessions rebuilt from communication patterns.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordSessionRebuilt
Example: gerbil_session_rebuilt_total{ifname="wg0"} 1

UpDownCounter

Active communication patterns.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordCommPattern
Example: gerbil_comm_pattern_active{ifname="wg0"} 2

Counter

Items removed by cleanup routines.

Show Details

Unit: 1
Labels: ifname, component
Emission path: metrics.RecordProxyCleanupRemoved
Example: gerbil_proxy_cleanup_removed_total{ifname="wg0",component="sessions"} 5

Counter

Proxy connection errors.

Show Details

Unit: 1
Labels: ifname, error_type
Emission path: metrics.RecordProxyConnectionError
Example: gerbil_proxy_connection_errors_total{ifname="wg0",error_type="timeout"} 1

Int64Gauge

Initial proxy mappings loaded.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordProxyInitialMappings
Example: gerbil_proxy_initial_mappings{ifname="wg0"} 8

Counter

Proxy mapping updates.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordProxyMappingUpdate
Example: gerbil_proxy_mapping_updates_total{ifname="wg0"} 2

Histogram (s)

Duration of idle cleanup cycles.

Show Details

Unit: seconds
Labels: ifname, component
Emission path: metrics.RecordProxyIdleCleanupDuration
Example: gerbil_proxy_idle_cleanup_duration_seconds_bucket{ifname="wg0",component="sessions",le="0.1"} 1

Show SNI Proxy

UpDownCounter

Active SNI proxy connections.

Show Details

Unit: 1
Labels: —
Emission path: metrics.RecordActiveProxyConnection
Example: gerbil_active_proxy_connections 2

Counter

Route lookups keyed by result.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordProxyRouteLookup
Example: gerbil_proxy_route_lookups_total{result="hit"} 6

Histogram (s)

TLS handshake duration for the SNI proxy.

Show Details

Unit: seconds
Labels: —
Emission path: metrics.RecordProxyTLSHandshake
Example: gerbil_proxy_tls_handshake_seconds_bucket{le="0.1"} 2

Counter (bytes)

Bytes sent or received by the SNI proxy.

Show Details

Unit: bytes
Labels: direction
Emission path: metrics.RecordProxyBytesTransmitted
Example: gerbil_proxy_bytes_transmitted_total{direction="egress"} 16384

Counter

Connections processed by the SNI proxy.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordSNIConnection
Example: gerbil_sni_connections_total{result="success"} 3

Histogram (s)

Lifetime distribution of proxied TLS connections.

Show Details

Unit: seconds
Labels: —
Emission path: metrics.RecordSNIConnectionDuration
Example: gerbil_sni_connection_duration_seconds_bucket{le="10"} 4

UpDownCounter

Active SNI tunnels.

Show Details

Unit: 1
Labels: —
Emission path: metrics.RecordSNIActiveConnection
Example: gerbil_sni_active_connections 2

Counter

SNI route cache hits and misses.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordSNIRouteCacheHit
Example: gerbil_sni_route_cache_hits_total{result="hit"} 10

Counter

SNI route API requests.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordSNIRouteAPIRequest
Example: gerbil_sni_route_api_requests_total{result="success"} 5

Histogram (s)

Route API call latency.

Show Details

Unit: seconds
Labels: —
Emission path: metrics.RecordSNIRouteAPILatency
Example: gerbil_sni_route_api_latency_seconds_bucket{le="0.25"} 4

Counter

Routes using local overrides.

Show Details

Unit: 1
Labels: hit
Emission path: metrics.RecordSNILocalOverride
Example: gerbil_sni_local_override_total{hit="true"} 1

Counter

PROXY protocol events.

Show Details

Unit: 1
Labels: event
Emission path: metrics.RecordSNITrustedProxyEvent
Example: gerbil_sni_trusted_proxy_events_total{event="parsed"} 2

Counter

PROXY protocol parse failures.

Show Details

Unit: 1
Labels: —
Emission path: metrics.RecordSNIProxyProtocolParseError
Example: gerbil_sni_proxy_protocol_parse_errors_total 1

Counter (bytes)

Bytes proxied through SNI tunnels.

Show Details

Unit: bytes
Labels: direction
Emission path: metrics.RecordSNIDataBytes
Example: gerbil_sni_data_bytes_total{direction="ingress"} 4096

Counter

SNI tunnel terminations keyed by reason.

Show Details

Unit: 1
Labels: reason
Emission path: metrics.RecordSNITunnelTermination
Example: gerbil_sni_tunnel_terminations_total{reason="client_close"} 1

Show HTTP API & Peer Management

Counter

HTTP requests to the management API.

Show Details

Unit: 1
Labels: endpoint, method, status_code
Emission path: metrics.RecordHTTPRequest
Example: gerbil_http_requests_total{endpoint="/peer",method="POST",status_code="200"} 1

Histogram (s)

HTTP request handling time.

Show Details

Unit: seconds
Labels: endpoint, method
Emission path: metrics.RecordHTTPRequestDuration
Example: gerbil_http_request_duration_seconds_bucket{endpoint="/peer",method="POST",le="0.1"} 3

Counter

Peer lifecycle operations.

Show Details

Unit: 1
Labels: operation, result
Emission path: metrics.RecordPeerOperation
Example: gerbil_peer_operations_total{operation="add",result="success"} 1

Counter

Proxy mapping update API calls.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordProxyMappingUpdateRequest
Example: gerbil_proxy_mapping_update_requests_total{result="success"} 1

Counter

Destination update API calls.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordDestinationsUpdateRequest
Example: gerbil_destinations_update_requests_total{result="success"} 1

Show Remote Config & Reporting

Counter

Remote configuration fetch attempts.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordRemoteConfigFetch
Example: gerbil_remote_config_fetches_total{result="success"} 1

Counter

Bandwidth report transmissions.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordBandwidthReport
Example: gerbil_bandwidth_reports_total{result="success"} 1

Counter (bytes)

Bytes per peer tracked by bandwidth calculation.

Show Details

Unit: bytes
Labels: peer, direction
Emission path: metrics.RecordPeerBandwidthBytes
Example: gerbil_peer_bandwidth_bytes_total{peer="abc",direction="rx"} 8192

Show System & Operations

Counter

Netlink events processed.

Show Details

Unit: 1
Labels: event_type
Emission path: metrics.RecordNetlinkEvent
Example: gerbil_netlink_events_total{event_type="link_up"} 1

Counter

Netlink or kernel errors.

Show Details

Unit: 1
Labels: component, error_type
Emission path: metrics.RecordNetlinkError
Example: gerbil_netlink_errors_total{component="wg",error_type="permission"} 1

Histogram (s)

Duration of reconciliation or sync loops.

Show Details

Unit: seconds
Labels: component
Emission path: metrics.RecordSyncDuration
Example: gerbil_sync_duration_seconds_bucket{component="remote_config",le="0.5"} 2

UpDownCounter

Current length of internal work queues.

Show Details

Unit: 1
Labels: queue
Emission path: metrics.RecordWorkqueueDepth
Example: gerbil_workqueue_depth{queue="relay"} 3

Counter

Kernel module load attempts.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordKernelModuleLoad
Example: gerbil_kernel_module_loads_total{result="success"} 1

Counter

Firewall rules applied.

Show Details

Unit: 1
Labels: result, chain
Emission path: metrics.RecordFirewallRuleApplied
Example: gerbil_firewall_rules_applied_total{result="success",chain="FORWARD"} 1

Counter

Configuration reloads.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordConfigReload
Example: gerbil_config_reloads_total{result="success"} 1

Counter

Process restart count.

Show Details

Unit: 1
Labels: —
Emission path: metrics.RecordRestart
Example: gerbil_restart_total 1

Counter

Authentication or peer validation failures.

Show Details

Unit: 1
Labels: peer, reason
Emission path: metrics.RecordAuthFailure
Example: gerbil_auth_failures_total{peer="abc",reason="invalid_key"} 1

Counter

Access-control denied events.

Show Details

Unit: 1
Labels: ifname, peer, policy
Emission path: metrics.RecordACLDenied
Example: gerbil_acl_denied_total{ifname="wg0",peer="abc",policy="deny"} 1

Float64Gauge

Days until certificate expiry.

Show Details

Unit: days
Labels: cert_name, ifname
Emission path: metrics.RecordCertificateExpiry
Example: gerbil_certificate_expiry_days{cert_name="server",ifname="wg0"} 42

Counter

Memory spikes detected by severity.

Show Details

Unit: 1
Labels: severity
Emission path: metrics.RecordMemorySpike
Example: gerbil_memory_spike_total{severity="warning"} 1

Counter

Heap profile files generated.

Show Details

Unit: 1
Labels: —
Emission path: metrics.RecordHeapProfileWritten
Example: gerbil_heap_profiles_written_total 1

Prometheus-style series for the same Gerbil metrics. Gerbil exposes these only when METRICS_BACKEND=prometheus.

In Prometheus mode, Gerbil registers a native Prometheus client and exposes the configured metrics path, default /metrics. In OTel mode, /metrics is not exposed.

Show WireGuard

gauge

Operational state of a WireGuard interface.

Show Details

Labels: ifname, instance • Unit: 1 • Path: metrics.RecordInterfaceUp
Example: gerbil_wg_interface_up{ifname="wg0",instance="gerbil-1"} 1

gauge

Number of configured peers per interface.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordPeersTotal
Example: gerbil_wg_peers_total{ifname="wg0"} 5

gauge

Current peer connection state.

Show Details

Labels: ifname, peer • Unit: 1 • Path: metrics.RecordPeerConnected
Example: gerbil_wg_peer_connected{ifname="wg0",peer="abc"} 1

gauge

Number of allowed IPs configured per peer.

Show Details

Labels: ifname, peer • Unit: 1 • Path: metrics.RecordAllowedIPsCount
Example: gerbil_allowed_ips_count{ifname="wg0",peer="abc"} 2

counter

Key rotation events.

Show Details

Labels: ifname, reason • Unit: 1 • Path: metrics.RecordKeyRotation
Example: gerbil_key_rotation_total{ifname="wg0",reason="scheduled"} 1

counter

WireGuard handshake attempts keyed by result.

Show Details

Labels: ifname, peer, result • Unit: 1 • Path: metrics.RecordHandshake
Example: gerbil_wg_handshakes_total{ifname="wg0",peer="abc",result="success"} 1

histogram

Distribution of WireGuard handshake latencies.

Show Details

Labels: ifname, peer • Unit: seconds • Path: metrics.RecordHandshakeLatency
Example: gerbil_wg_handshake_latency_seconds_bucket{ifname="wg0",peer="abc",le="0.1"} 3

histogram

Observed peer round-trip time.

Show Details

Labels: ifname, peer • Unit: seconds • Path: metrics.RecordPeerRTT
Example: gerbil_wg_peer_rtt_seconds_bucket{ifname="wg0",peer="abc",le="0.05"} 4

counter

Bytes received from a WireGuard peer.

Show Details

Labels: ifname, peer • Unit: bytes • Path: metrics.RecordBytesReceived
Example: gerbil_wg_bytes_received_total{ifname="wg0",peer="abc"} 8192

counter

Bytes transmitted to a WireGuard peer.

Show Details

Labels: ifname, peer • Unit: bytes • Path: metrics.RecordBytesTransmitted
Example: gerbil_wg_bytes_transmitted_total{ifname="wg0",peer="abc"} 16384

Show Relay

gauge

Active UDP relay sessions.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordActiveSession / metrics.RecordSession
Example: gerbil_active_sessions{ifname="wg0"} 3

counter

UDP packets processed by relay workers.

Show Details

Labels: ifname, type, direction • Unit: 1 • Path: metrics.RecordUDPPacket
Example: gerbil_udp_packets_total{ifname="wg0",type="data",direction="rx"} 42

histogram

Size distribution of packets forwarded through the relay.

Show Details

Labels: ifname, type • Unit: bytes • Path: metrics.RecordUDPPacketSize
Example: gerbil_udp_packet_size_bytes_bucket{ifname="wg0",type="data",le="1024"} 7

counter

Hole punch messages processed by result.

Show Details

Labels: ifname, result • Unit: 1 • Path: metrics.RecordHolePunchEvent
Example: gerbil_hole_punch_events_total{ifname="wg0",result="success"} 1

gauge

Active proxy mappings.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordProxyMapping
Example: gerbil_proxy_mapping_active{ifname="wg0"} 4

counter

Sessions rebuilt from communication patterns.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordSessionRebuilt
Example: gerbil_session_rebuilt_total{ifname="wg0"} 1

gauge

Active communication patterns.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordCommPattern
Example: gerbil_comm_pattern_active{ifname="wg0"} 2

counter

Items removed by cleanup routines.

Show Details

Labels: ifname, component • Unit: 1 • Path: metrics.RecordProxyCleanupRemoved
Example: gerbil_proxy_cleanup_removed_total{ifname="wg0",component="sessions"} 5

counter

Proxy connection errors.

Show Details

Labels: ifname, error_type • Unit: 1 • Path: metrics.RecordProxyConnectionError
Example: gerbil_proxy_connection_errors_total{ifname="wg0",error_type="timeout"} 1

gauge

Initial proxy mappings loaded.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordProxyInitialMappings
Example: gerbil_proxy_initial_mappings{ifname="wg0"} 8

counter

Proxy mapping updates.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordProxyMappingUpdate
Example: gerbil_proxy_mapping_updates_total{ifname="wg0"} 2

histogram

Duration of idle cleanup cycles.

Show Details

Labels: ifname, component • Unit: seconds • Path: metrics.RecordProxyIdleCleanupDuration
Example: gerbil_proxy_idle_cleanup_duration_seconds_bucket{ifname="wg0",component="sessions",le="0.1"} 1

Show SNI Proxy

gauge

Active SNI proxy connections.

Show Details

Labels: — • Unit: 1 • Path: metrics.RecordActiveProxyConnection
Example: gerbil_active_proxy_connections 2

counter

Route lookups keyed by result.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordProxyRouteLookup
Example: gerbil_proxy_route_lookups_total{result="hit"} 6

histogram

TLS handshake duration for the SNI proxy.

Show Details

Labels: — • Unit: seconds • Path: metrics.RecordProxyTLSHandshake
Example: gerbil_proxy_tls_handshake_seconds_bucket{le="0.1"} 2

counter

Bytes sent or received by the SNI proxy.

Show Details

Labels: direction • Unit: bytes • Path: metrics.RecordProxyBytesTransmitted
Example: gerbil_proxy_bytes_transmitted_total{direction="egress"} 16384

counter

Connections processed by the SNI proxy.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordSNIConnection
Example: gerbil_sni_connections_total{result="success"} 3

histogram

Lifetime distribution of proxied TLS connections.

Show Details

Labels: — • Unit: seconds • Path: metrics.RecordSNIConnectionDuration
Example: gerbil_sni_connection_duration_seconds_bucket{le="10"} 4

gauge

Active SNI tunnels.

Show Details

Labels: — • Unit: 1 • Path: metrics.RecordSNIActiveConnection
Example: gerbil_sni_active_connections 2

counter

SNI route cache hits and misses.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordSNIRouteCacheHit
Example: gerbil_sni_route_cache_hits_total{result="hit"} 10

counter

SNI route API requests.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordSNIRouteAPIRequest
Example: gerbil_sni_route_api_requests_total{result="success"} 5

histogram

Route API call latency.

Show Details

Labels: — • Unit: seconds • Path: metrics.RecordSNIRouteAPILatency
Example: gerbil_sni_route_api_latency_seconds_bucket{le="0.25"} 4

counter

Routes using local overrides.

Show Details

Labels: hit • Unit: 1 • Path: metrics.RecordSNILocalOverride
Example: gerbil_sni_local_override_total{hit="true"} 1

counter

PROXY protocol events.

Show Details

Labels: event • Unit: 1 • Path: metrics.RecordSNITrustedProxyEvent
Example: gerbil_sni_trusted_proxy_events_total{event="parsed"} 2

counter

PROXY protocol parse failures.

Show Details

Labels: — • Unit: 1 • Path: metrics.RecordSNIProxyProtocolParseError
Example: gerbil_sni_proxy_protocol_parse_errors_total 1

counter

Bytes proxied through SNI tunnels.

Show Details

Labels: direction • Unit: bytes • Path: metrics.RecordSNIDataBytes
Example: gerbil_sni_data_bytes_total{direction="ingress"} 4096

counter

SNI tunnel terminations keyed by reason.

Show Details

Labels: reason • Unit: 1 • Path: metrics.RecordSNITunnelTermination
Example: gerbil_sni_tunnel_terminations_total{reason="client_close"} 1

Show HTTP API & Peer Management

counter

HTTP requests to the management API.

Show Details

Labels: endpoint, method, status_code • Unit: 1 • Path: metrics.RecordHTTPRequest
Example: gerbil_http_requests_total{endpoint="/peer",method="POST",status_code="200"} 1

histogram

HTTP request handling time.

Show Details

Labels: endpoint, method • Unit: seconds • Path: metrics.RecordHTTPRequestDuration
Example: gerbil_http_request_duration_seconds_bucket{endpoint="/peer",method="POST",le="0.1"} 3

counter

Peer lifecycle operations.

Show Details

Labels: operation, result • Unit: 1 • Path: metrics.RecordPeerOperation
Example: gerbil_peer_operations_total{operation="add",result="success"} 1

counter

Proxy mapping update API calls.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordProxyMappingUpdateRequest
Example: gerbil_proxy_mapping_update_requests_total{result="success"} 1

counter

Destination update API calls.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordDestinationsUpdateRequest
Example: gerbil_destinations_update_requests_total{result="success"} 1

Show Remote Config & Reporting

counter

Remote configuration fetch attempts.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordRemoteConfigFetch
Example: gerbil_remote_config_fetches_total{result="success"} 1

counter

Bandwidth report transmissions.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordBandwidthReport
Example: gerbil_bandwidth_reports_total{result="success"} 1

counter

Bytes per peer tracked by bandwidth calculation.

Show Details

Labels: peer, direction • Unit: bytes • Path: metrics.RecordPeerBandwidthBytes
Example: gerbil_peer_bandwidth_bytes_total{peer="abc",direction="rx"} 8192

Show System & Operations

counter

Netlink events processed.

Show Details

Labels: event_type • Unit: 1 • Path: metrics.RecordNetlinkEvent
Example: gerbil_netlink_events_total{event_type="link_up"} 1

counter

Netlink or kernel errors.

Show Details

Labels: component, error_type • Unit: 1 • Path: metrics.RecordNetlinkError
Example: gerbil_netlink_errors_total{component="wg",error_type="permission"} 1

histogram

Duration of reconciliation or sync loops.

Show Details

Labels: component • Unit: seconds • Path: metrics.RecordSyncDuration
Example: gerbil_sync_duration_seconds_bucket{component="remote_config",le="0.5"} 2

gauge

Current length of internal work queues.

Show Details

Labels: queue • Unit: 1 • Path: metrics.RecordWorkqueueDepth
Example: gerbil_workqueue_depth{queue="relay"} 3

counter

Kernel module load attempts.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordKernelModuleLoad
Example: gerbil_kernel_module_loads_total{result="success"} 1

counter

Firewall rules applied.

Show Details

Labels: result, chain • Unit: 1 • Path: metrics.RecordFirewallRuleApplied
Example: gerbil_firewall_rules_applied_total{result="success",chain="FORWARD"} 1

counter

Configuration reloads.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordConfigReload
Example: gerbil_config_reloads_total{result="success"} 1

counter

Process restart count.

Show Details

Labels: — • Unit: 1 • Path: metrics.RecordRestart
Example: gerbil_restart_total 1

counter

Authentication or peer validation failures.

Show Details

Labels: peer, reason • Unit: 1 • Path: metrics.RecordAuthFailure
Example: gerbil_auth_failures_total{peer="abc",reason="invalid_key"} 1

counter

Access-control denied events.

Show Details

Labels: ifname, peer, policy • Unit: 1 • Path: metrics.RecordACLDenied
Example: gerbil_acl_denied_total{ifname="wg0",peer="abc",policy="deny"} 1

gauge

Days until certificate expiry.

Show Details

Labels: cert_name, ifname • Unit: days • Path: metrics.RecordCertificateExpiry
Example: gerbil_certificate_expiry_days{cert_name="server",ifname="wg0"} 42

counter

Memory spikes detected by severity.

Show Details

Labels: severity • Unit: 1 • Path: metrics.RecordMemorySpike
Example: gerbil_memory_spike_total{severity="warning"} 1

counter

Heap profile files generated.

Show Details

Labels: — • Unit: 1 • Path: metrics.RecordHeapProfileWritten
Example: gerbil_heap_profiles_written_total 1

Pangolin Kubernetes Controller metrics

OpenTelemetry (OTel)
Prometheus

Additional OpenTelemetry metric instruments exposed by the Pangolin Kubernetes Controller.

The controller exposes Prometheus-native metrics and additional OTel-backed metrics on the same /metrics endpoint. The OTel-backed series use the pangolin_controller_* prefix.

Show Reconcile

Histogram (s)

Duration of each reconcile phase.

Show Details

Unit: seconds
Labels: phase, result
Label values:
phase: middlewares | routers | serversTransports | services | tcp | udp
result: success | error
Emission path: OTel reconcile phase instrumentation
Example: pangolin_controller_reconcile_phase_duration_seconds_bucket{phase="routers",result="success",le="0.5"} 3

UpDownCounter

Number of active reconcile routines by phase.

Show Details

Unit: 1
Labels: phase
Label values: middlewares | routers | serversTransports | services | tcp | udp
Emission path: Parallel reconcile instrumentation
Example: pangolin_controller_active_reconcile_routines{phase="routers"} 1

Counter

Controller loop iterations by outcome.

Show Details

Unit: 1
Labels: outcome
Label values: success | nochange | error
Emission path: Controller loop instrumentation
Example: pangolin_controller_loop_iterations_total{outcome="success"} 10

Show Fetch & Config

Histogram (s)

Duration of remote fetch cycle HTTP requests.

Show Details

Unit: seconds
Labels: status_code, status_class
Label values:
status_code: 200 | 304 | 401 | 403 | 404 | 5xx
status_class: 2xx | 3xx | 4xx | 5xx
Emission path: Remote config fetch instrumentation
Example: pangolin_controller_fetch_duration_seconds_bucket{status_code="200",status_class="2xx",le="0.25"} 4

Histogram (s)

Duration of configuration parsing.

Show Details

Unit: seconds
Labels: section
Label values: full
Emission path: Config parse instrumentation
Example: pangolin_controller_config_parse_duration_seconds_bucket{section="full",le="0.1"} 2

Show Kubernetes API

Histogram (s)

Duration of Kubernetes API requests.

Show Details

pangolin_controller_k8s_request_duration_seconds_bucket{verb="patch",resource_kind="IngressRoute",result="success",forced="false",le="0.25"} 3

Counter

Total Kubernetes API requests.

Show Details

Unit: 1
Labels: verb, resource_kind, result, forced
Emission path: Kubernetes API request instrumentation
Example: pangolin_controller_k8s_requests_total{verb="patch",resource_kind="IngressRoute",result="success",forced="false"} 1

Counter

Retry attempts in the SSA apply loop.

Show Details

Unit: 1
Labels: reason, operation, resource_kind
Label values:
reason: conflict | transient | timeout
operation: get | create | patch | delete | apply
Emission path: SSA apply retry instrumentation
Example: pangolin_controller_retries_total{reason="conflict",operation="patch",resource_kind="IngressRoute"} 1

Show Garbage Collection

Histogram (s)

Duration of garbage collection runs.

Show Details

Unit: seconds
Labels: result
Label values: success | fail | dryrun
Emission path: GC instrumentation
Example: pangolin_controller_gc_run_duration_seconds_bucket{result="success",le="0.5"} 1

Prometheus-native metrics exposed by the Pangolin Kubernetes Controller. These use the pangolin_kube_controller_* prefix.

The metrics endpoint is exposed at GET /metrics on METRICS_ADDR, default :9090. The same HTTP server also exposes /healthz, /readyz, and optionally /debug/pprof/ when pprof is enabled.

Show Reconcile

histogram

Duration of a full successful reconcile loop.

Show Details

Labels: —
Unit: seconds
Path: Collector.ReconcileDuration
Example: pangolin_kube_controller_reconcile_seconds_bucket{le="0.5"} 3

counter

Total errors during reconcile steps.

Show Details

Labels: —
Unit: 1
Path: Collector.ReconcileErrors
Example: pangolin_kube_controller_reconcile_errors_total 2

gauge

Number of consecutive reconcile or fetch errors.

Show Details

Labels: —
Unit: 1
Path: Collector.ConsecutiveErrors
Example: pangolin_kube_controller_consecutive_errors 0

gauge

Unix timestamp of the last successful fetch.

Show Details

Labels: —
Unit: seconds since Unix epoch
Path: Collector.LastFetchSuccess
Example: pangolin_kube_controller_last_fetch_success_timestamp_seconds 1767225600

gauge

Controller readiness state.

Show Details

Labels: —
Unit: 1
Value: 1 = ready, 0 = not ready
Path: Collector.Ready
Example: pangolin_kube_controller_ready 1

Show Objects

counter

Applied objects by kind and action.

Show Details

Labels: kind, action
Label values:
kind: IngressRoute | Middleware | TraefikService | other managed Traefik CRDs
action: create | patch
Unit: 1
Path: Collector.AppliedObjects
Example: pangolin_kube_controller_objects_applied_total{kind="IngressRoute",action="patch"} 1

counter

Deleted objects by kind.

Show Details

Labels: kind
Unit: 1
Path: Collector.DeletedObjects
Example: pangolin_kube_controller_objects_deleted_total{kind="Middleware"} 1

gauge

Desired objects count by kind from the last processed config.

Show Details

Labels: kind
Unit: 1
Path: Collector.DesiredObjects
Example: pangolin_kube_controller_desired_objects_count{kind="IngressRoute"} 12

Show Garbage Collection

counter

Garbage-collected objects by kind and reason.

Show Details

Labels: kind, reason
Label values:
reason: immediate | grace
Unit: 1
Path: Collector.GCDeletedTotal
Example: pangolin_kube_controller_gc_deleted_total{kind="IngressRoute",reason="grace"} 1

counter

Garbage collection runs by result.

Show Details

Labels: result
Label values: start | success | fail | dryrun
Unit: 1
Path: Collector.GCRunsTotal
Example: pangolin_kube_controller_gc_runs_total{result="success"} 1

counter

Grace deletion queue items dropped due to queue overflow.

Show Details

Labels: —
Unit: 1
Path: Collector.GraceQueueDropped
Example: pangolin_kube_controller_grace_queue_dropped_total 0

gauge

Current depth of the grace deletion queue.

Show Details

Labels: —
Unit: 1
Path: Collector.GraceQueueDepth
Example: pangolin_kube_controller_grace_queue_depth 3

Show Instance Label

counter

Successful Traefik instance label resolutions.

Show Details

Labels: —
Unit: 1
Path: Collector.InstanceLabelDetectSuccess
Example: pangolin_kube_controller_instance_label_detect_success_total 1

counter

Failed Traefik instance label resolutions or verifications.

Show Details

Labels: —
Unit: 1
Path: Collector.InstanceLabelDetectFailures
Example: pangolin_kube_controller_instance_label_detect_failure_total 1

gauge

Unix timestamp of the last instance label verification.

Show Details

Labels: —
Unit: seconds since Unix epoch
Path: Collector.InstanceLabelLastCheck
Example: pangolin_kube_controller_instance_label_last_check_timestamp_seconds 1767225600

Show Leader Election

gauge

Current leader election state of the controller.

Show Details

Labels: —
Unit: 1
Value: 1 = leader, 0 = follower, -1 = leader election disabled
Path: Collector.LeaderState
Example: pangolin_kube_controller_leader_state 1

Show Runtime

runtime metrics

Go runtime metrics registered by the Prometheus Go collector.

Show Details

Labels: varies
Unit: varies
Path: collectors.NewGoCollector()
Example: go_goroutines 14

process metrics

Process metrics registered by the Prometheus process collector.

Show Details

Labels: varies
Unit: varies
Path: collectors.NewProcessCollector()
Example: process_resident_memory_bytes 52428800

References

Have improvements or a missing metric? Open an issue or PR referencing this page.

About

Manage Pangolin

Self-host Pangolin

Development

Metrics and Observability

Capability Matrix

Newt

Gerbil

Kubernetes Controller

Observability Capability Matrix

Supported Signals

Collection Patterns

Newt

Gerbil

Pangolin Kubernetes Controller

Alerting Examples

Community Metrics Guide

Newt Kubernetes Monitoring

Controller Monitoring Values

Versions

Full Metric Reference

Newt metrics

Gerbil metrics

Pangolin Kubernetes Controller metrics

References

Capability Matrix

Newt

Gerbil

Kubernetes Controller

​Observability Capability Matrix

​Supported Signals

​Collection Patterns

​Newt

​Gerbil

​Pangolin Kubernetes Controller

​Alerting Examples

Community Metrics Guide

Newt Kubernetes Monitoring

Controller Monitoring Values

​Versions

​Full Metric Reference

​Newt metrics

​Gerbil metrics

​Pangolin Kubernetes Controller metrics

​References

Observability Capability Matrix

Supported Signals

Collection Patterns

Newt

Gerbil

Pangolin Kubernetes Controller

Alerting Examples

Versions

Full Metric Reference

Newt metrics

Gerbil metrics

Pangolin Kubernetes Controller metrics

References