Metrics and Observability

Try free on Pangolin Cloud

Fastest way to get started with Pangolin using the hosted control plane. No credit card required.

Pangolin exposes observability signals across multiple components, but not every component provides the same telemetry surface. This page explains which metrics, traces, logs, and profiling endpoints are available today, how they are exposed, and which collection patterns are recommended for production deployments. Newt provides the broadest native observability support with Prometheus metrics, OTLP metrics, OTLP traces, health checks, and optional pprof. Gerbil supports metrics through either a Prometheus backend or an OTLP backend. The Pangolin Kubernetes Controller exposes a Prometheus-compatible scrape endpoint and includes additional OTel-backed metric instruments on that endpoint.

This page focuses on Pangolin-native observability. For a community walkthrough that collects Traefik metrics with Prometheus and Grafana, see the community metrics guide.All currently documented component metrics are listed in the Full Metric Reference.

Capability Matrix

Compare metrics, traces, logs, and profiling support across Pangolin components.

Newt

Native Prometheus metrics, OTLP metrics, OTLP traces, health checks, and optional pprof.

Gerbil

Metrics-only observability using either a Prometheus or OTLP backend.

Kubernetes Controller

Prometheus scraping, health probes, ServiceMonitor examples, and optional pprof.

Observability Capability Matrix

Component	Metrics	Traces	Logs	Profiling
`newt`	Prometheus, OTLP	OTLP	stdout	pprof optional
`gerbil`	Prometheus or OTLP	—	stdout	—
`pangolin-kube-controller`	Prometheus scrape	—	stdout	pprof optional

Supported Signals

Signal	What it means
Prometheus scrape metrics	Pull-based metrics served over HTTP on `/metrics`
OTLP metrics	Push-based OpenTelemetry metrics sent to an OTel Collector or compatible backend
OTLP traces	Distributed traces sent to an OTel Collector or trace backend
Application logs	stdout, file logs, audit logs, or platform logs
Profiling	Debug endpoints such as `pprof`

Collection Patterns

Use one or more of the following patterns depending on your deployment model. Prometheus scrape mode is the simplest option for local or Kubernetes monitoring. OTLP is useful when you already operate an OpenTelemetry Collector or want to forward telemetry to a managed backend such as Grafana Cloud, Mimir, or Tempo.

Prometheus scrape mode
OTel Collector
Kubernetes ServiceMonitor
Logs to Loki

prometheus.yml (fragment)

scrape_configs:
  - job_name: newt
    static_configs:
      - targets: ["newt:2112"]

  - job_name: gerbil
    metrics_path: /metrics
    static_configs:
      - targets: ["gerbil:3003"]

  - job_name: pangolin-kube-controller
    static_configs:
      - targets: ["pangolin-kube-controller:9090"]

otel-collector.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:
  prometheus:
    config:
      scrape_configs:
        - job_name: pangolin-kube-controller
          static_configs:
            - targets: ["pangolin-kube-controller:9090"]

processors:
  batch: {}

exporters:
  prometheusremotewrite:
    endpoint: https://mimir.example.com/api/v1/push
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Today the traces pipeline is relevant only for newt. pangolin-kube-controller exposes additional OTel instruments, but they are exported on the same Prometheus scrape endpoint rather than pushed through OTLP.

servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pangolin-kube-controller
spec:
  selector:
    matchLabels:
      app: pangolin-kube-controller
  endpoints:
    - port: http-metrics
      path: /metrics
      interval: 30s

helm-values.yaml

global:
  metrics:
    enabled: true
    service:
      enabled: true
    serviceMonitor:
      enabled: true

controller:
  monitoring:
    serviceMonitor:
      enabled: true

Helm values for Newt are documented on the Kubernetes Newt configuration page. Controller and Gerbil ServiceMonitor values are documented on the Kubernetes Pangolin configuration page.

promtail.yaml

server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: pangolin
    static_configs:
      - targets: [localhost]
        labels:
          job: pangolin
          __path__: /var/log/containers/*pangolin*.log

Use Grafana Alloy, Promtail, Fluent Bit, or your preferred SIEM forwarder to collect container and application logs.

Newt

Newt metrics are not typically enabled in default deployments. Turn them on explicitly and expose the admin address only where you intend to scrape or profile. Available metrics are listed in Newt metrics.

Configuration

Environment Variables
CLI Args
Helm Values

NEWT_METRICS_PROMETHEUS_ENABLED=true
NEWT_METRICS_OTLP_ENABLED=true
NEWT_ADMIN_ADDR=:2112
OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
OTEL_EXPORTER_OTLP_INSECURE=true
OTEL_METRIC_EXPORT_INTERVAL=15s
NEWT_PPROF_ENABLED=false

newt \
  --metrics=true \
  --otlp=true \
  --metrics-admin-addr=:2112 \
  --pprof=false \
  --endpoint=https://pangolin.example.com \
  --id=saz281jfa8z37zg \
  --secret=your-secret

global:
  metrics:
    enabled: true
    adminAddr: ":2112"
    service:
      enabled: true
    serviceMonitor:
      enabled: true
    pprofEnabled: false

Signals and endpoints

Prometheus scrape endpoint: /metrics on NEWT_ADMIN_ADDR
Health endpoint: /healthz on the same admin server
OTLP metrics: enabled with NEWT_METRICS_OTLP_ENABLED=true or --otlp=true
OTLP traces: initialized when OTLP export is enabled
Profiling: /debug/pprof/* when NEWT_PPROF_ENABLED=true

The Newt OTLP switch enables the OTLP telemetry pipeline. The current environment variable name contains METRICS, but the implementation also initializes OTLP tracing when OTLP is enabled.

Newt’s admin server defaults to a loopback bind address. In containers or Kubernetes, set NEWT_ADMIN_ADDR=:2112 or another non-loopback address only when you intentionally want Prometheus, pprof, or health checks to reach it.

Examples

Direct Prometheus
OTLP metrics and traces

prometheus.yml (fragment)

scrape_configs:
  - job_name: newt
    static_configs:
      - targets: ["newt:2112"]

otel-collector.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch: {}

exporters:
  prometheusremotewrite:
    endpoint: https://mimir.example.com/api/v1/push
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Gerbil

Gerbil supports metrics only. Choose either a native Prometheus backend or an OTLP metrics backend at runtime. Those backends are mutually exclusive. Available metrics are listed in Gerbil metrics.

Configuration

Environment Variables
CLI Args

METRICS_ENABLED=true
METRICS_BACKEND=prometheus
METRICS_PATH=/metrics
LISTEN=:3003

# OTel mode
OTEL_METRICS_PROTOCOL=grpc
OTEL_METRICS_ENDPOINT=otel-collector:4317
OTEL_METRICS_INSECURE=true
OTEL_METRICS_EXPORT_INTERVAL=60s
OTEL_METRICS_TIMEOUT=10s

./gerbil \
  --metrics-enabled \
  --metrics-backend=prometheus \
  --metrics-path=/metrics \
  --config=/etc/gerbil/config.json

./gerbil \
  --metrics-enabled \
  --metrics-backend=otel \
  --otel-metrics-protocol=grpc \
  --otel-metrics-endpoint=otel-collector:4317 \
  --otel-metrics-insecure \
  --otel-metrics-export-interval=10s \
  --otel-metrics-timeout=10s \
  --config=/etc/gerbil/config.json

Signals and endpoints

Prometheus metrics endpoint: METRICS_PATH, default /metrics
Metrics are served on Gerbil’s configured HTTP listen address
The Docker Compose metrics example commonly scrapes gerbil:3003
Health endpoint: /healthz
OTLP metrics: enabled when METRICS_BACKEND=otel
Traces: not supported yet
Profiling: not supported yet

Examples

Prometheus scrape
OTLP metrics

prometheus.yml (fragment)

scrape_configs:
  - job_name: gerbil
    metrics_path: /metrics
    static_configs:
      - targets: ["gerbil:3003"]

Replace gerbil:3003 with the actual Gerbil HTTP listen address in your deployment.

otel-collector.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch: {}

exporters:
  prometheusremotewrite:
    endpoint: https://mimir.example.com/api/v1/push

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite]

Pangolin Kubernetes Controller

The controller exposes a Prometheus-compatible /metrics endpoint and standard health probes. It also registers additional OpenTelemetry metric instruments, but those instruments are exported on the same scrape endpoint rather than pushed through OTLP. Available metrics are listed in Controller metrics.

pprof is available only when explicitly enabled via ENABLE_PPROF=true.

Configuration

Environment Variables
Helm Values

METRICS_ADDR=:9090
DISABLE_LIVEZ=false
ENABLE_PPROF=false

controller:
  service:
    enabled: true
    port: 9090
    portName: metrics
  monitoring:
    serviceMonitor:
      enabled: true
    podMonitor:
      enabled: false
    prometheusRule:
      enabled: false

Expose the controller metrics endpoint only inside trusted networks or through Kubernetes-native monitoring resources. If TLS or auth is required, terminate it with a Service mesh, Ingress, sidecar, or platform-specific monitoring gateway unless native TLS support is verified.

Endpoints

/metrics on METRICS_ADDR
/healthz and /readyz for readiness
/livez and /health/live for liveness unless DISABLE_LIVEZ=true
/debug/pprof/* when ENABLE_PPROF=true

Representative metrics and Kubernetes examples

service-and-servicemonitor.yaml

apiVersion: v1
kind: Service
metadata:
  name: pangolin-kube-controller
  labels:
    app: pangolin-kube-controller
spec:
  selector:
    app: pangolin-kube-controller
  ports:
    - name: http-metrics
      port: 9090
      targetPort: 9090
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pangolin-kube-controller
spec:
  selector:
    matchLabels:
      app: pangolin-kube-controller
  endpoints:
    - port: http-metrics
      path: /metrics
      interval: 30s

Alerting Examples

Newt
Gerbil
Controller

increase(newt_connection_errors_total[5m]) > 10

histogram_quantile(
  0.95,
  sum(rate(newt_tunnel_latency_seconds_bucket[5m])) by (le)
) > 1

increase(newt_tunnel_reconnects_total[10m]) > 20

max_over_time(gerbil_wg_interface_up[5m]) by (ifname) < 1

sum(gerbil_wg_peers_total) > 0
and
sum(gerbil_wg_peer_connected) == 0

min_over_time(pangolin_kube_controller_ready[5m]) == 0

increase(pangolin_kube_controller_reconcile_errors_total[10m]) > 5

Community Metrics Guide

Traefik and metrics collection with Prometheus and Grafana.

Newt Kubernetes Monitoring

Verified Newt chart values for metrics, Services, and ServiceMonitor resources.

Controller Monitoring Values

Verified chart values for controller Services, ServiceMonitor, PodMonitor, and PrometheusRule resources.

Versions

Component	Signal	Since version
Newt	Prometheus scrape metrics	`v1.6.0`
Newt	OTLP metrics	`v1.6.0`
Newt	OTLP traces	`v1.6.0`
Newt	pprof	`v1.10.4`
Gerbil	Prometheus scrape metrics	`v1.4.0`
Gerbil	OTLP metrics	`v1.4.0`
Pangolin Kubernetes Controller	Prometheus scrape metrics	`v0.1.0-alpha.1`

Full Metric Reference

The full reference below is grouped by component.

Newt: Prometheus metrics, OTLP metrics, OTLP traces
Gerbil: Prometheus or OTLP metrics
Pangolin Kubernetes Controller: Prometheus-native metrics and additional OTel-backed scrape metrics

Metric names, labels, and defaults can change between component releases.

Newt metrics

OpenTelemetry (OTel)
Prometheus

newt

OpenTelemetry metric instruments exposed by Newt. Expand each section to see individual metrics with labels, units, emission points, and examples.

Show Site & Build

newt_site_registrations_total

Counter

Counts Pangolin registration attempts keyed by result.

Show Details

Unit: 1
Labels: result (success|failure), site_id
Emission path: telemetry.IncSiteRegistration
Example: newt_site_registrations_total{result="success",site_id="abc"} 1

newt_site_online

ObservableGauge

0/1 heartbeat for the active site.

Show Details

Unit: 1
Labels: site_id
Emission path: state.TelemetryView (callback)
Example: newt_site_online{site_id="self"} 1

newt_site_last_heartbeat_seconds

ObservableGauge

Seconds since last Pangolin heartbeat.

Show Details

Unit: seconds
Labels: site_id
Emission path: TouchHeartbeat (callback)
Example: newt_site_last_heartbeat_seconds{site_id="self"} 3.2

newt_build_info

ObservableGauge

Constant 1 with build metadata labels.

Show Details

Unit: 1
Labels: version, commit
Emission path: Build info registration
Example: newt_build_info{version="1.2.3",commit="abc123"} 1

newt_restart_count_total

Counter

Process boot indicator (increments once per process start).

Show Details

Unit: 1
Labels: —
Emission path: RegisterBuildInfo
Example: newt_restart_count_total 1

newt_cert_rotation_total

Counter

Certificate rotation events keyed by result.

Show Details

Unit: 1
Labels: result
Emission path: IncCertRotation
Example: newt_cert_rotation_total{result="success"} 1

newt_config_reloads_total

Counter

Config reload attempts keyed by result.

Show Details

Unit: 1
Labels: result
Emission path: telemetry.IncConfigReload
Example: newt_config_reloads_total{result="success"} 1

newt_config_apply_seconds

Histogram (s)

Duration per config-apply phase keyed by phase and result.

Show Details

Unit: seconds
Labels: phase, result
Emission path: telemetry.ObserveConfigApply
Example: newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3

Show Tunnel

newt_tunnel_sessions

ObservableGauge

Active sessions per tunnel (or collapsed).

Show Details

Unit: 1
Labels: site_id, tunnel_id
Emission path: RegisterStateView
Example: newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2

newt_tunnel_bytes_total

Counter (bytes)

Traffic per tunnel, direction, and protocol.

Show Details

Unit: bytes
Labels: tunnel_id, direction (ingress|egress), protocol (tcp|udp)
Emission path: Proxy manager
Example: newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192

newt_tunnel_latency_seconds

Histogram (s)

RTT samples per tunnel/transport.

Show Details

Unit: seconds
Labels: tunnel_id, transport
Emission path: Health checks
Example: newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4

newt_tunnel_reconnects_total

Counter

Reconnect attempts keyed by initiator & reason.

Show Details

Unit: 1
Labels: tunnel_id, initiator (client|server), reason
Emission path: telemetry.IncReconnect
Example: newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3

Show Connection & Auth

newt_connection_attempts_total

Counter

Auth/WebSocket connection attempts keyed by transport & result.

Show Details

Unit: 1
Labels: transport, result
Emission path: telemetry.IncConnAttempt
Example: newt_connection_attempts_total{transport="websocket",result="failure"} 2

newt_connection_errors_total

Counter

Connection errors keyed by transport and type.

Show Details

Unit: 1
Labels: transport, error_type
Emission path: telemetry.IncConnError
Example: newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1

Show WebSocket

newt_websocket_connect_latency_seconds

Histogram (s)

Dial latency for Pangolin WebSocket.

Show Details

Unit: seconds
Labels: result, transport
Emission path: ObserveWSConnectLatency
Example: newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1

newt_websocket_disconnects_total

Counter

WebSocket disconnects keyed by reason.

Show Details

Unit: 1
Labels: reason, tunnel_id
Emission path: IncWSDisconnect
Example: newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2

newt_websocket_keepalive_failures_total

Counter

Ping/Pong failures observed by keepalive.

Show Details

Unit: 1
Labels: reason (e.g., ping_write, pong_timeout)
Emission path: telemetry.IncWSKeepaliveFailure(ctx, "ping_write")
Example: newt_websocket_keepalive_failures_total{reason="ping_write"} 1

newt_websocket_session_duration_seconds

Histogram (s)

Duration of established WS sessions keyed by result.

Show Details

Unit: seconds
Labels: result (success|error)
Emission path: telemetry.ObserveWSSessionDuration(ctx, time.Since(start).Seconds(), "error")
Example: newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3

newt_websocket_connected

ObservableGauge

Current WS connection state (0/1).

Show Details

Unit: 1
Labels: —
Emission path: telemetry.SetWSConnectionState(true|false)
Example: newt_websocket_connected 1

newt_websocket_reconnects_total

Counter

WebSocket reconnect attempts keyed by reason.

Show Details

Unit: 1
Labels: reason
Emission path: telemetry.IncWSReconnect(ctx, "ping_write")
Example: newt_websocket_reconnects_total{reason="ping_write"} 1

newt_websocket_messages_total

Counter

In/out WS messages keyed by direction & type.

Show Details

Unit: 1
Labels: direction (in|out), msg_type (ping|pong|text|…)
Emission path: IncWSMessage
Example: newt_websocket_messages_total{direction="out",msg_type="ping"} 4

Show Proxy

newt_proxy_active_connections

ObservableGauge

Active TCP/UDP proxy connections per tunnel/protocol.

Show Details

Unit: 1
Labels: protocol, tunnel_id
Emission path: Proxy callback
Example: newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3

newt_proxy_buffer_bytes

ObservableGauge (bytes)

Proxy buffer pool size.

Show Details

Unit: bytes
Labels: protocol, tunnel_id
Emission path: Proxy callback
Example: newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240

newt_proxy_async_backlog_bytes

ObservableGauge (bytes)

Unflushed async byte backlog.

Show Details

Unit: bytes
Labels: protocol, tunnel_id
Emission path: Proxy callback
Example: newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096

newt_proxy_drops_total

Counter

Proxy write drops keyed by protocol/tunnel.

Show Details

Unit: 1
Labels: protocol, tunnel_id
Emission path: IncProxyDrops
Example: newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2

newt_proxy_accept_total

Counter

Proxy accept events keyed by result/reason.

Show Details

Unit: 1
Labels: tunnel_id, protocol, result, reason
Emission path: telemetry.IncProxyAccept(ctx, tunnelID, "tcp", "failure", "timeout")
Example: newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1

newt_proxy_connections_total

Counter

Lifecycle events (opened/closed) per connection.

Show Details

Unit: 1
Labels: tunnel_id, protocol, event (opened|closed)
Emission path: telemetry.IncProxyConnectionEvent(ctx, tunnelID, "tcp", telemetry.ProxyConnectionOpened)
Example: newt_proxy_connections_total{protocol="tcp",event="opened"} 1

newt_proxy_connection_duration_seconds

Histogram (s)

Duration of completed proxy connections.

Show Details

Unit: seconds
Labels: tunnel_id, protocol, result
Emission path: telemetry.ObserveProxyConnectionDuration(ctx, tunnelID, "tcp", "success", seconds)
Example: newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3

newt

Prometheus-style series for the same Newt metrics. Names, labels, and examples mirror the OTel tab.

Show Site & Build

newt_site_registrations_total

counter

Counts Pangolin registration attempts keyed by result.

Show Details

Labels: result, site_id • Unit: 1 • Path: telemetry.IncSiteRegistration
Example: newt_site_registrations_total{result="success",site_id="abc"} 1

newt_site_online

gauge

0/1 heartbeat for the active site.

Show Details

Labels: site_id • Unit: 1 • Path: state.TelemetryView
Example: newt_site_online{site_id="self"} 1

newt_site_last_heartbeat_seconds

gauge

Seconds since last Pangolin heartbeat.

Show Details

Labels: site_id • Unit: seconds • Path: TouchHeartbeat
Example: newt_site_last_heartbeat_seconds{site_id="self"} 3.2

newt_build_info

gauge

Constant 1 with build metadata labels.

Show Details

Labels: version, commit • Unit: 1 • Path: Build info registration
Example: newt_build_info{version="1.2.3",commit="abc123"} 1

newt_restart_count_total

counter

Process boot indicator (increments once).

Show Details

Labels: — • Unit: 1 • Path: RegisterBuildInfo
Example: newt_restart_count_total 1

newt_cert_rotation_total

counter

Certificate rotation events keyed by result.

Show Details

Labels: result • Unit: 1 • Path: IncCertRotation
Example: newt_cert_rotation_total{result="success"} 1

newt_config_reloads_total

counter

Config reload attempts keyed by result.

Show Details

Labels: result • Unit: 1 • Path: telemetry.IncConfigReload
Example: newt_config_reloads_total{result="success"} 1

newt_config_apply_seconds

histogram

Duration per config-apply phase & result.

Show Details

Labels: phase, result • Unit: seconds • Path: telemetry.ObserveConfigApply
Example: newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3

Show Tunnel

newt_tunnel_sessions

gauge

Active sessions per tunnel (or collapsed).

Show Details

Labels: site_id, tunnel_id • Unit: 1 • Path: RegisterStateView
Example: newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2

newt_tunnel_bytes_total

counter

Traffic per tunnel/direction/protocol.

Show Details

Labels: tunnel_id, direction, protocol • Unit: bytes • Path: Proxy manager
Example: newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192

newt_tunnel_latency_seconds

histogram

RTT samples per tunnel/transport.

Show Details

Labels: tunnel_id, transport • Unit: seconds • Path: Health checks
Example: newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4

newt_tunnel_reconnects_total

counter

Reconnect attempts by initiator & reason.

Show Details

Labels: tunnel_id, initiator, reason • Unit: 1 • Path: telemetry.IncReconnect
Example: newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3

Show Connection & Auth

newt_connection_attempts_total

counter

Auth/WebSocket attempts by transport & result.

Show Details

Labels: transport, result • Unit: 1 • Path: telemetry.IncConnAttempt
Example: newt_connection_attempts_total{transport="websocket",result="failure"} 2

newt_connection_errors_total

counter

Connection errors by transport and type.

Show Details

Labels: transport, error_type • Unit: 1 • Path: telemetry.IncConnError
Example: newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1

Show WebSocket

newt_websocket_connect_latency_seconds

histogram

Dial latency for Pangolin WebSocket.

Show Details

Labels: result, transport • Unit: seconds • Path: ObserveWSConnectLatency
Example: newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1

newt_websocket_disconnects_total

counter

WS disconnects by reason.

Show Details

Labels: reason, tunnel_id • Unit: 1 • Path: IncWSDisconnect
Example: newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2

newt_websocket_keepalive_failures_total

counter

Keepalive Ping/Pong failures.

Show Details

Labels: reason • Unit: 1 • Path: telemetry.IncWSKeepaliveFailure(ctx, "ping_write")
Example: newt_websocket_keepalive_failures_total{reason="ping_write"} 1

newt_websocket_session_duration_seconds

histogram

Duration of established WebSocket sessions by result.

Show Details

Labels: result • Unit: seconds • Path: telemetry.ObserveWSSessionDuration(...)
Example: newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3

newt_websocket_connected

gauge

Current WS connection status (0/1).

Show Details

Labels: — • Unit: 1 • Path: telemetry.SetWSConnectionState(true|false)
Example: newt_websocket_connected 1

newt_websocket_reconnects_total

counter

Reconnect attempts by reason.

Show Details

Labels: reason • Unit: 1 • Path: telemetry.IncWSReconnect(ctx, "ping_write")
Example: newt_websocket_reconnects_total{reason="ping_write"} 1

newt_websocket_messages_total

counter

In/out WS messages by direction & type.

Show Details

Labels: direction, msg_type • Unit: 1 • Path: IncWSMessage
Example: newt_websocket_messages_total{direction="out",msg_type="ping"} 4

Show Proxy

newt_proxy_active_connections

gauge

Active TCP/UDP proxy connections per tunnel/protocol.

Show Details

Labels: protocol, tunnel_id • Unit: 1 • Path: Proxy callback
Example: newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3

newt_proxy_buffer_bytes

gauge

Proxy buffer pool size.

Show Details

Labels: protocol, tunnel_id • Unit: bytes • Path: Proxy callback
Example: newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240

newt_proxy_async_backlog_bytes

gauge

Unflushed async byte backlog.

Show Details

Labels: protocol, tunnel_id • Unit: bytes • Path: Proxy callback
Example: newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096

newt_proxy_drops_total

counter

Proxy write drops per protocol/tunnel.

Show Details

Labels: protocol, tunnel_id • Unit: 1 • Path: IncProxyDrops
Example: newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2

newt_proxy_accept_total

counter

Proxy accept events by result/reason.

Show Details

Labels: tunnel_id, protocol, result, reason • Unit: 1 • Path: telemetry.IncProxyAccept(...)
Example: newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1

newt_proxy_connections_total

counter

Connection lifecycle events (opened/closed).

Show Details

Labels: tunnel_id, protocol, event • Unit: 1 • Path: telemetry.IncProxyConnectionEvent(...)
Example: newt_proxy_connections_total{protocol="tcp",event="opened"} 1

newt_proxy_connection_duration_seconds

histogram

Duration of completed proxy connections.

Show Details

Labels: tunnel_id, protocol, result • Unit: seconds • Path: telemetry.ObserveProxyConnectionDuration(...)
Example: newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3

Gerbil metrics

OpenTelemetry (OTel)
Prometheus

gerbil

OpenTelemetry metric instruments exposed by Gerbil. Gerbil supports exactly one metrics backend at runtime: prometheus, otel, or none.

In otel mode, Gerbil pushes metrics to an OTLP collector. The /metrics endpoint is not exposed in this mode.

Show WireGuard

gerbil_wg_interface_up

Int64Gauge

Operational state of a WireGuard interface.

Show Details

Unit: 1
Labels: ifname, instance
Emission path: metrics.RecordInterfaceUp
Example: gerbil_wg_interface_up{ifname="wg0",instance="gerbil-1"} 1

gerbil_wg_peers_total

UpDownCounter

Number of configured peers per interface.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordPeersTotal
Example: gerbil_wg_peers_total{ifname="wg0"} 5

gerbil_wg_peer_connected

Int64Gauge

Current peer connection state.

Show Details

Unit: 1
Labels: ifname, peer
Emission path: metrics.RecordPeerConnected
Example: gerbil_wg_peer_connected{ifname="wg0",peer="abc"} 1

gerbil_allowed_ips_count

UpDownCounter

Number of allowed IPs configured per peer.

Show Details

Unit: 1
Labels: ifname, peer
Emission path: metrics.RecordAllowedIPsCount
Example: gerbil_allowed_ips_count{ifname="wg0",peer="abc"} 2

gerbil_key_rotation_total

Counter

Key rotation events.

Show Details

Unit: 1
Labels: ifname, reason
Emission path: metrics.RecordKeyRotation
Example: gerbil_key_rotation_total{ifname="wg0",reason="scheduled"} 1

gerbil_wg_handshakes_total

Counter

WireGuard handshake attempts keyed by result.

Show Details

Unit: 1
Labels: ifname, peer, result
Emission path: metrics.RecordHandshake
Example: gerbil_wg_handshakes_total{ifname="wg0",peer="abc",result="success"} 1

gerbil_wg_handshake_latency_seconds

Histogram (s)

Distribution of WireGuard handshake latencies.

Show Details

Unit: seconds
Labels: ifname, peer
Emission path: metrics.RecordHandshakeLatency
Example: gerbil_wg_handshake_latency_seconds_bucket{ifname="wg0",peer="abc",le="0.1"} 3

gerbil_wg_peer_rtt_seconds

Histogram (s)

Observed peer round-trip time.

Show Details

Unit: seconds
Labels: ifname, peer
Emission path: metrics.RecordPeerRTT
Example: gerbil_wg_peer_rtt_seconds_bucket{ifname="wg0",peer="abc",le="0.05"} 4

gerbil_wg_bytes_received_total

Counter (bytes)

Bytes received from a WireGuard peer.

Show Details

Unit: bytes
Labels: ifname, peer
Emission path: metrics.RecordBytesReceived
Example: gerbil_wg_bytes_received_total{ifname="wg0",peer="abc"} 8192

gerbil_wg_bytes_transmitted_total

Counter (bytes)

Bytes transmitted to a WireGuard peer.

Show Details

Unit: bytes
Labels: ifname, peer
Emission path: metrics.RecordBytesTransmitted
Example: gerbil_wg_bytes_transmitted_total{ifname="wg0",peer="abc"} 16384

Show Relay

gerbil_active_sessions

UpDownCounter

Number of active UDP relay sessions.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordActiveSession / metrics.RecordSession
Example: gerbil_active_sessions{ifname="wg0"} 3

gerbil_udp_packets_total

Counter

UDP packets processed by relay workers.

Show Details

Unit: 1
Labels: ifname, type, direction
Emission path: metrics.RecordUDPPacket
Example: gerbil_udp_packets_total{ifname="wg0",type="data",direction="rx"} 42

gerbil_udp_packet_size_bytes

Histogram (bytes)

Size distribution of packets forwarded through the relay.

Show Details

Unit: bytes
Labels: ifname, type
Emission path: metrics.RecordUDPPacketSize
Example: gerbil_udp_packet_size_bytes_bucket{ifname="wg0",type="data",le="1024"} 7

gerbil_hole_punch_events_total

Counter

Hole punch messages processed by result.

Show Details

Unit: 1
Labels: ifname, result
Emission path: metrics.RecordHolePunchEvent
Example: gerbil_hole_punch_events_total{ifname="wg0",result="success"} 1

gerbil_proxy_mapping_active

UpDownCounter

Active proxy mappings.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordProxyMapping
Example: gerbil_proxy_mapping_active{ifname="wg0"} 4

gerbil_session_rebuilt_total

Counter

Sessions rebuilt from communication patterns.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordSessionRebuilt
Example: gerbil_session_rebuilt_total{ifname="wg0"} 1

gerbil_comm_pattern_active

UpDownCounter

Active communication patterns.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordCommPattern
Example: gerbil_comm_pattern_active{ifname="wg0"} 2

gerbil_proxy_cleanup_removed_total

Counter

Items removed by cleanup routines.

Show Details

Unit: 1
Labels: ifname, component
Emission path: metrics.RecordProxyCleanupRemoved
Example: gerbil_proxy_cleanup_removed_total{ifname="wg0",component="sessions"} 5

gerbil_proxy_connection_errors_total

Counter

Proxy connection errors.

Show Details

Unit: 1
Labels: ifname, error_type
Emission path: metrics.RecordProxyConnectionError
Example: gerbil_proxy_connection_errors_total{ifname="wg0",error_type="timeout"} 1

gerbil_proxy_initial_mappings

Int64Gauge

Initial proxy mappings loaded.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordProxyInitialMappings
Example: gerbil_proxy_initial_mappings{ifname="wg0"} 8

gerbil_proxy_mapping_updates_total

Counter

Proxy mapping updates.

Show Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordProxyMappingUpdate
Example: gerbil_proxy_mapping_updates_total{ifname="wg0"} 2

gerbil_proxy_idle_cleanup_duration_seconds

Histogram (s)

Duration of idle cleanup cycles.

Show Details

Unit: seconds
Labels: ifname, component
Emission path: metrics.RecordProxyIdleCleanupDuration
Example: gerbil_proxy_idle_cleanup_duration_seconds_bucket{ifname="wg0",component="sessions",le="0.1"} 1

Show SNI Proxy

gerbil_active_proxy_connections

UpDownCounter

Active SNI proxy connections.

Show Details

Unit: 1
Labels: —
Emission path: metrics.RecordActiveProxyConnection
Example: gerbil_active_proxy_connections 2

gerbil_proxy_route_lookups_total

Counter

Route lookups keyed by result.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordProxyRouteLookup
Example: gerbil_proxy_route_lookups_total{result="hit"} 6

gerbil_proxy_tls_handshake_seconds

Histogram (s)

TLS handshake duration for the SNI proxy.

Show Details

Unit: seconds
Labels: —
Emission path: metrics.RecordProxyTLSHandshake
Example: gerbil_proxy_tls_handshake_seconds_bucket{le="0.1"} 2

gerbil_proxy_bytes_transmitted_total

Counter (bytes)

Bytes sent or received by the SNI proxy.

Show Details

Unit: bytes
Labels: direction
Emission path: metrics.RecordProxyBytesTransmitted
Example: gerbil_proxy_bytes_transmitted_total{direction="egress"} 16384

gerbil_sni_connections_total

Counter

Connections processed by the SNI proxy.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordSNIConnection
Example: gerbil_sni_connections_total{result="success"} 3

gerbil_sni_connection_duration_seconds

Histogram (s)

Lifetime distribution of proxied TLS connections.

Show Details

Unit: seconds
Labels: —
Emission path: metrics.RecordSNIConnectionDuration
Example: gerbil_sni_connection_duration_seconds_bucket{le="10"} 4

gerbil_sni_active_connections

UpDownCounter

Active SNI tunnels.

Show Details

Unit: 1
Labels: —
Emission path: metrics.RecordSNIActiveConnection
Example: gerbil_sni_active_connections 2

gerbil_sni_route_cache_hits_total

Counter

SNI route cache hits and misses.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordSNIRouteCacheHit
Example: gerbil_sni_route_cache_hits_total{result="hit"} 10

gerbil_sni_route_api_requests_total

Counter

SNI route API requests.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordSNIRouteAPIRequest
Example: gerbil_sni_route_api_requests_total{result="success"} 5

gerbil_sni_route_api_latency_seconds

Histogram (s)

Route API call latency.

Show Details

Unit: seconds
Labels: —
Emission path: metrics.RecordSNIRouteAPILatency
Example: gerbil_sni_route_api_latency_seconds_bucket{le="0.25"} 4

gerbil_sni_local_override_total

Counter

Routes using local overrides.

Show Details

Unit: 1
Labels: hit
Emission path: metrics.RecordSNILocalOverride
Example: gerbil_sni_local_override_total{hit="true"} 1

gerbil_sni_trusted_proxy_events_total

Counter

PROXY protocol events.

Show Details

Unit: 1
Labels: event
Emission path: metrics.RecordSNITrustedProxyEvent
Example: gerbil_sni_trusted_proxy_events_total{event="parsed"} 2

gerbil_sni_proxy_protocol_parse_errors_total

Counter

PROXY protocol parse failures.

Show Details

Unit: 1
Labels: —
Emission path: metrics.RecordSNIProxyProtocolParseError
Example: gerbil_sni_proxy_protocol_parse_errors_total 1

gerbil_sni_data_bytes_total

Counter (bytes)

Bytes proxied through SNI tunnels.

Show Details

Unit: bytes
Labels: direction
Emission path: metrics.RecordSNIDataBytes
Example: gerbil_sni_data_bytes_total{direction="ingress"} 4096

gerbil_sni_tunnel_terminations_total

Counter

SNI tunnel terminations keyed by reason.

Show Details

Unit: 1
Labels: reason
Emission path: metrics.RecordSNITunnelTermination
Example: gerbil_sni_tunnel_terminations_total{reason="client_close"} 1

Show HTTP API & Peer Management

gerbil_http_requests_total

Counter

HTTP requests to the management API.

Show Details

Unit: 1
Labels: endpoint, method, status_code
Emission path: metrics.RecordHTTPRequest
Example: gerbil_http_requests_total{endpoint="/peer",method="POST",status_code="200"} 1

gerbil_http_request_duration_seconds

Histogram (s)

HTTP request handling time.

Show Details

Unit: seconds
Labels: endpoint, method
Emission path: metrics.RecordHTTPRequestDuration
Example: gerbil_http_request_duration_seconds_bucket{endpoint="/peer",method="POST",le="0.1"} 3

gerbil_peer_operations_total

Counter

Peer lifecycle operations.

Show Details

Unit: 1
Labels: operation, result
Emission path: metrics.RecordPeerOperation
Example: gerbil_peer_operations_total{operation="add",result="success"} 1

gerbil_proxy_mapping_update_requests_total

Counter

Proxy mapping update API calls.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordProxyMappingUpdateRequest
Example: gerbil_proxy_mapping_update_requests_total{result="success"} 1

gerbil_destinations_update_requests_total

Counter

Destination update API calls.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordDestinationsUpdateRequest
Example: gerbil_destinations_update_requests_total{result="success"} 1

Show Remote Config & Reporting

gerbil_remote_config_fetches_total

Counter

Remote configuration fetch attempts.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordRemoteConfigFetch
Example: gerbil_remote_config_fetches_total{result="success"} 1

gerbil_bandwidth_reports_total

Counter

Bandwidth report transmissions.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordBandwidthReport
Example: gerbil_bandwidth_reports_total{result="success"} 1

gerbil_peer_bandwidth_bytes_total

Counter (bytes)

Bytes per peer tracked by bandwidth calculation.

Show Details

Unit: bytes
Labels: peer, direction
Emission path: metrics.RecordPeerBandwidthBytes
Example: gerbil_peer_bandwidth_bytes_total{peer="abc",direction="rx"} 8192

Show System & Operations

gerbil_netlink_events_total

Counter

Netlink events processed.

Show Details

Unit: 1
Labels: event_type
Emission path: metrics.RecordNetlinkEvent
Example: gerbil_netlink_events_total{event_type="link_up"} 1

gerbil_netlink_errors_total

Counter

Netlink or kernel errors.

Show Details

Unit: 1
Labels: component, error_type
Emission path: metrics.RecordNetlinkError
Example: gerbil_netlink_errors_total{component="wg",error_type="permission"} 1

gerbil_sync_duration_seconds

Histogram (s)

Duration of reconciliation or sync loops.

Show Details

Unit: seconds
Labels: component
Emission path: metrics.RecordSyncDuration
Example: gerbil_sync_duration_seconds_bucket{component="remote_config",le="0.5"} 2

gerbil_workqueue_depth

UpDownCounter

Current length of internal work queues.

Show Details

Unit: 1
Labels: queue
Emission path: metrics.RecordWorkqueueDepth
Example: gerbil_workqueue_depth{queue="relay"} 3

gerbil_kernel_module_loads_total

Counter

Kernel module load attempts.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordKernelModuleLoad
Example: gerbil_kernel_module_loads_total{result="success"} 1

gerbil_firewall_rules_applied_total

Counter

Firewall rules applied.

Show Details

Unit: 1
Labels: result, chain
Emission path: metrics.RecordFirewallRuleApplied
Example: gerbil_firewall_rules_applied_total{result="success",chain="FORWARD"} 1

gerbil_config_reloads_total

Counter

Configuration reloads.

Show Details

Unit: 1
Labels: result
Emission path: metrics.RecordConfigReload
Example: gerbil_config_reloads_total{result="success"} 1

gerbil_restart_total

Counter

Process restart count.

Show Details

Unit: 1
Labels: —
Emission path: metrics.RecordRestart
Example: gerbil_restart_total 1

gerbil_auth_failures_total

Counter

Authentication or peer validation failures.

Show Details

Unit: 1
Labels: peer, reason
Emission path: metrics.RecordAuthFailure
Example: gerbil_auth_failures_total{peer="abc",reason="invalid_key"} 1

gerbil_acl_denied_total

Counter

Access-control denied events.

Show Details

Unit: 1
Labels: ifname, peer, policy
Emission path: metrics.RecordACLDenied
Example: gerbil_acl_denied_total{ifname="wg0",peer="abc",policy="deny"} 1

gerbil_certificate_expiry_days

Float64Gauge

Days until certificate expiry.

Show Details

Unit: days
Labels: cert_name, ifname
Emission path: metrics.RecordCertificateExpiry
Example: gerbil_certificate_expiry_days{cert_name="server",ifname="wg0"} 42

gerbil_memory_spike_total

Counter

Memory spikes detected by severity.

Show Details

Unit: 1
Labels: severity
Emission path: metrics.RecordMemorySpike
Example: gerbil_memory_spike_total{severity="warning"} 1

gerbil_heap_profiles_written_total

Counter

Heap profile files generated.

Show Details

Unit: 1
Labels: —
Emission path: metrics.RecordHeapProfileWritten
Example: gerbil_heap_profiles_written_total 1

gerbil

Prometheus-style series for the same Gerbil metrics. Gerbil exposes these only when METRICS_BACKEND=prometheus.

In Prometheus mode, Gerbil registers a native Prometheus client and exposes the configured metrics path, default /metrics. In OTel mode, /metrics is not exposed.

Show WireGuard

gerbil_wg_interface_up

gauge

Operational state of a WireGuard interface.

Show Details

Labels: ifname, instance • Unit: 1 • Path: metrics.RecordInterfaceUp
Example: gerbil_wg_interface_up{ifname="wg0",instance="gerbil-1"} 1

gerbil_wg_peers_total

gauge

Number of configured peers per interface.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordPeersTotal
Example: gerbil_wg_peers_total{ifname="wg0"} 5

gerbil_wg_peer_connected

gauge

Current peer connection state.

Show Details

Labels: ifname, peer • Unit: 1 • Path: metrics.RecordPeerConnected
Example: gerbil_wg_peer_connected{ifname="wg0",peer="abc"} 1

gerbil_allowed_ips_count

gauge

Number of allowed IPs configured per peer.

Show Details

Labels: ifname, peer • Unit: 1 • Path: metrics.RecordAllowedIPsCount
Example: gerbil_allowed_ips_count{ifname="wg0",peer="abc"} 2

gerbil_key_rotation_total

counter

Key rotation events.

Show Details

Labels: ifname, reason • Unit: 1 • Path: metrics.RecordKeyRotation
Example: gerbil_key_rotation_total{ifname="wg0",reason="scheduled"} 1

gerbil_wg_handshakes_total

counter

WireGuard handshake attempts keyed by result.

Show Details

Labels: ifname, peer, result • Unit: 1 • Path: metrics.RecordHandshake
Example: gerbil_wg_handshakes_total{ifname="wg0",peer="abc",result="success"} 1

gerbil_wg_handshake_latency_seconds

histogram

Distribution of WireGuard handshake latencies.

Show Details

Labels: ifname, peer • Unit: seconds • Path: metrics.RecordHandshakeLatency
Example: gerbil_wg_handshake_latency_seconds_bucket{ifname="wg0",peer="abc",le="0.1"} 3

gerbil_wg_peer_rtt_seconds

histogram

Observed peer round-trip time.

Show Details

Labels: ifname, peer • Unit: seconds • Path: metrics.RecordPeerRTT
Example: gerbil_wg_peer_rtt_seconds_bucket{ifname="wg0",peer="abc",le="0.05"} 4

gerbil_wg_bytes_received_total

counter

Bytes received from a WireGuard peer.

Show Details

Labels: ifname, peer • Unit: bytes • Path: metrics.RecordBytesReceived
Example: gerbil_wg_bytes_received_total{ifname="wg0",peer="abc"} 8192

gerbil_wg_bytes_transmitted_total

counter

Bytes transmitted to a WireGuard peer.

Show Details

Labels: ifname, peer • Unit: bytes • Path: metrics.RecordBytesTransmitted
Example: gerbil_wg_bytes_transmitted_total{ifname="wg0",peer="abc"} 16384

Show Relay

gerbil_active_sessions

gauge

Active UDP relay sessions.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordActiveSession / metrics.RecordSession
Example: gerbil_active_sessions{ifname="wg0"} 3

gerbil_udp_packets_total

counter

UDP packets processed by relay workers.

Show Details

Labels: ifname, type, direction • Unit: 1 • Path: metrics.RecordUDPPacket
Example: gerbil_udp_packets_total{ifname="wg0",type="data",direction="rx"} 42

gerbil_udp_packet_size_bytes

histogram

Size distribution of packets forwarded through the relay.

Show Details

Labels: ifname, type • Unit: bytes • Path: metrics.RecordUDPPacketSize
Example: gerbil_udp_packet_size_bytes_bucket{ifname="wg0",type="data",le="1024"} 7

gerbil_hole_punch_events_total

counter

Hole punch messages processed by result.

Show Details

Labels: ifname, result • Unit: 1 • Path: metrics.RecordHolePunchEvent
Example: gerbil_hole_punch_events_total{ifname="wg0",result="success"} 1

gerbil_proxy_mapping_active

gauge

Active proxy mappings.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordProxyMapping
Example: gerbil_proxy_mapping_active{ifname="wg0"} 4

gerbil_session_rebuilt_total

counter

Sessions rebuilt from communication patterns.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordSessionRebuilt
Example: gerbil_session_rebuilt_total{ifname="wg0"} 1

gerbil_comm_pattern_active

gauge

Active communication patterns.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordCommPattern
Example: gerbil_comm_pattern_active{ifname="wg0"} 2

gerbil_proxy_cleanup_removed_total

counter

Items removed by cleanup routines.

Show Details

Labels: ifname, component • Unit: 1 • Path: metrics.RecordProxyCleanupRemoved
Example: gerbil_proxy_cleanup_removed_total{ifname="wg0",component="sessions"} 5

gerbil_proxy_connection_errors_total

counter

Proxy connection errors.

Show Details

Labels: ifname, error_type • Unit: 1 • Path: metrics.RecordProxyConnectionError
Example: gerbil_proxy_connection_errors_total{ifname="wg0",error_type="timeout"} 1

gerbil_proxy_initial_mappings

gauge

Initial proxy mappings loaded.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordProxyInitialMappings
Example: gerbil_proxy_initial_mappings{ifname="wg0"} 8

gerbil_proxy_mapping_updates_total

counter

Proxy mapping updates.

Show Details

Labels: ifname • Unit: 1 • Path: metrics.RecordProxyMappingUpdate
Example: gerbil_proxy_mapping_updates_total{ifname="wg0"} 2

gerbil_proxy_idle_cleanup_duration_seconds

histogram

Duration of idle cleanup cycles.

Show Details

Labels: ifname, component • Unit: seconds • Path: metrics.RecordProxyIdleCleanupDuration
Example: gerbil_proxy_idle_cleanup_duration_seconds_bucket{ifname="wg0",component="sessions",le="0.1"} 1

Show SNI Proxy

gerbil_active_proxy_connections

gauge

Active SNI proxy connections.

Show Details

Labels: — • Unit: 1 • Path: metrics.RecordActiveProxyConnection
Example: gerbil_active_proxy_connections 2

gerbil_proxy_route_lookups_total

counter

Route lookups keyed by result.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordProxyRouteLookup
Example: gerbil_proxy_route_lookups_total{result="hit"} 6

gerbil_proxy_tls_handshake_seconds

histogram

TLS handshake duration for the SNI proxy.

Show Details

Labels: — • Unit: seconds • Path: metrics.RecordProxyTLSHandshake
Example: gerbil_proxy_tls_handshake_seconds_bucket{le="0.1"} 2

gerbil_proxy_bytes_transmitted_total

counter

Bytes sent or received by the SNI proxy.

Show Details

Labels: direction • Unit: bytes • Path: metrics.RecordProxyBytesTransmitted
Example: gerbil_proxy_bytes_transmitted_total{direction="egress"} 16384

gerbil_sni_connections_total

counter

Connections processed by the SNI proxy.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordSNIConnection
Example: gerbil_sni_connections_total{result="success"} 3

gerbil_sni_connection_duration_seconds

histogram

Lifetime distribution of proxied TLS connections.

Show Details

Labels: — • Unit: seconds • Path: metrics.RecordSNIConnectionDuration
Example: gerbil_sni_connection_duration_seconds_bucket{le="10"} 4

gerbil_sni_active_connections

gauge

Active SNI tunnels.

Show Details

Labels: — • Unit: 1 • Path: metrics.RecordSNIActiveConnection
Example: gerbil_sni_active_connections 2

gerbil_sni_route_cache_hits_total

counter

SNI route cache hits and misses.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordSNIRouteCacheHit
Example: gerbil_sni_route_cache_hits_total{result="hit"} 10

gerbil_sni_route_api_requests_total

counter

SNI route API requests.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordSNIRouteAPIRequest
Example: gerbil_sni_route_api_requests_total{result="success"} 5

gerbil_sni_route_api_latency_seconds

histogram

Route API call latency.

Show Details

Labels: — • Unit: seconds • Path: metrics.RecordSNIRouteAPILatency
Example: gerbil_sni_route_api_latency_seconds_bucket{le="0.25"} 4

gerbil_sni_local_override_total

counter

Routes using local overrides.

Show Details

Labels: hit • Unit: 1 • Path: metrics.RecordSNILocalOverride
Example: gerbil_sni_local_override_total{hit="true"} 1

gerbil_sni_trusted_proxy_events_total

counter

PROXY protocol events.

Show Details

Labels: event • Unit: 1 • Path: metrics.RecordSNITrustedProxyEvent
Example: gerbil_sni_trusted_proxy_events_total{event="parsed"} 2

gerbil_sni_proxy_protocol_parse_errors_total

counter

PROXY protocol parse failures.

Show Details

Labels: — • Unit: 1 • Path: metrics.RecordSNIProxyProtocolParseError
Example: gerbil_sni_proxy_protocol_parse_errors_total 1

gerbil_sni_data_bytes_total

counter

Bytes proxied through SNI tunnels.

Show Details

Labels: direction • Unit: bytes • Path: metrics.RecordSNIDataBytes
Example: gerbil_sni_data_bytes_total{direction="ingress"} 4096

gerbil_sni_tunnel_terminations_total

counter

SNI tunnel terminations keyed by reason.

Show Details

Labels: reason • Unit: 1 • Path: metrics.RecordSNITunnelTermination
Example: gerbil_sni_tunnel_terminations_total{reason="client_close"} 1

Show HTTP API & Peer Management

gerbil_http_requests_total

counter

HTTP requests to the management API.

Show Details

Labels: endpoint, method, status_code • Unit: 1 • Path: metrics.RecordHTTPRequest
Example: gerbil_http_requests_total{endpoint="/peer",method="POST",status_code="200"} 1

gerbil_http_request_duration_seconds

histogram

HTTP request handling time.

Show Details

Labels: endpoint, method • Unit: seconds • Path: metrics.RecordHTTPRequestDuration
Example: gerbil_http_request_duration_seconds_bucket{endpoint="/peer",method="POST",le="0.1"} 3

gerbil_peer_operations_total

counter

Peer lifecycle operations.

Show Details

Labels: operation, result • Unit: 1 • Path: metrics.RecordPeerOperation
Example: gerbil_peer_operations_total{operation="add",result="success"} 1

gerbil_proxy_mapping_update_requests_total

counter

Proxy mapping update API calls.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordProxyMappingUpdateRequest
Example: gerbil_proxy_mapping_update_requests_total{result="success"} 1

gerbil_destinations_update_requests_total

counter

Destination update API calls.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordDestinationsUpdateRequest
Example: gerbil_destinations_update_requests_total{result="success"} 1

Show Remote Config & Reporting

gerbil_remote_config_fetches_total

counter

Remote configuration fetch attempts.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordRemoteConfigFetch
Example: gerbil_remote_config_fetches_total{result="success"} 1

gerbil_bandwidth_reports_total

counter

Bandwidth report transmissions.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordBandwidthReport
Example: gerbil_bandwidth_reports_total{result="success"} 1

gerbil_peer_bandwidth_bytes_total

counter

Bytes per peer tracked by bandwidth calculation.

Show Details

Labels: peer, direction • Unit: bytes • Path: metrics.RecordPeerBandwidthBytes
Example: gerbil_peer_bandwidth_bytes_total{peer="abc",direction="rx"} 8192

Show System & Operations

gerbil_netlink_events_total

counter

Netlink events processed.

Show Details

Labels: event_type • Unit: 1 • Path: metrics.RecordNetlinkEvent
Example: gerbil_netlink_events_total{event_type="link_up"} 1

gerbil_netlink_errors_total

counter

Netlink or kernel errors.

Show Details

Labels: component, error_type • Unit: 1 • Path: metrics.RecordNetlinkError
Example: gerbil_netlink_errors_total{component="wg",error_type="permission"} 1

gerbil_sync_duration_seconds

histogram

Duration of reconciliation or sync loops.

Show Details

Labels: component • Unit: seconds • Path: metrics.RecordSyncDuration
Example: gerbil_sync_duration_seconds_bucket{component="remote_config",le="0.5"} 2

gerbil_workqueue_depth

gauge

Current length of internal work queues.

Show Details

Labels: queue • Unit: 1 • Path: metrics.RecordWorkqueueDepth
Example: gerbil_workqueue_depth{queue="relay"} 3

gerbil_kernel_module_loads_total

counter

Kernel module load attempts.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordKernelModuleLoad
Example: gerbil_kernel_module_loads_total{result="success"} 1

gerbil_firewall_rules_applied_total

counter

Firewall rules applied.

Show Details

Labels: result, chain • Unit: 1 • Path: metrics.RecordFirewallRuleApplied
Example: gerbil_firewall_rules_applied_total{result="success",chain="FORWARD"} 1

gerbil_config_reloads_total

counter

Configuration reloads.

Show Details

Labels: result • Unit: 1 • Path: metrics.RecordConfigReload
Example: gerbil_config_reloads_total{result="success"} 1

gerbil_restart_total

counter

Process restart count.

Show Details

Labels: — • Unit: 1 • Path: metrics.RecordRestart
Example: gerbil_restart_total 1

gerbil_auth_failures_total

counter

Authentication or peer validation failures.

Show Details

Labels: peer, reason • Unit: 1 • Path: metrics.RecordAuthFailure
Example: gerbil_auth_failures_total{peer="abc",reason="invalid_key"} 1

gerbil_acl_denied_total

counter

Access-control denied events.

Show Details

Labels: ifname, peer, policy • Unit: 1 • Path: metrics.RecordACLDenied
Example: gerbil_acl_denied_total{ifname="wg0",peer="abc",policy="deny"} 1

gerbil_certificate_expiry_days

gauge

Days until certificate expiry.

Show Details

Labels: cert_name, ifname • Unit: days • Path: metrics.RecordCertificateExpiry
Example: gerbil_certificate_expiry_days{cert_name="server",ifname="wg0"} 42

gerbil_memory_spike_total

counter

Memory spikes detected by severity.

Show Details

Labels: severity • Unit: 1 • Path: metrics.RecordMemorySpike
Example: gerbil_memory_spike_total{severity="warning"} 1

gerbil_heap_profiles_written_total

counter

Heap profile files generated.

Show Details

Labels: — • Unit: 1 • Path: metrics.RecordHeapProfileWritten
Example: gerbil_heap_profiles_written_total 1

Pangolin Kubernetes Controller metrics

OpenTelemetry (OTel)
Prometheus

pangolin-kube-controller

Additional OpenTelemetry metric instruments exposed by the Pangolin Kubernetes Controller.

The controller exposes Prometheus-native metrics and additional OTel-backed metrics on the same /metrics endpoint. The OTel-backed series use the pangolin_controller_* prefix.

Show Reconcile

pangolin_controller_reconcile_phase_duration_seconds

Histogram (s)

Duration of each reconcile phase.

Show Details

Unit: seconds
Labels: phase, result
Label values:
phase: middlewares | routers | serversTransports | services | tcp | udp
result: success | error
Emission path: OTel reconcile phase instrumentation
Example: pangolin_controller_reconcile_phase_duration_seconds_bucket{phase="routers",result="success",le="0.5"} 3

pangolin_controller_active_reconcile_routines

UpDownCounter

Number of active reconcile routines by phase.

Show Details

Unit: 1
Labels: phase
Label values: middlewares | routers | serversTransports | services | tcp | udp
Emission path: Parallel reconcile instrumentation
Example: pangolin_controller_active_reconcile_routines{phase="routers"} 1

pangolin_controller_loop_iterations_total

Counter

Controller loop iterations by outcome.

Show Details

Unit: 1
Labels: outcome
Label values: success | nochange | error
Emission path: Controller loop instrumentation
Example: pangolin_controller_loop_iterations_total{outcome="success"} 10

Show Fetch & Config

pangolin_controller_fetch_duration_seconds

Histogram (s)

Duration of remote fetch cycle HTTP requests.

Show Details

Unit: seconds
Labels: status_code, status_class
Label values:
status_code: 200 | 304 | 401 | 403 | 404 | 5xx
status_class: 2xx | 3xx | 4xx | 5xx
Emission path: Remote config fetch instrumentation
Example: pangolin_controller_fetch_duration_seconds_bucket{status_code="200",status_class="2xx",le="0.25"} 4

pangolin_controller_config_parse_duration_seconds

Histogram (s)

Duration of configuration parsing.

Show Details

Unit: seconds
Labels: section
Label values: full
Emission path: Config parse instrumentation
Example: pangolin_controller_config_parse_duration_seconds_bucket{section="full",le="0.1"} 2

Show Kubernetes API

pangolin_controller_k8s_request_duration_seconds

Histogram (s)

Duration of Kubernetes API requests.

Show Details

pangolin_controller_k8s_request_duration_seconds_bucket{verb="patch",resource_kind="IngressRoute",result="success",forced="false",le="0.25"} 3

pangolin_controller_k8s_requests_total

Counter

Total Kubernetes API requests.

Show Details

Unit: 1
Labels: verb, resource_kind, result, forced
Emission path: Kubernetes API request instrumentation
Example: pangolin_controller_k8s_requests_total{verb="patch",resource_kind="IngressRoute",result="success",forced="false"} 1

pangolin_controller_retries_total

Counter

Retry attempts in the SSA apply loop.

Show Details

Unit: 1
Labels: reason, operation, resource_kind
Label values:
reason: conflict | transient | timeout
operation: get | create | patch | delete | apply
Emission path: SSA apply retry instrumentation
Example: pangolin_controller_retries_total{reason="conflict",operation="patch",resource_kind="IngressRoute"} 1

Show Garbage Collection

pangolin_controller_gc_run_duration_seconds

Histogram (s)

Duration of garbage collection runs.

Show Details

Unit: seconds
Labels: result
Label values: success | fail | dryrun
Emission path: GC instrumentation
Example: pangolin_controller_gc_run_duration_seconds_bucket{result="success",le="0.5"} 1

pangolin-kube-controller

Prometheus-native metrics exposed by the Pangolin Kubernetes Controller. These use the pangolin_kube_controller_* prefix.

The metrics endpoint is exposed at GET /metrics on METRICS_ADDR, default :9090. The same HTTP server also exposes /healthz, /readyz, and optionally /debug/pprof/ when pprof is enabled.

Show Reconcile

pangolin_kube_controller_reconcile_seconds

histogram

Duration of a full successful reconcile loop.

Show Details

Labels: —
Unit: seconds
Path: Collector.ReconcileDuration
Example: pangolin_kube_controller_reconcile_seconds_bucket{le="0.5"} 3

pangolin_kube_controller_reconcile_errors_total

counter

Total errors during reconcile steps.

Show Details

Labels: —
Unit: 1
Path: Collector.ReconcileErrors
Example: pangolin_kube_controller_reconcile_errors_total 2

pangolin_kube_controller_consecutive_errors

gauge

Number of consecutive reconcile or fetch errors.

Show Details

Labels: —
Unit: 1
Path: Collector.ConsecutiveErrors
Example: pangolin_kube_controller_consecutive_errors 0

pangolin_kube_controller_last_fetch_success_timestamp_seconds

gauge

Unix timestamp of the last successful fetch.

Show Details

Labels: —
Unit: seconds since Unix epoch
Path: Collector.LastFetchSuccess
Example: pangolin_kube_controller_last_fetch_success_timestamp_seconds 1767225600

pangolin_kube_controller_ready

gauge

Controller readiness state.

Show Details

Labels: —
Unit: 1
Value: 1 = ready, 0 = not ready
Path: Collector.Ready
Example: pangolin_kube_controller_ready 1

Show Objects

pangolin_kube_controller_objects_applied_total

counter

Applied objects by kind and action.

Show Details

Labels: kind, action
Label values:
kind: IngressRoute | Middleware | TraefikService | other managed Traefik CRDs
action: create | patch
Unit: 1
Path: Collector.AppliedObjects
Example: pangolin_kube_controller_objects_applied_total{kind="IngressRoute",action="patch"} 1

pangolin_kube_controller_objects_deleted_total

counter

Deleted objects by kind.

Show Details

Labels: kind
Unit: 1
Path: Collector.DeletedObjects
Example: pangolin_kube_controller_objects_deleted_total{kind="Middleware"} 1

pangolin_kube_controller_desired_objects_count

gauge

Desired objects count by kind from the last processed config.

Show Details

Labels: kind
Unit: 1
Path: Collector.DesiredObjects
Example: pangolin_kube_controller_desired_objects_count{kind="IngressRoute"} 12

Show Garbage Collection

pangolin_kube_controller_gc_deleted_total

counter

Garbage-collected objects by kind and reason.

Show Details

Labels: kind, reason
Label values:
reason: immediate | grace
Unit: 1
Path: Collector.GCDeletedTotal
Example: pangolin_kube_controller_gc_deleted_total{kind="IngressRoute",reason="grace"} 1

pangolin_kube_controller_gc_runs_total

counter

Garbage collection runs by result.

Show Details

Labels: result
Label values: start | success | fail | dryrun
Unit: 1
Path: Collector.GCRunsTotal
Example: pangolin_kube_controller_gc_runs_total{result="success"} 1

pangolin_kube_controller_grace_queue_dropped_total

counter

Grace deletion queue items dropped due to queue overflow.

Show Details

Labels: —
Unit: 1
Path: Collector.GraceQueueDropped
Example: pangolin_kube_controller_grace_queue_dropped_total 0

pangolin_kube_controller_grace_queue_depth

gauge

Current depth of the grace deletion queue.

Show Details

Labels: —
Unit: 1
Path: Collector.GraceQueueDepth
Example: pangolin_kube_controller_grace_queue_depth 3

Show Instance Label

pangolin_kube_controller_instance_label_detect_success_total

counter

Successful Traefik instance label resolutions.

Show Details

Labels: —
Unit: 1
Path: Collector.InstanceLabelDetectSuccess
Example: pangolin_kube_controller_instance_label_detect_success_total 1

pangolin_kube_controller_instance_label_detect_failure_total

counter

Failed Traefik instance label resolutions or verifications.

Show Details

Labels: —
Unit: 1
Path: Collector.InstanceLabelDetectFailures
Example: pangolin_kube_controller_instance_label_detect_failure_total 1

pangolin_kube_controller_instance_label_last_check_timestamp_seconds

gauge

Unix timestamp of the last instance label verification.

Show Details

Labels: —
Unit: seconds since Unix epoch
Path: Collector.InstanceLabelLastCheck
Example: pangolin_kube_controller_instance_label_last_check_timestamp_seconds 1767225600

Show Leader Election

pangolin_kube_controller_leader_state

gauge

Current leader election state of the controller.

Show Details

Labels: —
Unit: 1
Value: 1 = leader, 0 = follower, -1 = leader election disabled
Path: Collector.LeaderState
Example: pangolin_kube_controller_leader_state 1

Show Runtime

go_*

runtime metrics

Go runtime metrics registered by the Prometheus Go collector.

Show Details

Labels: varies
Unit: varies
Path: collectors.NewGoCollector()
Example: go_goroutines 14

process_*

process metrics

Process metrics registered by the Prometheus process collector.

Show Details

Labels: varies
Unit: varies
Path: collectors.NewProcessCollector()
Example: process_resident_memory_bytes 52428800

References

Have improvements or a missing metric? Open an issue or PR referencing this page.

About

Manage Pangolin

Self-host Pangolin

Development

Metrics and Observability

Try free on Pangolin Cloud

Capability Matrix

Newt

Gerbil

Kubernetes Controller

Observability Capability Matrix

Supported Signals

Collection Patterns

Newt

Gerbil

Pangolin Kubernetes Controller

Alerting Examples

Community Metrics Guide

Newt Kubernetes Monitoring

Controller Monitoring Values

Versions

Full Metric Reference

Newt metrics

Gerbil metrics

Pangolin Kubernetes Controller metrics

References

About

Manage Pangolin

Self-host Pangolin

Development

Documentation Index

Try free on Pangolin Cloud

Capability Matrix

Newt

Gerbil

Kubernetes Controller

​Observability Capability Matrix

​Supported Signals

​Collection Patterns

​Newt

​Gerbil

​Pangolin Kubernetes Controller

​Alerting Examples

Community Metrics Guide

Newt Kubernetes Monitoring

Controller Monitoring Values

​Versions

​Full Metric Reference

​Newt metrics

​Gerbil metrics

​Pangolin Kubernetes Controller metrics

​References

Observability Capability Matrix

Supported Signals

Collection Patterns

Newt

Gerbil

Pangolin Kubernetes Controller

Alerting Examples

Versions

Full Metric Reference

Newt metrics

Gerbil metrics

Pangolin Kubernetes Controller metrics

References