Metrics and Observability

Understand metrics, traces, logs, and profiling support across Pangolin components

Pangolin exposes observability signals across multiple components, but not every component provides the same telemetry surface. This page explains which metrics, traces, logs, and profiling endpoints are available today, how they are exposed, and which collection patterns are recommended for production deployments.

Newt provides the broadest native observability support with Prometheus metrics, OTLP metrics, OTLP traces, health checks, and optional pprof. Gerbil supports metrics through either a Prometheus backend or an OTLP backend. The Pangolin Kubernetes Controller exposes a Prometheus-compatible scrape endpoint and includes additional OTel-backed metric instruments on that endpoint.

This page focuses on Pangolin-native observability. For a community walkthrough that collects Traefik metrics with Prometheus and Grafana, see the community metrics guide.

All currently documented component metrics are listed in the Full Metric Reference.

Capability Matrix

Compare metrics, traces, logs, and profiling support across Pangolin components.

Newt

Native Prometheus metrics, OTLP metrics, OTLP traces, health checks, and optional pprof.

Gerbil

Metrics-only observability using either a Prometheus or OTLP backend.

Kubernetes Controller

Prometheus scraping, health probes, ServiceMonitor examples, and optional pprof.

Observability Capability Matrix

Component	Metrics	Traces	Logs	Profiling
`newt`	Prometheus, OTLP	OTLP	stdout	pprof optional
`gerbil`	Prometheus or OTLP	—	stdout	—
`pangolin-kube-controller`	Prometheus scrape	—	stdout	pprof optional

Supported Signals

Signal	What it means
Prometheus scrape metrics	Pull-based metrics served over HTTP on `/metrics`
OTLP metrics	Push-based OpenTelemetry metrics sent to an OTel Collector or compatible backend
OTLP traces	Distributed traces sent to an OTel Collector or trace backend
Application logs	stdout, file logs, audit logs, or platform logs
Profiling	Debug endpoints such as `pprof`

Collection Patterns

Use one or more of the following patterns depending on your deployment model. Prometheus scrape mode is the simplest option for local or Kubernetes monitoring. OTLP is useful when you already operate an OpenTelemetry Collector or want to forward telemetry to a managed backend such as Grafana Cloud, Mimir, or Tempo.

prometheus.yml (fragment)

scrape_configs:
  - job_name: newt
    static_configs:
      - targets: ["newt:2112"]

  - job_name: gerbil
    metrics_path: /metrics
    static_configs:
      - targets: ["gerbil:3003"]

  - job_name: pangolin-kube-controller
    static_configs:
      - targets: ["pangolin-kube-controller:9090"]

Newt

Newt metrics are not typically enabled in default deployments. Turn them on explicitly and expose the admin address only where you intend to scrape or profile. Available metrics are listed in Newt metrics.

Configuration

NEWT_METRICS_PROMETHEUS_ENABLED=true
NEWT_METRICS_OTLP_ENABLED=true
NEWT_ADMIN_ADDR=:2112
OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
OTEL_EXPORTER_OTLP_INSECURE=true
OTEL_METRIC_EXPORT_INTERVAL=15s
NEWT_PPROF_ENABLED=false

Signals and endpoints

Prometheus scrape endpoint: /metrics on NEWT_ADMIN_ADDR
Health endpoint: /healthz on the same admin server
OTLP metrics: enabled with NEWT_METRICS_OTLP_ENABLED=true or --otlp=true
OTLP traces: initialized when OTLP export is enabled
Profiling: /debug/pprof/* when NEWT_PPROF_ENABLED=true

The Newt OTLP switch enables the OTLP telemetry pipeline. The current environment variable name contains METRICS, but the implementation also initializes OTLP tracing when OTLP is enabled.

Newt's admin server defaults to a loopback bind address. In containers or Kubernetes, set NEWT_ADMIN_ADDR=:2112 or another non-loopback address only when you intentionally want Prometheus, pprof, or health checks to reach it.

Examples

prometheus.yml (fragment)

scrape_configs:
  - job_name: newt
    static_configs:
      - targets: ["newt:2112"]

Gerbil

Gerbil supports metrics only. Choose either a native Prometheus backend or an OTLP metrics backend at runtime. Those backends are mutually exclusive. Available metrics are listed in Gerbil metrics.

Configuration

METRICS_ENABLED=true
METRICS_BACKEND=prometheus
METRICS_PATH=/metrics
LISTEN=:3003

# OTel mode
OTEL_METRICS_PROTOCOL=grpc
OTEL_METRICS_ENDPOINT=otel-collector:4317
OTEL_METRICS_INSECURE=true
OTEL_METRICS_EXPORT_INTERVAL=60s
OTEL_METRICS_TIMEOUT=10s

Signals and endpoints

Prometheus metrics endpoint: METRICS_PATH, default /metrics
Metrics are served on Gerbil's configured HTTP listen address
The Docker Compose metrics example commonly scrapes gerbil:3003
Health endpoint: /healthz
OTLP metrics: enabled when METRICS_BACKEND=otel
Traces: not supported yet
Profiling: not supported yet

Examples

prometheus.yml (fragment)

scrape_configs:
  - job_name: gerbil
    metrics_path: /metrics
    static_configs:
      - targets: ["gerbil:3003"]

Replace gerbil:3003 with the actual Gerbil HTTP listen address in your deployment.

Pangolin Kubernetes Controller

The controller exposes a Prometheus-compatible /metrics endpoint and standard health probes. It also registers additional OpenTelemetry metric instruments, but those instruments are exported on the same scrape endpoint rather than pushed through OTLP. Available metrics are listed in Controller metrics.

pprof is available only when explicitly enabled via ENABLE_PPROF=true.

Configuration

METRICS_ADDR=:9090
DISABLE_LIVEZ=false
ENABLE_PPROF=false

Expose the controller metrics endpoint only inside trusted networks or through Kubernetes-native monitoring resources. If TLS or auth is required, terminate it with a Service mesh, Ingress, sidecar, or platform-specific monitoring gateway unless native TLS support is verified.

Endpoints

/metrics on METRICS_ADDR
/healthz and /readyz for readiness
/livez and /health/live for liveness unless DISABLE_LIVEZ=true
/debug/pprof/* when ENABLE_PPROF=true

Representative metrics and Kubernetes examples

service-and-servicemonitor.yaml

apiVersion: v1
kind: Service
metadata:
  name: pangolin-kube-controller
  labels:
    app: pangolin-kube-controller
spec:
  selector:
    app: pangolin-kube-controller
  ports:
    - name: http-metrics
      port: 9090
      targetPort: 9090
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: pangolin-kube-controller
spec:
  selector:
    matchLabels:
      app: pangolin-kube-controller
  endpoints:
    - port: http-metrics
      path: /metrics
      interval: 30s

Alerting Examples

increase(newt_connection_errors_total[5m]) > 10

histogram_quantile(
  0.95,
  sum(rate(newt_tunnel_latency_seconds_bucket[5m])) by (le)
) > 1

increase(newt_tunnel_reconnects_total[10m]) > 20

Community Metrics Guide

Traefik and metrics collection with Prometheus and Grafana.

Newt Kubernetes Monitoring

Verified Newt chart values for metrics, Services, and ServiceMonitor resources.

Controller Monitoring Values

Verified chart values for controller Services, ServiceMonitor, PodMonitor, and PrometheusRule resources.

Versions

Component	Signal	Since version
Newt	Prometheus scrape metrics	`v1.6.0`
Newt	OTLP metrics	`v1.6.0`
Newt	OTLP traces	`v1.6.0`
Newt	pprof	`v1.10.4`
Gerbil	Prometheus scrape metrics	`v1.4.0`
Gerbil	OTLP metrics	`v1.4.0`
Pangolin Kubernetes Controller	Prometheus scrape metrics	`v0.1.0-alpha.1`

Full Metric Reference

The full reference below is grouped by component.

Newt: Prometheus metrics, OTLP metrics, OTLP traces
Gerbil: Prometheus or OTLP metrics
Pangolin Kubernetes Controller: Prometheus-native metrics and additional OTel-backed scrape metrics

Metric names, labels, and defaults can change between component releases.

Newt metrics

newt

OpenTelemetry metric instruments exposed by Newt. Expand each section to see individual metrics with labels, units, emission points, and examples.

Show Site & BuildHide Site & Build

newt_site_registrations_totalCounter

Counts Pangolin registration attempts keyed by result.

Show DetailsHide Details

Unit: 1
Labels: result (success|failure), site_id
Emission path: telemetry.IncSiteRegistration
Example: newt_site_registrations_total{result="success",site_id="abc"} 1

newt_site_onlineObservableGauge

0/1 heartbeat for the active site.

Show DetailsHide Details

Unit: 1
Labels: site_id
Emission path: state.TelemetryView (callback)
Example: newt_site_online{site_id="self"} 1

newt_site_last_heartbeat_secondsObservableGauge

Seconds since last Pangolin heartbeat.

Show DetailsHide Details

Unit: seconds
Labels: site_id
Emission path: TouchHeartbeat (callback)
Example: newt_site_last_heartbeat_seconds{site_id="self"} 3.2

newt_build_infoObservableGauge

Constant 1 with build metadata labels.

Show DetailsHide Details

Unit: 1
Labels: version, commit
Emission path: Build info registration
Example: newt_build_info{version="1.2.3",commit="abc123"} 1

newt_restart_count_totalCounter

Process boot indicator (increments once per process start).

Show DetailsHide Details

Unit: 1
Labels: —
Emission path: RegisterBuildInfo
Example: newt_restart_count_total 1

newt_cert_rotation_totalCounter

Certificate rotation events keyed by result.

Show DetailsHide Details

Unit: 1
Labels: result
Emission path: IncCertRotation
Example: newt_cert_rotation_total{result="success"} 1

newt_config_reloads_totalCounter

Config reload attempts keyed by result.

Show DetailsHide Details

Unit: 1
Labels: result
Emission path: telemetry.IncConfigReload
Example: newt_config_reloads_total{result="success"} 1

newt_config_apply_secondsHistogram (s)

Duration per config-apply phase keyed by phase and result.

Show DetailsHide Details

Unit: seconds
Labels: phase, result
Emission path: telemetry.ObserveConfigApply
Example: newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3

Show TunnelHide Tunnel

newt_tunnel_sessionsObservableGauge

Active sessions per tunnel (or collapsed).

Show DetailsHide Details

Unit: 1
Labels: site_id, tunnel_id
Emission path: RegisterStateView
Example: newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2

newt_tunnel_bytes_totalCounter (bytes)

Traffic per tunnel, direction, and protocol.

Show DetailsHide Details

Unit: bytes
Labels: tunnel_id, direction (ingress|egress), protocol (tcp|udp)
Emission path: Proxy manager
Example: newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192

newt_tunnel_latency_secondsHistogram (s)

RTT samples per tunnel/transport.

Show DetailsHide Details

Unit: seconds
Labels: tunnel_id, transport
Emission path: Health checks
Example: newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4

newt_tunnel_reconnects_totalCounter

Reconnect attempts keyed by initiator & reason.

Show DetailsHide Details

Unit: 1
Labels: tunnel_id, initiator (client|server), reason
Emission path: telemetry.IncReconnect
Example: newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3

Show Connection & AuthHide Connection & Auth

newt_connection_attempts_totalCounter

Auth/WebSocket connection attempts keyed by transport & result.

Show DetailsHide Details

Unit: 1
Labels: transport, result
Emission path: telemetry.IncConnAttempt
Example: newt_connection_attempts_total{transport="websocket",result="failure"} 2

newt_connection_errors_totalCounter

Connection errors keyed by transport and type.

Show DetailsHide Details

Unit: 1
Labels: transport, error_type
Emission path: telemetry.IncConnError
Example: newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1

Show WebSocketHide WebSocket

newt_websocket_connect_latency_secondsHistogram (s)

Dial latency for Pangolin WebSocket.

Show DetailsHide Details

Unit: seconds
Labels: result, transport
Emission path: ObserveWSConnectLatency
Example: newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1

newt_websocket_disconnects_totalCounter

WebSocket disconnects keyed by reason.

Show DetailsHide Details

Unit: 1
Labels: reason, tunnel_id
Emission path: IncWSDisconnect
Example: newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2

newt_websocket_keepalive_failures_totalCounter

Ping/Pong failures observed by keepalive.

Show DetailsHide Details

Unit: 1
Labels: reason (e.g., ping_write, pong_timeout)
Emission path: telemetry.IncWSKeepaliveFailure(ctx, "ping_write")
Example: newt_websocket_keepalive_failures_total{reason="ping_write"} 1

newt_websocket_session_duration_secondsHistogram (s)

Duration of established WS sessions keyed by result.

Show DetailsHide Details

Unit: seconds
Labels: result (success|error)
Emission path: telemetry.ObserveWSSessionDuration(ctx, time.Since(start).Seconds(), "error")
Example: newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3

newt_websocket_connectedObservableGauge

Current WS connection state (0/1).

Show DetailsHide Details

Unit: 1
Labels: —
Emission path: telemetry.SetWSConnectionState(true|false)
Example: newt_websocket_connected 1

newt_websocket_reconnects_totalCounter

WebSocket reconnect attempts keyed by reason.

Show DetailsHide Details

Unit: 1
Labels: reason
Emission path: telemetry.IncWSReconnect(ctx, "ping_write")
Example: newt_websocket_reconnects_total{reason="ping_write"} 1

newt_websocket_messages_totalCounter

In/out WS messages keyed by direction & type.

Show DetailsHide Details

Unit: 1
Labels: direction (in|out), msg_type (ping|pong|text|...)
Emission path: IncWSMessage
Example: newt_websocket_messages_total{direction="out",msg_type="ping"} 4

Show ProxyHide Proxy

newt_proxy_active_connectionsObservableGauge

Active TCP/UDP proxy connections per tunnel/protocol.

Show DetailsHide Details

Unit: 1
Labels: protocol, tunnel_id
Emission path: Proxy callback
Example: newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3

newt_proxy_buffer_bytesObservableGauge (bytes)

Proxy buffer pool size.

Show DetailsHide Details

Unit: bytes
Labels: protocol, tunnel_id
Emission path: Proxy callback
Example: newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240

newt_proxy_async_backlog_bytesObservableGauge (bytes)

Unflushed async byte backlog.

Show DetailsHide Details

Unit: bytes
Labels: protocol, tunnel_id
Emission path: Proxy callback
Example: newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096

newt_proxy_drops_totalCounter

Proxy write drops keyed by protocol/tunnel.

Show DetailsHide Details

Unit: 1
Labels: protocol, tunnel_id
Emission path: IncProxyDrops
Example: newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2

newt_proxy_accept_totalCounter

Proxy accept events keyed by result/reason.

Show DetailsHide Details

Unit: 1
Labels: tunnel_id, protocol, result, reason
Emission path: telemetry.IncProxyAccept(ctx, tunnelID, "tcp", "failure", "timeout")
Example: newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1

newt_proxy_connections_totalCounter

Lifecycle events (opened/closed) per connection.

Show DetailsHide Details

Unit: 1
Labels: tunnel_id, protocol, event (opened|closed)
Emission path: telemetry.IncProxyConnectionEvent(ctx, tunnelID, "tcp", telemetry.ProxyConnectionOpened)
Example: newt_proxy_connections_total{protocol="tcp",event="opened"} 1

newt_proxy_connection_duration_secondsHistogram (s)

Duration of completed proxy connections.

Show DetailsHide Details

Unit: seconds
Labels: tunnel_id, protocol, result
Emission path: telemetry.ObserveProxyConnectionDuration(ctx, tunnelID, "tcp", "success", seconds)
Example: newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3

Gerbil metrics

gerbil

OpenTelemetry metric instruments exposed by Gerbil. Gerbil supports exactly one metrics backend at runtime: prometheus, otel, or none.

In otel mode, Gerbil pushes metrics to an OTLP collector. The /metrics endpoint is not exposed in this mode.

Show WireGuardHide WireGuard

gerbil_wg_interface_upInt64Gauge

Operational state of a WireGuard interface.

Show DetailsHide Details

Unit: 1
Labels: ifname, instance
Emission path: metrics.RecordInterfaceUp
Example: gerbil_wg_interface_up{ifname="wg0",instance="gerbil-1"} 1

gerbil_wg_peers_totalUpDownCounter

Number of configured peers per interface.

Show DetailsHide Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordPeersTotal
Example: gerbil_wg_peers_total{ifname="wg0"} 5

gerbil_wg_peer_connectedInt64Gauge

Current peer connection state.

Show DetailsHide Details

Unit: 1
Labels: ifname, peer
Emission path: metrics.RecordPeerConnected
Example: gerbil_wg_peer_connected{ifname="wg0",peer="abc"} 1

gerbil_allowed_ips_countUpDownCounter

Number of allowed IPs configured per peer.

Show DetailsHide Details

Unit: 1
Labels: ifname, peer
Emission path: metrics.RecordAllowedIPsCount
Example: gerbil_allowed_ips_count{ifname="wg0",peer="abc"} 2

gerbil_key_rotation_totalCounter

Key rotation events.

Show DetailsHide Details

Unit: 1
Labels: ifname, reason
Emission path: metrics.RecordKeyRotation
Example: gerbil_key_rotation_total{ifname="wg0",reason="scheduled"} 1

gerbil_wg_handshakes_totalCounter

WireGuard handshake attempts keyed by result.

Show DetailsHide Details

Unit: 1
Labels: ifname, peer, result
Emission path: metrics.RecordHandshake
Example: gerbil_wg_handshakes_total{ifname="wg0",peer="abc",result="success"} 1

gerbil_wg_handshake_latency_secondsHistogram (s)

Distribution of WireGuard handshake latencies.

Show DetailsHide Details

Unit: seconds
Labels: ifname, peer
Emission path: metrics.RecordHandshakeLatency
Example: gerbil_wg_handshake_latency_seconds_bucket{ifname="wg0",peer="abc",le="0.1"} 3

gerbil_wg_peer_rtt_secondsHistogram (s)

Observed peer round-trip time.

Show DetailsHide Details

Unit: seconds
Labels: ifname, peer
Emission path: metrics.RecordPeerRTT
Example: gerbil_wg_peer_rtt_seconds_bucket{ifname="wg0",peer="abc",le="0.05"} 4

gerbil_wg_bytes_received_totalCounter (bytes)

Bytes received from a WireGuard peer.

Show DetailsHide Details

Unit: bytes
Labels: ifname, peer
Emission path: metrics.RecordBytesReceived
Example: gerbil_wg_bytes_received_total{ifname="wg0",peer="abc"} 8192

gerbil_wg_bytes_transmitted_totalCounter (bytes)

Bytes transmitted to a WireGuard peer.

Show DetailsHide Details

Unit: bytes
Labels: ifname, peer
Emission path: metrics.RecordBytesTransmitted
Example: gerbil_wg_bytes_transmitted_total{ifname="wg0",peer="abc"} 16384

Show RelayHide Relay

gerbil_active_sessionsUpDownCounter

Number of active UDP relay sessions.

Show DetailsHide Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordActiveSession / metrics.RecordSession
Example: gerbil_active_sessions{ifname="wg0"} 3

gerbil_udp_packets_totalCounter

UDP packets processed by relay workers.

Show DetailsHide Details

Unit: 1
Labels: ifname, type, direction
Emission path: metrics.RecordUDPPacket
Example: gerbil_udp_packets_total{ifname="wg0",type="data",direction="rx"} 42

gerbil_udp_packet_size_bytesHistogram (bytes)

Size distribution of packets forwarded through the relay.

Show DetailsHide Details

Unit: bytes
Labels: ifname, type
Emission path: metrics.RecordUDPPacketSize
Example: gerbil_udp_packet_size_bytes_bucket{ifname="wg0",type="data",le="1024"} 7

gerbil_hole_punch_events_totalCounter

Hole punch messages processed by result.

Show DetailsHide Details

Unit: 1
Labels: ifname, result
Emission path: metrics.RecordHolePunchEvent
Example: gerbil_hole_punch_events_total{ifname="wg0",result="success"} 1

gerbil_proxy_mapping_activeUpDownCounter

Active proxy mappings.

Show DetailsHide Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordProxyMapping
Example: gerbil_proxy_mapping_active{ifname="wg0"} 4

gerbil_session_rebuilt_totalCounter

Sessions rebuilt from communication patterns.

Show DetailsHide Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordSessionRebuilt
Example: gerbil_session_rebuilt_total{ifname="wg0"} 1

gerbil_comm_pattern_activeUpDownCounter

Active communication patterns.

Show DetailsHide Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordCommPattern
Example: gerbil_comm_pattern_active{ifname="wg0"} 2

gerbil_proxy_cleanup_removed_totalCounter

Items removed by cleanup routines.

Show DetailsHide Details

Unit: 1
Labels: ifname, component
Emission path: metrics.RecordProxyCleanupRemoved
Example: gerbil_proxy_cleanup_removed_total{ifname="wg0",component="sessions"} 5

gerbil_proxy_connection_errors_totalCounter

Proxy connection errors.

Show DetailsHide Details

Unit: 1
Labels: ifname, error_type
Emission path: metrics.RecordProxyConnectionError
Example: gerbil_proxy_connection_errors_total{ifname="wg0",error_type="timeout"} 1

gerbil_proxy_initial_mappingsInt64Gauge

Initial proxy mappings loaded.

Show DetailsHide Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordProxyInitialMappings
Example: gerbil_proxy_initial_mappings{ifname="wg0"} 8

gerbil_proxy_mapping_updates_totalCounter

Proxy mapping updates.

Show DetailsHide Details

Unit: 1
Labels: ifname
Emission path: metrics.RecordProxyMappingUpdate
Example: gerbil_proxy_mapping_updates_total{ifname="wg0"} 2

gerbil_proxy_idle_cleanup_duration_secondsHistogram (s)

Duration of idle cleanup cycles.

Show DetailsHide Details

Unit: seconds
Labels: ifname, component
Emission path: metrics.RecordProxyIdleCleanupDuration
Example: gerbil_proxy_idle_cleanup_duration_seconds_bucket{ifname="wg0",component="sessions",le="0.1"} 1

Show SNI ProxyHide SNI Proxy

gerbil_active_proxy_connectionsUpDownCounter

Active SNI proxy connections.

Show DetailsHide Details

Unit: 1
Labels: —
Emission path: metrics.RecordActiveProxyConnection
Example: gerbil_active_proxy_connections 2

gerbil_proxy_route_lookups_totalCounter

Route lookups keyed by result.

Show DetailsHide Details

Unit: 1
Labels: result
Emission path: metrics.RecordProxyRouteLookup
Example: gerbil_proxy_route_lookups_total{result="hit"} 6

gerbil_proxy_tls_handshake_secondsHistogram (s)

TLS handshake duration for the SNI proxy.

Show DetailsHide Details

Unit: seconds
Labels: —
Emission path: metrics.RecordProxyTLSHandshake
Example: gerbil_proxy_tls_handshake_seconds_bucket{le="0.1"} 2

gerbil_proxy_bytes_transmitted_totalCounter (bytes)

Bytes sent or received by the SNI proxy.

Show DetailsHide Details

Unit: bytes
Labels: direction
Emission path: metrics.RecordProxyBytesTransmitted
Example: gerbil_proxy_bytes_transmitted_total{direction="egress"} 16384

gerbil_sni_connections_totalCounter

Connections processed by the SNI proxy.

Show DetailsHide Details

Unit: 1
Labels: result
Emission path: metrics.RecordSNIConnection
Example: gerbil_sni_connections_total{result="success"} 3

gerbil_sni_connection_duration_secondsHistogram (s)

Lifetime distribution of proxied TLS connections.

Show DetailsHide Details

Unit: seconds
Labels: —
Emission path: metrics.RecordSNIConnectionDuration
Example: gerbil_sni_connection_duration_seconds_bucket{le="10"} 4

gerbil_sni_active_connectionsUpDownCounter

Active SNI tunnels.

Show DetailsHide Details

Unit: 1
Labels: —
Emission path: metrics.RecordSNIActiveConnection
Example: gerbil_sni_active_connections 2

gerbil_sni_route_cache_hits_totalCounter

SNI route cache hits and misses.

Show DetailsHide Details

Unit: 1
Labels: result
Emission path: metrics.RecordSNIRouteCacheHit
Example: gerbil_sni_route_cache_hits_total{result="hit"} 10

gerbil_sni_route_api_requests_totalCounter

SNI route API requests.

Show DetailsHide Details

Unit: 1
Labels: result
Emission path: metrics.RecordSNIRouteAPIRequest
Example: gerbil_sni_route_api_requests_total{result="success"} 5

gerbil_sni_route_api_latency_secondsHistogram (s)

Route API call latency.

Show DetailsHide Details

Unit: seconds
Labels: —
Emission path: metrics.RecordSNIRouteAPILatency
Example: gerbil_sni_route_api_latency_seconds_bucket{le="0.25"} 4

gerbil_sni_local_override_totalCounter

Routes using local overrides.

Show DetailsHide Details

Unit: 1
Labels: hit
Emission path: metrics.RecordSNILocalOverride
Example: gerbil_sni_local_override_total{hit="true"} 1

gerbil_sni_trusted_proxy_events_totalCounter

PROXY protocol events.

Show DetailsHide Details

Unit: 1
Labels: event
Emission path: metrics.RecordSNITrustedProxyEvent
Example: gerbil_sni_trusted_proxy_events_total{event="parsed"} 2

gerbil_sni_proxy_protocol_parse_errors_totalCounter

PROXY protocol parse failures.

Show DetailsHide Details

Unit: 1
Labels: —
Emission path: metrics.RecordSNIProxyProtocolParseError
Example: gerbil_sni_proxy_protocol_parse_errors_total 1

gerbil_sni_data_bytes_totalCounter (bytes)

Bytes proxied through SNI tunnels.

Show DetailsHide Details

Unit: bytes
Labels: direction
Emission path: metrics.RecordSNIDataBytes
Example: gerbil_sni_data_bytes_total{direction="ingress"} 4096

gerbil_sni_tunnel_terminations_totalCounter

SNI tunnel terminations keyed by reason.

Show DetailsHide Details

Unit: 1
Labels: reason
Emission path: metrics.RecordSNITunnelTermination
Example: gerbil_sni_tunnel_terminations_total{reason="client_close"} 1

Show HTTP API & Peer ManagementHide HTTP API & Peer Management

gerbil_http_requests_totalCounter

HTTP requests to the management API.

Show DetailsHide Details

Unit: 1
Labels: endpoint, method, status_code
Emission path: metrics.RecordHTTPRequest
Example: gerbil_http_requests_total{endpoint="/peer",method="POST",status_code="200"} 1

gerbil_http_request_duration_secondsHistogram (s)

HTTP request handling time.

Show DetailsHide Details

Unit: seconds
Labels: endpoint, method
Emission path: metrics.RecordHTTPRequestDuration
Example: gerbil_http_request_duration_seconds_bucket{endpoint="/peer",method="POST",le="0.1"} 3

gerbil_peer_operations_totalCounter

Peer lifecycle operations.

Show DetailsHide Details

Unit: 1
Labels: operation, result
Emission path: metrics.RecordPeerOperation
Example: gerbil_peer_operations_total{operation="add",result="success"} 1

gerbil_proxy_mapping_update_requests_totalCounter

Proxy mapping update API calls.

Show DetailsHide Details

Unit: 1
Labels: result
Emission path: metrics.RecordProxyMappingUpdateRequest
Example: gerbil_proxy_mapping_update_requests_total{result="success"} 1

gerbil_destinations_update_requests_totalCounter

Destination update API calls.

Show DetailsHide Details

Unit: 1
Labels: result
Emission path: metrics.RecordDestinationsUpdateRequest
Example: gerbil_destinations_update_requests_total{result="success"} 1

Show Remote Config & ReportingHide Remote Config & Reporting

gerbil_remote_config_fetches_totalCounter

Remote configuration fetch attempts.

Show DetailsHide Details

Unit: 1
Labels: result
Emission path: metrics.RecordRemoteConfigFetch
Example: gerbil_remote_config_fetches_total{result="success"} 1

gerbil_bandwidth_reports_totalCounter

Bandwidth report transmissions.

Show DetailsHide Details

Unit: 1
Labels: result
Emission path: metrics.RecordBandwidthReport
Example: gerbil_bandwidth_reports_total{result="success"} 1

gerbil_peer_bandwidth_bytes_totalCounter (bytes)

Bytes per peer tracked by bandwidth calculation.

Show DetailsHide Details

Unit: bytes
Labels: peer, direction
Emission path: metrics.RecordPeerBandwidthBytes
Example: gerbil_peer_bandwidth_bytes_total{peer="abc",direction="rx"} 8192

Show System & OperationsHide System & Operations

gerbil_netlink_events_totalCounter

Netlink events processed.

Show DetailsHide Details

Unit: 1
Labels: event_type
Emission path: metrics.RecordNetlinkEvent
Example: gerbil_netlink_events_total{event_type="link_up"} 1

gerbil_netlink_errors_totalCounter

Netlink or kernel errors.

Show DetailsHide Details

Unit: 1
Labels: component, error_type
Emission path: metrics.RecordNetlinkError
Example: gerbil_netlink_errors_total{component="wg",error_type="permission"} 1

gerbil_sync_duration_secondsHistogram (s)

Duration of reconciliation or sync loops.

Show DetailsHide Details

Unit: seconds
Labels: component
Emission path: metrics.RecordSyncDuration
Example: gerbil_sync_duration_seconds_bucket{component="remote_config",le="0.5"} 2

gerbil_workqueue_depthUpDownCounter

Current length of internal work queues.

Show DetailsHide Details

Unit: 1
Labels: queue
Emission path: metrics.RecordWorkqueueDepth
Example: gerbil_workqueue_depth{queue="relay"} 3

gerbil_kernel_module_loads_totalCounter

Kernel module load attempts.

Show DetailsHide Details

Unit: 1
Labels: result
Emission path: metrics.RecordKernelModuleLoad
Example: gerbil_kernel_module_loads_total{result="success"} 1

gerbil_firewall_rules_applied_totalCounter

Firewall rules applied.

Show DetailsHide Details

Unit: 1
Labels: result, chain
Emission path: metrics.RecordFirewallRuleApplied
Example: gerbil_firewall_rules_applied_total{result="success",chain="FORWARD"} 1

gerbil_config_reloads_totalCounter

Configuration reloads.

Show DetailsHide Details

Unit: 1
Labels: result
Emission path: metrics.RecordConfigReload
Example: gerbil_config_reloads_total{result="success"} 1

gerbil_restart_totalCounter

Process restart count.

Show DetailsHide Details

Unit: 1
Labels: —
Emission path: metrics.RecordRestart
Example: gerbil_restart_total 1

gerbil_auth_failures_totalCounter

Authentication or peer validation failures.

Show DetailsHide Details

Unit: 1
Labels: peer, reason
Emission path: metrics.RecordAuthFailure
Example: gerbil_auth_failures_total{peer="abc",reason="invalid_key"} 1

gerbil_acl_denied_totalCounter

Access-control denied events.

Show DetailsHide Details

Unit: 1
Labels: ifname, peer, policy
Emission path: metrics.RecordACLDenied
Example: gerbil_acl_denied_total{ifname="wg0",peer="abc",policy="deny"} 1

gerbil_certificate_expiry_daysFloat64Gauge

Days until certificate expiry.

Show DetailsHide Details

Unit: days
Labels: cert_name, ifname
Emission path: metrics.RecordCertificateExpiry
Example: gerbil_certificate_expiry_days{cert_name="server",ifname="wg0"} 42

gerbil_memory_spike_totalCounter

Memory spikes detected by severity.

Show DetailsHide Details

Unit: 1
Labels: severity
Emission path: metrics.RecordMemorySpike
Example: gerbil_memory_spike_total{severity="warning"} 1

gerbil_heap_profiles_written_totalCounter

Heap profile files generated.

Show DetailsHide Details

Unit: 1
Labels: —
Emission path: metrics.RecordHeapProfileWritten
Example: gerbil_heap_profiles_written_total 1

Pangolin Kubernetes Controller metrics

pangolin-kube-controller

Additional OpenTelemetry metric instruments exposed by the Pangolin Kubernetes Controller.

The controller exposes Prometheus-native metrics and additional OTel-backed metrics on the same /metrics endpoint. The OTel-backed series use the pangolin_controller_* prefix.

Show ReconcileHide Reconcile

pangolin_controller_reconcile_phase_duration_secondsHistogram (s)

Duration of each reconcile phase.

Show DetailsHide Details

Unit: seconds
Labels: phase, result
Label values:
phase: middlewares | routers | serversTransports | services | tcp | udp
result: success | error
Emission path: OTel reconcile phase instrumentation
Example: pangolin_controller_reconcile_phase_duration_seconds_bucket{phase="routers",result="success",le="0.5"} 3

pangolin_controller_active_reconcile_routinesUpDownCounter

Number of active reconcile routines by phase.

Show DetailsHide Details

Unit: 1
Labels: phase
Label values: middlewares | routers | serversTransports | services | tcp | udp
Emission path: Parallel reconcile instrumentation
Example: pangolin_controller_active_reconcile_routines{phase="routers"} 1

pangolin_controller_loop_iterations_totalCounter

Controller loop iterations by outcome.

Show DetailsHide Details

Unit: 1
Labels: outcome
Label values: success | nochange | error
Emission path: Controller loop instrumentation
Example: pangolin_controller_loop_iterations_total{outcome="success"} 10

Show Fetch & ConfigHide Fetch & Config

pangolin_controller_fetch_duration_secondsHistogram (s)

Duration of remote fetch cycle HTTP requests.

Show DetailsHide Details

Unit: seconds
Labels: status_code, status_class
Label values:
status_code: 200 | 304 | 401 | 403 | 404 | 5xx
status_class: 2xx | 3xx | 4xx | 5xx
Emission path: Remote config fetch instrumentation
Example: pangolin_controller_fetch_duration_seconds_bucket{status_code="200",status_class="2xx",le="0.25"} 4

pangolin_controller_config_parse_duration_secondsHistogram (s)

Duration of configuration parsing.

Show DetailsHide Details

Unit: seconds
Labels: section
Label values: full
Emission path: Config parse instrumentation
Example: pangolin_controller_config_parse_duration_seconds_bucket{section="full",le="0.1"} 2

Show Kubernetes APIHide Kubernetes API

pangolin_controller_k8s_request_duration_secondsHistogram (s)

Duration of Kubernetes API requests.

Show DetailsHide Details

Unit: seconds
Labels: verb, resource_kind, result, forced
Label values:
verb: get | create | patch | update | delete | list
resource_kind: IngressRoute | Middleware | TraefikService | ServersTransport | ServersTransportTCP | Service | EndpointSlice
result: success | error | conflict
forced: true | false
Emission path: Kubernetes API request instrumentation
Example: pangolin_controller_k8s_request_duration_seconds_bucket{verb="patch",resource_kind="IngressRoute",result="success",forced="false",le="0.25"} 3

pangolin_controller_k8s_requests_totalCounter

Total Kubernetes API requests.

Show DetailsHide Details

Unit: 1
Labels: verb, resource_kind, result, forced
Emission path: Kubernetes API request instrumentation
Example: pangolin_controller_k8s_requests_total{verb="patch",resource_kind="IngressRoute",result="success",forced="false"} 1

pangolin_controller_retries_totalCounter

Retry attempts in the SSA apply loop.

Show DetailsHide Details

Unit: 1
Labels: reason, operation, resource_kind
Label values:
reason: conflict | transient | timeout
operation: get | create | patch | delete | apply
Emission path: SSA apply retry instrumentation
Example: pangolin_controller_retries_total{reason="conflict",operation="patch",resource_kind="IngressRoute"} 1

Show Garbage CollectionHide Garbage Collection

pangolin_controller_gc_run_duration_secondsHistogram (s)

Duration of garbage collection runs.

Show DetailsHide Details

Unit: seconds
Labels: result
Label values: success | fail | dryrun
Emission path: GC instrumentation
Example: pangolin_controller_gc_run_duration_seconds_bucket{result="success",le="0.5"} 1

References

Have improvements or a missing metric? Open an issue or PR referencing this page.

Metrics and Observability

Observability Capability MatrixCopy Anchor Link

Supported SignalsCopy Anchor Link

Collection PatternsCopy Anchor Link

NewtCopy Anchor Link

GerbilCopy Anchor Link

Pangolin Kubernetes ControllerCopy Anchor Link

Alerting ExamplesCopy Anchor Link

VersionsCopy Anchor Link

Full Metric ReferenceCopy Anchor Link

Newt metricsCopy Anchor Link

Gerbil metricsCopy Anchor Link

Pangolin Kubernetes Controller metricsCopy Anchor Link

ReferencesCopy Anchor Link

On this page

Observability Capability Matrix

Supported Signals

Collection Patterns

Newt

Gerbil

Pangolin Kubernetes Controller

Alerting Examples

Versions

Full Metric Reference

Newt metrics

Gerbil metrics

Pangolin Kubernetes Controller metrics

References