We provide metrics in the OpenTelemetry (OTel) format and additionally support the following vendor backends:
Prometheus (native scrape and via OTel Collector)
Why Metrics & OTel Observability enables:
Incident detection (latency spikes, reconnect storms)Capacity planning (bytes, active sessions)User‑experience SLAs (p95 tunnel latency, auth latency)Faster RCA (dimensions like error_type, result)
OpenTelemetry provides a vendor‑neutral pipeline so you can change backends without retouching instrumented code.
Availability Newt exposes metrics starting from specific releases, but metrics are disabled in their default configuration.
Newt: metrics implemented since Newt 1.6.0 (disabled by default)
Open Telemetry Push metrics and traces to an OTel Collector or any backend that accepts OTLP.
If you only enable Prometheus scrape, leave *_METRICS_OTLP_ENABLED=false and omit OTLP vars.
The OTel Collector commonly uses port 4317 for gRPC and 4318 for HTTP. Set OTEL_EXPORTER_OTLP_PROTOCOL to http/protobuf for HTTP or grpc for gRPC, and point OTEL_EXPORTER_OTLP_ENDPOINT accordingly.
For further customization, see the OTel Collector documentation .
Environment Variables
CLI Args
NEWT_METRICS_OTLP_ENABLED=true # enable OTLP exporter OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317 OTEL_EXPORTER_OTLP_INSECURE=true # or false + TLS vars OTEL_METRIC_EXPORT_INTERVAL=15s # Optional auth / TLS OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer%20XYZ OTEL_EXPORTER_OTLP_CERTIFICATE=/etc/otel/ca.pem
Newt Configuration Examples
CLI (gRPC)
Docker Compose
# Enable OTLP exporters and point to your Collector's gRPC receiver. export OTEL_EXPORTER_OTLP_ENDPOINT = "http://localhost:4317" export OTEL_EXPORTER_OTLP_PROTOCOL = "grpc" newt \ --otlp=true --id saz281jfa8z37zg --secret ssfdfsder33rrerrwe --endpoint http://pangolin.example.com Prometheus (without OTel Collector)
Each service listens on an admin HTTP address (example Newt default :2112). Environment Variables
CLI Args
NEWT_METRICS_PROMETHEUS_ENABLED=true # /metrics endpoint NEWT_ADMIN_ADDR=:2112 # admin HTTP address
Newt Configuration Examples
CLI
Docker Compose
Prometheus Scrape Config
newt \ --metrics-prometheus-enabled=true \ --admin-addr=:2112 \ --id saz281jfa8z37zg \ --secret ssfdfsder33rrerrwe \ --endpoint https://pangolin.example.com Full Metric Reference Version 1.0.0 from 2025-10-28 Below are currently implemented metrics for Newt .
Metric : exact metric nameInstrument & unit : OTel instrument type and canonical unitPurpose : what the metric conveys / recommended useEmission path : subsystem responsible (for troubleshooting missing data)Example series : representative sample including labels
Names/labels can change between major versions. Avoid hard‑coding full label sets in alerts; prefer existence checks and aggregate functions.
Newt metrics OpenTelemetry (OTel)
Prometheus
OpenTelemetry metric instruments exposed by Newt. Expand each section to see individual metrics with labels, units, emission points, and examples. newt_site_registrations_total
Counts Pangolin registration attempts keyed by result. Unit: 1Labels: result (success|failure), site_idEmission path: telemetry.IncSiteRegistrationExample: newt_site_registrations_total{result="success",site_id="abc"} 1
0/1 heartbeat for the active site. Unit: 1Labels: site_idEmission path: state.TelemetryView (callback)Example: newt_site_online{site_id="self"} 1
newt_site_last_heartbeat_seconds
Seconds since last Pangolin heartbeat. Unit: secondsLabels: site_idEmission path: TouchHeartbeat (callback)Example: newt_site_last_heartbeat_seconds{site_id="self"} 3.2
Constant 1 with build metadata labels. Unit: 1Labels: version, commitEmission path: Build info registrationExample: newt_build_info{version="1.2.3",commit="abc123"} 1
Process boot indicator (increments once per process start). Unit: 1Labels: —Emission path: RegisterBuildInfoExample: newt_restart_count_total 1
Certificate rotation events keyed by result. Unit: 1Labels: resultEmission path: IncCertRotationExample: newt_cert_rotation_total{result="success"} 1
newt_config_reloads_total
Config reload attempts keyed by result. Unit: 1Labels: resultEmission path: telemetry.IncConfigReloadExample: newt_config_reloads_total{result="success"} 1
newt_config_apply_seconds
Duration per config-apply phase keyed by phase and result. Unit: secondsLabels: phase, resultEmission path: telemetry.ObserveConfigApplyExample: newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3
Active sessions per tunnel (or collapsed). Unit: 1Labels: site_id, tunnel_idEmission path: RegisterStateViewExample: newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2
Traffic per tunnel, direction, and protocol. Unit: bytesLabels: tunnel_id, direction (ingress|egress), protocol (tcp|udp)Emission path: Proxy managerExample: newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192
newt_tunnel_latency_seconds
RTT samples per tunnel/transport. Unit: secondsLabels: tunnel_id, transportEmission path: Health checksExample: newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4
newt_tunnel_reconnects_total
Reconnect attempts keyed by initiator & reason. Unit: 1Labels: tunnel_id, initiator (client|server), reasonEmission path: telemetry.IncReconnectExample: newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3
newt_connection_attempts_total
Auth/WebSocket connection attempts keyed by transport & result. Unit: 1Labels: transport, resultEmission path: telemetry.IncConnAttemptExample: newt_connection_attempts_total{transport="websocket",result="failure"} 2
newt_connection_errors_total
Connection errors keyed by transport and type. Unit: 1Labels: transport, error_typeEmission path: telemetry.IncConnErrorExample: newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1
newt_websocket_connect_latency_seconds
Dial latency for Pangolin WebSocket. Unit: secondsLabels: result, transportEmission path: ObserveWSConnectLatencyExample: newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1
newt_websocket_disconnects_total
WebSocket disconnects keyed by reason. Unit: 1Labels: reason, tunnel_idEmission path: IncWSDisconnectExample: newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2
newt_websocket_keepalive_failures_total
Ping/Pong failures observed by keepalive. Unit: 1Labels: reason (e.g., ping_write, pong_timeout)Emission path: telemetry.IncWSKeepaliveFailure(ctx, "ping_write")Example: newt_websocket_keepalive_failures_total{reason="ping_write"} 1
newt_websocket_session_duration_seconds
Duration of established WS sessions keyed by result. Unit: secondsLabels: result (success|error)Emission path: telemetry.ObserveWSSessionDuration(ctx, time.Since(start).Seconds(), "error")Example: newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3
Current WS connection state (0/1). Unit: 1Labels: —Emission path: telemetry.SetWSConnectionState(true|false)Example: newt_websocket_connected 1
newt_websocket_reconnects_total
WebSocket reconnect attempts keyed by reason. Unit: 1Labels: reasonEmission path: telemetry.IncWSReconnect(ctx, "ping_write")Example: newt_websocket_reconnects_total{reason="ping_write"} 1
newt_websocket_messages_total
In/out WS messages keyed by direction & type. Unit: 1Labels: direction (in|out), msg_type (ping|pong|text|…)Emission path: IncWSMessageExample: newt_websocket_messages_total{direction="out",msg_type="ping"} 4
newt_proxy_active_connections
Active TCP/UDP proxy connections per tunnel/protocol. Unit: 1Labels: protocol, tunnel_idEmission path: Proxy callbackExample: newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3
Proxy buffer pool size. Unit: bytesLabels: protocol, tunnel_idEmission path: Proxy callbackExample: newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240
newt_proxy_async_backlog_bytes
Unflushed async byte backlog. Unit: bytesLabels: protocol, tunnel_idEmission path: Proxy callbackExample: newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096
Proxy write drops keyed by protocol/tunnel. Unit: 1Labels: protocol, tunnel_idEmission path: IncProxyDropsExample: newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2
Proxy accept events keyed by result/reason. Unit: 1Labels: tunnel_id, protocol, result, reasonEmission path: telemetry.IncProxyAccept(ctx, tunnelID, "tcp", "failure", "timeout")Example: newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1
newt_proxy_connections_total
Lifecycle events (opened/closed) per connection. Unit: 1Labels: tunnel_id, protocol, event (opened|closed)Emission path: telemetry.IncProxyConnectionEvent(ctx, tunnelID, "tcp", telemetry.ProxyConnectionOpened)Example: newt_proxy_connections_total{protocol="tcp",event="opened"} 1
newt_proxy_connection_duration_seconds
Duration of completed proxy connections. Unit: secondsLabels: tunnel_id, protocol, resultEmission path: telemetry.ObserveProxyConnectionDuration(ctx, tunnelID, "tcp", "success", seconds)Example: newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3
References Have improvements or a missing metric? Open an issue or PR referencing this page.
