- Prometheus (native scrape and via OTel Collector)
Why Metrics & OTel
Observability enables:- Incident detection (latency spikes, reconnect storms)
- Capacity planning (bytes, active sessions)
- User‑experience SLAs (p95 tunnel latency, auth latency)
- Faster RCA (dimensions like
error_type,result)
Availability
Newt exposes metrics starting from specific releases, but metrics are disabled in their default configuration.- Newt: metrics implemented since Newt 1.6.0 (disabled by default)
Open Telemetry
Push metrics and traces to an OTel Collector or any backend that accepts OTLP.If you only enable Prometheus scrape, leave
*_METRICS_OTLP_ENABLED=false and omit OTLP vars.The OTel Collector commonly uses port
4317 for gRPC and 4318 for HTTP. Set OTEL_EXPORTER_OTLP_PROTOCOL to http/protobuf for HTTP or grpc for gRPC, and point OTEL_EXPORTER_OTLP_ENDPOINT accordingly.
For further customization, see the OTel Collector documentation.Newt Configuration
Newt Configuration
- Environment Variables
- CLI Args
Copy
Ask AI
NEWT_METRICS_OTLP_ENABLED=true # enable OTLP exporter
OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
OTEL_EXPORTER_OTLP_INSECURE=true # or false + TLS vars
OTEL_METRIC_EXPORT_INTERVAL=15s
# Optional auth / TLS
OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer%20XYZ
OTEL_EXPORTER_OTLP_CERTIFICATE=/etc/otel/ca.pem
Copy
Ask AI
newt \
--metrics-otlp-enabled=true \ # alias for otel
--otel=true \
--otel-exporter-otlp-endpoint=otel-collector:4317 \
--otel-exporter-otlp-insecure=true \
--otel-metric-export-interval=15s \
--otel-exporter-otlp-headers=authorization=Bearer%20XYZ \
--otel-exporter-otlp-certificate=/etc/otel/ca.pem
Newt Configuration Examples
Newt Configuration Examples
- CLI (gRPC)
- Docker Compose
Copy
Ask AI
# Enable OTLP exporters and point to your Collector's gRPC receiver.
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
newt \
--otlp=true
--id saz281jfa8z37zg
--secret ssfdfsder33rrerrwe
--endpoint http://pangolin.example.com
docker-compose.metrics.yaml
Copy
Ask AI
services:
otel-collector:
image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:latest # DO NOT use 'latest' in production
command: ["--config=/etc/otel/config.yaml"]
volumes:
- ./otel-config.yaml:/etc/otel/config.yaml:ro
ports:
- "4317:4317" # gRPC
- "4318:4318" # HTTP
- "8888:8888" # Prometheus exporter (from the Collector) - Optional
newt:
image: fosrl/newt:latest # DO NOT use 'latest' in production
environment:
NEWT_METRICS_OTLP_ENABLED: "true"
OTEL_EXPORTER_OTLP_ENDPOINT: otel-collector:4317
OTEL_EXPORTER_OTLP_INSECURE: "true"
PANGOLIN_ENDPOINT: https://example.com
NEWT_ID: heresmynewtid
NEWT_SECRET: yoursupersecretkeyhere
otel-config.yaml
Copy
Ask AI
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors: {}
# Example exporters:
exporters:
otlp:
endpoint: otel-collector:4317
insecure: true
prometheus:
endpoint: "0.0.0.0:8889"
service:
pipelines:
metrics:
receivers: [otlp]
processors: []
exporters: [prometheus]
otel-config-remote.yaml
Copy
Ask AI
exporters:
prometheusremotewrite:
endpoint: https://prom-remote.example.com/api/v1/write
headers:
X-Scope-OrgID: tenant-a
tls:
insecure_skip_verify: false
service:
pipelines:
metrics/remote:
receivers: [otlp]
processors: [batch]
exporters: [prometheusremotewrite]
Combine exporters (e.g. local Prometheus + remote write) to retain fast local dashboards and ship long‑term retention externally.
Prometheus (without OTel Collector)
Newt Configuration
Newt Configuration
Each service listens on an admin HTTP address (example Newt default See the CLI reference for all available flags.
:2112).- Environment Variables
- CLI Args
Copy
Ask AI
NEWT_METRICS_PROMETHEUS_ENABLED=true # /metrics endpoint
NEWT_ADMIN_ADDR=:2112 # admin HTTP address
Copy
Ask AI
newt \
--metrics-prometheus-enabled=true \ # alias for metrics
--metrics=true
--admin-addr=:2112 \
Newt Configuration Examples
Newt Configuration Examples
- CLI
- Docker Compose
- Prometheus Scrape Config
Copy
Ask AI
newt \
--metrics-prometheus-enabled=true \
--admin-addr=:2112 \
--id saz281jfa8z37zg \
--secret ssfdfsder33rrerrwe \
--endpoint https://pangolin.example.com
docker-compose.metrics.yaml
Copy
Ask AI
services:
newt:
image: fosrl/newt:latest # DO NOT use 'latest' in production
environment:
NEWT_METRICS_OTLP_ENABLED: "true"
OTEL_EXPORTER_OTLP_ENDPOINT: otel-collector:4317
OTEL_EXPORTER_OTLP_INSECURE: "true"
PANGOLIN_ENDPOINT: https://example.com
NEWT_ID: saz281jfa8z37zg
NEWT_SECRET: ssfdfsder33rrerrwe
prometheus.yml (fragment)
Copy
Ask AI
scrape_configs:
- job_name: pangolin
static_configs: [{ targets: ["pangolin:2112"] }]
Full Metric Reference
Version 1.0.0 from 2025-10-28 Below are currently implemented metrics for Newt.- Metric: exact metric name
- Instrument & unit: OTel instrument type and canonical unit
- Purpose: what the metric conveys / recommended use
- Emission path: subsystem responsible (for troubleshooting missing data)
- Example series: representative sample including labels
Names/labels can change between major versions. Avoid hard‑coding full label sets in alerts; prefer existence checks and aggregate functions.
Newt metrics
- OpenTelemetry (OTel)
- Prometheus
newt
OpenTelemetry metric instruments exposed by Newt. Expand each section to see individual metrics with labels, units, emission points, and examples.
Show Site & Build
Show Site & Build
Counts Pangolin registration attempts keyed by result.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
result (success|failure), site_idEmission path:
telemetry.IncSiteRegistrationExample:
newt_site_registrations_total{result="success",site_id="abc"} 10/1 heartbeat for the active site.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
site_idEmission path:
state.TelemetryView (callback)Example:
newt_site_online{site_id="self"} 1Seconds since last Pangolin heartbeat.
Show Details
Show Details
Unit: seconds
Labels:
Emission path:
Example:
Labels:
site_idEmission path:
TouchHeartbeat (callback)Example:
newt_site_last_heartbeat_seconds{site_id="self"} 3.2Constant 1 with build metadata labels.
Show Details
Show Details
Unit: 1
Labels:
Emission path: Build info registration
Example:
Labels:
version, commitEmission path: Build info registration
Example:
newt_build_info{version="1.2.3",commit="abc123"} 1Process boot indicator (increments once per process start).
Show Details
Show Details
Unit: 1
Labels: —
Emission path:
Example:
Labels: —
Emission path:
RegisterBuildInfoExample:
newt_restart_count_total 1Certificate rotation events keyed by result.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
resultEmission path:
IncCertRotationExample:
newt_cert_rotation_total{result="success"} 1Config reload attempts keyed by result.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
resultEmission path:
telemetry.IncConfigReloadExample:
newt_config_reloads_total{result="success"} 1Duration per config-apply phase keyed by
phase and result.Show Details
Show Details
Unit: seconds
Labels:
Emission path:
Example:
Labels:
phase, resultEmission path:
telemetry.ObserveConfigApplyExample:
newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3Show Tunnel
Show Tunnel
Active sessions per tunnel (or collapsed).
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
site_id, tunnel_idEmission path:
RegisterStateViewExample:
newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2Traffic per tunnel, direction, and protocol.
Show Details
Show Details
Unit: bytes
Labels:
Emission path: Proxy manager
Example:
Labels:
tunnel_id, direction (ingress|egress), protocol (tcp|udp)Emission path: Proxy manager
Example:
newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192RTT samples per tunnel/transport.
Show Details
Show Details
Unit: seconds
Labels:
Emission path: Health checks
Example:
Labels:
tunnel_id, transportEmission path: Health checks
Example:
newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4Reconnect attempts keyed by initiator & reason.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
tunnel_id, initiator (client|server), reasonEmission path:
telemetry.IncReconnectExample:
newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3Show Connection & Auth
Show Connection & Auth
Auth/WebSocket connection attempts keyed by transport & result.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
transport, resultEmission path:
telemetry.IncConnAttemptExample:
newt_connection_attempts_total{transport="websocket",result="failure"} 2Connection errors keyed by transport and type.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
transport, error_typeEmission path:
telemetry.IncConnErrorExample:
newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1Show WebSocket
Show WebSocket
Dial latency for Pangolin WebSocket.
Show Details
Show Details
Unit: seconds
Labels:
Emission path:
Example:
Labels:
result, transportEmission path:
ObserveWSConnectLatencyExample:
newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1WebSocket disconnects keyed by reason.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
reason, tunnel_idEmission path:
IncWSDisconnectExample:
newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2Ping/Pong failures observed by keepalive.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
reason (e.g., ping_write, pong_timeout)Emission path:
telemetry.IncWSKeepaliveFailure(ctx, "ping_write")Example:
newt_websocket_keepalive_failures_total{reason="ping_write"} 1Duration of established WS sessions keyed by result.
Show Details
Show Details
Unit: seconds
Labels:
Emission path:
Example:
Labels:
result (success|error)Emission path:
telemetry.ObserveWSSessionDuration(ctx, time.Since(start).Seconds(), "error")Example:
newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3Current WS connection state (0/1).
Show Details
Show Details
Unit: 1
Labels: —
Emission path:
Example:
Labels: —
Emission path:
telemetry.SetWSConnectionState(true|false)Example:
newt_websocket_connected 1WebSocket reconnect attempts keyed by reason.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
reasonEmission path:
telemetry.IncWSReconnect(ctx, "ping_write")Example:
newt_websocket_reconnects_total{reason="ping_write"} 1In/out WS messages keyed by direction & type.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
direction (in|out), msg_type (ping|pong|text|…)Emission path:
IncWSMessageExample:
newt_websocket_messages_total{direction="out",msg_type="ping"} 4Show Proxy
Show Proxy
Active TCP/UDP proxy connections per tunnel/protocol.
Show Details
Show Details
Unit: 1
Labels:
Emission path: Proxy callback
Example:
Labels:
protocol, tunnel_idEmission path: Proxy callback
Example:
newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3Proxy buffer pool size.
Show Details
Show Details
Unit: bytes
Labels:
Emission path: Proxy callback
Example:
Labels:
protocol, tunnel_idEmission path: Proxy callback
Example:
newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240Unflushed async byte backlog.
Show Details
Show Details
Unit: bytes
Labels:
Emission path: Proxy callback
Example:
Labels:
protocol, tunnel_idEmission path: Proxy callback
Example:
newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096Proxy write drops keyed by protocol/tunnel.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
protocol, tunnel_idEmission path:
IncProxyDropsExample:
newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2Proxy accept events keyed by result/reason.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
tunnel_id, protocol, result, reasonEmission path:
telemetry.IncProxyAccept(ctx, tunnelID, "tcp", "failure", "timeout")Example:
newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1Lifecycle events (opened/closed) per connection.
Show Details
Show Details
Unit: 1
Labels:
Emission path:
Example:
Labels:
tunnel_id, protocol, event (opened|closed)Emission path:
telemetry.IncProxyConnectionEvent(ctx, tunnelID, "tcp", telemetry.ProxyConnectionOpened)Example:
newt_proxy_connections_total{protocol="tcp",event="opened"} 1Duration of completed proxy connections.
Show Details
Show Details
Unit: seconds
Labels:
Emission path:
Example:
Labels:
tunnel_id, protocol, resultEmission path:
telemetry.ObserveProxyConnectionDuration(ctx, tunnelID, "tcp", "success", seconds)Example:
newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3newt
Prometheus-style series for the same Newt metrics. Names, labels, and examples mirror the OTel tab.
Show Site & Build
Show Site & Build
Counts Pangolin registration attempts keyed by result.
Show Details
Show Details
Labels:
Example:
result, site_id • Unit: 1 • Path: telemetry.IncSiteRegistrationExample:
newt_site_registrations_total{result="success",site_id="abc"} 10/1 heartbeat for the active site.
Show Details
Show Details
Labels:
Example:
site_id • Unit: 1 • Path: state.TelemetryViewExample:
newt_site_online{site_id="self"} 1Seconds since last Pangolin heartbeat.
Show Details
Show Details
Labels:
Example:
site_id • Unit: seconds • Path: TouchHeartbeatExample:
newt_site_last_heartbeat_seconds{site_id="self"} 3.2Constant 1 with build metadata labels.
Show Details
Show Details
Labels:
Example:
version, commit • Unit: 1 • Path: Build info registrationExample:
newt_build_info{version="1.2.3",commit="abc123"} 1Process boot indicator (increments once).
Show Details
Show Details
Labels: — • Unit: 1 • Path:
Example:
RegisterBuildInfoExample:
newt_restart_count_total 1Certificate rotation events keyed by result.
Show Details
Show Details
Labels:
Example:
result • Unit: 1 • Path: IncCertRotationExample:
newt_cert_rotation_total{result="success"} 1Config reload attempts keyed by result.
Show Details
Show Details
Labels:
Example:
result • Unit: 1 • Path: telemetry.IncConfigReloadExample:
newt_config_reloads_total{result="success"} 1Duration per config-apply phase & result.
Show Details
Show Details
Labels:
Example:
phase, result • Unit: seconds • Path: telemetry.ObserveConfigApplyExample:
newt_config_apply_seconds_bucket{phase="peer",result="success",le="0.1"} 3Show Tunnel
Show Tunnel
Active sessions per tunnel (or collapsed).
Show Details
Show Details
Labels:
Example:
site_id, tunnel_id • Unit: 1 • Path: RegisterStateViewExample:
newt_tunnel_sessions{site_id="self",tunnel_id="wgpub"} 2Traffic per tunnel/direction/protocol.
Show Details
Show Details
Labels:
Example:
tunnel_id, direction, protocol • Unit: bytes • Path: Proxy managerExample:
newt_tunnel_bytes_total{direction="egress",protocol="tcp",tunnel_id="wgpub"} 8192RTT samples per tunnel/transport.
Show Details
Show Details
Labels:
Example:
tunnel_id, transport • Unit: seconds • Path: Health checksExample:
newt_tunnel_latency_seconds_bucket{transport="wireguard",le="0.05",tunnel_id="wgpub"} 4Reconnect attempts by initiator & reason.
Show Details
Show Details
Labels:
Example:
tunnel_id, initiator, reason • Unit: 1 • Path: telemetry.IncReconnectExample:
newt_tunnel_reconnects_total{initiator="client",reason="timeout",tunnel_id="wgpub"} 3Show Connection & Auth
Show Connection & Auth
Auth/WebSocket attempts by transport & result.
Show Details
Show Details
Labels:
Example:
transport, result • Unit: 1 • Path: telemetry.IncConnAttemptExample:
newt_connection_attempts_total{transport="websocket",result="failure"} 2Connection errors by transport and type.
Show Details
Show Details
Labels:
Example:
transport, error_type • Unit: 1 • Path: telemetry.IncConnErrorExample:
newt_connection_errors_total{transport="auth",error_type="auth_failed"} 1Show WebSocket
Show WebSocket
Dial latency for Pangolin WebSocket.
Show Details
Show Details
Labels:
Example:
result, transport • Unit: seconds • Path: ObserveWSConnectLatencyExample:
newt_websocket_connect_latency_seconds_bucket{result="success",transport="websocket",le="0.5"} 1WS disconnects by reason.
Show Details
Show Details
Labels:
Example:
reason, tunnel_id • Unit: 1 • Path: IncWSDisconnectExample:
newt_websocket_disconnects_total{reason="remote_close",tunnel_id="wgpub"} 2Keepalive Ping/Pong failures.
Show Details
Show Details
Labels:
Example:
reason • Unit: 1 • Path: telemetry.IncWSKeepaliveFailure(ctx, "ping_write")Example:
newt_websocket_keepalive_failures_total{reason="ping_write"} 1Duration of established WebSocket sessions by result.
Show Details
Show Details
Labels:
Example:
result • Unit: seconds • Path: telemetry.ObserveWSSessionDuration(...)Example:
newt_websocket_session_duration_seconds_bucket{result="error",le="60"} 3Current WS connection status (0/1).
Show Details
Show Details
Labels: — • Unit: 1 • Path:
Example:
telemetry.SetWSConnectionState(true|false)Example:
newt_websocket_connected 1Reconnect attempts by reason.
Show Details
Show Details
Labels:
Example:
reason • Unit: 1 • Path: telemetry.IncWSReconnect(ctx, "ping_write")Example:
newt_websocket_reconnects_total{reason="ping_write"} 1In/out WS messages by direction & type.
Show Details
Show Details
Labels:
Example:
direction, msg_type • Unit: 1 • Path: IncWSMessageExample:
newt_websocket_messages_total{direction="out",msg_type="ping"} 4Show Proxy
Show Proxy
Active TCP/UDP proxy connections per tunnel/protocol.
Show Details
Show Details
Labels:
Example:
protocol, tunnel_id • Unit: 1 • Path: Proxy callbackExample:
newt_proxy_active_connections{protocol="tcp",tunnel_id="wgpub"} 3Proxy buffer pool size.
Show Details
Show Details
Labels:
Example:
protocol, tunnel_id • Unit: bytes • Path: Proxy callbackExample:
newt_proxy_buffer_bytes{protocol="tcp",tunnel_id="wgpub"} 10240Unflushed async byte backlog.
Show Details
Show Details
Labels:
Example:
protocol, tunnel_id • Unit: bytes • Path: Proxy callbackExample:
newt_proxy_async_backlog_bytes{protocol="udp",tunnel_id="wgpub"} 4096Proxy write drops per protocol/tunnel.
Show Details
Show Details
Labels:
Example:
protocol, tunnel_id • Unit: 1 • Path: IncProxyDropsExample:
newt_proxy_drops_total{protocol="udp",tunnel_id="wgpub"} 2Proxy accept events by result/reason.
Show Details
Show Details
Labels:
Example:
tunnel_id, protocol, result, reason • Unit: 1 • Path: telemetry.IncProxyAccept(...)Example:
newt_proxy_accept_total{protocol="tcp",result="failure",reason="timeout"} 1Connection lifecycle events (opened/closed).
Show Details
Show Details
Labels:
Example:
tunnel_id, protocol, event • Unit: 1 • Path: telemetry.IncProxyConnectionEvent(...)Example:
newt_proxy_connections_total{protocol="tcp",event="opened"} 1Duration of completed proxy connections.
Show Details
Show Details
Labels:
Example:
tunnel_id, protocol, result • Unit: seconds • Path: telemetry.ObserveProxyConnectionDuration(...)Example:
newt_proxy_connection_duration_seconds_bucket{protocol="tcp",result="success",le="1"} 3References
Have improvements or a missing metric? Open an issue or PR referencing this page.