Skip to main content
S3 destinations upload batches of your organization’s audit logs as objects in a bucket you control. Use them for long-term archival, data lakes (Athena, Glue, BigQuery), or S3-compatible stores such as MinIO and Cloudflare R2.
Event streaming is only available in Pangolin Cloud or self-hosted Enterprise Edition.

​
Overview

An S3 destination writes one object per batch via PutObject. Each object contains up to 250 events of a single log type. There is no custom body template or field mapping; Pangolin serializes every event in a fixed shape and chooses the object key automatically. Configure:
  1. Settings: Name, credentials, region, bucket, optional prefix and custom endpoint.
  2. Format: File format (JSON array, NDJSON, or CSV) and optional gzip compression.
  3. Logs: Which log types are forwarded.

​
Settings tab

Pangolin uses static access keys only. There is no IAM role, instance profile, or OIDC picker in the UI.
S3 destination settings with credentials, region, and bucket
Uploads time out after 60 seconds per object.

​
Format tab

Gzip compression (optional): When enabled, the object body is gzip-compressed before upload, Content-Encoding: gzip is set, and the object key gets a .gz suffix (for example ….json.gz). Decompress before parsing unless your tool handles gzip automatically. File format:
Format tab with file format and gzip options

​
Logs tab

Choose which log categories are uploaded. Each enabled type is written to its own key prefix (request/, action/, etc.). Only log types enabled for your organization can be streamed.
Logs tab for selecting streamed log types

​
Object key layout

Every upload gets a unique key:
Without prefix:
With prefix pangolin/audit and gzip:
Enabling multiple log types on one destination produces separate object streams under different logType/ segments. A single object never mixes log types.

​
Event record shape

Each event in JSON and NDJSON objects uses this fixed structure:
Some columns are stored as JSON strings in the database (headers, query, and metadata on request logs, for example). In data, they appear as string values, not nested JSON objects. Parse them in your pipeline if you need structured fields.

​
File formats

​
JSON array (default)

  • One S3 object per batch; body is [{…}, {…}, …].
  • Up to 250 events per object.
  • Content-Type: application/json.

​
NDJSON

  • One S3 object per batch; body is one JSON record per line with no outer array.
  • Good for Athena, BigQuery load jobs, Spark, and similar line-oriented pipelines.
  • Content-Type: application/x-ndjson.

​
CSV format

  • Header row: event, timestamp, then all field names found in data across that batch (union of keys, in insertion order).
  • Each data row flattens event, timestamp, and spreads data fields into columns. There is no nested data column.
  • Missing fields in a given row leave an empty cell.
  • Object or array values in data are written as JSON.stringify strings inside the cell.
  • Content-Type: text/csv; charset=utf-8.
The column set can grow as new fields appear in later batches. Order is not guaranteed to stay identical across all objects over time.

​
Batching and throughput

  • Objects are written per batch (up to ~250 events), not one object per log line.
  • Pangolin polls for new logs on a regular interval and may write multiple objects during catch-up after a pause.
  • No backfill: New destinations start from the current log cursor. Historical logs already in Pangolin are not uploaded.
  • Extended outage: If the destination is unreachable for about 24 hours, the backlog may be discarded and streaming resumes from the present cursor (same behavior as HTTP streaming).

​
Gzip

When gzip is enabled:
  1. The serialized body is compressed before upload.
  2. The object key includes .gz (for example ….ndjson.gz).
  3. S3 stores Content-Encoding: gzip.
Consumers must decompress before parsing unless the tool auto-detects gzip (many Athena and Spark setups do when Content-Encoding is set). NDJSON plus gzip is a common choice for cost-sensitive archival.

​
S3-compatible storage

Set Custom endpoint to your vendor’s S3 API URL and provide access key credentials per that vendor’s documentation. Pangolin does not expose path-style vs virtual-hosted addressing, ACLs, SSE-KMS, storage class, or multipart tuning. Configure those in the vendor console or bucket policy.

​
IAM and bucket policy

Grant the access key permission to write under your prefix. A minimal AWS example:
Adjust bucket name and prefix to match your configuration. ListBucket is optional but useful when debugging missing objects. Block public access, encryption at rest, lifecycle rules, and object tags are configured in AWS or your vendor console, not in Pangolin.

​
Log type reference

The data object in each streamed event is the full stored log row. Field sets differ by log type. See the documentation for that log type under Logs & Analytics for the complete data shape.

​
Limitations and troubleshooting

  • No custom JSON shape: Fixed event record only. Use an HTTP destination if you need body templates or field remapping.
  • No per-event objects: Always batched (up to ~250 events per object).
  • No mixed log types in one object: Each upload contains a single log type.
  • Upload-time partitioning: Key date folders use upload time (UTC), not the event’s timestamp.
  • CSV columns: Automatic from batch contents; not user-selectable; column set may change over time.
  • Static credentials only: Rotate keys by updating the destination; credentials are stored encrypted server-side.
  • Historical logs: New destinations do not backfill.
  • Delivery errors: Check the destination’s last error in the dashboard. Common causes: AccessDenied, wrong bucket or region, bad endpoint URL, TLS issues, or expired credentials.
  • Missing objects: Confirm prefix, lifecycle rules, and that the log type is enabled on the Logs tab.
  • Athena/Glue parse errors: Verify format (JSON array vs NDJSON), gzip handling, and that the crawler/table schema matches flattened CSV columns if using CSV.