# Metrics

> Every metric series emitted by the DuckDB gateway, with its labels, for the Prometheus endpoint and the AWS, Azure and GCP sinks.

The manager registers these series through Micrometer. They are emitted to whichever sink is active (`QOD_METRICS_SINK`); under the default Prometheus sink they appear at `GET :20900/metrics`. For how to scrape, push to a cloud monitor, or import the Grafana dashboard, see [Observability](/qod/operating/observability).

## Application metrics

| Metric | Type | Labels | Meaning |
|---|---|---|---|
| `statements_total` | counter | `tenant`, `pool`, `status` | Statements executed, partitioned by outcome status. Drives QPS and error-rate panels. |
| `statement_duration_seconds` | histogram | `tenant`, `pool` | Statement execution latency; the source of the p50/p95/p99 percentiles. |
| `flightsql_sessions_active` | gauge | (none) | Currently open FlightSQL sessions. |
| `pool_nodes` | gauge | `tenant`, `pool`, `role` | Node count per pool, broken down by node role. |
| `node_healthy` | gauge | `tenant`, `pool`, `node_id`, `role` | 1 when the node is healthy, 0 otherwise. |
| `node_draining` | gauge | `tenant`, `pool`, `node_id`, `role` | 1 when the node is draining in-flight work before shutdown. |
| `node_in_flight` | gauge | `tenant`, `pool`, `node_id`, `role` | Statements currently executing on the node. |
| `node_ewma_latency_seconds` | gauge | `tenant`, `pool`, `node_id`, `role` | EWMA of completed-statement latency, the signal the router uses to pick the least-loaded node. |

## Routing metrics

Emitted per routed statement by the [cache-aware placement](/qod/concepts/routing) layer. The locality series (`routing_tables_total`, `routing_placements_total`) run on any routing policy and stay populated even with `QOD_ROUTING_CACHE_AWARE=false`; the decision and load series describe placement outcomes.

| Metric | Type | Labels | Meaning |
|---|---|---|---|
| `routing_tables_total` | counter | `tenant`, `pool`, `result` | Table references seen, split into `new` (first time this table is routed) and `repeat` (seen before). A low repeat rate means locality has little to exploit. |
| `routing_placements_total` | counter | `tenant`, `pool`, `result` | Repeat-table routings by whether the table `stay`ed on its last node or `switch`ed to another. The switch rate is the locality-loss (scatter) signal: high means placement is being destroyed. |
| `routing_decisions_total` | counter | `tenant`, `pool`, `outcome` | Placement decisions by outcome: `claim`, `sticky-fresh`, `sticky-stale`, `overflow-new-home`, `overflow-evict-home`, `pinned-sticky`, `pinned-move`, `no-refs-fallback`, `not-eligible`, `flag-off`. Frequent `overflow-evict-home` on a pool means three homes are not enough for its hottest tables. `pinned-sticky` / `pinned-move` mark statements a transaction pin or soft preferredNode placed rather than the scorer, so they are excluded from the overflow signal. |
| `routing_load_ratio` | summary | `tenant`, `pool` | Chosen-node in-flight count over the pool average. Should stay under `routing.loadCapFactor`; a sustained excess means the load cap is not binding. |

## Maintenance metrics

Emitted per finished [managed-maintenance](/qod/operating/maintenance) run. These series carry the `qod_` prefix; the legacy series above predate the prefix convention.

| Metric | Type | Labels | Meaning |
|---|---|---|---|
| `qod_maint_runs_total` | counter | `tenant`, `tenant_db`, `result` | Maintenance runs by outcome (`succeeded`, `failed`, `partial`). A rising `partial` count usually means the pinned-file guard is firing. |
| `qod_maint_bytes_reclaimed_total` | counter | `tenant`, `tenant_db` | Catalog bytes released by runs (physical deletion lags by the cleanup grace window). |
| `qod_maint_files_compacted_total` | counter | `tenant`, `tenant_db` | Files touched by the merge and rewrite steps. |
| `qod_maint_snapshots_expired_total` | counter | `tenant`, `tenant_db` | Snapshots expired (pinned snapshots are skipped, never counted here). |
| `qod_maint_duration_seconds` | timer | `tenant`, `tenant_db` | End-to-end run duration, including maintenance-node spawn time. |

## DuckDB engine metrics

Scraped from each node's DuckDB engine (`duckdb_memory()`, `duckdb_temporary_files()`) by the background health probe, one extra round-trip per node per `QOD_HEALTH_CHECK_INTERVAL_SEC` tick. A node that has never been scraped successfully publishes no row (rather than a misleading zero); a failed scrape keeps the previous sample until the next tick.

| Metric | Type | Labels | Meaning |
|---|---|---|---|
| `node_duckdb_memory_used_bytes` | gauge | `tenant`, `pool`, `node_id`, `role` | Buffer-manager memory in use, summed across all consumers (base tables, hash tables, parquet readers, ...). Compare against the node's `memory_limit` to spot memory pressure before latency degrades. |
| `node_duckdb_temp_storage_bytes` | gauge | `tenant`, `pool`, `node_id`, `role` | Bytes the buffer manager has moved to temporary storage. |
| `node_duckdb_spill_files` | gauge | `tenant`, `pool`, `node_id`, `role` | Live spill-to-disk files. Non-zero means queries are exceeding the memory budget and spilling. |
| `node_duckdb_spill_bytes` | gauge | `tenant`, `pool`, `node_id`, `role` | Total size of live spill files. |

## JVM and process metrics

Registered by the Micrometer JVM and process binders:

| Metric | Type | Meaning |
|---|---|---|
| `jvm_memory_used_bytes` | gauge | Heap and non-heap memory in use. |
| `jvm_gc_pause_seconds_sum` | counter | Cumulative GC pause time. |
| `jvm_threads_live_threads` | gauge | Live thread count. |
| `process_uptime_seconds` | gauge | Process uptime since manager start. |

## Common labels

Every series can carry static deployment labels when these are set, useful for separating environments in a shared Grafana:

| Variable | Label | Example |
|---|---|---|
| `QOD_METRICS_DEPLOYMENT` | `deployment` | `prod-eu` |
| `QOD_METRICS_REGION` | `region` | `eu-west-1` |