> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-postgresql-tls-support.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> Monitor ClickPipes using Prometheus metrics.

# Monitoring ClickPipes

In addition to in-console monitoring, ClickPipes exposes metrics to a [Prometheus-compatible endpoint](/products/cloud/features/monitoring/prometheus) for scraping. These metrics are published with other ClickHouse Cloud service metrics, and allow you to integrate ClickPipes monitoring with your existing observability stack (e.g., [Grafana](/products/cloud/features/monitoring/prometheus#integrating-with-grafana), [Datadog](/products/cloud/features/monitoring/prometheus#integrating-with-datadog)).

<h2 id="metric-labels">
  Metric labels
</h2>

ClickPipes metrics use the [standard service-level labels](/products/cloud/features/monitoring/prometheus#metric-labels) (`clickhouse_org`, `clickhouse_service`, `clickhouse_service_name`) plus the following ClickPipes-specific labels:

| Label              | Description                                                             |
| ------------------ | ----------------------------------------------------------------------- |
| `clickpipe_id`     | The unique identifier of the ClickPipe                                  |
| `clickpipe_name`   | The name of the ClickPipe                                               |
| `clickpipe_source` | The source type (e.g., `kafka`, `postgres`, `s3`)                       |
| `clickpipe_state`  | The state of the ClickPipe (e.g., `Stopped`, `Provisioning`, `Running`) |

<h3 id="sample-response">
  Sample response
</h3>

The following shows what ClickPipes metrics look like in a Prometheus scrape response for a Kafka ClickPipe. The set of exposed metrics varies by pipe type — see [Available metrics](#available-metrics) for which metrics apply to which pipe types:

```response theme={null}
# HELP ClickPipes_Info Always equal to 1. Label "clickpipe_state" contains the current state of the pipe: Stopped/Provisioning/Running/Paused/Failed
# TYPE ClickPipes_Info gauge
ClickPipes_Info{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent",clickpipe_state="Running"} 1

# HELP ClickPipes_FetchedEvents_Total Total number of records fetched from the source.
# TYPE ClickPipes_FetchedEvents_Total counter
ClickPipes_FetchedEvents_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 5535376

# HELP ClickPipes_SentEvents_Total Total number of records sent to ClickHouse
# TYPE ClickPipes_SentEvents_Total counter
ClickPipes_SentEvents_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 5534250

# HELP ClickPipes_FetchedBytes_Total Total uncompressed bytes fetched from the source.
# TYPE ClickPipes_FetchedBytes_Total counter
ClickPipes_FetchedBytes_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 873286202

# HELP ClickPipes_SentBytes_Total Total uncompressed bytes sent to ClickHouse.
# TYPE ClickPipes_SentBytes_Total counter
ClickPipes_SentBytes_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 477187967

# HELP ClickPipes_Errors_Total Total errors ingesting data.
# TYPE ClickPipes_Errors_Total counter
ClickPipes_Errors_Total{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 0

# HELP ClickPipes_Latency Time in milliseconds between when the record was produced to when it was written to ClickHouse.
# TYPE ClickPipes_Latency gauge
ClickPipes_Latency{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickhouse_service="82b83b6a-5568-4a82-aa78-fed9239db83f",clickhouse_service_name="my service",clickpipe_id="642bb967-940b-459e-9f63-a2833f62ec44",clickpipe_name="my kafka pipe",clickpipe_source="confluent"} 340
```

<h2 id="available-metrics">
  Available metrics
</h2>

Not every metric is emitted by every pipe type. In particular, CDC ClickPipes expose a different subset of metrics than streaming and object storage ClickPipes.

* **Streaming, object storage** — Kafka, Kinesis, Pub/Sub, S3, and GCS
* **CDC** — Postgres, MySQL, MongoDB and BigQuery
* **All** — every pipe type

<h3 id="metrics-data-transfer">
  Data transfer
</h3>

| Metric                                     | Type    | Available for             | Description                                                                                                                                                                                                                                                                                  |
| ------------------------------------------ | ------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ClickPipes_FetchedBytes_Total`            | Counter | Streaming, object storage | Total uncompressed bytes fetched from the source.                                                                                                                                                                                                                                            |
| `ClickPipes_FetchedBytesCompressed_Total`  | Counter | All                       | Total bytes fetched from the source, as read over the wire. If the source data is uncompressed, equivalent to `ClickPipes_FetchedBytes_Total`. For CDC ClickPipes, this excludes bytes fetched from initial load, which is emitted separately to `ClickPipes_FetchedBytesInitialLoad_Total`. |
| `ClickPipes_FetchedBytesInitialLoad_Total` | Counter | CDC                       | Total bytes fetched during the initial load phase, including resyncs.                                                                                                                                                                                                                        |
| `ClickPipes_SentBytes_Total`               | Counter | Streaming, object storage | Total uncompressed bytes sent to ClickHouse.                                                                                                                                                                                                                                                 |
| `ClickPipes_SentBytesCompressed_Total`     | Counter | Streaming, object storage | Total compressed bytes sent to ClickHouse.                                                                                                                                                                                                                                                   |

<h3 id="metrics-events">
  Events and records
</h3>

| Metric                              | Type    | Available for             | Description                                                                                                                  |
| ----------------------------------- | ------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `ClickPipes_FetchedEvents_Total`    | Counter | Streaming, object storage | Total number of records fetched from the source.                                                                             |
| `ClickPipes_SentEvents_Total`       | Counter | Streaming, object storage | Total number of records sent to ClickHouse.                                                                                  |
| `ClickPipes_FetchedEvents_PerTable` | Gauge   | CDC                       | Number of records fetched per destination table and event type. Uses additional `destination_table` and `event_type` labels. |

<h3 id="metrics-errors">
  Errors
</h3>

| Metric                    | Type    | Available for | Description                                                                                 |
| ------------------------- | ------- | ------------- | ------------------------------------------------------------------------------------------- |
| `ClickPipes_Errors_Total` | Counter | All           | Total [errors](/integrations/clickpipes/home#error-reporting) encountered during ingestion. |

<h3 id="metrics-latency">
  Latency
</h3>

| Metric                                    | Type          | Available for             | Description                                                                                                  |
| ----------------------------------------- | ------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `ClickPipes_Latency`                      | Gauge (*avg*) | Streaming, object storage | Time in milliseconds between when a record was produced at the source and when it was written to ClickHouse. |
| `ClickPipes_SourceReplicationLatency_MiB` | Gauge (*avg*) | CDC (Postgres only)       | Replication slot lag at the source in MiB. Only emitted by Postgres ClickPipes.                              |

<h3 id="metrics-resources">
  Resource utilization
</h3>

`ClickPipes_Replica_*` metrics report resource utilization for streaming and object storage ClickPipes. Each ClickPipe has **dedicated compute**, so these values reflect utilization for individual ClickPipes.

| Metric                           | Type           | Available for             | Description                                                    |
| -------------------------------- | -------------- | ------------------------- | -------------------------------------------------------------- |
| `ClickPipes_Replica_CPUUsage`    | Gauge (*avg*)  | Streaming, object storage | Average CPU usage of the ingestion replicas, in millicores.    |
| `ClickPipes_Replica_CPULimit`    | Gauge (*last*) | Streaming, object storage | Configured CPU limit of the ingestion replicas, in millicores. |
| `ClickPipes_Replica_MemoryUsage` | Gauge (*avg*)  | Streaming, object storage | Average memory usage of the ingestion replicas, in bytes.      |
| `ClickPipes_Replica_MemoryLimit` | Gauge (*last*) | Streaming, object storage | Configured memory limit of the ingestion replicas, in bytes.   |

`ClickPipes_CDC_*` metrics report resource utilization for CDC ClickPipes. All CDC ClickPipes in the same service **share compute**, so these values reflect the combined usage of every CDC ClickPipe in the service.

| Metric                               | Type           | Available for | Description                                                                                                     |
| ------------------------------------ | -------------- | ------------- | --------------------------------------------------------------------------------------------------------------- |
| `ClickPipes_CDC_CPUUsage`            | Gauge (*avg*)  | CDC           | Average CPU usage, in millicores. Shared across all CDC ClickPipes in the service.                              |
| `ClickPipes_CDC_CPULimit`            | Gauge (*last*) | CDC           | Configured CPU limit, in millicores. Shared across all CDC ClickPipes in the service.                           |
| `ClickPipes_CDC_MemoryUsage`         | Gauge (*avg*)  | CDC           | Average memory usage, in bytes. Shared across all CDC ClickPipes in the service.                                |
| `ClickPipes_CDC_MemoryLimit`         | Gauge (*last*) | CDC           | Configured memory limit, in bytes. Shared across all CDC ClickPipes in the service.                             |
| `ClickPipes_CDC_NetworkReceiveBytes` | Gauge          | CDC           | Inbound network bandwidth, in bytes (delta per scrape window). Shared across all CDC ClickPipes in the service. |

<h3 id="metrics-state">
  State and progress
</h3>

| Metric                          | Type  | Available for | Description                                                                            |
| ------------------------------- | ----- | ------------- | -------------------------------------------------------------------------------------- |
| `ClickPipes_Info`               | Gauge | All           | Always equal to `1`. Use the `clickpipe_state` label to filter or alert on pipe state. |
| `ClickPipes_LastFetchedBatchId` | Gauge | CDC           | ID of the last batch fetched from the source.                                          |
| `ClickPipes_LastSentBatchId`    | Gauge | CDC           | ID of the last batch sent to ClickHouse.                                               |

<h2 id="related">
  Related pages
</h2>

* [Prometheus endpoint](/products/cloud/features/monitoring/prometheus) — Endpoint reference, authentication, and setup for external tools (e.g., Grafana, Datadog)
* [Error reporting](/integrations/clickpipes/home#error-reporting) — Where record and system errors are stored