Prometheus can be configured to read from and write to remote storage, in addition to its local time series database. This is intended to support long-term storage of monitoring data.
When configured, Prometheus storage queries (e.g. via the HTTP API) are sent to both local and remote storage, and results are merged.
Note that to maintain reliability in the face of remote storage issues, alerting and recording rule evaluation use only the local TSDB.
You configure the remote storage read path in the remote_read section of the Prometheus configuration file.
At its simplest, you will just specify the read endpoint URL for your remote storage, plus an authentication method. You can use either HTTP basic or bearer token authentication.
You might want to use the read_recent flag: when set to true, all queries will be answered from remote as well as local storage. When false (the default), any queries that can be answered completely from local storage will not be sent to the remote endpoint.
You can specify a set of required_matchers (label, value pairs) to restrict remote reads to some subset of queries. This is useful if e.g. you write only a subset of your metrics to remote storage (see below).
For more complex configurations, there are also options for request timeouts, TLS configuration, and proxy setup.
You can read from multiple remote endpoints by having one remote_read section for each.
When configured, Prometheus forwards its scraped samples to one or more remote stores.
Remote writes work by "tailing" time series samples written to local storage, and queuing them up for write to remote storage.
The queue is actually a dynamically-managed set of "shards": all of the samples for any particular time series (i.e. unique metric) will end up on the same shard.
The queue automatically scales up or down the number of shards writing to remote storage to keep up with the rate of incoming data.
This allows Prometheus to manage remote storage while using only the resources necessary to do so, and with minimal configuration.
You configure the remote storage write path in the remote_write section of the Prometheus configuration file.
Like for remote_read, the simplest configuration is just a remote storage write URL, plus an authentication method. You can use either HTTP basic or bearer token authentication.
You can use write_relabel_configs to relabel or restrict the metrics you write to remote storage. For example, a common use is to drop some subset of metrics:
The queue_config section gives you some control over the dynamic queue described above. Usually, you won't need to make changes here and can rely on Prometheus' defaults.
Like for remote_read, you can also configure options for request timeouts, TLS configuration, and proxy setup.
You can write to multiple remote endpoints by having one remote_write section for each.
You may see some messages from the remote storage subsystem in your logs:
The remote storage subsystem exports lots of metrics, prefixed with prometheus_remote_storage_ or prometheus_wal_watcher_.Here's a selection you might find interesting: