Enabling PromQL queries with Erlenmeyer
In this post, we will review what Erlenmeyer brings and how to handle Clever Cloud services metrics with PromQL.
Erlenmeyer protocols
On Clever Cloud, we deployed an Erlenmeyer in front of our Warp10 backend. This means all these open source protocols can be used to query our application Metrics. The host is https://PROTO-c1-warp10-clevercloud-customers.services.clever-cloud.com/PROTO with PROTO being:
- for PromQL:
prometheus, - for OpenTSDB:
opentsdb, - for InfluxQL:
influxql, - for Graphite:
graphite.
Retrieve the memory data with PromQL
READ TOKEN and an APP_ID as an environment variable:export TOKEN=MY_AWESOME_TOKEN # Replace the value by your Clever Cloud Warp10 token
export APP_ID=app_test # Replace the value by one of your own Clever Cloud application
Get raw data
mem.used_percent in PromQL, we need to send an HTTP GET request to Erlenmeyer, with the TOKEN as basic AUTH. In that case, the PromQL GET request requires four parameters: query, start, end and step. Similarly, to learn more about the PromQL language, you can read the Prometheus documentation. In our case we will create a simple query to get the raw data:api/v1/query_range?
query=mem.used_percent{app_id="$APP_ID"}&
start=1623289800&
end=1623311400&
step=120squery parameter is used for the PromQL queries: start and end are milliseconds timestamps corresponding to the time limit to get data. And the step one is the sampling parameter.120s.cURL url encoded:curl --request GET --url "https://u:${TOKEN}@prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus/api/v1/query_range?query=mem.used_percent%7Bapp_id%3D%22${APP_ID}%22%7D&start=1623289800&end=1623311400&step=120s" --header 'Content-Type: application/json'
Group by example
A group by query can be useful in the Clever Cloud case to merge some series (multiple deployments or instances). Thus we will group data based on the app_id labels in the following example.
READ_TOKEN and APP_ID. To compute a group by in our case we apply the max PromQL function grouped by the series app_id label:api/v1/query_range?
query=max(mem.used_percent{app_id="$APP_ID"}) by (app_id)&
start=1623289800&
end=1623311400&
step=120s
cURL url encoded:curl --request GET --url "https://u:${TOKEN}@prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus/api/v1/query_range?query=max(mem.used_percent%7Bapp_id%3D%22${APP_ID}%22%7D)%20by%20(app_id)&start=1623289800&end=1623311400&step=120s" --header 'Content-Type: application/json'
Grafana PromQL data source
You may use Grafana to create you own dashboards, which is an open source interactive visualisation web application (to deploy a Grafana as a Clever Cloud application, we provide a github example and you can also follow this French blog post). Of course you can configure a PromQL data source to get Clever Cloud applications metrics:
- create a Prometheus source,
- then set Prometheus source host to
https://prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus, - in addition activate the basic authentication with a user set to
metricsand a password containing yourread Warp10 token, - to conclude, click on
Save and Test, your Prometheus data source should now be working!
Prometheus UI with remote_read endpoint
Like with Grafana, you can use Prometheus and plot Clever Cloud metrics in the Web UI. And as a matter of fact, simply fill your Prometheus configuration with a remote_read source:
api/v1/query_range?
remote_read:
- url: "https://prometheus-c1-warp10-clevercloud-customers.services.clever-cloud.com/prometheus/remote_read/"
basic_auth:
username: "metrics"
password: "TOKEN"
{ __name__= "mem.used_percent", app_id="$APP_ID" }
. which are invalid prometheus characters. Nonetheless the backend response will always translate those . values in series data (name or labels) into valid _.To Sum Up
As you can see with this post, you can get metrics of Clever Cloud applications and add-ons. Presently you now have one more way to query them, as Erlenmeyer allows the use of PromQL. However the drawback is that you will have to update regularly your token, as they expire in a few days. We are working on a solution to improve the user experience. First, we will soon provide our customers with access to a Grafana with configured data source and already pre-filled dashboards. The token will be handled on our side and you will then be able to build your own graphs with PromQL or WarpScript queries.
Aurélien Hebert
Software engineer working on distributed systems and data storage. Analytics systems enthusiast.
Follow me
