In this tutorial, we will address the common monitoring use case of filtering metrics within observIQ’s distribution of the OpenTelemetry (OTEL) collector. Whether the metrics are deemed unnecessary, or they are filtered for security concerns, the process is fairly straightforward.
For our sample environment, we will use MySQL on Red Hat Enterprise Linux 8. The destination exporter will be to Google Cloud Operations, but the process is exporter agnostic. We are using this exporter to provide the visual charts showing the metric before and after filtering.
- Suitable operating system
- observIQ Distro for OTEL Collector installed
- MySQL installed
- MySQL Least Privilege User (LPU) setup
- OTEL configured to collect metrics from MySQL
Once configured using the LPU I created, MySQL metrics should be flowing. For our purposes, we will focus on the specific metric
mysql.buffer_pool.limit. Currently, our config.yaml MySQL section looks like this:
mysql: endpoint: localhost:3306 username: otel password: otelPassword collection_interval: 60s
After waiting for at least five minutes to get a good amount of data, metrics will look something like this in Google’s Metrics Explorer:
Now that metrics are flowing, we can filter them. First, let us discuss the reasons to filter this specific metric. The answer is simple: It isn’t really all that useful or important. It will, barring a configuration change by the DBA, be a flat line. Even after a configuration change, it would simply step that flat line up or down.
To do the filtering, we first need to look at the metadata file for the MySQL receiver. In this file, we find a listing of the attributes and metrics associated with this receiver. If we go to the metrics section of the file, and find our pool limit metric, we learn it looks like this:
mysql.buffer_pool.limit: enabled: true description: The configured size of the InnoDB buffer pool. unit: By sum: value_type: int input_type: string monotonic: false aggregation: cumulative
This lets us know that it is enabled by default, gives a description, and some other important data about the metric. As these are the defaults, we can interpret from it that if we set the
enabled parameter to false, then it should disable — aka filter — this metric. It will not be collected, and since it isn’t collected, it also will not be sent to the exporter.
To achieve this in our configuration file, we make the following changes:
mysql: endpoint: localhost:3306 username: otel password: otelPassword collection_interval: 60s metrics: mysql.buffer_pool.limit: enabled: false
This replicates the structure from the metadata file, but with everything else trimmed other than the bare minimum number of lines needed to achieve our goal.
Once this has been changed, and the collector restarted, I again wait at least five minutes and check Google’s Metrics Explorer to see what has changed:
As shown in the screenshot, data was last sent to Google at 10:48, and it is now 11:13.
While the information needed is located in a few different places, filtering is very easy to do. Don’t forget that the metadata we looked at also provides other information that can be useful in understanding your data.