Prometheus exporter: struggle to compile

Dremio University
1

Hello,

I’m trying to compile the Prometheus exporter (GitHub - dremio-hub/dremio-prometheus-exporter)

I get the following error:

[WARNING] The POM for com.dremio.tools:errorprone-dremio:jar:24.0.0-202302100528110223-3a169b7c is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.220 s
[INFO] Finished at: 2023-03-15T21:42:57Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:compile (default-compile) on project dremio-telemetry-prometheus: Execution default-compile of goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:compile failed: Plugin org.apache.maven.plugins:maven-compiler-plugin:3.10.1 or one of its dependencies could not be resolved: Failure to find com.dremio.tools:errorprone-dremio:jar:24.0.0-202302100528110223-3a169b7c in https://maven.dremio.com/free/ was cached in the local repository, resolution will not be reattempted until the update interval of dremio-free has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

I think I’m failing to update the pom.xml correctly (I have 0 experience in Java)
what version should I put ? and where ?

Is there another Prometheus exporter ?

Thank you

2

@jjournet Sorry for the delay, any chance you tried to run it again with the -e switch?

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

3

@jjournet

We dont need to do this any more. We automatically expose prometheus metrics via http endpoint.

Can you please try and let us know

Thanks
Bali

4

I’ve also been trying to build this plugin as I was not aware that Dremio now exposed the metrics natively. Can you share the scrape endpoint? Also, is there any updated documentation that explains how to configure the build-in metrics?

5

@rs_trey

Would the below 2 help?

6

Whoa, thank you! The “Monitoring System Telemetry” page I kept reading still referenced the third-party exporter. I modified my StatefulSets and ConfigMap and can now see the metrics. Now to update my copy of the helm chart (since PRs aren’t accepted). Thanks again!

7

Thanks for the update @rs_trey Glad you are unblocked

8

I am trying to make prometheus work in my single node dremio. I added services.web-admin.port: 32020 line to dremio.conf file and the run upgrade helm chart.
But I get
curl: (7) Failed to connect to 172.28.1.6 port 32020 after 0 ms: Connection refused Error

rs_trey says he modified ConfigMap and StatefulSets but I am not sure what I need to change.
Besides, in dremio logs I can see
2023-11-21 08:24:54,598 [config-refresh1] WARN com.dremio.telemetry.api.Telemetry - Failure reading telemetry configuration. Leaving telemetry as is.
java.lang.IllegalArgumentException: resource dremio-telemetry.yaml not found.

  •    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:220)*

Error and also:
2023-11-21 08:25:04,110 [main] INFO c.dremio.dac.server.LivenessService - Started liveness service on port 32020
2023-11-21 08:25:04,130 [main] INFO c.d.s.j.s.LocalJobTelemetryServer - LocalJobTelemetryServer is up

Does this mean metrics is working but the port is not accessible? In that case, how can I expose port 32020 to be accessible from grafana?

Thanks

9

@txalaparta Does telnet 172.28.1.6 32020 work?

10

It doesn´t not. But neither does 172.28.1.6:30010 for example and it works.
The error is:
telnet: could not resolve 172.28.1.6:32020/telnet: Name or service not known

11

Another thing I can see is that telemetry.yml never gets copied to /opt/dremio/conf folder. I even created a docker image to copy it with no success.

12

@txalaparta I have asked someone who have worked on monitoring tools to help out, stay tuned, ping, in case you do not hear back in a few days

13

Unable to upload a Doc here for monitoring on k8s. Please let me know your email so that i can forward to you.

14

Here´s my email:
oesparza@bilbomatica.es

Looking forward to receiving it.
Thanks