Kubernetes setup: DremioOSS cannot connect to minio as datasource

1

Hi I’ve succesfully setup dremio oss on kubernetes with minio as distributed storage. The dremio bucket is populated so dremio can connect to minio.

I have an extra bucket on that minio, in which I have ingested an iceberg table with embedded metadata (no nessie).

Now I’m trying to connect S3 datasource pointing to that bucket. I’ve followed the documentation and added compatibility mode and added the properties fs.s3a.path.style.access and fs.s3a.endpoint taking care not to add the http or s3 (e.g. datalake-minio:9000)

I’m experiencing all sorts of issues. When saving I get something like “This may take a few minutes , then it fails”. Logs on the dremio pod shows what seem to be lock issues and ssl issues.

com.dremio.common.exceptions.UserException: Unable to acquire source change lock for source [asd] within timeout.
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:984)
at com.dremio.exec.catalog.CatalogServiceImpl.getDistributedLock(CatalogServiceImpl.java:671)
at com.dremio.exec.catalog.CatalogServiceImpl.updateSource(CatalogServiceImpl.java:557)
at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.updateSource(CatalogServiceImpl.java:974)
at com.dremio.exec.catalog.CatalogImpl.updateSource(CatalogImpl.java:2526)
at com.dremio.exec.catalog.SourceAccessChecker.updateSource(SourceAccessChecker.java:480)

Also
Unable to execute HTTP request: Unsupported or unrecognized SSL message

What is going on? How can i debug this? Why is dremio able to connect minio as distributed storage, but not as s3 datasource.

2

It looks like its trying to connect via https to my minio endpoint, but i don’t want this. Adding fs.s3a.connection.ssl.enabled=false to connection properties didn’t work

3

Hi Max,
I dont know if this helps but here’s my configuration which works well, however, it is using https.

What happened when you applied the s3a-connectionssl.enabled=false?

4

fs.s3a.connection.ssl.enabled didn’t seem to do much. Still trying to do a SSL handshake to http endpoint. It’s interesting because dremio doesn’t allow you to put http(s) in the fs.s3a.endpoint which is allowed in regular s3a configurations (e.g. in spark you set it as http(s)://:, which tells me dremio is doing some inferences.

Anyway I managed to get it to work by creating an ingress with tls to my minio kubernetes service and use that as the host in the endpoint in connection properties.
It was still failing with the typical PKIX error because SSL self-signed cerificates weren’t known to dremio trust store , so I just disabled SSL checks for now. I’ll import a certificate later. Not sure if both -D options are needed , but adding this to my bitnami chart disabled SSL cer checking:

  masterCoordinator:
    extraJavaOpts: "-Dcom.amazonaws.sdk.disableCertChecking=true -Dfs.s3a.connection.ssl.trustAllCertificates=true"
  coordinator:
    extraJavaOpts: "-Dcom.amazonaws.sdk.disableCertChecking=true -Dfs.s3a.connection.ssl.trustAllCertificates=true"
  executor:
    common:
      extraJavaOpts: "-Dcom.amazonaws.sdk.disableCertChecking=true -Dfs.s3a.connection.ssl.trustAllCertificates=true"