Hi,
I’m following the documentation to stand up Dremio in an on-premise k8s kluster, using the helm chart dremio_v2. Since Dremio version 21 doesn’t support “local” as an alternative for distributed storage, I’m going with S3 since we have that available as an on-premise offering through StorageGrid. The settings are the same as when using Minio.
I’ve set up a bucket per the documentation and named it aptly ‘dremio’.
However, I’m getting the following error UnknownHostException: dremio.<redacted>.com.
To me it looks like Dremio is going for a domain-style access pattern which neither Minio nor StorageGrid supports. I was under the impression that S3-like interfaces was supported for distributed storage, since there is a special section for it in the documentation.
I’ve added all core-site.xml properties in the key distStorage.aws.extraProperties
Has anyone had any luck with using either Minio or StorageGrid or any other S3-like interfaces as a distributed storage? Or is it just the K8s version of Dremio that doesn’t fully support this?
Kind regards,
Jesper
@ramworks both StorageGrid and MinIO should work fine with Dremio. Last I tested was with v20.0. Common mistakes are putting http(s):// prefix for the endpoint or missing out to put the port.
Here’s a sample values.yaml that work for me (only relevant sections):
#... Line #287
distStorage:
type: "aws"
#... Line #344
aws:
bucketName: "dremio"
path: "/"
authentication: "accessKeySecret"
credentials:
accessKey: "ACCESS_KEY"
secret: "SECRET_KEY"
extraProperties: |
<property>
<name>fs.s3a.endpoint</name>
<value>175.1.2.3:9000</value>
</property>
<property>
<name>fs.s3a.path.style.access</name>
<value>true</value>
</property>
<property>
<name>dremio.s3.compat</name>
<value>true</value>
</property>
<property>
<name>fs.s3a.connection.ssl.enabled</name>
<value>false</value>
</property>
2 Likes
@lenoyjacob Our vaules.yaml files look identical in that regards. Only difference is that we have https for our StorageGrid endpoint. Here’s mine, for reference:
distStorage:
type: "aws"
aws:
bucketName: "dremio"
path: "/"
authentication: "accessKeySecret"
credentials:
accessKey: "ACCESS_KEY"
secret: "SECRET_KEY"
extraProperties: |
<property>
<name>fs.s3a.endpoint</name>
<value>storagegrid.endpoint.company.com</value>
</property>
<property>
<name>fs.s3a.path.style.access</name>
<value>true</value>
</property>
<property>
<name>dremio.s3.compat</name>
<value>true</value>
</property>
<property>
<name>fs.s3a.connection.ssl.enabled</name>
<value>true</value>
</property>
The annoying thing is, I got it to work (yay!) when having the following image:
image: dremio/dremio-oss
imageTag: 20.1.0
But not any later tags/versions. Later versions always result in the error stated in the OP. Probably should file this as a bug somewhere. Anyhow, thanks for your help in confirming that at least my settings were correct!
Cheers!
@ramworks Are you blocked on your upgrade to v21.x due to this?
@balaji.ramaswamy Yes, unfortunately. Seems that neither versions of 21.x respects the additional properties fully.
I saw that version 22.0 popped up on hub.docker.com a couple of days ago. Any info if distributed storage now respects path style access to enable Minio or StorageGrid?
@ramworks Is your question if Dremio supports Minio? Can you please elaborate “path style access”?
Ok, @balaji.ramaswamy . Feels like we lost context here. I created this thread about a month ago since I wasn’t able to specify a S3-like interface as a distributed storage in a kubernetes installation.
It turned out that neither versions of 21 respected the additional properties outlined in Dremios documentation in regards to setting up an S3-like interface as a Distributed Storage. Only version 20 did.
After my findings you asked me if this effectively blocked me from upgrading to version 21, to which I responded a heartfelt Yes.
So now when version 22 is out I am asking if this issue might have been resolved.
Kind regards,
Jesper
@ramworks Apologies, I should have gone through the post again. I have again pinged the right person for his availability, once I do, will update here
1 Like
Hi again,
Unfortunately, the issue with distributed storage settings not respecting fs.s3s.endpoint and fs.s3a.path.style.access outlined in the documentation is still there in version 22.0.0.
I am getting the following error when querying a dataset in version 22.0.0. I’ve set dremio as the bucket for distributed storage and made sure that it is created.
Edit: It’s actually when querying a physical dataset that is comprised of several files with identical schemas residing in the same catalog, i.e. the catalog is formatted as a dataset:
com.dremio.common.exceptions.UserException: UnknownHostException: dremio.<redacted hostname>.com
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:890)
at com.dremio.service.jobs.JobResultsStore.loadJobData(JobResultsStore.java:152)
at com.dremio.service.jobs.JobResultsStore$LateJobLoader.load(JobResultsStore.java:331)
at com.dremio.service.jobs.JobDataImpl.range(JobDataImpl.java:50)
at com.dremio.service.jobs.LocalJobsService.getJobData(LocalJobsService.java:1136)
at com.dremio.service.jobs.JobsFlightProducer.getStream(JobsFlightProducer.java:78)
at com.dremio.dac.service.flight.CoordinatorFlightProducer.getStream(CoordinatorFlightProducer.java:63)
at org.apache.arrow.flight.FlightService.doGetCustom(FlightService.java:111)
at org.apache.arrow.flight.FlightBindingService$DoGetMethod.invoke(FlightBindingService.java:144)
at org.apache.arrow.flight.FlightBindingService$DoGetMethod.invoke(FlightBindingService.java:134)
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
at io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)
at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
at io.opentracing.contrib.grpc.TracingServerInterceptor$2.onHalfClose(TracingServerInterceptor.java:231)
at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
at io.grpc.util.TransmitStatusRuntimeExceptionInterceptor$1.onHalfClose(TransmitStatusRuntimeExceptionInterceptor.java:74)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:340)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:866)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Regards,
Jesper
@ramworks Did you find a solution to this problem?
I’m installing on K8s Dremio 24.1 with same settings as you. When fs.s3a.connection.ssl.enabled is set to ‘true’ the installation fails. Set to ‘false’ it succeeds
@humanSupafly Well in the end I found a solution to my business problem 
I was tasked to facilitate queries on a dataset comprised of > 50.000 parquet files. Roughly 4 billion records. To get any kind of reasonable speed out of Dremio, this data first had to have it’s format specified and then in a timely manner converted into a reflection.
I stared at the problem for a few days after running into the issues outlined in this thread and realized that playing Lake House was just a very weird way of for unknown reasons avoiding an MPP database. So I installed ClickHouse on an identical set of hardware and ingested all the data into a table. Both query speeds and storage requirements were noticeably improved.
Since the team behind ClickHouse also ships an official python library, a Kafka Connect client, a dbt adapter, ODBC and JDBC-drivers and comes with a very easy to use REST API all integration-bits that we hadn’t really figured out with Dremio yet also kinda resolved themselves.
So for us the choice was obvious. We didn’t need a Lake House. We needed the best OSS columnar database on the market.
Kind regards,
Jesper Bagge
Oke, thanks for the answer @ramworks
I managed to configure ssl enabled distributed storage with our onprem s3. in our case the problem was the certificate and policy settings on the s3 endpoints.