S3-like storage for Distributed Storage

Hi,

I’m following the documentation to stand up Dremio in an on-premise k8s kluster, using the helm chart dremio_v2. Since Dremio version 21 doesn’t support “local” as an alternative for distributed storage, I’m going with S3 since we have that available as an on-premise offering through StorageGrid. The settings are the same as when using Minio.

I’ve set up a bucket per the documentation and named it aptly ‘dremio’.

However, I’m getting the following error UnknownHostException: dremio.<redacted>.com.

To me it looks like Dremio is going for a domain-style access pattern which neither Minio nor StorageGrid supports. I was under the impression that S3-like interfaces was supported for distributed storage, since there is a special section for it in the documentation.

I’ve added all core-site.xml properties in the key distStorage.aws.extraProperties

Has anyone had any luck with using either Minio or StorageGrid or any other S3-like interfaces as a distributed storage? Or is it just the K8s version of Dremio that doesn’t fully support this?

Kind regards,
Jesper

@ramworks both StorageGrid and MinIO should work fine with Dremio. Last I tested was with v20.0. Common mistakes are putting http(s):// prefix for the endpoint or missing out to put the port.

Here’s a sample values.yaml that work for me (only relevant sections):

#... Line #287

distStorage:
  type: "aws"

#... Line #344

  aws:
    bucketName: "dremio"
    path: "/"
    authentication: "accessKeySecret" 
    credentials:
      accessKey: "ACCESS_KEY"
      secret: "SECRET_KEY"

    extraProperties: |
      <property>
        <name>fs.s3a.endpoint</name>
        <value>175.1.2.3:9000</value>
      </property>
      <property>
        <name>fs.s3a.path.style.access</name>
        <value>true</value>
      </property>
      <property>
        <name>dremio.s3.compat</name>
        <value>true</value>
      </property>
      <property>
        <name>fs.s3a.connection.ssl.enabled</name>
        <value>false</value>
      </property>

2 Likes

@lenoyjacob Our vaules.yaml files look identical in that regards. Only difference is that we have https for our StorageGrid endpoint. Here’s mine, for reference:

distStorage:
  type: "aws"
  aws:
    bucketName: "dremio"
    path: "/"
    authentication: "accessKeySecret"
    credentials:
      accessKey: "ACCESS_KEY"
      secret: "SECRET_KEY"
    extraProperties: |
      <property>
        <name>fs.s3a.endpoint</name>
        <value>storagegrid.endpoint.company.com</value>
      </property>
      <property>
        <name>fs.s3a.path.style.access</name>
        <value>true</value>
      </property>
      <property>
        <name>dremio.s3.compat</name>
        <value>true</value>
      </property>
      <property>
        <name>fs.s3a.connection.ssl.enabled</name>
        <value>true</value>
      </property>

The annoying thing is, I got it to work (yay!) when having the following image:

image: dremio/dremio-oss
imageTag: 20.1.0

But not any later tags/versions. Later versions always result in the error stated in the OP. Probably should file this as a bug somewhere. Anyhow, thanks for your help in confirming that at least my settings were correct!

Cheers!

@ramworks Are you blocked on your upgrade to v21.x due to this?

@balaji.ramaswamy Yes, unfortunately. Seems that neither versions of 21.x respects the additional properties fully.

I saw that version 22.0 popped up on hub.docker.com a couple of days ago. Any info if distributed storage now respects path style access to enable Minio or StorageGrid?

@ramworks Is your question if Dremio supports Minio? Can you please elaborate “path style access”?

Ok, @balaji.ramaswamy . Feels like we lost context here. I created this thread about a month ago since I wasn’t able to specify a S3-like interface as a distributed storage in a kubernetes installation.

It turned out that neither versions of 21 respected the additional properties outlined in Dremios documentation in regards to setting up an S3-like interface as a Distributed Storage. Only version 20 did.

After my findings you asked me if this effectively blocked me from upgrading to version 21, to which I responded a heartfelt Yes.

So now when version 22 is out I am asking if this issue might have been resolved.

Kind regards,
Jesper

@ramworks Apologies, I should have gone through the post again. I have again pinged the right person for his availability, once I do, will update here

1 Like

Hi again,

Unfortunately, the issue with distributed storage settings not respecting fs.s3s.endpoint and fs.s3a.path.style.access outlined in the documentation is still there in version 22.0.0.

I am getting the following error when querying a dataset in version 22.0.0. I’ve set dremio as the bucket for distributed storage and made sure that it is created.

Edit: It’s actually when querying a physical dataset that is comprised of several files with identical schemas residing in the same catalog, i.e. the catalog is formatted as a dataset:

com.dremio.common.exceptions.UserException: UnknownHostException: dremio.<redacted hostname>.com
	at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:890)
	at com.dremio.service.jobs.JobResultsStore.loadJobData(JobResultsStore.java:152)
	at com.dremio.service.jobs.JobResultsStore$LateJobLoader.load(JobResultsStore.java:331)
	at com.dremio.service.jobs.JobDataImpl.range(JobDataImpl.java:50)
	at com.dremio.service.jobs.LocalJobsService.getJobData(LocalJobsService.java:1136)
	at com.dremio.service.jobs.JobsFlightProducer.getStream(JobsFlightProducer.java:78)
	at com.dremio.dac.service.flight.CoordinatorFlightProducer.getStream(CoordinatorFlightProducer.java:63)
	at org.apache.arrow.flight.FlightService.doGetCustom(FlightService.java:111)
	at org.apache.arrow.flight.FlightBindingService$DoGetMethod.invoke(FlightBindingService.java:144)
	at org.apache.arrow.flight.FlightBindingService$DoGetMethod.invoke(FlightBindingService.java:134)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at io.opentracing.contrib.grpc.TracingServerInterceptor$2.onHalfClose(TracingServerInterceptor.java:231)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at io.grpc.util.TransmitStatusRuntimeExceptionInterceptor$1.onHalfClose(TransmitStatusRuntimeExceptionInterceptor.java:74)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:340)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:866)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Regards,
Jesper

@ramworks Did you find a solution to this problem?
I’m installing on K8s Dremio 24.1 with same settings as you. When fs.s3a.connection.ssl.enabled is set to ‘true’ the installation fails. Set to ‘false’ it succeeds

@humanSupafly Well in the end I found a solution to my business problem :smirk:

I was tasked to facilitate queries on a dataset comprised of > 50.000 parquet files. Roughly 4 billion records. To get any kind of reasonable speed out of Dremio, this data first had to have it’s format specified and then in a timely manner converted into a reflection.

I stared at the problem for a few days after running into the issues outlined in this thread and realized that playing Lake House was just a very weird way of for unknown reasons avoiding an MPP database. So I installed ClickHouse on an identical set of hardware and ingested all the data into a table. Both query speeds and storage requirements were noticeably improved.

Since the team behind ClickHouse also ships an official python library, a Kafka Connect client, a dbt adapter, ODBC and JDBC-drivers and comes with a very easy to use REST API all integration-bits that we hadn’t really figured out with Dremio yet also kinda resolved themselves.

So for us the choice was obvious. We didn’t need a Lake House. We needed the best OSS columnar database on the market.

Kind regards,
Jesper Bagge

Oke, thanks for the answer @ramworks

I managed to configure ssl enabled distributed storage with our onprem s3. in our case the problem was the certificate and policy settings on the s3 endpoints.