Setting up S3 connection in Dremio via Proxy

Hi Dremio Community,

We are trying to set up an S3 connection in Dremio via a corporate proxy connection. We have defined the AWS access and secret key and the assumed role.

In addition, we also added the following property in the “Advanced Options” tab.

  • fs.s3a.proxy.port
  • fs.s3a.session.token
  • fs.s3a.server-side-encryption-algorithm
  • fs.s3a.endpoint

We set the ‘’ as ‘org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider’.

When we disable the ‘Compatibility Mode’, we are getting an ‘Illegal State Exception’ (see below). java.lang.IllegalStateException: org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider

When we enable the ‘Compatibility Mode’, we are getting a NullPointerException.

What could be the issue here? And how should we create a S3 source in Dremio with Proxy?

@librian Would like to look at the full stack, are you able to send us the server.log from the coordinator?

The following stacktrace is from the coordinator log when we attempt to create a new S3 source.

Caused by: java.util.concurrent.TimeoutException: null
        at java.util.concurrent.CompletableFuture.timedGet( ~[na:1.8.0_121]
        at java.util.concurrent.CompletableFuture.get( ~[na:1.8.0_121]
        at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin( ~[dremio-sabot-kernel-4.1.8-202003120636020140-9c2a6b13.jar:4.1.8-202003120636020140-9c2a6b13]
        at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource( ~[dremio-sabot-kernel-4.1.8-202003120636020140-9c2a6b13.jar:4.1.8-202003120636020140-9c2a6b13]
        ... 70 common frames omitted
        Suppressed: java.lang.NullPointerException: null
                at ~[dremio-sabot-kernel-4.1.8-202003120636020140-9c2a6b13.jar:4.1.8-202003120636020140-9c2a6b13]
                at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals( ~[dremio-sabot-kernel-4.1.8-202003120636020140-9c2a6b13.jar:4.1.8-202003120636020140-9c2a6b13]
                at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin( ~[dremio-sabot-kernel-4.1.8-202003120636020140-9c2a6b13.jar:4.1.8-202003120636020140-9c2a6b13]
                ... 71 common frames omitted

@librian Are you able to add the below 2 loggers to logback.xml under the conf folder (logger section), restart the coordinator, retry and then once it fails, send us the server.log?

Hi Balaji,

I reset all the configurations and have the following options during s3 data source creation. We are still getting “Connect to .com:443 failed: connect timed out”.

The value for fs.s3a.endpoint is ‘’. I can also see from the code that it defaults to US_EAST_1 if the pattern match fails. dremio-oss/ at master · dremio/dremio-oss · GitHub

Is there a way to tell Dremio to connect to the above custom STS endpoint?

@librian Please add the loggers mentioned, restart Dremio, try to add the source and then send us the server.log from the coordinator