Setting up S3 connection in Dremio via Proxy

1

Hi Dremio Community,

We are trying to set up an S3 connection in Dremio via a corporate proxy connection. We have defined the AWS access and secret key and the assumed role.

In addition, we also added the following property in the “Advanced Options” tab.

  • fs.s3a.proxy.host
  • fs.s3a.proxy.port
  • fs.s3a.session.token
  • fs.s3a.server-side-encryption-algorithm
  • fs.s3a.endpoint
  • fs.s3a.aws.credentials.provider

We set the ‘fs.s3a.aws.credentials.provider’ as ‘org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider’.

When we disable the ‘Compatibility Mode’, we are getting an ‘Illegal State Exception’ (see below).

com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider

When we enable the ‘Compatibility Mode’, we are getting a NullPointerException.

What could be the issue here? And how should we create a S3 source in Dremio with Proxy?

2

@librian Would like to look at the full stack, are you able to send us the server.log from the coordinator?

3

The following stacktrace is from the coordinator log when we attempt to create a new S3 source.

Caused by: java.util.concurrent.TimeoutException: null
        at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) ~[na:1.8.0_121]
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) ~[na:1.8.0_121]
        at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:822) ~[dremio-sabot-kernel-4.1.8-202003120636020140-9c2a6b13.jar:4.1.8-202003120636020140-9c2a6b13]
        at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:327) ~[dremio-sabot-kernel-4.1.8-202003120636020140-9c2a6b13.jar:4.1.8-202003120636020140-9c2a6b13]
        ... 70 common frames omitted
        Suppressed: java.lang.NullPointerException: null
                at com.dremio.exec.store.dfs.FileSystemPlugin.getSourceCapabilities(FileSystemPlugin.java:242) ~[dremio-sabot-kernel-4.1.8-202003120636020140-9c2a6b13.jar:4.1.8-202003120636020140-9c2a6b13]
                at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:726) ~[dremio-sabot-kernel-4.1.8-202003120636020140-9c2a6b13.jar:4.1.8-202003120636020140-9c2a6b13]
                at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:837) ~[dremio-sabot-kernel-4.1.8-202003120636020140-9c2a6b13.jar:4.1.8-202003120636020140-9c2a6b13]
                ... 71 common frames omitted
4

@librian Are you able to add the below 2 loggers to logback.xml under the conf folder (logger section), restart the coordinator, retry and then once it fails, send us the server.log?

5

Hi Balaji,

I reset all the configurations and have the following options during s3 data source creation. We are still getting “Connect to sts.us-east-1.amazonaws .com:443 failed: connect timed out”.

fs.s3a.proxy.host
fs.s3a.proxy.port
fs.s3a.endpoint

The value for fs.s3a.endpoint is ‘bucket.vpc-xxxx-xxxx.s3.us-east-1.vpc.amazonaws.com’. I can also see from the code that it defaults to US_EAST_1 if the pattern match fails. dremio-oss/S3FileSystem.java at master · dremio/dremio-oss · GitHub

Is there a way to tell Dremio to connect to the above custom STS endpoint?

6

@librian Please add the loggers mentioned, restart Dremio, try to add the source and then send us the server.log from the coordinator