Dremio setup error : Failed to get policy document to validate permissions

1

I am trying to setup dremio on ec2 in eu-central-1 region using market place AMI - “ami-054babf5407cc1456”.
I have setup IAM profile for the instance and security groups as described in the documentation - Configuring AWS IAM Permissions and IAM Roles | Dremio Documentation

I see this error,

2023-09-05 09:00:33,176 [qtp332498651-20] INFO  c.d.d.validator.AwsProjectValidator - Failed to get policy document to validate permissions, Unable to execute HTTP request: Received close_notify during handshake

I suspect dremio tries to access AWS api via global endpoints like - sts.amazonaws.com, but I would like to know if I can start dremio with extra arguments or env variables to enforce regional endpoints like - sts.eu-central-1.amazonaws.com ?

image
image976×494 26.6 KB

2

@hchoudhary Did you try to refresh the page? There is currently no way to configure or enforce the regional endpoints

3

Hi @balaji.ramaswamy , thanks for your response. i have tried to refresh couple of times but I have the same error.
Is it possible to enable extra verbose logging so I can see some more info at /var/log/dremio/server.log

4

@hchoudhary Not sure if the verbose logging will help, as I said there is currently no way to configure or enforce the regional endpoints

You can open logback.xml under the conf folder and enable both com.dremio and root logger to debug, make sure to set back to original value after the troubleshooting is done

Note; Good idea to make a copy of logback.xml so you can just put it back

5

@hchoudhary can you please send me over your core-site.xml. I have just spoken to your colleague Jacob about this and I would be keen to understand more about the problem

6

Hi @Ashley_Farrugia, Thanks for reaching out. here are contents of core-site.xml file found under /etc/dremio

<?xml version="1.0"?>
<!--

    Copyright (C) 2017-2019 Dremio Corporation. This file is confidential and private property.

-->
<configuration>
  <property>
    <name>fs.dremioS3.impl</name>
    <value>com.dremio.plugins.s3.store.S3FileSystem</value>
  </property>
  <property>
    <name>fs.s3a.aws.credentials.provider</name>
    <value>com.amazonaws.auth.InstanceProfileCredentialsProvider</value>
  </property>
  <property>
    <name>fs.s3a.buffer.dir</name>
    <value>/var/lib/dremio/data/buffer</value>
  </property>
  <property>
    <name>dremio.s3.async</name>
    <value>true</value>
  </property>
 </configuration>
7

Thank you, can you please try adding the following properties? and restarting your Dremio cluster.

<property>
   <name>fs.s3a.endpoint</name>
   <value>s3.eu-central-1.amazonaws.com</value>
 </property>
<property>
   <name>dremio.s3.region</name>
   <value>eu-central-1</value>
 </property>
<property>
       <name>fs.s3a.assumed.role.sts.endpoint</name>
       <value>sts.eu-central-1.amazonaws.com </value>
</property>
<property>

Many thanks,
Ashley

8

I have added the properties suggested, and now core-site.xml file looks like this

<?xml version="1.0"?>
<!--

    Copyright (C) 2017-2019 Dremio Corporation. This file is confidential and private property.

-->
<configuration>
  <property>
    <name>fs.dremioS3.impl</name>
    <value>com.dremio.plugins.s3.store.S3FileSystem</value>
  </property>
  <property>
    <name>fs.s3a.aws.credentials.provider</name>
    <value>com.amazonaws.auth.InstanceProfileCredentialsProvider</value>
  </property>
  <property>
    <name>fs.s3a.buffer.dir</name>
    <value>/var/lib/dremio/data/buffer</value>
  </property>
  <property>
    <name>dremio.s3.async</name>
    <value>true</value>
  </property>
  <property>
   <name>fs.s3a.endpoint</name>
   <value>s3.eu-central-1.amazonaws.com</value>
  </property>
  <property>
    <name>dremio.s3.region</name>
    <value>eu-central-1</value>
  </property>
  <property>
     <name>fs.s3a.assumed.role.sts.endpoint</name>
     <value>sts.eu-central-1.amazonaws.com </value>
 </property>
 </configuration>

Then I restart dremio on ec2, its the same error, confirmed by logs also.

        - jar:file:/opt/dremio/jars/dremio-sysflight-plugin-24.1.4-202308151830020521-3ba6a783.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/dremio-yarn-24.1.4-202308151830020521-3ba6a783.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/3rdparty/dremio-client-base-24.1.4-202308151830020521-3ba6a783.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/3rdparty/dremio-client-jdbc-24.1.4-202308151830020521-3ba6a783.jar!/sabot-module.conf
        - jar:file:/opt/dremio/jars/3rdparty/dremio-hive2-exec-shaded-24.1.4-202308151830020521-3ba6a783.jar!/sabot-module.conf


2023-09-26 17:23:32,451 [qtp332498651-15] INFO  c.d.d.validator.AwsProjectValidator - Failed to get policy document to validate permissions, Unable to execute HTTP request: Received close_notify during handshake
2023-09-26 17:23:39,383 [qtp332498651-22] INFO  c.d.d.validator.AwsProjectValidator - Failed to get policy document to validate permissions, Unable to execute HTTP request: Received close_notify during handshake

is the config file correctly updated by me? what else can I try ?
please let me know if you need any additional details about the setup.

9

No problem, I think this might actually be related to trying to access the IAM global endpoint, there is currently no VPC Endpoint for this, to overcome this can you please insert the following into your dremio.conf file?

provisioning.validation.enabled: false