Dremio was not able to connect with Elasticsearch hosted in Ec2

1

I have installed Elasticsearch in AWS EC2 from the following link :-

https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.4.rpm and modified my /etc/elasticsearch/elasticsearch.yml file to point to my Ec2 private IP address

and I could see that Elasticsearch is installed and up and running and working fine by running curl as below :-

curl myec2instance_private_ip-adress:9200/_cluster/health?pretty . its returning the expected result .

. But when I am trying to connect my EC2 hosted Elasticsearch from dremio elasticsearch by typing private IP address into Dremio elastic search source host and choosing no authentication in authentication option and while I tried to save it I am getting the following error :-

Encountered a problem while executing com.dremio.plugins.elastic.ElasticActions$NodesInfo@27c0cb30.

Cannot gather Elasticsearch nodes information. Please make sure that the user has [cluster:monitor/nodes/info] privilege.

.

Please note I wide opened all the port and connection from any ip and any source in my ec2 security group and NACL and subnet to make sure there is no blocking but still no luck . Please advise .

2

Hi @Sayed_Rahman

What version of Dremio are you using? We have support for Elasticsearch on AWS only from Dremio version 3.0.5.

If you are already on 3.0.5 and still getting the error, then kindly check the below things

  1. AWS on Elasticsearch is a separate source, see attached image
    AWS on elasticsearch
  2. When adding the hostname, do not give the http

aws-es-source-form
aws-es-source-form.png1738×1598 195 KB
aws-es-source-type
aws-es-source-type.png1126×572 42.7 KB

3

I am not using aws managed elastic search .As I see in your snapshot you are showing managed aws ES .I am using EC2 hosted elastic search .Yes ,I am using latest version of dremio .

Thanks,

4

Thanks @Sayed_Rahman

Kindly send us the server.log from the Dremio coordinator

5

sayed_personal_account_personal_dremio_server_log.zip (57.2 KB)

6

I recreated the same environment (EC2 hosted elastic search 6.5.4 with latest version of Dremio) today and tried again to connect and got the same error message and sharing the server log with you .Thanks

7

Hi @Sayed_Rahman

Can you please share your elastic source settings on Dremio? the General and advanced tab?, you can mark “XXX” for the values. More interested in the check boxes. attached screenshots

managed-es-advanced
managed-es-advanced.png1906×1442 248 KB
managed-es-general
managed-es-general.png1924×1200 190 KB

8

screen_two_my_personal-account_dremio
screen_two_my_personal-account_dremio.jpg3360×2100 905 KB

9

screen_one_my_personal-account_dremio
screen_one_my_personal-account_dremio.jpg3360×2100 838 KB

10

Hi balaji, Any update on my error ?
Thanks,
Sayed Rahman

11

Hi @Sayed_Rahman

Can you please check the box for “Managed Elasticsearch service” and try again ?

12

Hi @Sayed_Rahman

I just got to know that your Dremio and ES are on the same box and problem is resolved

Thanks
@balaji.ramaswamy

13

Luckily Naren sankar (solution architect) from Dremio came to our office yesterday and today we connected and he helped me to figure it out . The issue is in host field I need to type “localhost” not the private IP of EC2 instance and it worked fine . Thank you for your help though .