ElasticSearch 6.X with Cognito

1

Hi,

I’m using AWS ES 6.2 with Cognito enabled. The ES service is https with a AWS domain and not exposed to the internet, i.e., it can only be accessed by services in the same VPC and has no public ip. The service is a simple cluster of two t2.small.elasticsearch instances.

Both the Cognito users (via Kibana - user/password) and the IAM users (Logstash - key/secret) have been able to successfully access ES as they are in the access policy:

{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Principal”: {
“AWS”: [
“arn:aws:iam::849805908400:user/bureau.dremio.homolog”,
“arn:aws:sts::849805908400:assumed-role/Cognito_bureau_identitypool_homoloAuth_Role/CognitoIdentityCredentials”,
“arn:aws:iam::849805908400:user/bureau.elk.homolog”
]
},
“Action”: “es:",
“Resource”: "arn:aws:es:us-east-2:849805908400:domain/bureau-elk-homolog/”
}
]
}

However, when I try to create a new ElasticSearch Source, using for instance the Cognito user ‘bureau.dremio.homolog’, the connection fails with a puzzling ‘cannot get cluster health information’. Has anyone ever been through this?

2

Hi @Matheus_Azevedo

Even before we look into Cognito enabled, we are working on supporting ES6 on AWS. We should have the support by the next few releases (Kindly watch for the release notes)

Thanks
@balaji.ramaswamy

1 Like
3

Thanks Balaji

So 6.X is still under development, ok. Is there anything planned for the Cognito Architecture? I think Cognito is intended to provide a login (user/password) to access ES via Kibana only. I use a pair of AWS IAM Access Key + Access Secret to authenticate via Logstash for example (using the Logstash amazon_es plugin).

Do you foresee an Access Key+Access Secret support in a future release?

TIA,
Matheus

4

Hi @Matheus_Azevedo

Let me get back to you on your second request. Again ES 6.0 is supported just the AWS version is under development. Are you able to connect Dremio to a non-AWS ES 6.0 and test?

Thanks
@balaji.ramaswamy

5

Hi @balaji.ramaswamy thanks for your reply.

I ran a Proof of Concept several months ago with an older version of Dremio, and yes, we were able to retrieve data from a ES 6.2.2. However, we moved to AWS ES, and Kibana authentication via Cognito is now a mandatory requirement for my project, so I think I will have to wait until Dremio supports AWS ES.

Thanks,
Matheus

6

Hi @Matheus_Azevedo

We have enhanced Dremio to support elasticsearch on AWS. If this is still a requirement, kindly download our latest version 3.0.5 from https://www.dremio.com/download/, upgrade, test and let us know. Not sure about the Cognito part, kindly test and let us know

Thanks
@balaji.ramaswamy