Dremio Connector Issue with Amazon OpenSearch Service - 404 Error when Querying Index Content

1

Hello fellow Dremio users,

I wanted to share an issue I encountered while trying to connect Dremio with Amazon OpenSearch Service and see if anyone has experienced a similar problem or found a solution. Specifically, I can establish a connection successfully, but I’m encountering a 404 error when attempting to query the index content.

Here are some details about my setup:

  1. Dremio version: 23.0.1
  2. Amazon OpenSearch Service version: 2.5.0
  3. Connector used: Amazon OpenSearch Service Source

The problem arises when I try to execute a query on the index content within Dremio. Instead of returning the expected results, I receive a 404 error response. I have double-checked the index name, the query syntax, and the permissions, but everything seems to be configured correctly.

image
image919×291 9.67 KB

I have already attempted the following troubleshooting steps without success:

  1. Verified the connection settings: I have confirmed that the connection details in Dremio are accurate, including the host, port, and access credentials.
  2. Checked index availability: I ensured that the index I’m querying is present and available in Amazon OpenSearch Service.
  3. Examined query syntax: I reviewed the query syntax for any potential mistakes or discrepancies, but it appears to be correct.
  4. Reviewed access permissions: I verified that the user credentials being used to connect Dremio to OpenSearch Service have sufficient permissions to query the index content.

Despite these efforts, the 404 error persists, preventing me from retrieving the desired data. I have searched extensively for similar issues or any documentation addressing this problem, but I haven’t found a definitive solution.

If anyone has encountered this issue before or has any insights on how to resolve it, I would greatly appreciate your assistance. Your input could be valuable not only to me but also to other community members who might face a similar challenge in the future.

Please feel free to share any suggestions, workarounds, or potential solutions that you have come across. Any advice, links to relevant resources, or personal experiences would be immensely helpful.

Thank you in advance for your time and assistance. I look forward to your responses!

Best regards, samuel2pb

2

Hey Samuel–

This is a newer area for me so my apologies if these are dumb questions.

Are you all using Shield at all? If so, have you verified the steps listed in the two sections here have been completed? Dremio

It sounds like you have at least validated you have the expected read and view_index_metadata permissions. Are these set against the correct index?

Finally, if nothing helps the above, you may be able to try the debug logging listed on the same page to see if that sheds some light on what is happening. Dremio

I hope something here helps. If not, let us know if anything pops in the logging and I’ll be happy to see if I can dig anything else up.

Dan

3

Hi Samuel,

The error you are seeing is exactly what you see when you try and connect Dremio to version 8.x of Elasticsearch, and though I don’t know the specifics of the internals of the OpenSearch connector, I suspect it’s due to the same reason.

The Elasticsearch connector only supports versions 6 and 7 of ES, and though OpenSearch originally forked from 7.x of ES, I assume it has undergone the same sorts of changes to it as ES has done over the versions. I.e. a query that used to be /myindex/_doc/_count now needs to be /myindex/_count
So I would assume that the OpenSearch connector only supports earlier versions of OS.
If you can capture network traffic to confirm this, then we can be sure that that is where the problem lies.

Best regards,

Andy