SQL issue for more than 1500 values in "IN" clause

Hi,

We have a customized plugin.

When user run the SQL including IN condition with more than 1500 value. Seems this condition cannot give to our plugin. Do you have any idea?

SQL example
Select * from Table where id in (1, 2, 3, …, 1500, 1501)

Dremio version: 4.9.1-202010230218060541-2e764ed0

@popejune

What do you mean by customized plugin? Is this a connector that you have developed? Is there a job profile you can share?

Hi @balaji.ramaswamy

We build a plugin to connect our data source via REST API. So we will receive the query from Dremio and send the corresponding query to data source.

@popejune

Are you able to send us the job profile?

Hi @balaji.ramaswamy

5842368d-ff1f-4c27-9e83-d24bd05165c0.zip (153.2 KB)

@popejune

Looks like the error is coming from your customized plugin, so it would be hard to help as we have not written the plugin

Is TDS3 the plugin you have written?

Thanks
Bali

Hi @balaji.ramaswamy ,

Yes, this is our customized plugin. But this plugin is based on the dremio RelOptRule, which should give the query filter to our plugin.

If the IN counts < 1500, our plugin could get the filters.

If the IN counts >= 1500, our plugin cannot get the filters.

1 Like