SQL issue for more than 1500 values in "IN" clause

popejune · February 10, 2021, 6:22am

Hi,

We have a customized plugin.

When user run the SQL including IN condition with more than 1500 value. Seems this condition cannot give to our plugin. Do you have any idea?

SQL example
Select * from Table where id in (1, 2, 3, …, 1500, 1501)

Dremio version: 4.9.1-202010230218060541-2e764ed0

balaji.ramaswamy · February 11, 2021, 7:40am

What do you mean by customized plugin? Is this a connector that you have developed? Is there a job profile you can share?

popejune · February 15, 2021, 12:59pm

We build a plugin to connect our data source via REST API. So we will receive the query from Dremio and send the corresponding query to data source.

balaji.ramaswamy · February 16, 2021, 6:32am

Are you able to send us the job profile?

popejune · February 17, 2021, 2:56am

balaji.ramaswamy · February 20, 2021, 7:19am

Looks like the error is coming from your customized plugin, so it would be hard to help as we have not written the plugin

Is TDS3 the plugin you have written?

Thanks
Bali

popejune · February 22, 2021, 3:09am

Yes, this is our customized plugin. But this plugin is based on the dremio RelOptRule, which should give the query filter to our plugin.

If the IN counts < 1500, our plugin could get the filters.

If the IN counts >= 1500, our plugin cannot get the filters.