Hi,
We have a customized plugin.
When user run the SQL including IN condition with more than 1500 value. Seems this condition cannot give to our plugin. Do you have any idea?
SQL example
Select * from Table where id in (1, 2, 3, …, 1500, 1501)
Dremio version: 4.9.1-202010230218060541-2e764ed0
What do you mean by customized plugin? Is this a connector that you have developed? Is there a job profile you can share?
We build a plugin to connect our data source via REST API. So we will receive the query from Dremio and send the corresponding query to data source.
Looks like the error is coming from your customized plugin, so it would be hard to help as we have not written the plugin
Is TDS3 the plugin you have written?
Thanks
Bali
Hi @balaji.ramaswamy ,
Yes, this is our customized plugin. But this plugin is based on the dremio RelOptRule, which should give the query filter to our plugin.
If the IN counts < 1500, our plugin could get the filters.
If the IN counts >= 1500, our plugin cannot get the filters.
This requires someone from our Engineering team to review, I have pinged them and will keep you posted
Thanks
Bali
Hi, @popejune for in clause with more than 1500 values, we actually generate a values rel and convert it into a Join, that’s why you no longer get the filter.
Do you mean it is executed on Dremio backend framework, but plugin could not get it?
Is there any way our plugin could get it? Could you show any sample code?
seems your rule is at physical planning phase. we already rewrite the plan and eliminate the filter for the inlist, so it’s no longer available in FilterPrel at the time you are trying to get it.
if you really want to catch the filter, you can update the support key: exec.operator.orfast.threshold.max, default value is 1500, you can change to a number larger than the number of values in your IN clause and it will direct planner to keep the in list. Be aware this could cause performance degradations.
