How to check runtime filtering is working?

Hi @balaji.ramaswamy ,

I am testing how runtime filtering works. For that i have ran below query once with planner.filter.runtime_filter and exec.non_partitioned_parquet.enable_runtime_filter support key enabled and once disabled. I found no such difference in job profile. I have both job profile for reference.

SELECT sum(price) as sum_price,custumer_state,product_category_name,EXTRACT(YEAR FROM order_purchase_timestamp) as order_year from minio.dremio.olist.order_items join (select order_id as order_id,customer_unique_id as customer_unique_id,TO_TIMESTAMP(order_purchase_timestamp, 'YYYY-MM-DD HH24:MI:SS', 1) as order_purchase_timestamp ,customer_state as custumer_state from minio.dremio.olist.orders join minio.dremio.olist.customer  on orders.customer_id = customer.customer_id) new_orders on order_items.order_id = new_orders.order_id join minio.dremio.olist.products on order_items.product_id = products.product_id group by custumer_state,price,product_category_name,EXTRACT(YEAR FROM order_purchase_timestamp); 

19c18ee3-7e6c-4c33-8595-4d8b635a6273.zip (26.8 KB)
28e3f84e-3ad6-4520-9e39-957833f66bdf.zip (26.6 KB)

@Ayush.goyal For Job ID# 1f63e09e-7c90-7761-677f-2f3372f9f600, Open the profile, click on planning and inside “Final Physical Transformation”, you can see below, 04-02 is the scan that got benefited by the runtime filter,

01-05 HashJoin(condition=[=($0, $3)], joinType=[inner], runtimeFilter=[[04-02 order_id]])

I do not find runtime filter used in the other profile, Job ID# 1f63e039-8920-6103-ecc2-4e06e7b19100