Reflection not picked up via Arrow Flight, but used via REST API and UI

1

Hi,

I’m encountering an issue where a reflection that works correctly when querying through the Dremio UI or REST API is not picked up when executing the same SQL query via Arrow Flight (Flight SQL).

In the query profile for the Flight execution, under the Acceleration tab, I see this message:

algebraic_matching: Skipping agg join matching due to limit. Number of joins: 33, Limit: 16

This message does not appear in the UI or REST execution of the same query, and in those cases, the reflection is applied successfully with AGGREGATE_REWRITE.

I’m wondering:

  • Why would the optimizer behavior differ between Arrow Flight and other access methods?
  • Is this join limit (16) specific to the execution context used by Flight SQL?
  • Can this limit be configured or increased to allow reflection matching in more complex queries via Flight?
  • Are there any planner settings or context options I should be setting explicitly when using Flight?

It seems like the number of joins is causing the reflection matching logic to skip, but only when using Arrow Flight. I’d need help to know whether this is expected behavior, and how to work around or resolve it.

Profiles:
rest-api-plan-with-reflections.zip (113.8 KB)
arrow-flight-without-reflections.zip (242.4 KB)
arrow-flight-with-reflections.zip (107.4 KB)

Build Info:

  • Build: 25.1.6-202501021803480419-127757a4
  • Edition: AWS Edition (activated)

Thanks in advance!

3

First of all, thank you for your reply, and apologies for the late response.

Yes, I experienced the issue consistently after the first occurrence.

Completely recreating the reflection resolved the problem — I created an identical reflection and dropped the original.

Regards,
Misi

4

@Mihaly This looks like a recent issue that Benny fixed. So you see an intermittent issue where sometimes the reflection is not listed but running same query via UI works?

5

Yes, we see such issue in version 25.1.6. Its random in a way that recreating (not rebuilding/refreshing but dropping/creating) helps. It affects only arrow flight while reflection is picked up when running same query in UI at the same time.

Here’s a fresh example. Two profiles of the same query executed using arrow flight before reflection recreated and after. First one is slow and is not used reflection over vds and scond one is fast.
(btw in jobs list we can see two rows: create prepared statement and execute prepared statement - i exported execute prepared statement profile, tell me if you need profiels for create prepared statement )
Can you confirm that this issue might be fixed already or it is a different issue? UI and REST always pick up reflection, its only affecting arrow flight

Arrow flight (execute prepared statement) - Reflection used.zip (109,0 КБ)
Arrow flight (execute prepared statement) - Reflection not used.zip (444,7 КБ)

6

If restating the coordinator also fixes the problem (possibly temporarily), then it’s definitely the same issue that was fixed in 25.2.11. See the third bullet here 25.x Release Notes | Dremio Documentation

1 Like