How to switch projects when using arrow flight?


As recommended, we use projects in Dremio to isolate each of our environments, such as development, production, etc. We have created a Python pipeline in development using Arrow Flight, which is working great. However, when we come to promote it to production and switch out the development catalog for a production one in the query, Arrow Flight cannot find the dataset/view and errors with “Unknown table…”.

After scouring the documentation we found Developing Client Applications with Apache Arrow Flight | Dremio Documentation and the following statement:

This seems like a pretty huge limitation, and I hope that I am just missing something, as I can see in the sequence diagram in the docs there is indeed a project-id being passed.

  1. I assume this is a limitation given I cannot query a non-default project. Do you know when this will be resolved?
  2. What would you suggest as a workaround? The current Python documentation suggests using Apache Arrow Python | Dremio Documentation


Here’s an example using pyarrow that passes in the project-id:

1 Like

Amazing! works like a charm. Thank you @hope