As recommended, we use projects in Dremio to isolate each of our environments, such as development, production, etc. We have created a Python pipeline in development using Arrow Flight, which is working great. However, when we come to promote it to production and switch out the development catalog for a production one in the query, Arrow Flight cannot find the dataset/view and errors with “Unknown table…”.
After scouring the documentation we found Developing Client Applications with Apache Arrow Flight | Dremio Documentation and the following statement:
This seems like a pretty huge limitation, and I hope that I am just missing something, as I can see in the sequence diagram in the docs there is indeed a project-id being passed.
- I assume this is a limitation given I cannot query a non-default project. Do you know when this will be resolved?
- What would you suggest as a workaround? The current Python documentation suggests using Apache Arrow Python | Dremio Documentation