How to switch projects when using arrow flight?

Dremio Cloud
1

Hi,

As recommended, we use projects in Dremio to isolate each of our environments, such as development, production, etc. We have created a Python pipeline in development using Arrow Flight, which is working great. However, when we come to promote it to production and switch out the development catalog for a production one in the query, Arrow Flight cannot find the dataset/view and errors with “Unknown table…”.

After scouring the documentation we found Developing Client Applications with Apache Arrow Flight | Dremio Documentation and the following statement:

image (41)
image (41)961×489 70.5 KB

This seems like a pretty huge limitation, and I hope that I am just missing something, as I can see in the sequence diagram in the docs there is indeed a project-id being passed.

  1. I assume this is a limitation given I cannot query a non-default project. Do you know when this will be resolved?
  2. What would you suggest as a workaround? The current Python documentation suggests using Apache Arrow Python | Dremio Documentation

Thanks,
Jonathan

2

Here’s an example using pyarrow that passes in the project-id:

1 Like
3

Amazing! works like a charm. Thank you @hope

4

@hope - How do you specify the project id? Is that set in the config.yaml file?