Question, we are trying to determine if we can utilize Dremio for our architecture. How do you get large queries from Dremio? For a quick example with only 10 million rows of results, I tried using python/ODBC endpoint but this query takes over 3 minutes to get the results. Utilizing Dremio’s REST API took 30 minutes. This is because the API has a query limit of 500 rows, so I had to request 20,000 times just to get the entire 10 million results. The query itself took 15 seconds only though, to determine the results, its really just getting the results over the network that took time. I also tried utilizing the flight endpoint, which I thought should be faster than ODBC but that also took 3+ minutes. With bigger query results (40+ million rows), I get an error
RpcException: Timeout while waiting for client to be in ready state.
In short, if I have a query that can be 1+ billion rows of results, what is the best fastest way for me to get this back to the python client?