We are trying to create a Python client to execute a SQL query and read the result from Dremio cluster. We were able to fetch the desired result successfully from a single endpoint.
If the data is available at multiple endpoints, do we need to iterate through all the “flight_info.endpoints” and get the tickets and and fetch the result using each ticket? And if so, then how do we do that, any reference for that scenario would be helpful.
@anupam When you referred to “multiple endpoints”, did you mean the Dremio cluster has multiple coordinators? Could you briefly describe what your Dremio cluster setup is like? Thank you.
In my current use case, the Python client is reading the query result from a single endpoint which is returned by the FlightInfo object. This is working fine.
However I want to explore the scenario described in this section --> Message Flow: Non-Distributed Client + Distributed Server [ https://www.dremio.com/is-time-to-replace-odbc-jdbc/ ].
As per the description, the FlightInfo object returns multiple Endpoint locations and their respective tickets.
So in context of this scenario,
does the client connect to multiple endpoint locations using the respective ticket?
should the client take care of reading the streams in parallel from each location?
Or should the client only talk to coordinator and read the stream from one endpoint?
In the scenario where there are multiple coordinators, you do not need to iterate through all the endpoints. If the query is executed on a coordinator different from the one that generated the query plan, the query plan will be generated again before the query is executed. This incurs double query planning work but does not prevent the query from being executed.
To prevent incurring double query planning work, we recommend session affinity to be set to true when Dremio is deployed with Kubernetes with multiple coordinators. Such that all the TCP connections from a particular client IP will be routed to a specific Dremio coordinator. This is to prevent the query plan to be generated twice. You can refer to the recommendation written in dremio-cloud-tools Github repository: https://github.com/dremio/dremio-cloud-tools/blob/master/charts/dremio_v2/docs/setup/Important-Setup-Considerations.md. Hope this helps.
Thanks for your response.
So the client will always connect to a single coordinator and will read the result from that coordinator end point.
Does that mean
The client will always read the streams sequentially from the coordinator end point OR
The flight can be internally composed of parallel streams OR
The client can read the streams in parallel; if so then how
It seems there is some confusion here.
What I understood from your comment is that, we can’t read the streams in parallel from one single end point, this part is clear.
But I was asking for reading the result streams in parallel from different executor locations by the Python client application as depicted in the attached screenshot. Please confirm if this scenario is supported today and if so, I would like to get some additional reference or a sample client program.
