# Arrow flight protocol **URL:** \ **Category:** Uncategorized\ **Created:** [August 12, 2019, 1:55pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855 "2019-08-12T13:55:07Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![dfleckinger](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/dfleckinger/32/370_2.png) [@dfleckinger](https://community.dremio.com/u/dfleckinger)\ **Post date:** [August 12, 2019, 1:55pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/1 "2019-08-12T13:55:07Z")
According to @tshiran blog post ([https://www.dremio.com/is-time-to-replace-odbc-jdbc/](https://www.dremio.com/is-time-to-replace-odbc-jdbc/)) , it’s time to replace ODBC/JDBC by arrow flight. Any insights when we will be able to call Dremio using this protocol instead of ODBC, for instance using pyarrow as shown in the example ? Thanks ---
**Author:** ![tshiran](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/tshiran/32/15_2.png) [@tshiran](https://community.dremio.com/u/tshiran)\ **Post date:** [August 14, 2019, 7:52am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/2 "2019-08-14T07:52:47Z")
@rymurr implemented Arrow Flight access to Dremio here: [https://github.com/dremio-hub/dremio-flight-connector](https://github.com/dremio-hub/dremio-flight-connector) Try it out and let us know what you think. ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [August 14, 2019, 9:23am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/3 "2019-08-14T09:23:44Z")
@dfleckinger thanks for your interest! Please let us know how you get on with the Flight plugin. It is still in beta so any feedback is extremely useful! ---
**Author:** ![dfleckinger](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/dfleckinger/32/370_2.png) [@dfleckinger](https://community.dremio.com/u/dfleckinger)\ **Post date:** [August 14, 2019, 11:00am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/4 "2019-08-14T11:00:26Z")
Thanks, I saw this implementation. However I could not make it work and I was wondering which versions of Dremio have implemented the arrow flight server protocol and on which port flight server is supposed to be listening. I have tested on Dremio CE version 3.1.11 and 3.3.1 , i don’t see 47470 in the list of listening ports. ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [August 15, 2019, 11:27am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/5 "2019-08-15T11:27:19Z")
The flight plugin currently works on 3.3.1 and above only. You would have to manually: - checkout the repo: `git clone https://github.com/dremio-hub/dremio-flight-connector.git` - build the plugin: `mvn clean install` - move the plugin (the shaded jar that is created by maven) to `dremio/jars` After those steps you should see the flight connector listening on 47470. This manual process is only while the flight connectivity is in beta. We hope to make it a part of the product by the end of this year. ---
**Author:** ![dfleckinger](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/dfleckinger/32/370_2.png) [@dfleckinger](https://community.dremio.com/u/dfleckinger)\ **Post date:** [August 16, 2019, 8:20am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/6 "2019-08-16T08:20:16Z")
hi @rymurr, thanks a lot for your contribution ! After reading the blog post, I thought that Flight server was included in Dremio build out of the box, which is apparently not the case. So thanks for your clarification on that. ---
**Author:** ![david.lee](https://avatars.discourse-cdn.com/v4/letter/d/ed655f/32.png) [@david.lee](https://community.dremio.com/u/david.lee)\ **Post date:** [August 20, 2019, 11:49pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/7 "2019-08-20T23:49:12Z")
I gave this a try today and it is failing because Dremio isn’t supporting a http2 listener. Is there a way to turn that on? ``` --------------------------------------------------------------------------- ArrowIOError Traceback (most recent call last) in 51 52 client = flight.FlightClient.connect('grpc+tcp://' + dremio_coordinator + ':47470') ---> 53 client.authenticate(HttpDremioClientAuthHandler(username, password)) 54 start_time = time.time() 55 cmd = Command(query=sql, parallel=False, coalesce=False, ticket=b'') /usr/local/Anaconda3-5.3.1-Linux-x86_64/envs/pyspark-3-6/lib/python3.6/site-packages/pyarrow/_flight.pyx in pyarrow._flight.FlightClient.authenticate() /usr/local/Anaconda3-5.3.1-Linux-x86_64/envs/pyspark-3-6/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status() ArrowIOError: gRPC failed with error code 14 and message: Trying to connect an http1.x server ``` ​ ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [August 22, 2019, 7:56am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/8 "2019-08-22T07:56:08Z")
Hey David, Thanks for trying out Flight! Are you by any chance running behind a firewall or proxy? Perhaps on a kubernetes cluster with ingress set up? Not all reverse proxies and ingress controllers play nice with http2. By default the Flight listener (which uses grpc under the hood) is set up to listen on the same port for both http2/http1.x but not all reverse proxies handle that well. If you share a bit of info on your environment I can point you towards the relevant documentation. ---
**Author:** ![josephwinston](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/josephwinston/32/1844_2.png) [@josephwinston](https://community.dremio.com/u/josephwinston)\ **Post date:** [August 22, 2019, 12:48pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/9 "2019-08-22T12:48:07Z")
I have a use case where I want to construct my own Apache Arrow Table and then give this table to Dremio. Is this use case one that will be supported? If this is request in scope, is there an example that I can look at. ---
**Author:** ![ORo](https://avatars.discourse-cdn.com/v4/letter/o/e19b73/32.png) [@ORo](https://community.dremio.com/u/ORo)\ **Post date:** [August 22, 2019, 1:14pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/10 "2019-08-22T13:14:53Z")
Hello, David, I ran into the same problem. For me it was the SQL statement. I had the VDS in my private space and didn’t address the table correctly. Then I created a new space (e.g. analytics) and copied the VDS to it. And then it went as expected: select \* from analytics.table Could you try that? Best regards Oliver ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [August 22, 2019, 1:17pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/11 "2019-08-22T13:17:37Z")
Hey Joseph. Welcome! Currently the best way to do that is to write the Arrow Table to a parquet file ([https://arrow.apache.org/docs/python/parquet.html](https://arrow.apache.org/docs/python/parquet.html)) in your chosen storage location (data lake, NAS, etc). You can then manually add it as a Datasource to Dremio or alternatively use the REST API (docs here: [https://docs.dremio.com/rest-api/](https://docs.dremio.com/rest-api/)) to let Dremio know you added some data. Adding support to the Flight connector for creating datasets (`doPut` in flight notation) is on the roadmap. That will save the steps above but first we want to get a GA version of flight released. ---
**Author:** ![josephwinston](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/josephwinston/32/1844_2.png) [@josephwinston](https://community.dremio.com/u/josephwinston)\ **Post date:** [August 22, 2019, 1:27pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/12 "2019-08-22T13:27:33Z")
Thanks for a quick response. I was trying to save the parquet step if at all possible. Would you be willing to take a pull request that implements the feature? And would such a change even be possible without digging into [https://github.com/dremio/dremio-oss](https://github.com/dremio/dremio-oss)? ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [August 22, 2019, 1:53pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/13 "2019-08-22T13:53:23Z")
The Dremio flight connector is currently Apache-2 licensed on our Dremio Hub [https://github.com/dremio-hub/dremio-flight-connector](https://github.com/dremio-hub/dremio-flight-connector) and distinct from the dremio-oss project. Technically speaking one should be able to implement `doPut` fairly easily on that connector however it will require some knowledge of the `CTAS` functionality in Dremio [https://docs.dremio.com/sql-reference/sql-commands/tables.html](https://docs.dremio.com/sql-reference/sql-commands/tables.html) to make sure the Arrow buffer from `doPut` ends up in the right place. Note also that the signature for `doPut` is changing [https://www.mail-archive.com/dev@arrow.apache.org/msg13191.html](https://www.mail-archive.com/dev@arrow.apache.org/msg13191.html) ---
**Author:** ![josephwinston](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/josephwinston/32/1844_2.png) [@josephwinston](https://community.dremio.com/u/josephwinston)\ **Post date:** [August 22, 2019, 1:59pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/14 "2019-08-22T13:59:05Z")
Thank you for the overview. I’ll start with the parquet path you suggested and if I can get time allocated, I’ll work on the creation of Arrow Tables as this unblocks a large amount of our stories. ---
**Author:** ![smora](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@smora](https://community.dremio.com/u/smora)\ **Post date:** [October 25, 2019, 8:41pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/15 "2019-10-25T20:41:21Z")
Hi @rymurr i have similar issue, running dremio in a k8 cluster with ingress setup I do see flight connector running in dremiio-master container on port 47470, but its not accessible from external python client i did port forward k8 dremio-master pod as below - created tcp/47470 of dremio-master-pod in service disovery as a nodeport - configured cluster tcp ingress service for above service/47470 when i connect form external python client getting below error > pyarrow.\_flight.FlightUnavailableError: gRPC returned unavailable error, with message: Connect Failed ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [October 28, 2019, 8:24am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/16 "2019-10-28T08:24:56Z")
Hey smora, Have you made sure flight is listening on the external interface for the docker container? I realised I have introduced a bug which requires you set `-e 'JAVA_EXTRA_OPTS="$JAVA_EXTRA_OPTS -Ddremio.flight.host=0.0.0.0'`. This will be fixed in the next release. For reference the dockerfile I test with is: ```auto FROM dremio/dremio-oss:latest COPY target/dremio-flight-connector-0.11.0-SNAPSHOT-shaded.jar /opt/dremio/jars/ ENV JAVA_EXTRA_OPTS="$JAVA_EXTRA_OPTS -Ddremio.flight.host=0.0.0.0" ENTRYPOINT ["bin/dremio", "start-fg"] ``` ---
**Author:** ![smora](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@smora](https://community.dremio.com/u/smora)\ **Post date:** [October 28, 2019, 3:15pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/17 "2019-10-28T15:15:05Z")
Hi @rymurr thanks for the suggestion, now i see the port is accessible via telnet, but python clinet is still not able to connect, any suggestion on further debugging? ``` remio@dremio-master-0:/opt/dremio$ ss -tuln Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port tcp LISTEN 0 128 :::31010 :::* tcp LISTEN 0 1 ::ffff:127.0.0.1:41610 :::* tcp LISTEN 0 128 :::47470 :::* tcp LISTEN 0 128 :::45678 :::* tcp LISTEN 0 50 :::9047 :::* dremio@dremio-master-0:/opt/dremio$ ``` python client error: ``` Traceback (most recent call last): File "C:\Program Files\JetBrains\PyCharm 2018.1\helpers\pydev\pydevd.py", line 1664, in main() File "C:\Program Files\JetBrains\PyCharm 2018.1\helpers\pydev\pydevd.py", line 1658, in main globals = debugger.run(setup['file'], None, None, is_module) File "C:\Program Files\JetBrains\PyCharm 2018.1\helpers\pydev\pydevd.py", line 1068, in run pydev_imports.execfile(file, globals, locals) # execute the script File "C:\Program Files\JetBrains\PyCharm 2018.1\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "C:/Users/smora/PycharmProjects/o2p/flighttest.py", line 45, in client.authenticate(HttpDremioClientAuthHandler(username, password)) File "pyarrow\_flight.pyx", line 660, in pyarrow._flight.FlightClient.authenticate File "pyarrow\error.pxi", line 87, in pyarrow.lib.check_status pyarrow.lib.ArrowIOError: gRPC failed with error code 16 and message: ``` on linux ``` Traceback (most recent call last): File "./flighttest.py", line 47, in client.authenticate(HttpDremioClientAuthHandler(username, password)) File "pyarrow/_flight.pyx", line 944, in pyarrow._flight.FlightClient.authenticate File "pyarrow/_flight.pyx", line 68, in pyarrow._flight.check_flight_status pyarrow._flight.FlightUnavailableError: gRPC returned unavailable error, with message: Connect Failed ``` ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [October 28, 2019, 3:36pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/18 "2019-10-28T15:36:02Z")
Hey @smora it looks like an authentication problem. How are you authenticating? I recommend using the python client [https://github.com/rymurr/dremio\_client:](https://github.com/rymurr/dremio_client:) ```auto from dremio_client.flight import query query(sql, hostname=hostname, port=port, username=username, password=password) ``` ---
**Author:** ![smora](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@smora](https://community.dremio.com/u/smora)\ **Post date:** [October 28, 2019, 4:28pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/19 "2019-10-28T16:28:55Z")
yes @rymurr Thank you for looking into it, i did use the class provided by you in dremio\_client, connection fails but can’t get any more error details other than “error code 16” log from k8 ingress: `[28/Oct/2019:15:55:44 +0000]TCP2001304490.073` code: ``` from pyarrow import flight import pyarrow as pa import base64 from pyarrow.flight import ClientAuthHandler from pyarrow.compat import tobytes class HttpDremioClientAuthHandler(ClientAuthHandler): def __init__ (self, username, password): ClientAuthHandler. __init__ (self) self.username = tobytes(username) self.password = tobytes(password) self.token = None def authenticate(self, outgoing, incoming): outgoing.write(base64.b64encode(self.username + b':' + self.password)) self.token = incoming.read() print(self.token) def get_token(self): return self.token username = 'user' password = 'pwd sql = '''select * from sysobj.options''' hostname = 'host' port = 47470 client = flight.FlightClient.connect('grpc+tcp://:47470') client.authenticate(HttpDremioClientAuthHandler(username, password)) info = client.get_flight_info(flight.FlightDescriptor.for_command(sql)) reader = client.do_get(info.endpoints[0].ticket) batches = [] while True: try: batch, metadata = reader.read_chunk() batches.append(batch) except StopIteration: break data = pa.Table.from_batches(batches) df = data.to_pandas() ``` using dremio\_client library: ``` from dremio_client.flight import query output = query(sql =sql, hostname=hostname, port=port, username=username, password=password) print(output) ``` ---
**Author:** ![martindut](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@martindut](https://community.dremio.com/u/martindut)\ **Post date:** [October 28, 2019, 5:01pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/20 "2019-10-28T17:01:40Z")
I am using dremio\_client, but get this error: ArrowException: Unknown error: gRPC returned unknown error, with message: Stream removed I have no idea how to further debug this. [Next page](https://community.dremio.com/t/arrow-flight-protocol/3855.md?page=2)