# Arrow flight protocol **URL:** \ **Category:** Uncategorized\ **Created:** [August 12, 2019, 1:55pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855 "2019-08-12T13:55:07Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![smora](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@smora](https://community.dremio.com/u/smora)\ **Post date:** [October 29, 2019, 1:36pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/21 "2019-10-29T13:36:13Z")
hi @rymurr i can confirm that issue is when connecting to flight service running inside a k8 cluster. i am able to connect to it if i run a container inside k8 cluster and by using explicit pod ip. even though dremio-master pod/port is exposed, not able to connect to flight service from outside the cluster. Any specific protocol i should be using when exposing the 47470 port and flight service ? ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [October 30, 2019, 10:36am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/22 "2019-10-30T10:36:53Z")
Hi Both, I have just released a new version of dremio\_client (0.5.0) [1]. I hope this fixes the auth issue. Please give it a try and let me know how it goes. As for the k8s port. The flight port is tcp 47470 and is a http/2 service. Nothing fancy so normal methods to expose in k8s should work fine. And have in the past for me. See [2] for how we currently expose Dremio ports [1] [https://github.com/rymurr/dremio\_client](https://github.com/rymurr/dremio_client) [2] [https://github.com/dremio/dremio-cloud-tools](https://github.com/dremio/dremio-cloud-tools) ---
**Author:** ![smora](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@smora](https://community.dremio.com/u/smora)\ **Post date:** [November 4, 2019, 3:48pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/23 "2019-11-04T15:48:53Z")
hi @rymurr confirmed, i am able to use flight service. i am using 0.4.3 dremio\_client, will test it out with 0.5.0. wrt to k8, issue might be related to certificates, after using ca signed cert’s ingress controller is resolving the request correctly. ---
**Author:** ![smora](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@smora](https://community.dremio.com/u/smora)\ **Post date:** [November 4, 2019, 3:56pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/24 "2019-11-04T15:56:56Z")
@rymurr any plans to add refresh reflection api support to this client ? dremio api refresh call is asynchronous and currently i am using custom code to trigger a refresh and query sysobj tables to figure out reflection refresh is complete based on a new row with new materialization id to be populated(not perfect but works most of the cases) ---
**Author:** ![jackyjiang](https://avatars.discourse-cdn.com/v4/letter/j/e47774/32.png) [@jackyjiang](https://community.dremio.com/u/jackyjiang)\ **Post date:** [November 12, 2019, 7:51pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/25 "2019-11-12T19:51:17Z")
Hello, Rymurr We have the same issue Traceback (most recent call last): File “DremioFlight.py”, line 28, in ``` client.authenticate(HttpDremioClientAuthHandler(username, password)) ``` File “pyarrow/\_flight.pyx”, line 944, in pyarrow.\_flight.FlightClient.authenticate File “pyarrow/\_flight.pyx”, line 68, in pyarrow.\_flight.check\_flight\_status pyarrow.\_flight.FlightUnavailableError: gRPC returned unavailable error, with message: Trying to connect an http1.x server We also created the new space for VDS, it still doesn’t work. Do you have any idea how to fix the issue? Thank you ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [November 13, 2019, 10:55am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/26 "2019-11-13T10:55:06Z")
Hey @smora just to confirm your flight connector is working now? As for further support for the flight plugin it is likely to only ever be for data transfer. We don’t foresee adding anything that is available via the REST api to the flight plugin. The reflection api is async by design and your approach is one of the more straightforward ways to find out if a reflection has refreshed. You may also use the REST api to query the status of a reflection. If you post your problem on a new community thread we may be able to assist further (reference me in the ticket) as its usually possible to pivot away from waiting for reflections to refresh by tweaking the problem definition. See: [https://www.dremio.com/lp/whitepaper-dremio-semantic-layer-best-practices](https://www.dremio.com/lp/whitepaper-dremio-semantic-layer-best-practices) for more details Best, Ryan ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [November 13, 2019, 11:00am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/27 "2019-11-13T11:00:55Z")
Hey @jackyjiang The error http1.x server appears to me it could be a problem w/ the connectivity between Dremio and the client. I have seen this before with nginx Ingress controllers on k8s. Check to make sure that `JAVA_EXTRA_OPTS="$JAVA_EXTRA_OPTS -Ddremio.flight.host=0.0.0.0"` is set, that you can connect to flight from localhost and that you can telnet to the flight port remotely. Also ensuire you are using dremio client 0.5.0 and the most recent version of the flight connector ---
**Author:** ![jackyjiang](https://avatars.discourse-cdn.com/v4/letter/j/e47774/32.png) [@jackyjiang](https://community.dremio.com/u/jackyjiang)\ **Post date:** [November 13, 2019, 4:16pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/28 "2019-11-13T16:16:22Z")
Hello @rymurr We just set up this in dremio-env DREMIO\_JAVA\_EXTRA\_OPTS="-Ddremio.flight.host=0.0.0.0" Actually, it is strange, after we set above parameters in all our nodes and rebooted them, it didn’t work at that time, and it suddenly works this morning. I am suspecting the enterprise admin did some change for our proxy. So, shall we still have to set the JAVA\_EXTRA\_OPTS in the server? Thank you ---
**Author:** ![smora](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@smora](https://community.dremio.com/u/smora)\ **Post date:** [November 13, 2019, 7:59pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/29 "2019-11-13T19:59:58Z")
Hi @rymurr confirmed, flight protocol is working in my environment, tested with latest ver of dremio\_client too. Thanks for your support. Few observations - UI shows job type as ODBC - not able cancel the query from UI. Ex: when transferring large data set to client, master node RAM gets filled up and client receives an exception, but the UI still shows as query running and no way to cancel it and it never completes. - possibly arrow bug with outer join queries Ex: select a\_col, b\_col from a left join b on a\_col = b\_col for unmatched row right side table col will be null, and recordBatch to arrowTable conversion fails with error, value out of range ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [November 14, 2019, 8:44am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/30 "2019-11-14T08:44:47Z")
Thanks for the very valuable feedback @smora !!! The first issue is a known one and will be fixed when flight is a permanent part of Dremio. The other two are new and I will take a look at them! We always welcome feedback like this! Best, Ryan ---
**Author:** ![smora](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@smora](https://community.dremio.com/u/smora)\ **Post date:** [February 10, 2020, 4:55pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/31 "2020-02-10T16:55:26Z")
Hi @rymurr wondering any further enhancements on flight connector? definitely one issue is memory build up on master node. even though reading in chunks, master seems to keep all data in memory, apparently reaching the limit and crashes. ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [February 11, 2020, 3:56pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/32 "2020-02-11T15:56:02Z")
Hey @smora I am still working on it and hope to have some improvements out soon, been a busy few months tho! ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [February 19, 2020, 8:55am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/33 "2020-02-19T08:55:44Z")
Hey @smora the new code pushed to github today has some fixes which should fix most of the memory issues. It should also fix the cancel issue. Note the new feature flag for flight (in prep for moving to the product) and that although parallel delivery is now present it isn’t functional yet until we push a patch to the oss repo. best, Ryan ---
**Author:** ![yogesh](https://avatars.discourse-cdn.com/v4/letter/y/49beb7/32.png) [@yogesh](https://community.dremio.com/u/yogesh)\ **Post date:** [February 27, 2020, 10:23am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/34 "2020-02-27T10:23:20Z")
Hey, I am new to this. I am running this locally on mac. After copying dremio-flight-connector jar to dremio’s jar folder, dremio is failed to start. it is giving error : ``` Dremio is exiting. Failure while starting services. java.lang.NoClassDefFoundError: org/apache/arrow/flight/auth/BasicServerAuthHandler$BasicAuthValidator at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.dremio.common.scanner.persistence.ScanResult.getImplementations(ScanResult.java:139) at com.dremio.service.InitializerRegistry.start(InitializerRegistry.java:55) at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137) at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:88) at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33) at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:184) at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:190) at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:104) Caused by: java.lang.ClassNotFoundException: org.apache.arrow.flight.auth.BasicServerAuthHandler$BasicAuthValidator at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ``` Could somebody please help me with this. ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [March 2, 2020, 8:10pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/35 "2020-03-02T20:10:10Z")
Hey @yogesh welcome to the site! Can I double check that you are using the `shaded` version of the `dremio-flight-connector` jar? Best, Ryan ---
**Author:** ![yogesh](https://avatars.discourse-cdn.com/v4/letter/y/49beb7/32.png) [@yogesh](https://community.dremio.com/u/yogesh)\ **Post date:** [March 3, 2020, 6:00am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/36 "2020-03-03T06:00:01Z")
Thanks @rymurr… I was using non shaded vesrion. With shaded version it is runing fine… Thanks for your quick help!!! ---
**Author:** ![smora](https://avatars.discourse-cdn.com/v4/letter/s/dec6dc/32.png) [@smora](https://community.dremio.com/u/smora)\ **Post date:** [March 6, 2020, 8:02pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/37 "2020-03-06T20:02:18Z")
Hi @rymurr thanks for the update, does this version of jar works with dremio ver 4.1.3? after adding new jar and > extraStartParams =“-Ddremio.flight.enable=true -Ddremio.flight.parallel.enable=true -Ddremio.flight.host=0.0.0.0” dremio-master log shows this message, am i missing something? > INFO com.dremio.flight.FlightInitializer - Flight plugin is not enabled, skipping initialization ---
**Author:** ![peschu](https://avatars.discourse-cdn.com/v4/letter/p/74df32/32.png) [@peschu](https://community.dremio.com/u/peschu)\ **Post date:** [March 6, 2020, 9:49pm UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/38 "2020-03-06T21:49:44Z")
Hi together, in the [article](https://www.dremio.com/is-time-to-replace-odbc-jdbc/) is only a benchmark/comparison of durations for queries. Would be nice to also have a benchmark which takes cpu load and memory consumption into account. That would make the decision easier depending on the setup/use-case… KR, Peter ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [March 9, 2020, 9:39am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/39 "2020-03-09T09:39:46Z")
Hey @smora, I have added a new flag: [https://github.com/dremio-hub/dremio-flight-connector#enable-flight](https://github.com/dremio-hub/dremio-flight-connector#enable-flight) in preparation for integrating the flight connector into the product. Apologies for the change, hope it isn’t too much trouble. Thanks again for all your testing! ---
**Author:** ![rymurr](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rymurr/32/1739_2.png) [@rymurr](https://community.dremio.com/u/rymurr)\ **Post date:** [March 9, 2020, 9:42am UTC](https://community.dremio.com/t/arrow-flight-protocol/3855/40 "2020-03-09T09:42:51Z")
Hey @peschu, thanks for the suggestion. I will take a look at preparing these comparisons. My gut instinct is that there will not be a huge difference on the Dremio side and savings on the client side will be application specific: converting to pandas via flight will require much lower memory for example compared to odbc. [Previous page](https://community.dremio.com/t/arrow-flight-protocol/3855.md?page=1) [Next page](https://community.dremio.com/t/arrow-flight-protocol/3855.md?page=3)