hi @rymurr
i can confirm that issue is when connecting to flight service running inside a k8 cluster.
i am able to connect to it if i run a container inside k8 cluster and by using explicit pod ip.
even though dremio-master pod/port is exposed, not able to connect to flight service from outside the cluster. Any specific protocol i should be using when exposing the 47470 port and flight service ?
Hi Both,
I have just released a new version of dremio_client (0.5.0) [1]. I hope this fixes the auth issue. Please give it a try and let me know how it goes.
As for the k8s port. The flight port is tcp 47470 and is a http/2 service. Nothing fancy so normal methods to expose in k8s should work fine. And have in the past for me. See [2] for how we currently expose Dremio ports
[1] https://github.com/rymurr/dremio_client
[2] https://github.com/dremio/dremio-cloud-tools
1 Like
hi @rymurr
confirmed, i am able to use flight service. i am using 0.4.3 dremio_client, will test it out with 0.5.0.
wrt to k8, issue might be related to certificates, after using ca signed cert’s ingress controller is resolving the request correctly.
@rymurr
any plans to add refresh reflection api support to this client ? dremio api refresh call is asynchronous and currently i am using custom code to trigger a refresh and query sysobj tables to figure out reflection refresh is complete based on a new row with new materialization id to be populated(not perfect but works most of the cases)
Hello, Rymurr
We have the same issue
Traceback (most recent call last):
File “DremioFlight.py”, line 28, in
client.authenticate(HttpDremioClientAuthHandler(username, password))
File “pyarrow/_flight.pyx”, line 944, in pyarrow._flight.FlightClient.authenticate
File “pyarrow/_flight.pyx”, line 68, in pyarrow._flight.check_flight_status
pyarrow._flight.FlightUnavailableError: gRPC returned unavailable error, with message: Trying to connect an http1.x server
We also created the new space for VDS, it still doesn’t work. Do you have any idea how to fix the issue? Thank you
Hey @smora just to confirm your flight connector is working now?
As for further support for the flight plugin it is likely to only ever be for data transfer. We don’t foresee adding anything that is available via the REST api to the flight plugin.
The reflection api is async by design and your approach is one of the more straightforward ways to find out if a reflection has refreshed. You may also use the REST api to query the status of a reflection. If you post your problem on a new community thread we may be able to assist further (reference me in the ticket) as its usually possible to pivot away from waiting for reflections to refresh by tweaking the problem definition. See: https://www.dremio.com/lp/whitepaper-dremio-semantic-layer-best-practices for more details
Best,
Ryan
Hey @jackyjiang
The error http1.x server appears to me it could be a problem w/ the connectivity between Dremio and the client. I have seen this before with nginx Ingress controllers on k8s. Check to make sure that JAVA_EXTRA_OPTS="$JAVA_EXTRA_OPTS -Ddremio.flight.host=0.0.0.0" is set, that you can connect to flight from localhost and that you can telnet to the flight port remotely. Also ensuire you are using dremio client 0.5.0 and the most recent version of the flight connector
Hello @rymurr
We just set up this in dremio-env
DREMIO_JAVA_EXTRA_OPTS="-Ddremio.flight.host=0.0.0.0"
Actually, it is strange, after we set above parameters in all our nodes and rebooted them, it didn’t work at that time, and it suddenly works this morning. I am suspecting the enterprise admin did some change for our proxy.
So, shall we still have to set the JAVA_EXTRA_OPTS in the server?
Thank you
Hi @rymurr
confirmed, flight protocol is working in my environment, tested with latest ver of dremio_client too. Thanks for your support.
Few observations
-
UI shows job type as ODBC
-
not able cancel the query from UI.
Ex: when transferring large data set to client, master node RAM gets filled up and client receives an exception, but the UI still shows as query running and no way to cancel it and it never completes. -
possibly arrow bug with outer join queries
Ex: select a_col, b_col from a left join b on a_col = b_col
for unmatched row right side table col will be null, and recordBatch to arrowTable conversion fails with error, value out of range
1 Like
Thanks for the very valuable feedback @smora !!!
The first issue is a known one and will be fixed when flight is a permanent part of Dremio. The other two are new and I will take a look at them!
We always welcome feedback like this!
Best,
Ryan
1 Like
Hi @rymurr
wondering any further enhancements on flight connector?
definitely one issue is memory build up on master node. even though reading in chunks, master seems to keep all data in memory, apparently reaching the limit and crashes.
Hey @smora I am still working on it and hope to have some improvements out soon, been a busy few months tho!
Hey @smora the new code pushed to github today has some fixes which should fix most of the memory issues. It should also fix the cancel issue.
Note the new feature flag for flight (in prep for moving to the product) and that although parallel delivery is now present it isn’t functional yet until we push a patch to the oss repo.
best,
Ryan
Hey, I am new to this.
I am running this locally on mac.
After copying dremio-flight-connector jar to dremio’s jar folder, dremio is failed to start.
it is giving error :
Dremio is exiting. Failure while starting services.
java.lang.NoClassDefFoundError: org/apache/arrow/flight/auth/BasicServerAuthHandler$BasicAuthValidator
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.dremio.common.scanner.persistence.ScanResult.getImplementations(ScanResult.java:139)
at com.dremio.service.InitializerRegistry.start(InitializerRegistry.java:55)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:88)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:184)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:190)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:104)
Caused by: java.lang.ClassNotFoundException: org.apache.arrow.flight.auth.BasicServerAuthHandler$BasicAuthValidator
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
Could somebody please help me with this.
Hey @yogesh welcome to the site!
Can I double check that you are using the shaded version of the dremio-flight-connector jar?
Best,
Ryan
Thanks @rymurr… I was using non shaded vesrion. With shaded version it is runing fine… Thanks for your quick help!!!
Hi @rymurr thanks for the update, does this version of jar works with dremio ver 4.1.3?
after adding new jar and
extraStartParams =“-Ddremio.flight.enable=true -Ddremio.flight.parallel.enable=true -Ddremio.flight.host=0.0.0.0”
dremio-master log shows this message, am i missing something?
INFO com.dremio.flight.FlightInitializer - Flight plugin is not enabled, skipping initialization
Hi together,
in the article is only a benchmark/comparison of durations for queries. Would be nice to also have a benchmark which takes cpu load and memory consumption into account.
That would make the decision easier depending on the setup/use-case…
KR,
Peter
Hey @smora,
I have added a new flag: https://github.com/dremio-hub/dremio-flight-connector#enable-flight in preparation for integrating the flight connector into the product. Apologies for the change, hope it isn’t too much trouble. Thanks again for all your testing!
Hey @peschu,
thanks for the suggestion. I will take a look at preparing these comparisons. My gut instinct is that there will not be a huge difference on the Dremio side and savings on the client side will be application specific: converting to pandas via flight will require much lower memory for example compared to odbc.