Getting error while connecting Dremio to Spark through JDBC Driver for Arrow Flight SQL

gakshat1107 · September 13, 2024, 7:28am

Hi,

I’m following the link to establish a connection with Dremio:JDBC Driver for Arrow Flight SQL | Dremio Documentation
I’m connecting to spark while giving the jar for apache flight : flight-sql-jdbc-driver-16.1.0.jar
Spark version : 3.4.3
Java version : jdk-1.8
Using the below code to connect to Dremio:

val connectionProperties = new java.util.Properties()
connectionProperties.setProperty(“user”, username)
connectionProperties.setProperty(“password”, password)
connectionProperties.setProperty(“driver”, “org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver”)
val jdbcUrl = “jdbc:arrow-flight-sql://10.61.196.**:32010/?useEncryption%3Dfalse&disableCertificateVerification%3Dtrue&useSSL%3Dfalse”
spark.read.jdbc(jdbcUrl, tableName_nessie_bronze, connectionProperties).show()

Getting the below error:

Caused by: org.apache.arrow.driver.jdbc.shaded.io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 00001204000000000000037fffffff000400100000000600002000000004080000000000000f00010000860700000000007fffffff00000001485454502f3220636c69656e74207072656661636520737472696e67206d697373696e67206f7220636f72727570742e204865782064756d7020666f722072656365697665642062797465733a20313630333033303132643031303030313239303330336235396538306634303464313130643435366332346339353363

    at org.apache.arrow.driver.jdbc.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1314)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at org.apache.arrow.driver.jdbc.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:750)

24/09/13 11:19:50 ERROR ClientHandshakeWrapper: Failed with unknown
org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.flight.FlightRuntimeException: UNAVAILABLE: io exception

I’ve followed all the steps even then I’m getting the error, am I missing something?

gakshat1107 · September 13, 2024, 10:06am

Hi,

I updated the Java version to jdk-11
I updated the JDBC Url to : jdbc:arrow-flight-sql://10.61.196.**:32010/?useEncryption=false&disableCertificateVerification=true&SSL=false
Now I’m getting the below error:

spark.read.jdbc(jdbcUrl, tableName_nessie_bronze, connectionProperties).show()
java.sql.SQLException: while preparing SQL: SELECT * FROM “NessieCatalog”.“Bronze”.“drembronzetable” WHERE 1=0
at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:361)
at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:178)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.getQueryOutputSchema(JDBCRDD.scala:65)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:58)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation$.getSchema(JDBCRelation.scala:241)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:37)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:346)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:229)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:211)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:172)
at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:249)
… 47 elided
Caused by: org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.flight.FlightRuntimeException:
at org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.flight.CallStatus.toRuntimeException(CallStatus.java:131)
at org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.flight.grpc.StatusUtils.fromGrpcRuntimeException(StatusUtils.java:165)
at org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.flight.grpc.StatusUtils$1.next(StatusUtils.java:251)
at org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.flight.sql.FlightSqlClient$PreparedStatement.(FlightSqlClient.java:959)
at org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.flight.sql.FlightSqlClient.prepare(FlightSqlClient.java:734)
at org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.flight.sql.FlightSqlClient.prepare(FlightSqlClient.java:714)
at org.apache.arrow.driver.jdbc.client.ArrowFlightSqlClientHandler.prepare(ArrowFlightSqlClientHandler.java:258)
at org.apache.arrow.driver.jdbc.ArrowFlightMetaImpl.prepareForHandle(ArrowFlightMetaImpl.java:165)
at org.apache.arrow.driver.jdbc.ArrowFlightMetaImpl.prepare(ArrowFlightMetaImpl.java:176)
at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:357)
… 59 more

gakshat1107 · September 13, 2024, 11:54am

Hi,

I changed my java version from 1.8 to 1.11 and then restarted my spark cluster, now it’s working fine.

balaji.ramaswamy · September 14, 2024, 12:05am

Thanks for updating the steps @gakshat1107 as this will be super useful for someone who has the same issue

Topic		Replies	Views
I can't connect on DBeaver	3	1676	March 21, 2023
Reg: Apache Arrow flight Sql JDBC	5	1506	January 3, 2023
How to use Arrow Flight to connect to Dremio	4	3898	December 18, 2020
Connections to Dremio Dremio Cloud	4	1125	February 5, 2024
Issue with SSL connection from Docker	13	4293	May 4, 2023

Getting error while connecting Dremio to Spark through JDBC Driver for Arrow Flight SQL

Related topics