Inserting data into Iceberg table using Arrow Flight client

1

Hello,

Is it possible to insert (or append) data to an Apache Iceberg table using an Arrow Flight client and Dremio?

For example, could an Arrow Flight client send a stream of record batches as part of a DoPut operation that ends up getting written to a Dremio data source?

Thanks!

2

yes if you are using iceberg tables you can make insert, update or merge sentences

3

I have just attempt to use the “DoPut” operation to write a RecordBatch, but it seems like its unsupported.
The server returns the following message:

‘Status(StatusCode=“Unimplemented”, Detail=“acceptPut is not implemented.”)’

4

@Nisden What are you exactly trying to do? Are you trying to insert into an Iceberg table?

5

Yes, we are trying to insert a large amount of records into an Iceberg table

6

@Nisden Just to narrow down the issue. Can you please try to run it as a pplain insert via the Dremio UI?

7

Yah, I got no issues using a regular “INSERT INTO” command. Its just the Arrow Flight Bulk insert endpoint that is throwing Unimplemented errors.

8

Hello,

I am facing similar issue using flight-sql-jdbc-driver-17.0.0.jar as recommended here

My need is to request Dremio from a Java program.
I haven"t found any mention of such limitations in the doc.

Thanks for your help

9

@lbourgeois @Nisden Are you able to send the profile which would tell us exactly where the error is happening? If no job found then it is failing before it hits dremio

10

For now we found a workaround by using hive-jdbc-3.1.3-standalone.jar driver
Stacktrace I can share with Arrow Flight driver is

[FATAL] 09:13:28 etl_uat.iceberg_components_0_2.iceberg_components- tIcebergOutput_1 acceptPut is not implemented.
java.sql.SQLException: acceptPut is not implemented.
at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.Helper.createException(Helper.java:56) ~[flight-sql-jdbc-driver-17.0.0.jar:17.0.0]
at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.Helper.createException(Helper.java:41) ~[flight-sql-jdbc-driver-17.0.0.jar:17.0.0]
at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:559) ~[flight-sql-jdbc-driver-17.0.0.jar:17.0.0]
at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.AvaticaPreparedStatement.executeLargeUpdate(AvaticaPreparedStatement.java:152) ~[flight-sql-jdbc-driver-17.0.0.jar:17.0.0]
at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.AvaticaPreparedStatement.executeUpdate(AvaticaPreparedStatement.java:147) ~[flight-sql-jdbc-driver-17.0.0.jar:17.0.0]
at org.talend.bigdata.core.di.components.iceberg.TIcebergOutput.write(TIcebergOutput.java:55) ~[talend-bigdata-core-di-1.12.0.jar:?]
at etl_uat.iceberg_components_0_2.iceberg_components.tFileInputDelimited_1Process(iceberg_components.java:1548) [classes/:?]
at etl_uat.iceberg_components_0_2.iceberg_components.runJobInTOS(iceberg_components.java:2967) [classes/:?]
at etl_uat.iceberg_components_0_2.iceberg_components.main(iceberg_components.java:2583) [classes/:?]