Dremio Apache Arrow writer using Java

1

Hi Team ,
I am trying to write data into tabular format : CSV file after processing the data retrieved from dremio using apache arrow flight connector using JAVA .

How can I write(batch) the processed data using apache arrow in tabular format - CSV (or parquet file) using JAVA. Plan is to store these data to azure data storage further post processing .

Could you please help.

Thanks,
Sunil

2

For CSV, VectorSchemaRoot has a method to generate a TSV string, convertToTSVString(). This doesn’t seem very efficient to store back though.

For parquet, there are examples in C++ and Python in the Arrow documentation:
https://arrow.apache.org/docs/python/parquet.html
https://arrow.apache.org/docs/cpp/parquet.html

Dremio itself lets you download results as Parquet or CSV files in its web UI.

1 Like
3

Thanks for response. Is there any java package to write dremio data into parquet file using flight connector client.