Hi, Our company try to use Dremio on our production system and currently testing arrow flight data transfer. (One of the reasons we choose Dremio out of many data connector)
I might set up something wrong but when I read 500 mb data(1,802,900 rows, 9 columns), both arrow flight connection and odbc connection take around 45 secs.
I’m expecting arrow flight connection would be a lot faster than odbc connection but apparently, there is no big difference.
I copied code from python arrow flight example in dremio git hub page
The data source is on s3 and file format is csv file.
I tested different file sizes and all show a similar pattern.
My questions are
- Is this performance expected because I used csv file?
- Should I use parquet file to take advantage of arrow flight?