As far as I know, data scientist often exports data into csv file, then uses it to train his model
However dremio only supports to export a million records, so I just wonder what is the way dremio proposes to support data scientists
Data scientists can query Dremio over ODBC/REST/JDBC today, and via Arrow Flight in the future. There is no limit on dataset size using these methods. Our philosophy is that users shouldn’t need to download copies of data - this creates many governance-related risks.
Here’s a quick example in Jupyter:
We are considering an option for downloading as Feather (based on Arrow) which would be far better than CSV in terms of data access speed and memory efficiency.