When i am trying to execute CTAS query from Dremio UI, it is working fine as expected and parquet file is getting distributed properly in different chunks.
The same query when i am trying to execute it from Python is getting failed, One workaround i was able to do that. After applying limit to select query of CTAS, it works. but it generates only single parquet file.
I have tried using pyodbc & pyarrow, same results when trying to run in from python end.
- Need a proper python package/library which can execute same as dremio ui from python end.
- pyarrow fails if query time goes more then 10 minutes.
Note - Cant provide the SQL Query.
Thanks in advance.