Hi,
Thanks for very interesting product.
I have a question - can I get somehow the same performance with Power BI or Qlik integration as I have with their internal columnar storages?
My setup is Windows Dremio.
Am using sample SF incidents dataset (which is very small, it is about 31 MB uncompressed).
Both reflections (raw and cube) are enabled. Sorting and partitions are not enabled, could not found the documentation for these options.
I tested the performance in Power BI (load and direct) and Qliksense - direct.
Performance is much slower for direct connections, then it would be for loaded data.
Load model in Power BI works fasts (because data is loaded into internal columnar storage of SSAS), but refresh time is very slow thru ODBC. I think that MySQL table with imported dataset works faster, without any optimizations.
When I test Apache Arrow implementation for Python/Pandas (called Feather) - read and write speeds are incredible. Feather writes dataframes at the speed approx 250MB/sec and reads at approx 550MB/sec on my pc with average specs. Basically, i can use the SSD as RAM extension for Pandas, especially if i switch to these newish SSDs with 3200 MB/sec support.
For example, I can read/write SF incidents file to/from disk in fraction of second, using Feather.
Qlik format, called qvd also works great for reads/writes (it similar to Feather, but compressed, thus a little bit slower). I can load csv to Qlik, then export it to compressed qvd and after that refreshes will be almost instant.
I understand, that Dremio for Windows is not intended for production. But assuming I have a powerful Linux machine(s) and 10 gbit network to Windows machine with Tableau, Power BI, Qliksense or Pandas - will I ever achieve the speed of Feather reads/writes?
In another words, there are three great file formats/technologies - Qlik QVD, Arrow (Feather) and internal columnar storages in BI tools, which work with huge speed on my machine.
Is there any way/setup/approach to achieve the same speed with Dremio-BI integration?
Thanks for your opinions and suggestions!
.