A
B
executing both does not work.
Have to comment/uncomment/wait etc. What is the flow?
This is expected behavior. Btw you have the ability to chain various VDS together if you want to develop more of a flow where one query is based on the results of another.
Yes, want flow. How do you run both? Do you have multiple windows open? Do you use a REPL with the API. What do you do to be productive with Dremio?
You might want to checkout something like dbeaver:
https://docs.dremio.com/client-applications/dbeaver.html
That may be more inline with the sort of flow you are looking for.
A flow that works for me is to use the Dremio UI to add sources, do the initial data exploration, and then some transformations/cleansing of the data. Then move to dbeaver for further analysis.
The IO is too bad tbh.
In my view Dremio is a line of defense against non-technical folks who want to have a look at a small view of data.
It is not where you explore data, fix it, do ML etc. There should always be an out … once the parquet dumping is fixed it could be a good initial connector layer that prepares HDFS fast-read, partitioned collections.
I think it is good to distinguish between Dremio’s web app and the experience that provides, vs Dremio as the query engine that executes queries from different tools like Tableau and Python.
Our goal is that Dremio provides the fastest and easiest way to analyze your data, no matter the source, and no matter the end user tool. I agree ML is going to be executed in a specific runtime environment like Python or the JVM. But that process would still read data through Dremio into memory for processing, which is why Arrow is so important and core to our engine. See Arrow Flight for more on where we are headed in this area.
I also agree we have a lot more we can do to make our SQL console world class. Having a searchable catalog, data lineage features, and the “last mile” ETL features in the same environment is very useful for many users.