Hi,
Here is the screenshot of the visualized (query) plan for a simple select * from table (table = hive external table built on parquet files on s3)
Can someone direct me to the documentation of the visualized plan?
Also, can someone explain what is the Writer, Union Exchange and Write Committer stages? And why are they in a simple SELECT * FROM query?
Currently we do not have any specific documentation on the visualized plan but to answer your question, the 3 phases you described as the phases to prepare to write to screen. In other words before we write to screen (UI), we need to write these into temporary locations in ARROW format. If you do the same SQL via a JDBC call, you will not see these, for example see the below query, it is a simple select * from employees in Oracle,
SELECT * FROM oracle_docker.HR.EMPLOYEES
See the attached visualized plans for the Dremio UI query and one from JDBC (I used DBeaver)
In general,
writer is the arrow writer that’s writing the query results to disk
union exchange just merges the result
writer committer is a way for the query to only commit the results once the query succeeds otherwise it would clean up after itself
Kindly let me know if you have any additional questions
