While running external queries to create tables, the row counts displayed in the dremio UI “Jobs” tab are wrong, while the query is running. For example, loading one table it will display 34M “Rows Returned” in the UI, but when the table finishes the number jumps down to 8.7M. This appears to be that because the UI is taking records loaded / total process time (8.7M / .252. = 34.5M), but I could be wrong. My rule of thumb to check on progress is to divide the displayed number by 4 and that seems to hold true.
In this screenshot, the 3rd job stitch_landing.stripe.events should only have 29M rows total. It went all the way up to 119M instead. Then when finished, it showed the correct row count.
I suspect it’s displaying the record processing rate for some reason while the query is running.
Query profile is attached for stitch_landing.stripe.events.
bd0c99b6-1e0a-49c1-94da-551d80c4bfb8.zip (23.1 KB)