Dremio displays incorrect Rows Returned count while query is running

1

While running external queries to create tables, the row counts displayed in the dremio UI “Jobs” tab are wrong, while the query is running. For example, loading one table it will display 34M “Rows Returned” in the UI, but when the table finishes the number jumps down to 8.7M. This appears to be that because the UI is taking records loaded / total process time (8.7M / .252. = 34.5M), but I could be wrong. My rule of thumb to check on progress is to divide the displayed number by 4 and that seems to hold true.

image
image935×249 30.8 KB

In this screenshot, the 3rd job stitch_landing.stripe.events should only have 29M rows total. It went all the way up to 119M instead. Then when finished, it showed the correct row count.
image
image915×216 26 KB

I suspect it’s displaying the record processing rate for some reason while the query is running.

image
image1512×248 12.5 KB

Query profile is attached for stitch_landing.stripe.events.

bd0c99b6-1e0a-49c1-94da-551d80c4bfb8.zip (23.1 KB)

2

@alex_a Since you are doing a CTAS, the PARQUET_WRITER (after the job is complete) tell us the # of records written which is 29,754,268 for nessie."stitch_landing"."stripe"."events"

Open the job id 1b1c6cfd-4e07-6165-84ed-387c82e29800 profile and expand PARQUET_WRITER 00-xx-05 and you should see the number

When the job is running the count can keep changing (I assume you are looking into the job profile?)