Aggregate Reflection not used

I have an Oracle database table with 20 million rows. The following simple aggregation takes 9 seconds to run:

select object_type, count() nobjs from big_table group by object_type order by count() desc

I created a reflection but it is not used:

Query and Planning

Query
Visualized Plan
Planning
Acceleration
Reflection Outcome

Query was NOT accelerated

b7ac073f-2559-4aea-aa1b-ac3a378bc536 (agg): considered, not matched.
Time To Find Reflections: 0 ms
Time To Canonicalize: 2 ms
Time To Match: 0 ms

I can upload a profile if required.

HI @druthven,

Happy to help. What version of Dremio are you using? Can you please upload the profile?

Thanks,
@balaji.ramaswamy

Latest community edition downloaded yesterday.

Have uploaded the profile.

-David

b8046d8a-9d92-4d2e-bc25-803810281a96.zip (3.6 KB)

Hi @druthven

Can you please send me the profile for the reflection you created?

Thanks
@balaji.ramaswamy

How do I send a profile for the reflection itself?

Attached is the profile for the aggregate reflection.da9cd178-bbdf-45ec-b74d-c2f0d9f27574.zip (4.2 KB)

Thanks @druthven, let me try this on my local Oracle Database and get back to you. I will create the reflection on the below SQL and get back to you on the acceleration result

SELECT  OBJECT_TYPE object_type,
        COUNT(1) nobjs
FROM    DELPHIXDB.BIG_TABLE
GROUP   BY OBJECT_TYPE

Thanks,
@balaji.ramaswamy

Here is how I created the table:

drop table big_table;

create table big_table
as
select rownum id,
OWNER, OBJECT_NAME, SUBOBJECT_NAME,
OBJECT_ID, DATA_OBJECT_ID,
OBJECT_TYPE, CREATED, LAST_DDL_TIME,
TIMESTAMP, STATUS, TEMPORARY,
GENERATED, SECONDARY
from all_objects a
where 1=0
/
alter table big_table nologging;

declare
l_cnt number;
l_rows number := &1;
begin
insert /*+ append */
into big_table
select rownum,
OWNER, OBJECT_NAME, SUBOBJECT_NAME,
OBJECT_ID, DATA_OBJECT_ID,
OBJECT_TYPE, CREATED, LAST_DDL_TIME,
TIMESTAMP, STATUS, TEMPORARY,
GENERATED, SECONDARY
from all_objects a
where rownum <= &1;

l_cnt := sql%rowcount;

commit;

while (l_cnt < l_rows)
loop
    insert /*+ APPEND */ into big_table
    select rownum+l_cnt,
           OWNER, OBJECT_NAME, SUBOBJECT_NAME,
           OBJECT_ID, DATA_OBJECT_ID,
           OBJECT_TYPE, CREATED, LAST_DDL_TIME,
           TIMESTAMP, STATUS, TEMPORARY,
           GENERATED, SECONDARY
      from big_table
     where rownum <= l_rows-l_cnt;
    l_cnt := l_cnt + sql%rowcount;
    commit;
end loop;

end;
/

alter table big_table add constraint
big_table_pk primary key(id)
/

begin
dbms_stats.gather_table_stats
( ownname => user,
tabname => ‘BIG_TABLE’,
cascade => TRUE );
end;
/
select count(*) from big_table;

select object_type, count() from big_table group by object_type order by count() desc;

Hi @druthven,

I was able to create the exact same table on Oracle 11GR2 but the acceleration seems to work fine for me. The only difference is I am on the Enterprise Edition. I am also going to try the same on the Community Edition. Please see attached file that has the SQL statements I ran inside of Oracle. I have also attached 2 screenshots

  1. Create-Agg-Reflection - Show the Agg Reflection columns I used
  2. Query-Accelarated-View - Screen shot from the job profile that says Query was accelarated

The query I ran on Dremio is below,

select object_type, count(object_name) from "Oracle-XE ".OPERATIONAL.big_table group by object_type order by count(object_name) desc

Will try the same on Community Edition and let you know the results

Thanks
@balaji.ramaswamy

Your query is slightly different to mine, mine uses ‘count(*)’ where yours uses ‘count(object_name)’. Anyway I tried your query. My aggregation screen does not offer the same options:

So I cannot add the measure ‘object_name’ the only choice is ‘EXPR$1’

Does this work with the enterprise edition? If so, is that the only method to test dremio with Oracle sources?

Hi @druthven,

Let me get back to you on this. I will have the steps done on the community edition

Thanks,
@balaji.ramaswamy

Hi @druthven

I was able to quickly test this on community edition and acceleration works as expected. Please see screenshots attached. For my example, BIG_TABLE has “100,000” rows

When you click on “Aggregate Reflections”, does it show all columns of “BIG_TABLE”?

Thanks,
@balaji.ramaswamyQueryAccelaration

When you click on “Aggregate Reflections”, does it show all columns of “BIG_TABLE”?
No, is there a new version of the community edition? I am running 1.2.2-201710100154510864-d40e3

Hi @druthven,

That should not change, Have you created the VDS on the entire dataset “select * from big_table” or on the aggregate query?

Thanks,
@balaji.ramaswamy

I only used the aggregate query. Can you send me an explicit list of the steps you took?

Oracle backend steps remain same

Step 1: On the Dremio UI, click on the Oracle Data Source-Schema-Table. This would run a select * on the table like below

SELECT * FROM "DELPHIXDB".BIG_TABLE

Step 2: Save the result set as a Virtual Data Set (VDS) by clicking “Save As” and give a name like “big_table”

Step 3: Click on the settings of the VDS and click on reflections

Step 4: Enable Aggregate reflections

Step 5: From the table columns listed on the left drag object_type to the “Dimensions” section

Step 6: From the table columns listed on the left drag object_id to the “Measures” section and Save the reflection

Step 7: Click on a new query and run the aggregate query of the VDS “big_table”

Step 8: Open the profile of the jib and see if query is accelerated

If you gt stuck, let me know and I can send you a screen recording

Thanks,
@balaji.ramaswamy

That worked, thank you. It generated the following SQL when creating the aggregation accelerator:

SELECT “OBJECT_TYPE”, SUM(“OBJECT_ID”) “agg-4-0”, COUNT(“OBJECT_ID”) “agg-4-1”, MAX(“OBJECT_ID”) “agg-4-2”, MIN(“OBJECT_ID”) “agg-4-3”, COUNT(1) "CONVERT_COUNT_STAR"
FROM “DELPHIXDB”."BIG_TABLE"
GROUP BY “OBJECT_TYPE”

I assume that query was used to create the materialized view? How can I check the contents of the materialized view? The following query

SELECT * FROM “__materialization”.“e9dc52cb-6dbd-458e-8741-7adfef5c2837”

gets: “Table ‘__materialization.e9dc52cb-6dbd-458e-8741-7adfef5c2837’ not found”

There may be a way to do this, I’m unsure and will defer to one of my colleagues.

We haven’t really designed Dremio for reflections to be user accessible. Can you say more about what you’re trying to accomplish? Or are you simply trying to verify the contents of the reflection?

If you look at your job history you can see whether the query was accelerated by the presence of the flame:

image

As you noted in your original post in this thread, you can see whether it was accelerated by looking at the query profile. If the query was accelerated then the reflection was used to generate the results. I suppose you can verify the contents of the reflection by checking the results of accelerated queries.