Error while creating reflections in Dremio OSS 25.0

I have built dremio oss 25.0 from source code as per the instructions in the GitHub README. I have used the flag -Ddremio.oss.only=true since i want OSS instead of community edition. The full command i used was. ./mvnw install -DskipTests -Ddremio.oss-only=true -Dlicense.skip=true. After building and running dremio i created a Hive 3.x source to connect to my iceberg tables that i have written using Hive catalog of iceberg. However i am hitting an error that says The option 'reflections.planning.exclude.file_based_incremental.iceberg.accelerations' does not exist.

I tried using the publicly available community image of dremio 25. Here i am able to create reflections. On further debugging i can see this property is available as a support key in community edition whereras it is absent in the dremio oss i built. Did i miss something while building? Please let me know what i can do to resolve this

@sanchitsk Your community post title is unable to create reflections, So you are able to query the table but only hitting this error when creating a reflection on top of it. Are you able to run the below query and send the profile

select now()

@balaji.ramaswamy I have sent you both the query profiles for the select now() and that of the failed reflection refresh. Like you mentioned i am able to run queries but not able to create reflections on top of iceberg tables

@balaji.ramaswamy : Any update on this ?

I found the bug and this problem only affects OSS.

Since you can modify and rebuild the files.

Add the following:

  BooleanValidator EXCLUDE_INCREMENTAL_ICEBERG =
      new BooleanValidator(
          "reflections.planning.exclude.file_based_incremental.iceberg.accelerations", true);

To:

Rebuild that jar and you should be good. The issue is that the option cannot be found here: dremio-oss/services/accelerator/src/main/java/com/dremio/service/reflection/refresh/RefreshHandler.java at master · dremio/dremio-oss · GitHub

I’ve logged a bug internally to fix this.

Thanks a lot @Benny_Chow. That worked!.

1 Like