How work with branch using DBT + Dremio + Nessie

1

@balaji.ramaswamy I have a project where I use DBT + Dremio and Nessie. All my tables are created in branch “main” on Nessie. How I do to use other branch using Dbt?

1 Like
2

if you want select table with branch use select * from <source>.<table or view > AT BRANCH <branchname>, Currently ,dbt-dremio not support with branch directly, but you can defination one
custom macro with branch like this

{%- macro refv2(schema,table,branch ) -%}
    {%- set relation = api.Relation.create(schema=schema,identifier=table)  -%}
    {{return (relation ~ ' AT ' ~  'branch ' ~ branch)  }}
{%- endmacro %}

when using with dbt model , you can write like this

select * from  {{refv2("dbtv4","dalongdemoapp","prod")}};

dbt will compile like this

select * from  "dbtv4"."dalongdemoapp" AT branch prod;

maybe need more custom macros for better support nessie branch feature

1 Like
3

I need that dbt create new table in the my branch. I can read from many branchs, but I can’t create a new table in my custom branch. Help me?

4

@Ferrarini the same as source and ref macro . if you wan’t create table in custom branch . write one custom macro for create_table_as , Currently dbt for dremio adapter not support with branch
you can try dbt/include/dremio/macros/materializations/table/create_table_as.sql this and modify it add branch support. for dbt project use dbt dispatch https://docs.getdbt.com/reference/dbt-jinja-functions/dispatch with your own create_table_as macro

dremio nessie data source create table with branch can like this CREATE table dbtv4.myappv5 AT BRANCH prod as select * from pg.public.sensor

5

Hi, @rongfengliang!

Thanks for posting this snippet. Very helpful to quickly solve the incompatibility issue with Nessie.

Found out that the issue was recently filed as a bug against the adapter: [Bug]: Validation of view sql failed. Version context for table nessie.names must be specified using AT SQL syntax · Issue #241 · dremio/dbt-dremio · GitHub

Thanks, Tim