Dbt-dremio - 'dict object' has no attribute 'CompilationError'

Ken · May 25, 2024, 5:44pm

I raised an issue regarding dbt-dremio.

[Bug]: 'dict object' has no attribute 'CompilationError' for incremental + merge strategy

opened 05:34PM - 25 May 24 UTC

bug

### Is there an existing issue for this? - [X] I have searched the existing i…ssues ### Current Behavior I have the below model which illustrates the error ``` {{ config( enabled=True, materialized='incremental', incremental_strategy='merge' ) }} {% if is_incremental() %} -- get few more days for lag() function select 2 as id, 99999 as number {% else %} select 1 as id, 1 as number UNION ALL select 2 as id, 2 as number UNION ALL select 3 as id, 3 as number {% endif %} ``` I got the below error ``` 17:30:12 Compilation Error in model delete_me (models/l1-hk-trading/rollup/delete_me.sql) 'dict object' has no attribute 'CompilationError' > in macro materialization_incremental_dremio (macros/materializations/incremental/incremental.sql) > called by model delete_me (models/l1-hk-trading/rollup/delete_me.sql) ``` However, I note that, if I switch to the default `append` strategy, the error does not appear. i.e. the below model works ``` {{ config( enabled=True, materialized='incremental', incremental_strategy='append' ) }} {% if is_incremental() %} -- get few more days for lag() function select 2 as id, 99999 as number {% else %} select 1 as id, 1 as number UNION ALL select 2 as id, 2 as number UNION ALL select 3 as id, 3 as number {% endif %} ``` ### Expected Behavior It should not throw Compilation error ``` 'dict object' has no attribute 'CompilationError' ``` ### Steps To Reproduce This is my dremio build. OSS dremio running in kubernetes ![image_2024_05_25T17_35_35_749Z](https://github.com/dremio/dbt-dremio/assets/57944769/81118eee-7913-4327-9fc3-15f830d8b258) ### Environment ```markdown - OS: - dbt-dremio: - Dremio Software: - Dremio Cloud: ``` ### Relevant log output ```shell dbt log (.venv) ➜ dremio git:(master) ✗ dbt run --select ./models/l1-hk-trading/rollup/delete_me.sql 17:30:07 Running with dbt=1.7.13 17:30:07 Registered adapter: dremio=1.7.0 17:30:08 Found 62 models, 7 seeds, 21 tests, 44 sources, 0 exposures, 0 metrics, 455 macros, 0 groups, 0 semantic models 17:30:08 17:30:10 Concurrency: 5 threads (target='prod') 17:30:10 17:30:10 1 of 1 START sql incremental model Minio.finance.l1.delete_me .................. [RUN] 17:30:12 1 of 1 ERROR creating sql incremental model Minio.finance.l1.delete_me ......... [ERROR in 2.34s] 17:30:12 17:30:12 Finished running 1 incremental model in 0 hours 0 minutes and 4.42 seconds (4.42s). 17:30:12 17:30:12 Completed with 1 error and 0 warnings: 17:30:12 17:30:12 Compilation Error in model delete_me (models/l1-hk-trading/rollup/delete_me.sql) 'dict object' has no attribute 'CompilationError' > in macro materialization_incremental_dremio (macros/materializations/incremental/incremental.sql) > called by model delete_me (models/l1-hk-trading/rollup/delete_me.sql) 17:30:12 17:30:12 Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1 ``` ```

It is related to some CompilationError when I use incremental model PLUS merge strategy.

I have looked into some similar issues with CompilationError, but it seems it was related to something else.

Thank you

Ken · June 4, 2024, 1:06am

Just to follow up on my own thread.

The issue is just that the code which supports incremental merge strategy is not yet in the latest release package (1.7.0).

As commented in the PR above, the code was already merged, but not yet in the latest release.

I resolved the issue by simply building the wheel from the latest commit in githuh (via pip installing from the git URL)

Topic		Replies	Views
Dremio 4.0 compileError at execution	0	995	September 19, 2019
Failed to create Incremental Reflection on ElasticSearch	10	2240	July 13, 2018
Dremio error with dbt	5	444	May 2, 2024
Error Incremental refresh from HDFS using CONVERT_FROM(JSON)	0	1083	April 30, 2020
Dremio fails pushdown	7	2037	August 14, 2022

Dbt-dremio - 'dict object' has no attribute 'CompilationError'

Related Topics