ReflectionManager - failed to update materialization cache

1

Hello guys,

We’ve identified a problem when the reflection manager was trying to automatically update some materializations. The LOAD MATERIALIZATION jobs were returning ok, but the Reflection was not actually created. Prints bellow. We’d tried to manually drop the reflection, then create again, but didn’t work.

We solved this problem by recreating the VDS and adding the Raw Reflection back again to this particular query, but this way we can have some availability problems.

  • Was Dremio supposed to clear this cache when it updates the metadata of the VDS (source configuration)?
  • How to clear this cache without recreating the VDS?

image
image.png1886×360 28.6 KB

image%20(1)
image (1).png850×678 41 KB 

    2019-09-23 16:06:49,565 [dremio-general-7] WARN  c.d.s.reflection.ReflectionManager - failed to update materialization cache for 5a0bcbab-4c32-4cf6-9aab-08315ddcc9b5/562efd82-61e0-42c9-b377-a406702876c1
    com.dremio.exec.planner.acceleration.MaterializationExpander$ExpansionException: Materialization 562efd82-61e0-42c9-b377-a406702876c1 have different row types for its table and query rels.
    table row type RecordType(TIMESTAMP(3) dth_alteracao_reg, VARCHAR(65536) cod_ordem_producao, VARCHAR(65536) idc_pedido, VARCHAR(65536) cod_item, VARCHAR(65536) plant, VARCHAR(65536) roe_idc_registro, VARCHAR(65536) roe_cod_ordem_producao, DOUBLE roe_qtd_emprego, TIMESTAMP(3) roe_dth_emprego, TIMESTAMP(3) roe_dth_criacao_reg, INTEGER res_cmp_pecas_prod, TIMESTAMP(3) res_dth_criacao_reg, VARCHAR(65536) res_flg_qualificacao, VARCHAR(65536) res_idc_registro, VARCHAR(65536) res_num_pecas, VARCHAR(65536) res_peca_cmp, VARCHAR(65536) res_peca_pes, INTEGER res_peso_sucatas_curt, INTEGER res_peso_sucatas_long, INTEGER res_pes_pecas_prod, INTEGER res_qtd_pecas_prod, INTEGER res_qtd_sucacas_curt, INTEGER res_qtd_sucatas_long, INTEGER pla_cmp_cortes, DOUBLE pla_cmp_max_luva, DOUBLE pla_cmp_min_luva, VARCHAR(65536) pla_cod_aco, VARCHAR(65536) pla_cod_aqa, VARCHAR(65536) pla_cod_corrida, VARCHAR(65536) pla_cod_material_consumido, VARCHAR(65536) pla_cod_material_produzido, VARCHAR(65536) pla_cod_norma, VARCHAR(65536) pla_cod_ordem_producao, VARCHAR(65536) pla_cod_pedido_critico, INTEGER pla_cod_projeto, VARCHAR(65536) pla_dim_ext_max, VARCHAR(65536) pla_dim_ext_min, VARCHAR(65536) pla_dim_ext_segmento, VARCHAR(65536) pla_dim_ext_tubo, VARCHAR(65536) pla_dsc_aco, VARCHAR(65536) pla_dsc_cliente, VARCHAR(65536) pla_dsc_mat_prod, VARCHAR(65536) pla_dsc_norma, VARCHAR(65536) pla_dsc_pedido_critico, VARCHAR(65536) pla_dsc_projeto, VARCHAR(65536) pla_dsc_tip_rosca, TIMESTAMP(3) pla_dth_alteracao_reg, TIMESTAMP(3) pla_dth_criacao_reg, TIMESTAMP(3) pla_dth_prevista_producao, VARCHAR(65536) pla_esp_par_max, VARCHAR(65536) pla_esp_par_min, VARCHAR(65536) pla_esp_par_segmento, VARCHAR(65536) pla_firma_insp, VARCHAR(65536) pla_idc_celula, VARCHAR(65536) pla_parede_tubo_para_luva, VARCHAR(65536) pla_peso_mat_prod, VARCHAR(65536) pla_pes_pecas_pla, VARCHAR(65536) pla_qtd_pecas_pla, VARCHAR(65536) pla_tipo_rosca, VARCHAR(65536) pla_tip_rosqueamento, VARCHAR(65536) pla_tmo_ciclo_por_equipamento, VARCHAR(65536) pla_txt_marc_punc)
    query row type RecordType(TIMESTAMP(3) dth_alteracao_reg, VARCHAR(65536) cod_ordem_producao, VARCHAR(65536) idc_pedido, VARCHAR(65536) cod_item, VARCHAR(3) plant, VARCHAR(65536) roe_idc_registro, VARCHAR(65536) roe_cod_ordem_producao, DOUBLE roe_qtd_emprego, TIMESTAMP(3) roe_dth_emprego, TIMESTAMP(3) roe_dth_criacao_reg, DOUBLE res_cmp_pecas_prod, TIMESTAMP(3) res_dth_criacao_reg, VARCHAR(65536) res_flg_qualificacao, VARCHAR(65536) res_idc_registro, VARCHAR(65536) res_num_pecas, VARCHAR(65536) res_peca_cmp, VARCHAR(65536) res_peca_pes, DOUBLE res_peso_sucatas_curt, DOUBLE res_peso_sucatas_long, DOUBLE res_pes_pecas_prod, DOUBLE res_qtd_pecas_prod, DOUBLE res_qtd_sucacas_curt, DOUBLE res_qtd_sucatas_long, DOUBLE pla_cmp_cortes, DOUBLE pla_cmp_max_luva, DOUBLE pla_cmp_min_luva, VARCHAR(65536) pla_cod_aco, VARCHAR(65536) pla_cod_aqa, VARCHAR(65536) pla_cod_corrida, VARCHAR(65536) pla_cod_material_consumido, VARCHAR(65536) pla_cod_material_produzido, VARCHAR(65536) pla_cod_norma, VARCHAR(65536) pla_cod_ordem_producao, VARCHAR(65536) pla_cod_pedido_critico, DOUBLE pla_cod_projeto, VARCHAR(65536) pla_dim_ext_max, VARCHAR(65536) pla_dim_ext_min, VARCHAR(65536) pla_dim_ext_segmento, VARCHAR(65536) pla_dim_ext_tubo, VARCHAR(65536) pla_dsc_aco, VARCHAR(65536) pla_dsc_cliente, VARCHAR(65536) pla_dsc_mat_prod, VARCHAR(65536) pla_dsc_norma, VARCHAR(65536) pla_dsc_pedido_critico, VARCHAR(65536) pla_dsc_projeto, VARCHAR(65536) pla_dsc_tip_rosca, TIMESTAMP(3) pla_dth_alteracao_reg, TIMESTAMP(3) pla_dth_criacao_reg, TIMESTAMP(3) pla_dth_prevista_producao, VARCHAR(65536) pla_esp_par_max, VARCHAR(65536) pla_esp_par_min, VARCHAR(65536) pla_esp_par_segmento, VARCHAR(65536) pla_firma_insp, VARCHAR(65536) pla_idc_celula, VARCHAR(65536) pla_parede_tubo_para_luva, VARCHAR(65536) pla_peso_mat_prod, VARCHAR(65536) pla_pes_pecas_pla, VARCHAR(65536) pla_qtd_pecas_pla, VARCHAR(65536) pla_tipo_rosca, VARCHAR(65536) pla_tip_rosqueamento, VARCHAR(65536) pla_tmo_ciclo_por_equipamento, VARCHAR(65536) pla_txt_marc_punc)
1 Like
2

@caiounderscore

We have addressed the issue where you are not required to recreate VDS in Dremio 4.0? What version of Dremio are you using?

3

We use version 3.1.6

4

@balaji.ramaswamy we may be getting this error too. Can you give more specifics about what was fixed? Maybe link to release note section?