Hi!
TL;DR - looking to get raw and aggregate reflections working.
My use case requires creating a raw and aggregate reflection (count) of a view consisting of 3 iceberg tables. These three tables are LEFT OUTER joined, which is pretty heavy operation to achieve query latency under a few seconds.
I tried creating raw reflections on the UI, and jobs show SUCCEEDED status, but my queries on the view don’t seem to be accelerated.
it seems that all the raw reflections have CANNOT_ACCELERATE_SCHEDULED status.
Here is a query profile of a SELECT query on this dataset I tried creating the reflections for
e80cbfc1-3393-40bf-b3bd-265d519aa794.zip (75.2 KB)
I also noticed that creating aggregate reflection on this dataset succeeded, but COUNT query does not seem to have accelerated.
edf71120-e257-4c0b-bd7b-c00e5f54d9b7.zip (50.0 KB)
Finally, I trailed master node’s logs, and saw this error.
2023-08-19 08:32:16,544 [dremio-general-1] WARN c.d.s.reflection.ReflectionManager - Failed to update materialization cache for materialization 4165c171-84db-4499-ba3f-b8c9b370257b/95d0f33a-f0c4-4d36-9164-9617cf85e2ba
com.dremio.exec.planner.acceleration.MaterializationExpander$ExpansionException: Materialization 95d0f33a-f0c4-4d36-9164-9617cf85e2ba have different row types for its table and query rels.
table row type RecordType(TIMESTAMP(3) timestamp, TIMESTAMP(3) log_timestamp, VARCHAR(65536) event_id, VARCHAR(65536) join_id, VARCHAR(65536) batch_id, VARCHAR(65536) data_node_66b2c676_0d68_4e9f_a0c0_c113ac1dc956, VARCHAR(65536) data_node_c88cee02_463d_4e4c_95a0_5f0fcceeadac, VARCHAR(65536) data_node_5c87182b_4e2d_4195_b980_4d822b4e1995, VARCHAR(65536) data_node_c765de66_f45c_4f0a_be5a_93f5b8f13129, VARCHAR(65536) data_node_458173fd_c598_4bf1_b710_eb9615baf2ad, VARCHAR(65536) data_node_d2127726_f509_4fc2_a9f4_afbad36d4c85, VARCHAR(65536) data_node_08275966_f923_4ad4_8cc8_256081982293, BIGINT data_node_453c1af8_0792_48b2_b3f1_13590d48bacb, BOOLEAN data_node_59a8d6c3_be1f_4753_89af_a098b94e4a7e, VARCHAR(65536) data_node_0aa36d14_2525_491a_9165_4a61497687dd, VARCHAR(65536) ARRAY data_node_17e5dcda_d3d3_4516_ab0f_f26f530a1b04, VARCHAR(65536) data_node_7a88dd60_3695_42cd_babd_ec47e3b864da, VARCHAR(65536) data_node_0554d4ba_20c3_4175_aa2b_487fb3eec38a, BIGINT data_node_408707b7_7496_4e02_96b6_75d81c834e19, VARCHAR(65536) data_node_18d203f7_c049_45f5_9312_46c77f1a4412, BIGINT data_node_26fd552c_d25e_462a_95e1_165038b39371, VARCHAR(65536) data_node_b4d1144e_f6ff_4fa6_aea9_47fc1b6c9c90, VARCHAR(65536) data_node_7ac9e31f_e340_4538_ae70_d3d24a287b0e, VARCHAR(65536) data_node_12b38859_2a46_4755_957d_66aba5c0231e, BIGINT data_node_f45e8112_a314_4b1e_96bc_4bb2217cedbd, VARCHAR(65536) data_node_7c4f2c88_5410_4f0b_861a_89e5f644707d, VARCHAR(65536) data_node_30a3b29c_e77d_4596_bff5_0df45d79d5c5, BIGINT data_node_71cf0401_065a_4647_93ec_f0f272f956bc, VARCHAR(65536) ARRAY data_node_29cc0221_ced3_4820_a906_3c4afc26d7fd, VARCHAR(65536) data_node_97ed84dd_ad4a_477d_b0d1_c28d90d1d50f, VARCHAR(65536) data_node_d1b063a2_4612_4a78_8595_8ce37d7d8542, VARCHAR(65536) data_node_fba52029_6854_4bf6_bc46_d313f1bbea5e, BIGINT data_node_0a165a2f_1f84_439f_9e2d_557d45c361b8, BIGINT data_node_2804df4b_8b76_4a2d_9213_8173d38f92f4, VARCHAR(65536) data_node_5fb02ef8_a5cc_45fe_bcb8_33fa7f4279c4, DOUBLE data_node_c898e1a7_fca9_4ec2_b1a8_966c3c8ec0aa, VARCHAR(65536) data_node_bf702015_63b1_4961_9d58_3acda15ed3f8, BIGINT data_node_7cdcce4e_55f3_46b7_845f_26f54d2c29be, BOOLEAN data_node_deb5d35a_8268_445f_9750_cc2d87097849, VARCHAR(65536) data_node_5ca91cc2_e25b_4681_ac05_bf91518b3d65, VARCHAR(65536) data_node_30a333a5_f02e_47ca_9dd1_58bcc2e9c427, VARCHAR(65536) data_node_4becb6a9_0393_4299_85f2_fef7a8ad32e0, VARCHAR(65536) data_node_6430c569_dc7d_4e29_a382_a400cc542990, VARCHAR(65536) data_node_147b24dc_5cdd_45c3_beb9_d5f2b97b7841, BIGINT data_node_8c5df338_bac2_49ef_ae36_77bf6e8fa510, BIGINT data_node_c99aa22d_3af4_4dfb_b159_c82e6b19f10a, VARCHAR(65536) data_node_b9eb1124_525b_4b4b_ad3f_154b14f29c15, BIGINT data_node_cb46c44d_0851_4e18_b028_48a5861f52b4, VARCHAR(65536) data_node_cf39abb4_1976_4b87_bc84_1b880e667c22, DOUBLE data_node_628de604_765d_4c9a_a2f4_dcdc2074e5e9, BOOLEAN data_node_dde9fead_e271_42d3_a52a_e14579ec1267, DOUBLE data_node_fb2f1ef3_8eec_4f82_b37e_ab9dd443b7c4, BIGINT data_node_df0b010d_1eee_41bf_a7a6_5be253a17b8c, BIGINT data_node_8ca23def_fd1a_403f_9235_3943dac47e6f, VARCHAR(65536) data_node_e5fe0bc8_846b_4162_8a94_f72c3738c786, VARCHAR(65536) tag_8e5fc946_04cd_4b79_8ce9_9ca6bba9e2a8, VARCHAR(65536) tag_7ca6b0eb_1245_4903_b623_dec8f0974fe8, VARCHAR(65536) tag_fbae855c_a104_460c_824e_a74d0b6450c3, VARCHAR(65536) tag_f623eacb_157d_4ccb_a7a2_50d227c7855e, VARCHAR(65536) tag_449686d1_823f_4066_8c63_ca829928faa5, VARCHAR(65536) tag_0a7d623b_f586_4991_b615_5d7e81514bb0, VARCHAR(65536) tag_0527be96_deeb_4c77_a058_91dc2b164ea1, VARCHAR(65536) tag_f7ddf94a_3ed4_4f3b_9500_3cf769591a7f, VARCHAR(65536) tag_4e279108_ecf7_400d_85db_2c3f5d96ffc5, VARCHAR(65536) tag_eaecd94c_c139_4596_8346_78c3a98cc48d, VARCHAR(65536) tag_58dfc4dc_2e9c_4859_98c1_386079e6ff08, VARCHAR(65536) tag_176553b6_8ab5_47f1_a83f_53ad1f381fb9, VARCHAR(65536) tag_bbbba43d_ca6e_48a4_ad33_193198005c79, VARCHAR(65536) data_node_99034222_24c7_47a2_851c_0dbf5bc845c9, VARCHAR(65536) data_node_f7718cb9_9b70_405e_bbd0_2119cf313c7b, VARCHAR(65536) data_node_64b962a8_7507_4a1f_9da9_1ac06068ae3e, VARCHAR(65536) data_node_0d057d5c_94ce_4d50_8e8d_eca7e1dddcf9, VARCHAR(65536) data_node_a45fd4c0_1bdb_41fb_b5ee_8cda1a7a379a, VARCHAR(65536) data_node_59ea9196_38dd_435a_b220_081b52f018a7, VARCHAR(65536) data_node_aef5a0c5_5be7_4be6_841a_57622ad91dfb, VARCHAR(65536) data_node_b2a7fea2_63b8_4885_b684_2c9b012d0d60, VARCHAR(65536) data_node_0f6a586e_d332_4a0f_baf9_8f311f61e2fd, VARCHAR(65536) data_node_3d5132d4_5ede_403a_90df_26cf3e342654, VARCHAR(65536) data_node_119c720b_7a68_4b5d_bfa8_fa144a68feae, VARCHAR(65536) data_node_9e08526c_e947_4c8f_bd95_b1a71e0ede38, VARCHAR(65536) data_node_2ddfb8d1_a731_4271_8fb2_99fddaac0cef, VARCHAR(65536) data_node_e363e419_ce5e_45cf_adce_5fb9cb8efd7c, VARCHAR(65536) data_node_cd5a4acb_dc34_48cb_acf4_2f77eb6b9f41, VARCHAR(65536) data_node_7a37d310_cde8_446b_9cc9_9db971b87c72, VARCHAR(65536) data_node_8a4e72a5_0f35_4198_a210_f1257728fbce, VARCHAR(65536) data_node_4bae8d06_e256_42e4_9fea_db860b676836, BOOLEAN data_node_b44aa941_9acf_4f5b_bbad_8e1ec2332b18, VARCHAR(65536) data_node_a4e8074e_8c36_4539_adb9_f8e7176104c6, VARCHAR(65536) data_node_9b655018_6978_45cd_bc24_f31809cacfe8, VARCHAR(65536) data_node_dc7fe00a_e854_4af5_862b_659034d4981c, VARCHAR(65536) data_node_56b7020a_a776_4ffa_af03_bc29622f6c75, VARCHAR(65536) data_node_a989f14f_06f7_4add_b403_98c10ff4cdf1, VARCHAR(65536) data_node_b49783bc_57c8_4f2b_bbce_1b5b5632dee9, BOOLEAN data_node_164954a8_cf0a_4b8b_9929_bc0880f7dc63, VARCHAR(65536) data_node_95165e8d_990b_4723_8c56_750dc457613d, VARCHAR(65536) data_node_d49a502b_9917_41eb_810e_59527a40bd9c, VARCHAR(65536) data_node_15bc6356_a158_46f0_90bf_1254abd32701, VARCHAR(65536) data_node_11bab3f7_8d09_4357_8857_759a78716616, VARCHAR(65536) data_node_fce4a1ad_4d16_4ece_bbd0_646a6e972e63, VARCHAR(65536) data_node_42ac5da0_09a8_43c7_9fa9_e79b1865ca30, VARCHAR(65536) data_node_40703402_d722_4e15_9762_e858dec23eff, VARCHAR(65536) data_node_7c0c653f_9ba3_4f42_983f_d8f540bb54ee, VARCHAR(65536) data_node_11cd4bf3_b874_4e32_98ea_97b5ca3104de, VARCHAR(65536) data_node_cf2c752e_a136_42ce_b4dd_7585138fb855, VARCHAR(65536) data_node_2a040569_a30b_4521_80a9_baa2f4770ee2, VARCHAR(65536) data_node_a21d4ceb_b498_49fe_af9c_758f41147d74, VARCHAR(65536) data_node_cb4c1066_6c73_48df_96de_5e2d411b183a, VARCHAR(65536) src_prediction_event_id, VARCHAR(65536) data_node_bc45c8b5_b14d_497c_aa90_1f5509afa3b6, VARCHAR(65536) join_id0)
query row type RecordType(TIMESTAMP(3) timestamp, TIMESTAMP(3) log_timestamp, VARCHAR(65536) event_id, VARCHAR(65536) join_id, VARCHAR(65536) batch_id, VARCHAR(65536) data_node_66b2c676_0d68_4e9f_a0c0_c113ac1dc956, VARCHAR(65536) data_node_c88cee02_463d_4e4c_95a0_5f0fcceeadac, VARCHAR(65536) data_node_5c87182b_4e2d_4195_b980_4d822b4e1995, VARCHAR(65536) data_node_c765de66_f45c_4f0a_be5a_93f5b8f13129, VARCHAR(65536) data_node_458173fd_c598_4bf1_b710_eb9615baf2ad, VARCHAR(65536) data_node_d2127726_f509_4fc2_a9f4_afbad36d4c85, VARCHAR(65536) data_node_08275966_f923_4ad4_8cc8_256081982293, BIGINT data_node_453c1af8_0792_48b2_b3f1_13590d48bacb, BOOLEAN data_node_59a8d6c3_be1f_4753_89af_a098b94e4a7e, VARCHAR(65536) data_node_0aa36d14_2525_491a_9165_4a61497687dd, VARCHAR(65536) ARRAY data_node_17e5dcda_d3d3_4516_ab0f_f26f530a1b04, VARCHAR(65536) data_node_7a88dd60_3695_42cd_babd_ec47e3b864da, VARCHAR(65536) data_node_0554d4ba_20c3_4175_aa2b_487fb3eec38a, BIGINT data_node_408707b7_7496_4e02_96b6_75d81c834e19, VARCHAR(65536) data_node_18d203f7_c049_45f5_9312_46c77f1a4412, BIGINT data_node_26fd552c_d25e_462a_95e1_165038b39371, VARCHAR(65536) data_node_b4d1144e_f6ff_4fa6_aea9_47fc1b6c9c90, VARCHAR(65536) data_node_7ac9e31f_e340_4538_ae70_d3d24a287b0e, VARCHAR(65536) data_node_12b38859_2a46_4755_957d_66aba5c0231e, BIGINT data_node_f45e8112_a314_4b1e_96bc_4bb2217cedbd, VARCHAR(65536) data_node_7c4f2c88_5410_4f0b_861a_89e5f644707d, VARCHAR(65536) data_node_30a3b29c_e77d_4596_bff5_0df45d79d5c5, BIGINT data_node_71cf0401_065a_4647_93ec_f0f272f956bc, VARCHAR(65536) ARRAY data_node_29cc0221_ced3_4820_a906_3c4afc26d7fd, VARCHAR(65536) data_node_97ed84dd_ad4a_477d_b0d1_c28d90d1d50f, VARCHAR(65536) data_node_d1b063a2_4612_4a78_8595_8ce37d7d8542, VARCHAR(65536) data_node_fba52029_6854_4bf6_bc46_d313f1bbea5e, BIGINT data_node_0a165a2f_1f84_439f_9e2d_557d45c361b8, BIGINT data_node_2804df4b_8b76_4a2d_9213_8173d38f92f4, VARCHAR(65536) data_node_5fb02ef8_a5cc_45fe_bcb8_33fa7f4279c4, DOUBLE data_node_c898e1a7_fca9_4ec2_b1a8_966c3c8ec0aa, VARCHAR(65536) data_node_bf702015_63b1_4961_9d58_3acda15ed3f8, BIGINT data_node_7cdcce4e_55f3_46b7_845f_26f54d2c29be, BOOLEAN data_node_deb5d35a_8268_445f_9750_cc2d87097849, VARCHAR(65536) data_node_5ca91cc2_e25b_4681_ac05_bf91518b3d65, VARCHAR(65536) data_node_30a333a5_f02e_47ca_9dd1_58bcc2e9c427, VARCHAR(65536) data_node_4becb6a9_0393_4299_85f2_fef7a8ad32e0, VARCHAR(65536) data_node_6430c569_dc7d_4e29_a382_a400cc542990, VARCHAR(65536) data_node_147b24dc_5cdd_45c3_beb9_d5f2b97b7841, BIGINT data_node_8c5df338_bac2_49ef_ae36_77bf6e8fa510, BIGINT data_node_c99aa22d_3af4_4dfb_b159_c82e6b19f10a, VARCHAR(65536) data_node_b9eb1124_525b_4b4b_ad3f_154b14f29c15, BIGINT data_node_cb46c44d_0851_4e18_b028_48a5861f52b4, VARCHAR(65536) data_node_cf39abb4_1976_4b87_bc84_1b880e667c22, DOUBLE data_node_628de604_765d_4c9a_a2f4_dcdc2074e5e9, BOOLEAN data_node_dde9fead_e271_42d3_a52a_e14579ec1267, DOUBLE data_node_fb2f1ef3_8eec_4f82_b37e_ab9dd443b7c4, BIGINT data_node_df0b010d_1eee_41bf_a7a6_5be253a17b8c, BIGINT data_node_8ca23def_fd1a_403f_9235_3943dac47e6f, VARCHAR(65536) data_node_e5fe0bc8_846b_4162_8a94_f72c3738c786, VARCHAR(65536) tag_8e5fc946_04cd_4b79_8ce9_9ca6bba9e2a8, VARCHAR(65536) tag_7ca6b0eb_1245_4903_b623_dec8f0974fe8, VARCHAR(65536) tag_fbae855c_a104_460c_824e_a74d0b6450c3, VARCHAR(65536) tag_f623eacb_157d_4ccb_a7a2_50d227c7855e, VARCHAR(65536) tag_449686d1_823f_4066_8c63_ca829928faa5, VARCHAR(65536) tag_0a7d623b_f586_4991_b615_5d7e81514bb0, VARCHAR(65536) tag_0527be96_deeb_4c77_a058_91dc2b164ea1, VARCHAR(65536) tag_f7ddf94a_3ed4_4f3b_9500_3cf769591a7f, VARCHAR(65536) tag_4e279108_ecf7_400d_85db_2c3f5d96ffc5, VARCHAR(65536) tag_eaecd94c_c139_4596_8346_78c3a98cc48d, VARCHAR(65536) tag_58dfc4dc_2e9c_4859_98c1_386079e6ff08, VARCHAR(65536) tag_176553b6_8ab5_47f1_a83f_53ad1f381fb9, VARCHAR(65536) tag_bbbba43d_ca6e_48a4_ad33_193198005c79, VARCHAR(65536) data_node_99034222_24c7_47a2_851c_0dbf5bc845c9, VARCHAR(65536) data_node_f7718cb9_9b70_405e_bbd0_2119cf313c7b, VARCHAR(65536) data_node_64b962a8_7507_4a1f_9da9_1ac06068ae3e, VARCHAR(65536) data_node_0d057d5c_94ce_4d50_8e8d_eca7e1dddcf9, VARCHAR(65536) data_node_a45fd4c0_1bdb_41fb_b5ee_8cda1a7a379a, VARCHAR(65536) data_node_59ea9196_38dd_435a_b220_081b52f018a7, VARCHAR(65536) data_node_aef5a0c5_5be7_4be6_841a_57622ad91dfb, VARCHAR(65536) data_node_b2a7fea2_63b8_4885_b684_2c9b012d0d60, VARCHAR(65536) data_node_0f6a586e_d332_4a0f_baf9_8f311f61e2fd, VARCHAR(65536) data_node_3d5132d4_5ede_403a_90df_26cf3e342654, VARCHAR(65536) data_node_119c720b_7a68_4b5d_bfa8_fa144a68feae, VARCHAR(65536) data_node_9e08526c_e947_4c8f_bd95_b1a71e0ede38, VARCHAR(65536) data_node_2ddfb8d1_a731_4271_8fb2_99fddaac0cef, VARCHAR(65536) data_node_e363e419_ce5e_45cf_adce_5fb9cb8efd7c, VARCHAR(65536) data_node_cd5a4acb_dc34_48cb_acf4_2f77eb6b9f41, VARCHAR(65536) data_node_7a37d310_cde8_446b_9cc9_9db971b87c72, VARCHAR(65536) data_node_8a4e72a5_0f35_4198_a210_f1257728fbce, VARCHAR(65536) data_node_4bae8d06_e256_42e4_9fea_db860b676836, BOOLEAN data_node_b44aa941_9acf_4f5b_bbad_8e1ec2332b18, VARCHAR(65536) data_node_a4e8074e_8c36_4539_adb9_f8e7176104c6, VARCHAR(65536) data_node_9b655018_6978_45cd_bc24_f31809cacfe8, VARCHAR(65536) data_node_dc7fe00a_e854_4af5_862b_659034d4981c, VARCHAR(65536) data_node_56b7020a_a776_4ffa_af03_bc29622f6c75, VARCHAR(65536) data_node_a989f14f_06f7_4add_b403_98c10ff4cdf1, VARCHAR(65536) data_node_b49783bc_57c8_4f2b_bbce_1b5b5632dee9, BOOLEAN data_node_164954a8_cf0a_4b8b_9929_bc0880f7dc63, VARCHAR(65536) data_node_95165e8d_990b_4723_8c56_750dc457613d, VARCHAR(65536) data_node_d49a502b_9917_41eb_810e_59527a40bd9c, VARCHAR(65536) data_node_15bc6356_a158_46f0_90bf_1254abd32701, VARCHAR(65536) data_node_11bab3f7_8d09_4357_8857_759a78716616, VARCHAR(65536) data_node_fce4a1ad_4d16_4ece_bbd0_646a6e972e63, VARCHAR(65536) data_node_42ac5da0_09a8_43c7_9fa9_e79b1865ca30, VARCHAR(65536) data_node_40703402_d722_4e15_9762_e858dec23eff, VARCHAR(65536) data_node_7c0c653f_9ba3_4f42_983f_d8f540bb54ee, VARCHAR(65536) data_node_11cd4bf3_b874_4e32_98ea_97b5ca3104de, VARCHAR(65536) data_node_cf2c752e_a136_42ce_b4dd_7585138fb855, VARCHAR(65536) data_node_2a040569_a30b_4521_80a9_baa2f4770ee2, VARCHAR(65536) data_node_a21d4ceb_b498_49fe_af9c_758f41147d74, VARCHAR(65536) data_node_cb4c1066_6c73_48df_96de_5e2d411b183a, VARCHAR(65536) src_prediction_event_id, VARCHAR(65536) data_node_bc45c8b5_b14d_497c_aa90_1f5509afa3b6, VARCHAR(65536) join_id0)
at com.dremio.exec.planner.acceleration.MaterializationExpander.expand(MaterializationExpander.java:116)
at com.dremio.exec.planner.acceleration.MaterializationDescriptor.getMaterializationFor(MaterializationDescriptor.java:160)
at com.dremio.service.reflection.ReflectionServiceImpl$CacheHelperImpl.expand(ReflectionServiceImpl.java:1139)
at com.dremio.service.reflection.ReflectionServiceImpl$CacheHelperImpl.expand(ReflectionServiceImpl.java:1120)
at com.dremio.service.reflection.MaterializationCache.update(MaterializationCache.java:272)
at com.dremio.service.reflection.ReflectionServiceImpl$DescriptorCacheImpl.update(ReflectionServiceImpl.java:1227)
at com.dremio.service.reflection.ReflectionManager.metadataRefreshJobSucceeded(ReflectionManager.java:1033)
at com.dremio.service.reflection.ReflectionManager.handleSuccessfulJob(ReflectionManager.java:844)
at com.dremio.service.reflection.ReflectionManager.handleRefreshingEntry(ReflectionManager.java:508)
at com.dremio.service.reflection.ReflectionManager.handleEntry(ReflectionManager.java:428)
at com.dremio.service.reflection.ReflectionManager.handleEntries(ReflectionManager.java:372)
at com.dremio.service.reflection.ReflectionManager.sync(ReflectionManager.java:237)
at com.dremio.service.reflection.ReflectionManager.run(ReflectionManager.java:210)
at com.dremio.common.WakeupHandler$1.run(WakeupHandler.java:65)
at com.dremio.context.RequestContext.run(RequestContext.java:96)
at com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$3(ContextMigratingExecutorService.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
