# Acceleration refresh policies, always shows expired

**URL:** <https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878>\
**Category:** Uncategorized\
**Created:** [March 15, 2018, 4:40pm UTC](https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878 "2018-03-15T16:40:22Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![dealercrm](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/dealercrm/32/627_2.png) [@dealercrm](https://community.dremio.com/u/dealercrm)\
**Post date:** [March 15, 2018, 4:40pm UTC](https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878/1 "2018-03-15T16:40:22Z")

</div>

A data sources table is set to refresh every 2 hours and always shows expired in the reflection window, even though the refreshes are successful.

Help me understand please. 🙂

At the MySQL data source level

![46%20PM](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/1X/eeb3f11ef0623e4f2d9921ad2dc9aafdbae95b1a.png)

Then at the data source table level, if you hover over the orange triangle with the error, it states “Expired”  
 ![17%20PM](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/1X/92309d32e1eb922e81a51c0411cd6ace59918634.png)

Then we click on jobs to review when the last time the table was queried for incremental updates. This shows that Accelerators are being created every 2 hours, as planned.

 ![54%20PM](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/1X/f0f9a73b5643ce785a4a7eabc6f99a2ac33f0810.png)

We click on the settings of the data sources table and then click on refresh policy. We then see that the refresh policy is set at 2 hours and to expire after 4 hours. We are looking at the error of “Expired” within the hour of when the Accelerator was created.  
 ![31%20PM](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/1X/2d3f30036450f05266b72c10be12865f1a65c8af.png)

What am I missing?

I jumped into the server.log, then I find a “Unable to cache materialzation” error. However, when I search for the job, it shows completed.

 ![07%20PM](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/1X/4b97ab8cfc1b84d5652cec2ddcf4a05f7c19d6dd.png)

Here is the server log

```
2018-03-15 16:30:16,860 [pool-11-thread-1] WARN c.d.s.a.CachedMaterializationProvider - Unable to cache materialization a4fd6347-f865-47c1-ac37-a909b06632b8
java.lang.AssertionError: null
	at org.apache.calcite.util.Pair.zip(Pair.java:202) ~[calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
	at org.apache.calcite.rex.RexUtil.generateCastExpressions(RexUtil.java:177) ~[calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
	at org.apache.calcite.rex.RexUtil.generateCastExpressions(RexUtil.java:159) ~[calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
	at com.dremio.exec.planner.common.MoreRelOptUtil.createCastRel(MoreRelOptUtil.java:239) ~[dremio-sabot-kernel-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.exec.planner.common.MoreRelOptUtil.createCastRel(MoreRelOptUtil.java:214) ~[dremio-sabot-kernel-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.exec.planner.sql.MaterializationExpander.expand(MaterializationExpander.java:90) ~[dremio-sabot-kernel-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.exec.planner.sql.MaterializationDescriptor.getMaterializationFor(MaterializationDescriptor.java:148) ~[dremio-sabot-kernel-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.service.accelerator.CachedMaterializationProvider.expand(CachedMaterializationProvider.java:288) [dremio-services-accelerator-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.service.accelerator.CachedMaterializationProvider.updateCache(CachedMaterializationProvider.java:239) [dremio-services-accelerator-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.service.accelerator.CachedMaterializationProvider.compareAndSetCache(CachedMaterializationProvider.java:182) [dremio-services-accelerator-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.service.accelerator.CachedMaterializationProvider.refresh(CachedMaterializationProvider.java:164) [dremio-services-accelerator-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.service.accelerator.CachedMaterializationProvider.access$000(CachedMaterializationProvider.java:68) [dremio-services-accelerator-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.service.accelerator.CachedMaterializationProvider$1.run(CachedMaterializationProvider.java:155) [dremio-services-accelerator-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_151]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_151]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
2018-03-15 16:30:16,875 [pool-11-thread-1] WARN c.d.s.a.CachedMaterializationProvider - Unable to cache materialization 6732821f-973b-47fe-8887-785e1a149469
java.lang.AssertionError: null
	at org.apache.calcite.util.Pair.zip(Pair.java:202) ~[calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
	at org.apache.calcite.rex.RexUtil.generateCastExpressions(RexUtil.java:177) ~[calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
	at org.apache.calcite.rex.RexUtil.generateCastExpressions(RexUtil.java:159) ~[calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
	at com.dremio.exec.planner.common.MoreRelOptUtil.createCastRel(MoreRelOptUtil.java:239) ~[dremio-sabot-kernel-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.exec.planner.common.MoreRelOptUtil.createCastRel(MoreRelOptUtil.java:214) ~[dremio-sabot-kernel-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.exec.planner.sql.MaterializationExpander.expand(MaterializationExpander.java:90) ~[dremio-sabot-kernel-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.exec.planner.sql.MaterializationDescriptor.getMaterializationFor(MaterializationDescriptor.java:148) ~[dremio-sabot-kernel-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.service.accelerator.CachedMaterializationProvider.expand(CachedMaterializationProvider.java:288) [dremio-services-accelerator-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.service.accelerator.CachedMaterializationProvider.updateCache(CachedMaterializationProvider.java:239) [dremio-services-accelerator-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.service.accelerator.CachedMaterializationProvider.compareAndSetCache(CachedMaterializationProvider.java:182) [dremio-services-accelerator-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.service.accelerator.CachedMaterializationProvider.refresh(CachedMaterializationProvider.java:164) [dremio-services-accelerator-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.service.accelerator.CachedMaterializationProvider.access$000(CachedMaterializationProvider.java:68) [dremio-services-accelerator-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.service.accelerator.CachedMaterializationProvider$1.run(CachedMaterializationProvider.java:155) [dremio-services-accelerator-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_151]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_151]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
2018-03-15 16:30:16,893 [pool-11-thread-1] WARN c.d.s.a.CachedMaterializationProvider - Unable to cache materialization 9504017d-533b-4d7a-86ab-0ddeb3d98903
java.lang.AssertionError: null
	at org.apache.calcite.util.Pair.zip(Pair.java:202) ~[calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
	at org.apache.calcite.rex.RexUtil.generateCastExpressions(RexUtil.java:177) ~[calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
	at org.apache.calcite.rex.RexUtil.generateCastExpressions(RexUtil.java:159) ~[calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
	at com.dremio.exec.planner.common.MoreRelOptUtil.createCastRel(MoreRelOptUtil.java:239) ~[dremio-sabot-kernel-1.4.9-201802191836310213-7195059.jar:1.4.9-201802191836310213-7195059]
	at com.dremio.exec.planner.common.MoreRelOptUtil.c
```

---

<div class="post-metadata">

**Author:** ![dealercrm](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/dealercrm/32/627_2.png) [@dealercrm](https://community.dremio.com/u/dealercrm)\
**Post date:** [March 15, 2018, 6:20pm UTC](https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878/2 "2018-03-15T18:20:04Z")

</div>

Definately has something to do with the incremental update. As I switched back the datasource table to NOT have a reflection. Then switched it to FULL UPDATE.

Then I wrote a query with all the fields in the table 70% of the data in the table, created a reflection on the query and now the reflection is created.

So I still need to figure out why it is failing during the incremental update of the reflection at the datasource table level.

---

<div class="post-metadata">

**Author:** ![dealercrm](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/dealercrm/32/627_2.png) [@dealercrm](https://community.dremio.com/u/dealercrm)\
**Post date:** [March 21, 2018, 2:36pm UTC](https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878/3 "2018-03-21T14:36:54Z")

</div>

Bump. 🙂

Still not able to get any table or query to auto update, always have to choose full update. Which is not going to work long term. Need auto update to provide incremental updates to large data sources that are append only.

Anyone have any examples I could walk through?

---

<div class="post-metadata">

**Author:** ![doron](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/doron/32/294_2.png) [@doron](https://community.dremio.com/u/doron)\
**Post date:** [March 21, 2018, 5:41pm UTC](https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878/4 "2018-03-21T17:41:40Z")

</div>

Hi,

Sorry for the late response. If you remove the partitioning from the reflection does that make it work correctly with the incremental refreshes? Also can you give us the version of Dremio you are using?

thanks,  
Doron

---

<div class="post-metadata">

**Author:** ![dealercrm](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/dealercrm/32/627_2.png) [@dealercrm](https://community.dremio.com/u/dealercrm)\
**Post date:** [May 3, 2018, 12:28pm UTC](https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878/5 "2018-05-03T12:28:24Z")

</div>

No removing partitioning does not resolve the issue.

I have tried many different data sources that have the BIGINT data type as an incremental field. None of them work.

Version is 1.4.9-201802191836310213-7195059

---

<div class="post-metadata">

**Author:** ![hakim](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/hakim/32/35_2.png) [@hakim](https://community.dremio.com/u/hakim)\
**Post date:** [May 3, 2018, 2:09pm UTC](https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878/6 "2018-05-03T14:09:42Z")

</div>

did you try on 2.0.3 ? we fixed a lot of issues around reflection management and we also improved how we report the state. In your case, either the problem will be resolved or the UI and log will give you more information about the problem (we improved the log message for such failures to report additional information instead of just AssertionError:null).

Some clarifications about the issue: when we are done refreshing the reflection we do some additional internal work (materialization cache being one of them), so it’s always possible to fail even if the job itself was successful. We are aware that the job history can be confusing in such situations and we are working on improving how to better report those issues.

---

<div class="post-metadata">

**Author:** ![dealercrm](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/dealercrm/32/627_2.png) [@dealercrm](https://community.dremio.com/u/dealercrm)\
**Post date:** [May 3, 2018, 3:12pm UTC](https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878/7 "2018-05-03T15:12:51Z")

</div>

Not yet.  
I read all of the release notes.  
Looking at migrating, as there are many items that have been resolved, that we have been experiencing pain around. Looks great!

My initial post was trying to elicit some response from the community on an example of a working incremental update.

Create a table with this structure in the db.  
Add data.  
Create reflection with incremental update.  
Show initial reflection creation success.  
Add more data.  
See update of reflection as successful.

I did something similar with a MySQL table with a incremental id field and a text field, with no success.

---

<div class="post-metadata">

**Author:** ![dfleckinger](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/dfleckinger/32/370_2.png) [@dfleckinger](https://community.dremio.com/u/dfleckinger)\
**Post date:** [May 4, 2018, 7:31am UTC](https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878/8 "2018-05-04T07:31:58Z")

</div>

Hi @hakim, you mention 2.0.3, that fixes issues about reflection management.  
Is it released, where can it be found ?

---

<div class="post-metadata">

**Author:** ![hakim](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/hakim/32/35_2.png) [@hakim](https://community.dremio.com/u/hakim)\
**Post date:** [May 4, 2018, 2:15pm UTC](https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878/9 "2018-05-04T14:15:49Z")

</div>

sorry, that was a typo. The fixes I was referring to are all in 2.0.1 already

---

<div class="post-metadata">

**Author:** ![dfleckinger](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/dfleckinger/32/370_2.png) [@dfleckinger](https://community.dremio.com/u/dfleckinger)\
**Post date:** [May 4, 2018, 2:19pm UTC](https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878/10 "2018-05-04T14:19:19Z")

</div>

ok. This issue seem similar to this one : [dremio-2-0-1rc-some-reflections-fail-at-creation](https://community.dremio.com/t/dremio-2-0-1rc-some-reflections-fail-at-creation/1007/6)

---

<div class="post-metadata">

**Author:** ![hakim](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/hakim/32/35_2.png) [@hakim](https://community.dremio.com/u/hakim)\
**Post date:** [May 4, 2018, 2:32pm UTC](https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878/11 "2018-05-04T14:32:09Z")

</div>

it’s similar in a sense that the refresh job succeeded but the the “internal” work failed after that, but looking at the stacktrace, it’s 2 different issues

---

<div class="post-metadata">

**Author:** ![dealercrm](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/dealercrm/32/627_2.png) [@dealercrm](https://community.dremio.com/u/dealercrm)\
**Post date:** [May 17, 2018, 6:10pm UTC](https://community.dremio.com/t/acceleration-refresh-policies-always-shows-expired/878/12 "2018-05-17T18:10:33Z")

</div>

Upgrading to 2.x solved this issue.

Thanks.