# Metadata Refresh process for Hive is taking long time

**URL:** <https://community.dremio.com/t/metadata-refresh-process-for-hive-is-taking-long-time/6791>\
**Category:** Uncategorized\
**Created:** [January 4, 2021, 1:17pm UTC](https://community.dremio.com/t/metadata-refresh-process-for-hive-is-taking-long-time/6791 "2021-01-04T13:17:14Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![PranayKamuni](https://avatars.discourse-cdn.com/v4/letter/p/9fc29f/32.png) [@PranayKamuni](https://community.dremio.com/u/PranayKamuni)\
**Post date:** [January 4, 2021, 1:17pm UTC](https://community.dremio.com/t/metadata-refresh-process-for-hive-is-taking-long-time/6791/1 "2021-01-04T13:17:14Z")

</div>

We have Hive source with 1 day metadata full refresh configured and every full refresh is taking almost 7 to 9 hours time.

Below are the logs for metadata full refresh,

Source ‘XXXXX’ refreshed in 24666 seconds. Details:  
Shallow probed 102798 datasets: 1 added, 102795 unchanged, 2 deleted  
Deep probed 1030 queried datasets: 786 changed, 242 unchanged, 0 deleted, 2 unreadable

There are few tables which has around 1600 partitions, which are taking approximately 10 mins time to complete refresh for that table.

Is there any optimisations in latest dremio version? Or is it normal behaviour?  
Also suggest if there is any way to reduce the process time.

we are using dremio 3.3.1.

Please respond.

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [January 5, 2021, 7:57am UTC](https://community.dremio.com/t/metadata-refresh-process-for-hive-is-taking-long-time/6791/2 "2021-01-05T07:57:20Z")

</div>

@PranayKamuni

Later this year, we are coming up with more optimizations to reduce time taken for metadata refresh. BY any chance in your source metadata settings, have you configured the metadata setting “Fetch mode” to “All Datasets”, see attached screenshot

 ![metadata](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/7/7c32ece9f2d52dc2e090bf0e127eed4e355c321a.png)

---

<div class="post-metadata">

**Author:** ![PranayKamuni](https://avatars.discourse-cdn.com/v4/letter/p/9fc29f/32.png) [@PranayKamuni](https://community.dremio.com/u/PranayKamuni)\
**Post date:** [January 5, 2021, 8:09am UTC](https://community.dremio.com/t/metadata-refresh-process-for-hive-is-taking-long-time/6791/3 "2021-01-05T08:09:45Z")

</div>

Thanks for the response.

No , we have not configured as ‘All Datasets’, It is only ‘Queried Datasets’.

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [January 6, 2021, 7:24am UTC](https://community.dremio.com/t/metadata-refresh-process-for-hive-is-taking-long-time/6791/4 "2021-01-06T07:24:03Z")

</div>

@PranayKamuni

Try this, add the below logger to logback.xml and restart coordinator, after a day, review metadata\_refresh.log and you will see dataset wise how much time was taken. You can bring that file into Dremio as CSV (space as delimiter) and find out the datasets that takes most of the time. If there are any datasets that you do not require then you can do “ALTER PDS FORGET METADATA”

```
<logger name="com.dremio.exec.catalog.MetadataSynchronizer" additivity="false">
   <level value="debug" />
    <if condition='isDefined("dremio.log.path")'>
      <then>
        <appender-ref ref="metadata_refresh"/>
      </then>
    </if>
 </logger>

```

Note: The ones that are listed as 0 ms can be ignored

---

<div class="post-metadata">

**Author:** ![PranayKamuni](https://avatars.discourse-cdn.com/v4/letter/p/9fc29f/32.png) [@PranayKamuni](https://community.dremio.com/u/PranayKamuni)\
**Post date:** [February 2, 2021, 8:08am UTC](https://community.dremio.com/t/metadata-refresh-process-for-hive-is-taking-long-time/6791/5 "2021-02-02T08:08:57Z")

</div>

Thanks Balaji. We will try this and validate