# Unable to refresh metadata for the dataset (due to concurrent updates). Please retry

**URL:** <https://community.dremio.com/t/unable-to-refresh-metadata-for-the-dataset-due-to-concurrent-updates-please-retry/11245>\
**Category:** Dremio Cloud\
**Created:** [December 6, 2023, 10:01pm UTC](https://community.dremio.com/t/unable-to-refresh-metadata-for-the-dataset-due-to-concurrent-updates-please-retry/11245 "2023-12-06T22:01:28Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jdwills](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/jdwills/32/5230_2.png) [@jdwills](https://community.dremio.com/u/jdwills)\
**Post date:** [December 6, 2023, 10:01pm UTC](https://community.dremio.com/t/unable-to-refresh-metadata-for-the-dataset-due-to-concurrent-updates-please-retry/11245/1 "2023-12-06T22:01:28Z")

</div>

Hi,

We have many files generated in the following example S3 folder structure. Sometimes in bursts of 100 over a few seconds and sometimes a couple every few minutes. We use partition column inference on the source.

```auto
└── day=10
    ├── hour=08
         ├── second=19
         │ └── data.parquet
         ├── second=20
         │ └── data.parquet
         ├── second=21
         │ └── data.parquet

```

After each file is written to disk, we run the Metadata Refresh via the API for each file similar to the following:

```auto
ALTER TABLE "xyz"."table1" REFRESH METADATA FOR PARTITIONS ( "day"='10', "hour"='08', "second"='19');

```

On average we are seeing refreshes take between 30-60 seconds per file. Which seems surprising for a 2MB parquet file with ~400 columns and ~1000 rows, but maybe not?

After 2 or more concurrent REFRESH METADATA calls (within a few seconds of each other) we start seeing jobs fail with the error:

> Unable to refresh metadata for the dataset (due to concurrent updates). Please retry.

and sometimes from the same refresh calls we have seen other errors:

> com.dremio.common.exceptions.UserRemoteException: CONCURRENT\_MODIFICATION ERROR: Unable to refresh metadata for the dataset (due to concurrent updates). Please retry.

> NessieReferenceConflictException: Values of existing and expected content for key ‘dremio.internal./dremio-b14a93fa-6981-4b9e-998e-c036056fc230/metadata/6e859ad3-dc84-4555-af79-78025d289af4’ are different.

We need the data to be present in Dremio asap, hence why we cant rely on scheduled refreshes.

Any help understanding how we can best refresh the metadata other than just keep retrying until it eventually everything succeeds?

Any help would be most appreciated. 🙏

Cheers,  
Jonathan

---

<div class="post-metadata">

**Author:** ![cindy.la](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/cindy.la/32/6730_2.png) [@cindy.la](https://community.dremio.com/u/cindy.la)\
**Post date:** [December 6, 2023, 10:34pm UTC](https://community.dremio.com/t/unable-to-refresh-metadata-for-the-dataset-due-to-concurrent-updates-please-retry/11245/2 "2023-12-06T22:34:35Z")

</div>

> [@jdwills](#):
>
> CONCURRENT\_MODIFICATION ERROR: Unable to refresh metadata for the dataset (due to concurrent updates). Please retry.

Hi @jdwills ,

I’d be happy to look into this for you further. Would you mind private messaging me your organization ID, as well as any job IDs for these failed metadata refreshes?

Cindy

---

<div class="post-metadata">

**Author:** ![jdwills](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/jdwills/32/5230_2.png) [@jdwills](https://community.dremio.com/u/jdwills)\
**Post date:** [December 6, 2023, 10:58pm UTC](https://community.dremio.com/t/unable-to-refresh-metadata-for-the-dataset-due-to-concurrent-updates-please-retry/11245/3 "2023-12-06T22:58:23Z")

</div>

Thanks @cindy.la PM’d you.