# Syntax of Refresh Metadata for partitions **URL:** \ **Category:** Dremio University\ **Created:** [November 8, 2021, 8:00pm UTC](https://community.dremio.com/t/syntax-of-refresh-metadata-for-partitions/8304 "2021-11-08T20:00:39Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![eduardoslopes](https://avatars.discourse-cdn.com/v4/letter/e/a183cd/32.png) [@eduardoslopes](https://community.dremio.com/u/eduardoslopes)\ **Post date:** [November 8, 2021, 8:00pm UTC](https://community.dremio.com/t/syntax-of-refresh-metadata-for-partitions/8304/1 "2021-11-08T20:00:39Z")
Hello, I’m testing the usage of manual refresh metadata for specific partitions. My datasets has four partition levels (dir0, dir1, dir2, dir3 - year, month, day, hour). There is any way to refresh metadata for a specific month or day? For example like this: `ALTER TABLE "mydataset" REFRESH METADATA FOR PARTITIONS ( "dir0" = '2021' AND "dir1" = 'November' AND "dir2" = '8');` or `ALTER TABLE "mydataset" REFRESH METADATA FOR PARTITIONS ( "dir0" = '2021' AND "dir1" = 'November' AND "dir1" = 'October');` I have tested, but this is not the correct syntax and I don’t found any example of usage with more than one partition level. ---
**Author:** ![lenoyjacob](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/lenoyjacob/32/3474_2.png) [@lenoyjacob](https://community.dremio.com/u/lenoyjacob)\ **Post date:** [November 8, 2021, 11:28pm UTC](https://community.dremio.com/t/syntax-of-refresh-metadata-for-partitions/8304/2 "2021-11-08T23:28:02Z")
Try comma separated values, like: ```auto ALTER TABLE "mydataset" REFRESH METADATA FOR PARTITIONS ("dir0" = '2021', "dir1" = 'November', "dir2" = '01', "dir3" = '01'); ALTER TABLE "mydataset" REFRESH METADATA FOR PARTITIONS ("dir0" = '2021', "dir1" = 'October', "dir2" = '01', "dir3" = '01'); ``` Note that a command must include **all partition levels**. ---
**Author:** ![eduardoslopes](https://avatars.discourse-cdn.com/v4/letter/e/a183cd/32.png) [@eduardoslopes](https://community.dremio.com/u/eduardoslopes)\ **Post date:** [November 9, 2021, 2:09pm UTC](https://community.dremio.com/t/syntax-of-refresh-metadata-for-partitions/8304/3 "2021-11-09T14:09:58Z")
Thanks, @lenoyjacob! This syntax did not give an execution error, but seems this is refreshing metadata for all dataset. For example, I have run this command: ```auto ALTER TABLE "mydataset" REFRESH METADATA FOR PARTITIONS ( "dir0" = '2021', "dir1" = 'November', "dir2" = '9', "dir3" = '11'); ``` But the metadata for partition `2021/November/9/12` was also refreshed. Furthermore, the execution time for this refresh was very like to a commom refresh (without specify the partition). This behavior was repeated 3 times with another partitions, so that seems not the expected behavior. ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [November 10, 2021, 6:43am UTC](https://community.dremio.com/t/syntax-of-refresh-metadata-for-partitions/8304/4 "2021-11-10T06:43:29Z")
@eduardoslopes The background refresh would refresh all new partitions and all changed partitions. Is it possible the background metadata refresh would have completed before your ALTER PDS? ---
**Author:** ![eduardoslopes](https://avatars.discourse-cdn.com/v4/letter/e/a183cd/32.png) [@eduardoslopes](https://community.dremio.com/u/eduardoslopes)\ **Post date:** [November 10, 2021, 11:48am UTC](https://community.dremio.com/t/syntax-of-refresh-metadata-for-partitions/8304/5 "2021-11-10T11:48:03Z")
@balaji.ramaswamy I think this is not happening, because I have changed the metadata refresh interval to 24 hours to avoid this possibility. As I have tried to do manual refresh several times, I think this is not possible. ---
**Author:** ![eduardoslopes](https://avatars.discourse-cdn.com/v4/letter/e/a183cd/32.png) [@eduardoslopes](https://community.dremio.com/u/eduardoslopes)\ **Post date:** [November 12, 2021, 6:33pm UTC](https://community.dremio.com/t/syntax-of-refresh-metadata-for-partitions/8304/6 "2021-11-12T18:33:04Z")
@lenoyjacob @balaji.ramaswamy Have you any idea how to solve that? ---
**Author:** ![lenoyjacob](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/lenoyjacob/32/3474_2.png) [@lenoyjacob](https://community.dremio.com/u/lenoyjacob)\ **Post date:** [November 12, 2021, 10:07pm UTC](https://community.dremio.com/t/syntax-of-refresh-metadata-for-partitions/8304/7 "2021-11-12T22:07:41Z")
Hey @eduardoslopes. I’m unable to reproduce what you are seeing. I’ve got a dataset with two sets of partitions `2021/09/29/14` and `2021/09/30/14`. Check count of existing records: ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/7/70299dedcbc96c4e9b58fccbe11c9e3cd778d4b5.png) It’s got 5000 records per partition. Let’s add a new parquet file to both partitions via AWS S3. Let’s refresh metadata only `2021/09/29/14`. Expectation is to see only the count for this partition to increase. ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/8/89c7a1790ca3860d8be710f7366dccb3915e5884.png) Check count again. ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/a/a5f1a4a0d6b2334550409be9f9b54c25b299353a.png) As you can see only the count in `2021/09/29/14` changed. The other partition count remained the same despite adding a new file. ---
**Author:** ![tid](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/tid/32/5965_2.png) [@tid](https://community.dremio.com/u/tid)\ **Post date:** [November 15, 2021, 9:07am UTC](https://community.dremio.com/t/syntax-of-refresh-metadata-for-partitions/8304/8 "2021-11-15T09:07:32Z")
Hello @eduardoslopes, I had similar issues. For me, the root cause was that I was testing the functionality in a local container with distStorage set to “local”. Incremental metadata refresh only works if you’re using a non-local distStorage location (Dremio stores the Iceberg metadata there). Unfortunately, the syntax works fine, there is no error thrown, but the behavior is just not what you’d expect. Thanks, Tim ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [November 15, 2021, 2:01pm UTC](https://community.dremio.com/t/syntax-of-refresh-metadata-for-partitions/8304/9 "2021-11-15T14:01:20Z")
@tid Thanks a lot for catching that, @eduardoslopes That is correct, to benefit all features of unlimited splits and Iceberg data, the metadata or dist store needs to got to a distributed storage liek S3/HDFS/Azure Storage ---
**Author:** ![eduardoslopes](https://avatars.discourse-cdn.com/v4/letter/e/a183cd/32.png) [@eduardoslopes](https://community.dremio.com/u/eduardoslopes)\ **Post date:** [November 16, 2021, 2:40pm UTC](https://community.dremio.com/t/syntax-of-refresh-metadata-for-partitions/8304/10 "2021-11-16T14:40:44Z")
Thanks @lenoyjacob @tid @balaji.ramaswamy, My test enviroment are already using distStorage to a S3 bucket. Maybe my issue is because my data are json formatted. That makes sense? Are this features available only for iceberg formatted datasets? ---
**Author:** ![tid](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/tid/32/5965_2.png) [@tid](https://community.dremio.com/u/tid)\ **Post date:** [November 16, 2021, 9:41pm UTC](https://community.dremio.com/t/syntax-of-refresh-metadata-for-partitions/8304/11 "2021-11-16T21:41:55Z")
Hi @eduardoslopes, that is most likely the root cause. The release notes ([Dremio](http://docs.dremio.com/advanced-administration/metadata-caching/#improved-metadata-refreshes-preview)) say: (quoting) When preview access is activated, the split limitation is removed for the following types of datasets: - FileSystem sources (S3, ADLS, HDFS) using: - Parquet formatted tables - Iceberg formatted tables - Delta Lake formatted tables - Hive sources (Hive 2 and Hive 3) using: - Parquet formatted tables - Avro formatted tables - ORC formatted tables (non-transactional only) (end of quote) Best regards, Tim ---
**Author:** ![eduardoslopes](https://avatars.discourse-cdn.com/v4/letter/e/a183cd/32.png) [@eduardoslopes](https://community.dremio.com/u/eduardoslopes)\ **Post date:** [November 17, 2021, 12:07pm UTC](https://community.dremio.com/t/syntax-of-refresh-metadata-for-partitions/8304/12 "2021-11-17T12:07:21Z")
Thanks @tid, This is probably the problem, but the release notes did not correlate the section Unlimited Splits with the section that explain about manual metadata refresh. So I thought there was no dependency between the features. @balaji.ramaswamy @tid @lenoyjacob there is any plan to implement this metadata refresh improvements also for json formatted datasets on future versions?