# Unable to do METADATA REFRESH FOR PARTITIONS **URL:** \ **Category:** Dremio Cloud\ **Created:** [October 18, 2023, 4:12pm UTC](https://community.dremio.com/t/unable-to-do-metadata-refresh-for-partitions/11097 "2023-10-18T16:12:55Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![rohitshetty](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rohitshetty](https://community.dremio.com/u/rohitshetty)\ **Post date:** [October 18, 2023, 4:12pm UTC](https://community.dremio.com/t/unable-to-do-metadata-refresh-for-partitions/11097/1 "2023-10-18T16:12:55Z")
I am running Dremio cloud (Not software) - and using S3 as a source. I formatted a folder with a parquet that looks like this. ```auto └── year=2023 ├── month=08 │ ├── day=19 │ │ └── userdata.parquet │ ├── day=20 │ │ └── userdata.parquet │ └── day=21 │ └── userdata.parquet └── month=09 (Added later after first ingestion - I am trying to refresh metadata for this) └── day=15 └── userdata.parquet ``` Then queried to see all my data correctly. Then added new data to month=09/day=15. Running this gives me the errors in the screenshot. ```auto ALTER TABLE "xyz"."standalone-test_1" REFRESH METADATA FOR PARTITIONS ( dir0='year=2023', dir1 = 'month=09', dir2 = 'day=15', "year"='2023', "month"='09', "day"='15' ); ``` ![ksnip_20231018-220417](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/5/5942b9e042df5c91b17b001bc70e5017f5148468.png) “Input error. Expected partition dir0” Also, another screenshot (Cannot attach it due to new signup restrictions - which I got by accidentally misnaming one of the partitions) shows me the list of partitions - and I can confirm, that I have included all partitions too, but I still am unable to refresh the metadata for the partition. Although REFRESH METADATA without “FOR PARTITIONS” works as intended. What am I doing wrong? Please help. I am attaching the folder for reference. [test\_parquet.zip](https://community.dremio.com/uploads/short-url/oMh5e08eJemL10jnAseTBhnvRRj.zip) (154.9 KB) ---
**Author:** ![lenoyjacob](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/lenoyjacob/32/3474_2.png) [@lenoyjacob](https://community.dremio.com/u/lenoyjacob)\ **Post date:** [October 18, 2023, 7:55pm UTC](https://community.dremio.com/t/unable-to-do-metadata-refresh-for-partitions/11097/2 "2023-10-18T19:55:32Z")
@rohitshetty, Welcome to Dremio Community! Try the following command. Works for me on your sample dataset: ```auto ALTER TABLE path.to.dataset REFRESH METADATA FOR PARTITIONS ( dir0 = 'year=2023', dir1 = 'month=08', dir2 = 'day=21' ); ``` ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/6/6b787c6122994191bc665961a566110637524a27.png) ---
**Author:** ![jdwills](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/jdwills/32/5230_2.png) [@jdwills](https://community.dremio.com/u/jdwills)\ **Post date:** [October 18, 2023, 9:08pm UTC](https://community.dremio.com/t/unable-to-do-metadata-refresh-for-partitions/11097/3 "2023-10-18T21:08:36Z")
Hi @lenoyjacob, One thing I think @rohitshetty forgot to mention (we work together) is that the source requires “Enable partition column inference” to be set, which creates partitions for the inferred columns too. If we do not include those partitions in the REFRESH command we get the following error: ![Screen Shot 2023-10-18 at 1.54.08 PM](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/0/04a9d5ed197e805511da6fa20cfe34f22e1f5290.png) When we do include them we get the original error “Input error. Expected partition dir0”. Do you think you can run your test again with the partition inference enabled? All the help is very much appreciated. Cheers, Jonathan ---
**Author:** ![lenoyjacob](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/lenoyjacob/32/3474_2.png) [@lenoyjacob](https://community.dremio.com/u/lenoyjacob)\ **Post date:** [October 19, 2023, 6:26am UTC](https://community.dremio.com/t/unable-to-do-metadata-refresh-for-partitions/11097/4 "2023-10-19T06:26:39Z")
Yup, looks like a bug. I’ve raised a ticket internally. As a workaround disable partition inference and use the metadata refresh [query](https://community.dremio.com/t/unable-to-do-metadata-refresh-for-partitions/11097/2) I posted above. For regular queries, you can create a View to abstract away the prefixed “dir0=”, “dir1=” and “dir2=” using something like [split\_part()](https://docs.dremio.com/cloud/reference/sql/sql-functions/functions/SPLIT_PART/). ---
**Author:** ![rohitshetty](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rohitshetty](https://community.dremio.com/u/rohitshetty)\ **Post date:** [October 19, 2023, 10:42am UTC](https://community.dremio.com/t/unable-to-do-metadata-refresh-for-partitions/11097/5 "2023-10-19T10:42:24Z")
Thank you @lenoyjacob , and thank you for raising the ticket too. I disabled partition inference, and used metadata refresh just as you did, and can confirm it works. I am wondering what are the implications of not having a partition inference. Would there be performance penalties? ---
**Author:** ![lenoyjacob](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/lenoyjacob/32/3474_2.png) [@lenoyjacob](https://community.dremio.com/u/lenoyjacob)\ **Post date:** [October 19, 2023, 3:41pm UTC](https://community.dremio.com/t/unable-to-do-metadata-refresh-for-partitions/11097/6 "2023-10-19T15:41:45Z")
@rohitshetty @jdwills Quick update. Turns out this has been fixed in [24.2.3](https://docs.dremio.com/current/release-notes/version-240-release#2423-release-notes-october-2023-enterprise:~:text=Fixed%20a%20number%20of%20issues%20that%20were%20affecting%20proper%20handling%20of%20inferred%20partition%20columns%2C%20specifically%20FOR%20PARTITIONS%20(...)%20was%20not%20working%20for%20inferred%20partition%20columns.) and [23.2.3](https://docs.dremio.com/current/release-notes/version-230-release#2323-release-notes-october-2023-enterprise:~:text=Fixed%20a%20number%20of%20issues%20that%20were%20affecting%20proper%20handling%20of%20inferred%20partition%20columns%2C%20specifically%20FOR%20PARTITIONS%20(...)%20was%20not%20working%20for%20inferred%20partition%20columns.). And should be fixed in the next release cycle for Dremio Cloud. > [@rohitshetty](#): > > I am wondering what are the implications of not having a partition inference. Would there be performance penalties? There shouldn’t be a performance implication using the workaround. You should be able to see partition pruning happening in the raw profile of the query. IMO, partition inference is more of a convenience feature. ---
**Author:** ![rohitshetty](https://avatars.discourse-cdn.com/v4/letter/r/b2d939/32.png) [@rohitshetty](https://community.dremio.com/u/rohitshetty)\ **Post date:** [October 19, 2023, 6:42pm UTC](https://community.dremio.com/t/unable-to-do-metadata-refresh-for-partitions/11097/7 "2023-10-19T18:42:24Z")
Thanks! > And should be fixed in the next release cycle for Dremio Cloud. That is great to hear! Do you know the approximate time when that would be? Thank you again for all your guidance so far! ---
**Author:** ![lenoyjacob](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/lenoyjacob/32/3474_2.png) [@lenoyjacob](https://community.dremio.com/u/lenoyjacob)\ **Post date:** [November 17, 2023, 12:07am UTC](https://community.dremio.com/t/unable-to-do-metadata-refresh-for-partitions/11097/8 "2023-11-17T00:07:54Z")
This should be now fixed. It was part of the November 16th, 2023 update of Dremio Cloud: [Changelog | Dremio Documentation](https://docs.dremio.com/cloud/changelog/#:~:text=Handling%20of%20inferred%20partition%20columns%20is%20improved.%20Specifically%2C%20FOR%20PARTITIONS%20(...)%20now%20works%20properly%20for%20inferred%20partition%20columns). ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/1/1dbee6e88a6428bd7b36d5f84e0060a1d1006c4d.png)