# AWS S3 costs caused ListBucket request on metadata refresh
**URL:** https://community.dremio.com/t/aws-s3-costs-caused-listbucket-request-on-metadata-refresh/8234
**Category:** Dremio University
**Created:** [October 29, 2021, 7:53pm UTC](https://community.dremio.com/t/aws-s3-costs-caused-listbucket-request-on-metadata-refresh/8234 "2021-10-29T19:53:26Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@eduardoslopes](https://community.dremio.com/u/eduardoslopes)
#### Post date: [October 29, 2021, 7:53pm UTC](https://community.dremio.com/t/aws-s3-costs-caused-listbucket-request-on-metadata-refresh/8234/1 "2021-10-29T19:53:26Z")
Hi,
I’m using dremio to read data from a S3 source. Our data are partitioned in four levels (year, month, day and hour) to increase query performance. I have noticed that the AWS costs caused by the ListBucket request increases a lot, according with Metadata Refresh interval decreases or when new datasets are created.
There is some way to reduce this cost without decreasing too much the metadata refresh frequency?
---
### Author:  [@eduardoslopes](https://community.dremio.com/u/eduardoslopes)
#### Post date: [November 1, 2021, 12:32pm UTC](https://community.dremio.com/t/aws-s3-costs-caused-listbucket-request-on-metadata-refresh/8234/2 "2021-11-01T12:32:55Z")
I have forgot to say, but I’m using Dremio version 4.7.3.
---
### Author:  [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)
#### Post date: [November 7, 2021, 4:37pm UTC](https://community.dremio.com/t/aws-s3-costs-caused-listbucket-request-on-metadata-refresh/8234/3 "2021-11-07T16:37:53Z")
@eduardoslopes Dremio 19.0 should do minimal metadata requests, please review release notes and documentation starting Dremio 18.0 on how to turn on Iceberg and unlimited splits
[https://docs.dremio.com/release-notes/1800-release-notes/](https://docs.dremio.com/release-notes/1800-release-notes/)
---
### Author:  [@eduardoslopes](https://community.dremio.com/u/eduardoslopes)
#### Post date: [November 8, 2021, 2:08pm UTC](https://community.dremio.com/t/aws-s3-costs-caused-listbucket-request-on-metadata-refresh/8234/4 "2021-11-08T14:08:23Z")
Thanks, @balaji.ramaswamy!
I’m starting tests with Dremio 18.2.0 on dev environment. Do you know if metadata refresh improvements works for json formatted tables or only for iceberg formatted tables?
---
### Author:  [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)
#### Post date: [November 10, 2021, 6:57am UTC](https://community.dremio.com/t/aws-s3-costs-caused-listbucket-request-on-metadata-refresh/8234/5 "2021-11-10T06:57:47Z")
@eduardoslopes The new flow is only for PARQUET/ORC and AVRO format files
---
### Author:  [@eduardoslopes](https://community.dremio.com/u/eduardoslopes)
#### Post date: [November 10, 2021, 12:01pm UTC](https://community.dremio.com/t/aws-s3-costs-caused-listbucket-request-on-metadata-refresh/8234/6 "2021-11-10T12:01:45Z")
@balaji.ramaswamy Thanks!
Can you explain me if there is any other way to reduce this costs and continue using json data format? Our data has complex schemas, and is hard for us to use formats like Parquet.
---
### Author:  [@chulucninh09](https://community.dremio.com/u/chulucninh09)
#### Post date: [November 10, 2021, 2:01pm UTC](https://community.dremio.com/t/aws-s3-costs-caused-listbucket-request-on-metadata-refresh/8234/7 "2021-11-10T14:01:58Z")
@eduardoslopes I faced the same issue when promote our cloudtrail logs bucket to dataset. I’m currently work around to run a batch job to transform cloudtrail json logs into parquet with iceberg table format. That was clunky to setup but after that, the performance is much better and also the cost of ListBucket too.
About latency, I experience the same because if you don’t refresh Dremio metadata, you also get no information about new json files.