# S3 : streaming JSON files in a bucket as new documents / records
**URL:** https://community.dremio.com/t/s3-streaming-json-files-in-a-bucket-as-new-documents-records/3511
**Category:** Uncategorized
**Created:** [June 5, 2019, 10:35am UTC](https://community.dremio.com/t/s3-streaming-json-files-in-a-bucket-as-new-documents-records/3511 "2019-06-05T10:35:05Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@mlb](https://community.dremio.com/u/mlb)
#### Post date: [June 5, 2019, 10:35am UTC](https://community.dremio.com/t/s3-streaming-json-files-in-a-bucket-as-new-documents-records/3511/1 "2019-06-05T10:35:05Z")
Hi,
I created a datasource out of a S3 bucket, which contains JSON files (records or documents).
New JSON files are continuously incoming into the bucket -
However, when querying the datasource with Dremio, new records are not appearing.
Is this pattern applicable with Dremio + S3 ?
Or should the datasource be MongoDB or something else ?
---
### Author:  [@Venugopal\_Menda](https://community.dremio.com/u/Venugopal_Menda)
#### Post date: [June 6, 2019, 5:03am UTC](https://community.dremio.com/t/s3-streaming-json-files-in-a-bucket-as-new-documents-records/3511/2 "2019-06-06T05:03:07Z")
Hi @mlb
How frequent the new files are coming into the bucket?. What is the Metadata refresh policy for the source you have?. Default is 1 hr, meaning metadata for the source will get update every hour, within that hour if there is a new file creates in the source it will not be visible in Dremio.
So you need to set the Metadata refresh time according to your work load,i.e how frequently new files writing and how frequently newly created files queried from Dremio.
@Venugopal_Menda
---
### Author:  [@mlb](https://community.dremio.com/u/mlb)
#### Post date: [June 6, 2019, 7:13am UTC](https://community.dremio.com/t/s3-streaming-json-files-in-a-bucket-as-new-documents-records/3511/3 "2019-06-06T07:13:32Z")
Thanks @Venugopal_Menda -
I figured this out and reduced it to 1mn. I am considering MongoDB for a realtime alternative datasource.
---
### Author:  [@koolay](https://community.dremio.com/u/koolay)
#### Post date: [September 4, 2020, 2:23am UTC](https://community.dremio.com/t/s3-streaming-json-files-in-a-bucket-as-new-documents-records/3511/4 "2020-09-04T02:23:19Z")
Is heavy if the metadata is refreshed thousands times per second?
---
### Author:  [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)
#### Post date: [September 8, 2020, 6:17am UTC](https://community.dremio.com/t/s3-streaming-json-files-in-a-bucket-as-new-documents-records/3511/5 "2020-09-08T06:17:29Z")
@koolay
What is the reason behind refreshing metadata 1000 times per second, if the entire source need not be refreshed, then you can only refresh the datasets you need by using the below command
[https://docs.dremio.com/sql-reference/sql-commands/datasets.html#managing-physical-datasets](https://docs.dremio.com/sql-reference/sql-commands/datasets.html#managing-physical-datasets)
---
### Author:  [@koolay](https://community.dremio.com/u/koolay)
#### Post date: [September 8, 2020, 8:15am UTC](https://community.dremio.com/t/s3-streaming-json-files-in-a-bucket-as-new-documents-records/3511/6 "2020-09-08T08:15:50Z")
@balaji.ramaswamy
Because records of PDS is from the streaming of kafka, and we want to query it by real time.
---
### Author:  [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)
#### Post date: [September 26, 2020, 7:18am UTC](https://community.dremio.com/t/s3-streaming-json-files-in-a-bucket-as-new-documents-records/3511/7 "2020-09-26T07:18:36Z")
@koolay
We would be coming up with features that do not require so much metadata requests. Will come as a feature in near time future releases
---
### Author:  [@koolay](https://community.dremio.com/u/koolay)
#### Post date: [September 29, 2020, 6:45am UTC](https://community.dremio.com/t/s3-streaming-json-files-in-a-bucket-as-new-documents-records/3511/8 "2020-09-29T06:45:51Z")
@balaji.ramaswamy
Thanks for your reply.
Is there a timeline about it?
Dremio is a greate product with best performance.
It is unfortunately that dremio can not support streaming like `Delta lake` and `Hudi`.
And the streaming is more and more important.
---
### Author:  [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)
#### Post date: [October 1, 2020, 3:13am UTC](https://community.dremio.com/t/s3-streaming-json-files-in-a-bucket-as-new-documents-records/3511/10 "2020-10-01T03:13:48Z")
@koolay
Expect V1 by EOY 2020
Thanks
Bali