# Dremio fails to recognize new folders

**URL:** <https://community.dremio.com/t/dremio-fails-to-recognize-new-folders/7682>\
**Category:** Uncategorized\
**Created:** [June 30, 2021, 3:20pm UTC](https://community.dremio.com/t/dremio-fails-to-recognize-new-folders/7682 "2021-06-30T15:20:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![cklar](https://avatars.discourse-cdn.com/v4/letter/c/b3f665/32.png) [@cklar](https://community.dremio.com/u/cklar)\
**Post date:** [June 30, 2021, 3:20pm UTC](https://community.dremio.com/t/dremio-fails-to-recognize-new-folders/7682/1 "2021-06-30T15:20:57Z")

</div>

Hi,

In line with the question here ([Dremio with reflection on updated (append) dataset](https://community.dremio.com/t/dremio-with-reflection-on-updated-append-dataset/7680)) I wanted to ask the following:

I’m referencing the following structure:

dir0

- date1
- date2  
-…

And within each date folder I have the same parquet file.

I want to be able to query for the most recent date with below query which works when setting it up. However when adding new folders it doesn’t take into account the new folders when I rerun the query. Somehow dremio fails to recognize them. I’m not running any reflection.

SELECT rundate,variable,“value”,source,dir1 as dealkey,dir0 as filetimestamp FROM mergeperdealkey WHERE dir0 IN (  
SELECT TO\_CHAR(timeconverted,‘YYYYMMDDHH24MISS’) FROM (  
SELECT to\_timestamp(timestring,‘YYYYMMDDHH24MISS’) as timeconverted FROM (SELECT DISTINCT(dir0) as “timestring” FROM mergeperdealkey) ORDER BY timeconverted desc LIMIT 1  
)  
)

Regards,  
Christian

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [July 2, 2021, 8:16am UTC](https://community.dremio.com/t/dremio-fails-to-recognize-new-folders/7682/2 "2021-07-02T08:16:39Z")

</div>

@cklar Have you refreshed the dataset?

[https://docs.dremio.com/sql-reference/sql-commands/datasets/](https://docs.dremio.com/sql-reference/sql-commands/datasets/)

You can also wait for the background refresh to complete, check metadata\_refresh.log on the coordinator

---

<div class="post-metadata">

**Author:** ![cklar](https://avatars.discourse-cdn.com/v4/letter/c/b3f665/32.png) [@cklar](https://community.dremio.com/u/cklar)\
**Post date:** [July 9, 2021, 11:48am UTC](https://community.dremio.com/t/dremio-fails-to-recognize-new-folders/7682/3 "2021-07-09T11:48:07Z")

</div>

Thanks - I’ll try it.

Right now I switched the refresh cycle to 1 min which seems to show me folder updates every 90 seconds. Do you know if setting the refresh cycle so low has any performance impacts?

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [July 12, 2021, 8:29am UTC](https://community.dremio.com/t/dremio-fails-to-recognize-new-folders/7682/4 "2021-07-12T08:29:04Z")

</div>

@cklar Yes, depending how often the datasets change, the coordinator can be under severe heap pressure causing full GC cycles