How to automatically transform a directory with json files into a virtual dataset

1

I have formatted a folder with several json files in a Minio object store into a virtual dataset. I have created a view on this dataset. When I remove a file from the object store, the view gets updated - the corresponding row is no longer returned by the view.
However when I add a file, the new row is not returned by the view.
When I navigate to the folder in the Dremio source and save the format again, the new row does get returned.

Is this intentional, and if so, what is the best way to automatically update the view every time files get added?

2

@serra after adding the new file, you have to refresh metadata

ALTER TABLE <PDS_NAME> REFRESH METADATA

1 Like
3

Thank you for the references @balaji.ramaswamy, that works well.

I opted to increase the refresh frequency in the source settings too.

image
image1205×791 36.3 KB

Any insight on why deleting a file does not require a metadata refresh, but adding a file does?

4

@serra Dremio would have done an inline refresh for the delete, can you see select job profile that ran after delete, it would have 2 attempts, one for file not found and the attempt 2 would be the successful run

1 Like
5

Ah, yes. And adding a file does not result in an error on query and hence no inline refresh?

6

To reflect the added file, either you need to explicitly refresh metadata or wait for the background refresh to complete