NAS Source - Missing Files

1

I have set up a NAS Source with files that contain json data.

The files are uniquely named using a guid.

I’ve re-created the source data, and obviously the guid is different. A file is in the appropriate directory, when I edit the dataset the preview keeps showing stale data (it was in an old file that I deleted). The error is:

Failure reading JSON file - File file:/C:/public-knowledge-library/datalake/data/normalmeasurement/com.daiwacm.botech.knowledge.client.corporate.houseaccount.HouseAccountMeasurement/29ccc3da-0613-4c88-a195-8b445e1035b8.json does not exist

Restarting Dremio does not fix the problem.

I’ve tried playing around with the Source metadata caching (does not work), and the acceleration (does not work).

I’ve also tried to mess about with reflections and this does not work either.

How can I force a complete reload of the dataset?

Regards.

Dave.

2

If your old VDS query was pointing to that exact file (29ccc3da-0613-4c88-a195-8b445e1035b8.json) which has now been deleted, the behavior is expected since Dremio is searching for that exact file to query. Maybe you can copy/paste the old query into a new VDS and just change the source?

3

Thanks Anthony.

The NAS Source is a directory not a file.

Does this mean that when a NAS Source is a directory, we can add files, but never remove them?

Regards.

Dave.

4

Ah ok makes sense. Here is documentation how to delete & force metadata refresh on a PDS (I assume you made the entire dir into a source) - https://docs.dremio.com/sql-reference/sql-commands/datasets.html#managing-physical-datasets