Hello, I have installed Dremio community version 18.2 in production environment and 19.1 in test envrionment.
But recently I found a problem for API catalog refresh : REST API Documentation Dremio
I have a same script in 2 environments to trigger catalog refresh for a PDS, after sending the request and refresh this PDS, it will relaunch all the VDS’s reflections associated with this pds. it works well with version 18.2 , but in my test envrionment with version 19.1 it doesn’t rerun the associated reflections of VDS, I tried another version 19.3 too but still have the same result
If I click button “Refresh Now” on that pds, it works fine with all versions.
The VDS I have is based on this PDS, normally if I refresh PDS it will implement more data (Refresh Method:Incremental update based on new files ) so the related reflctions on VDS shoud be triggered.
Can you help me with this issue ?
Thanks
Clicking “Refresh Now” and calling the REST API do exactly the same thing. If you look at the network requests coming out of your browser, you’ll see that it hits the same REST API endpoint.
Here are a couple of other pointers:
- Clicking on Refresh Now won’t update dependent reflection in the FAILED state. (i.e. Red icon in the Reflection Admin UI or sys.reflections.status=FAILED) To fix the FAILED reflection, toggle to disable, save and then re-enable.
- For file based incremental refreshes, refresh metadata has to happen first on the PDS so that Dremio knows about the new files.
- If you are not getting a new Job after calling the REST API, then there could be a dependency issue. You can query sys.reflection_dependencies to check what your VDS’s reflection depends on. It needs to depend on the same PDS that you are passing in in the REST API. (It can also depend on the PDS through an intermediary reflection).
1 Like
Thanks @bennychow
@alpha94511 Let us know if you are able to find the issue
Thanks for the update @alpha94511
