I am using this API’s catalog/{id}/refresh to refresh a PDS. but I am running into the following error.
response code - 400
{“errorMessage”:“Something went wrong. Please check the log file for details, see https://docs.dremio.com/advanced-administration/log-files.html",“moreInfo”:"Can only refresh datasets but found [com.dremio.service.namespace.source.proto.SourceConfig].”}
Is there a different way to refresh the PDS?
Seems your id is pointing at a source and not a dataset.
Hi Doron,
I just realized that. I was able to refresh the dataset using a catalog id that is tied to the dataset.
Can you please tell me how to know whether the dataset is refreshed or not? is there a separate api that I can hit to check the completion of refresh?
We currently do not have an API for that, the best you can do is look at running jobs but there is no way to easily figure out which job it is (since a refresh can cause many jobs to be run if there are many reflections.
Thank you for confirming it. When i execute the the catalog refresh end point, I dont see any refresh queries under job section and yet I get a 204 back. where should i start troubleshooting this?
/api/v3/catalog/611e7a1f-1bed-4a08-ac62-66ffc939c500/refresh
@sraavi Is it possible you did not select all job types? Make sure to choose acceleration as shown below
Hi Balaji,
Thank you for your response.
All the options are checked. I still do not see any load queries being fired except the scheduled ones. What is a best way to start debugging such this issue?
Thank you
Shailendar
@sraavi Do you see a REFRESH REFLECTION and not a LOAD MATERIALIZATION job? This can happen if the REFRESH returns zero rows. Can you let it refresh once via background and see if both REFRESH and LOAD jobs show as successful?
Hello balaji,
I am refreshing a PDS. This does not have any reflections set up. I am able to do it via sql api using the below command.
alter pds “PDS.path” refresh metadata auto promotion force update;
I was unable to refresh it through the refresh api - /api/v3/catalog/611e7a1f-1bed-4a08-ac62-66ffc939c500/refresh
I get a 204 back but I don’t see the alter command being executed in the background under jobs tab. I hope this provides a better context?
please let me know if you need more info.
Thank you for helping us out.
The API Refresh a Catalog is not to refresh metadata for a PDS but to refresh for any VDS/PDS but you pass the PDS ID, if there is. VDS joining 2 PDS then you can pass the catalog id of any one of the PDS
The above API has nothing to do with metadata refresh
Hello Balaji,
Please there’s this issue I’m having with reflections of VDS failing during refreshes, how do I resolve this issue because there’s so many failures during monitoring.
Thank you for your swift response.
I’m suspecting the longer wait time between load-refresh-drop of the jobs even though refreshes happens every 3hrs but some reflections goes into “in-progress” state and drops/fails afterwards
