Is there any rest api end point to enable/disable reflections.
Yes, you can do that. See here: https://docs.dremio.com/rest-api/reflections/
The common pattern is to disable automatic reflection maintenance and to control that externally, especially when using External Reflections.
Let us know if you need more info in this area.
1 Like
Thanks Kelly,
Could you please share a curl example for the same ( enable /disable reflection )
First thing to do is login to get the authorization token:
curl --request POST \
--url http://localhost:9047/apiv2/login \
--header 'authorization : _dremio6ro5jfgsbue98eeh0vmn9utnoa' \
--header 'content-type: application/json' \
--data '{
"userName": "username",
"password": "password"
}'
That will return a JSON object that contains a token value.
Next you have to get the id for the reflection - you can list all reflections using:
curl --request GET \
--url http://localhost:9047/api/v3/reflection \
--header 'authorization : _dremio${authorization}' \
--header 'content-type: application/json'
This gives you a list of all reflections:
{
"data": [
{
"id": "e7f924ca-0014-4288-85d9-d698c4e342a1",
"type": "AGGREGATION",
"name": "Aggregation Reflection",
"tag": "5",
"datasetId": "77d43461-bf46-47c0-9d04-d8b82b6794e5",
"enabled": true,
"partitionDistributionStrategy": "CONSOLIDATED",
"entityType": "reflection"
}
]
}
The enabled field controls the enabling/disabling of a reflection. To update the reflection, you would PUT the reflection like this (in this case we disable the reflection):
curl --request PUT \
--url http://localhost:9047/api/v3/reflection \
--header 'authorization : _dremio6ro5jfgsbue98eeh0vmn9utnoa' \
--header 'content-type: application/json' \
--data '{
"data": [
{
"id": "e7f924ca-0014-4288-85d9-d698c4e342a1",
"type": "AGGREGATION",
"name": "Aggregation Reflection",
"tag": "5",
"datasetId": "77d43461-bf46-47c0-9d04-d8b82b6794e5",
"enabled": false,
"measureFields": [
{
"name": "PdId",
"measureTypeList": [
"COUNT",
"SUM",
"MIN",
"MAX"
]
},
{
"name": "Descript",
"measureTypeList": [
"COUNT",
"MIN",
"MAX"
]
}
],
"partitionDistributionStrategy": "CONSOLIDATED",
"entityType": "reflection"
}
]
}'
2 Likes
Thanks a lot Doron
minor changes in your last curl
Post -> replace by PUT
Url -> should contain reflection id ( id should not be part of body data)
curl --request PUT
–url http://localhost:9047/api/v3/reflection/61db937a-26c6-4024-b577-ab82ae80427a
–header ‘authorization : _dremioxxx0ic1jnc0iju719blhia8’
–header ‘content-type: application/json’
–data ‘{
“type”: “RAW”,
“name”: “Raw Reflection”,
“tag”: “0”,
“datasetId”: “f8c064c0-e9cc-4715-ad3e-485d85b8e5d7”,
“enabled”: false,
“displayFields”: [
{
“name”: “111”
},
{
“name”: “DateTime_SK”
},
{
“name”: “Stores_SK”
}
],
“partitionDistributionStrategy”: “CONSOLIDATED”,
“entityType”: “reflection”
}’
@doron using your code above returns an error when I try it. The first authorization call, I tried it both ways with the authorization header and without it and it returns the same error.
curl --request PUT \
--url http://localhost:9047/apiv2/login \
--header 'content-type: application/json' \
--data '{
"userName": "user",
"password": "pswd"
}'
This returns
{“errorMessage”:“Something went wrong”,“moreInfo”:“HTTP 405 Method Not Allowed”}
I mistakenly made the login a PUT, it should be a POST. I fixed up the post, sorry about that!
1 Like
@doron Could you tell me how to create the Reflection/Dataset using Rest api, without Dataset (Physical dataset) creation using Rest Api .How we can create the Reflection?
Hi @doron,
Even with your curl API call,
I’m facing this problem when I try to request reflections list on my dremio server.
Why is that? I followed this simple procedure:
curl --request GET \
–url http://${SERVER}:9047/api/v3/reflection \
–header ‘authorization : _dremio$tokenID’ \
–header ‘content-type: application/json’ \
–header ‘Accept: /’ \
–header ‘Connection: keep-alive’
but I’m getting:
{“errorMessage”:“Something went wrong”,“moreInfo”:“HTTP 405 Method Not Allowed”}
I’m stucked =(
Thank you in advance
oirasor
OK, I got it. That Endpoint is available only in Enterprise Edition. =(
Listing all reflections is enterprise edition only, creating reflections APIs are available in all editions.
Exactly, reading deeper the documentation always helps. =)
Just one more thing… refreshing a RAW reflection, as I see in the docs, could be possible with this endpoint:
PUT /api/v3/reflection/{id}
but what I need to add as data parameter?
Could you provide an example with curl?
I already tried providing in --data parameter the output of GET /api/v3/reflection/{id}
Thank you in advance.
Rosario
data parameter should be a reflection model - https://docs.dremio.com/rest-api/reflections/models.html
Was there a particular error you received when you passed it value of /api/v3/reflection/{id} ?
This is the output with GET method:
{
“id”: “74d74055-650c-490b-9acf-6bfefedc93e8”,
“type”: “RAW”,
“name”: “Raw Reflection”,
“tag”: “7”,
“createdAt”: “2018-12-06T14:14:22.365Z”,
“updatedAt”: “2018-12-06T14:14:22.365Z”,
“datasetId”: “aa60ac10-233e-4b2a-9f06-a6f94c01b840”,
“currentSizeBytes”: 2455147228,
“totalSizeBytes”: 2455147228,
“enabled”: true,
“status”: {
“config”: “OK”,
“refresh”: “SCHEDULED”,
“availability”: “AVAILABLE”,
“combinedStatus”: “CAN_ACCELERATE_WITH_FAILURES”,
“failureCount”: 1,
“lastDataFetch”: “2018-12-10T10:44:30.872Z”,
“expiresAt”: “2018-12-11T10:44:30.872Z”
},
“displayFields”: [
{
“name”: “dataIn”
},
{
“name”: “dataOut”
},
{
“name”: “ClasseIn”
},
{
“name”: “ClasseOut”
},
{
“name”: “DescrizioneIn”
},
{
“name”: “DescrizioneOut”
},
{
“name”: “latitudineIn”
},
{
“name”: “longitudineIn”
},
{
“name”: “latitudineOut”
},
{
“name”: “longitudineOut”
}
],
“partitionFields”: [
{
“name”: “dataIn”
}
],
“partitionDistributionStrategy”: “CONSOLIDATED”,
“entityType”: “reflection”
}
And this the output with PUT method … but I don’t know what to add to achieve an Update of reflection…
curl --request PUT --url ‘http://localhost:9047/api/v3/reflection/74d74055-650c-490b-9acf-6bfefedc93e8’ --header ‘authorization : _dremio1jv6rf43j8tmrr7f5g7e2aq5hq’ | jq .
{
“errorMessage”: “Something went wrong”
}
Any suggestions?
For the PUT you should send the entire message you have from the GET - just change enabled to false. Then do the same thing and switch enabled back to true.
Or I miss something or it doesn’t work…
I tried:
curl --silent --request PUT --url ‘http://localhost:9047/api/v3/reflection/74d74055-650c-490b-9acf-6bfefedc93e8’ --header ‘authorization : _dremio1jv6rf43j8tmrr7f5g7e2aq5hq’ --data ‘{“id”:“74d74055-650c-490b-9acf-6bfefedc93e8”,“type”:“RAW”,“name”:“Raw Reflection”,“tag”:“7”,“createdAt”:“2018-12-06T14:14:22.365Z”,“updatedAt”:“2018-12-06T14:14:22.365Z”,“datasetId”:“aa60ac10-233e-4b2a-9f06-a6f94c01b840”,“currentSizeBytes”:2474656982,“totalSizeBytes”:4929804210,“enabled”:false,“status”:{“config”:“OK”,“refresh”:“SCHEDULED”,“availability”:“AVAILABLE”,“combinedStatus”:“CAN_ACCELERATE”,“failureCount”:0,“lastDataFetch”:“2018-12-10T16:44:30.963Z”,“expiresAt”:“2018-12-11T16:44:30.963Z”},“displayFields”:[{“name”:“dataIn”},{“name”:“dataOut”},{“name”:“ClasseIn”},{“name”:“ClasseOut”},{“name”:“DescrizioneIn”},{“name”:“DescrizioneOut”},{“name”:“latitudineIn”},{“name”:“longitudineIn”},{“name”:“latitudineOut”},{“name”:“longitudineOut”}],“partitionFields”:[{“name”:“dataIn”}],“partitionDistributionStrategy”:“CONSOLIDATED”,“entityType”:“reflection”}’ | jq .
{
“errorMessage”: “Something went wrong”,
“moreInfo”: “HTTP 415 Unsupported Media Type”
}
You probably need to set the content type header:
--header 'content-type: application/json'
to JSON, curl I believe defaults to plain text.
Thank you very much. It worked fine. =)