Hi Laurent - Can you please provide a few external schedular tools which can be connected to Dremio.
Also it would be helpful if you can give me any community link where this already being discussed.
Hi Laurent - I need to Query an Elastic search index which is created on a daily basis. Is there any way i can Schedule a job say every 4 hours so that the query runs and pulls the data from index at specified interval.
is it a new question, unrelated to the current topic? Make sure to open a new topic next time so it’s easier for people to find answers for similar questions. As for now, Dremio has no way to run a query periodically, but that should be something easy to do with an external tool connecting to Dremio.
I don’t think there is a community link for this yet. My request was more about opening a new one if you have a new question so that people can discover it more easily.
As for schedulers, I cannot make any recommendation as it really depends on your need, but it can be as simple as a tool like sqlline to run JDBC queries and crontab to run a script periodically.
Hi Laurent - I am running a Dremio sql query on an ElasticSearch index which has my application log details (pls see sample below). Basically i am extracting required details for my daily metrics from the logs. I can rename the daily index files (ES-196"."logstash-order-2018.03.04-log) which is created on a daily basis to a common name like ‘test-alias’ as described in one of your forum.
My requirement is to shcedule this query at 1:00 AM early morning. Are you saying Dremio supports Crontab ? If so how to configure it. I understand you cannot comment on the specific external tools, but how to integrate external scheduler tools to Dremio. Is there any documenation available ?
Any help on this is highly appreciated.
SELECT operation, serviceName, COUNT() AS Count_Star
SELECT extract_pattern(log_message, '(?<=Label=)(.)(?=,LastValue)’, 0, ‘INDEX’) AS operation,extract_pattern(log_message, ‘\d+’, 0, ‘INDEX’) AS lastvalue, extract_pattern(log_message, ‘(?<=Last_Access=)(.)(?=))’, 0, ‘INDEX’) AS lastaccess, serviceName, cspUserId, serviceVersion
FROM “ES-196”.“logstash-order-2018.03.04-log”.syslog AS syslog
WHERE regexp_like(log_message, '.?\QLabel=\E.*?’) ) nested_0
GROUP BY operation , serviceName
Sorry, but it’s not very feasible for me to reply to private messages. Could you please open a new topic at https://community.dremio.com?