Logs - best practices on AWS

1

My goal is to enable quick troubleshooting without having to SSH into the dremio instances on AWS. I am considering building a method to copy the dremio logs mentioned here https://docs.dremio.com/advanced-administration/log-files.html

Do I need to track logs from the executor nodes? What do they offer that the master coordinator does not log itself?

I plan to implement something similar to this AWS blog on exporting logs to S3 https://aws.amazon.com/blogs/mt/how-to-export-ec2-instance-execution-logs-to-an-s3-bucket-using-cloudwatch-logs-lambda-and-cloudformation/

I assume adding logging into a custom cloudformation template will add some maintenance burden. Are there any existing methods or best practices to use?

2

@datocrats-org, you should definitely track all logs generated by the executor processes as you can make detailed observations about what work they are doing for which query, and identify garbage collection and zookeeper issues when they arise.

However, we do not currently have any best practices for persisting the logs in a standalone AWS deployment. As you observed, they are currently written to the local filesystem of each node.

3

Thanks, good to know. Do you know any existing methods for getting the executor instance ids using dremio APIs or logs from the master coordinator?