How to set new coordinador nodes or change deploy? Master 100% CPU AWS Edition

1

I’m currently runing Dremio 21.x on AWS Edition, but mt coordinator runs almost on 100% CPU in some cases.

My datadog monitor above

image
image1605×531 70.3 KB

This has crashing the entire operation and i read in the docs that AWS can’t have more than one coodinator, it’s possible to migrate from AWS Editon to YARN or other solution?

2

@hl-nahuz Let us first find out why your CPU is at 100%, if we are able to address this then we do not have to migrate,

  • How many sources
  • Types of sources
  • Refresh frequency of each source
  • All GC and server logs from coordinator when this issue happened
  • thread dumps, script below when problem happened
  • ps -ef | grep dremio
  • login as the the unix id running the Dremio process
  • mkdir jstack
  • cd jstack
  • vi jstack.sh (run jstack more frequently if the time to capture is very less)

#!/bin/bash

for i in seq -w 3 1 300
do
jstack -l dremio-pid > ThreadDump$i.txt
sleep 1
done

  • chmod +x jstack.sh
  • ./jstack.sh
  • tar -cvf jstack.tar *.txt
  • gzip jstack.tar