Yarn Log Handling Only Partially Works

1

We are using YARN for our cluster provisioning. I can see in the code that YarnTwillLogHandler is supposed to process and re-log messages coming from the Yarn job.

In practice, this seems to only receive a very small subset of log messages from Yarn. I can see the messages coming in from the various hosts, but they are always high level instead of the full log you see when you check STDOUT on the containers logs.

Is there a configuration/code tweak I can make to get the full logs of the remote containers to show up instead of just partials?

2

@patricker

Is this still an issue? i think we can redirect it to a different location so the stdout will remain the same but the missing message would go to the redirected log

3

I haven’t tested this in anything newer than Dremio 4.0, but we worked around this a while ago by adding Splunk logging over TCP, with a different TCP port for each Dremio cluster.