Info: Dremio on EMR - Failed to set up YARN

I was attempting to set up dremio on AWS EMR and couldn’t get the master nodes to submit the Yarn job. It would fail immediately with this error:

Exception in thread “main” java.lang.NoClassDefFoundError:
at java.lang.Class.getDeclaredMethods0(Native Method) .
at java.lang.Class.privateGetDeclaredMethods(
at java.lang.Class.privateGetMethodRecursive(
at java.lang.Class.getMethod0(
at java.lang.Class.getMethod(
at org.apache.twill.launcher.TwillLauncher.main(
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at java.lang.ClassLoader.loadClass(
at java.lang.ClassLoader.loadClass(
… 6 more

After a lot of debugging, I discovered the yarn-site.xml file I was providing to dremio contained new lines in the yarn.application.classpath value. These new lines caused a failure in twill (a yarn helper) because twill expects the configuration to be one line. Hope this helps someone out there.

1 Like

Hello, is there a “guide” to install Dremio on EMR?