How to set up development environment for dremio-oss with IntelliJ

#1

Hi,

I tried to set up development environments for dremio-oss with IntelliJ.

  1. mvn clean install -DskipTests
  2. Import source code with pom.xml to IntelliJ

But some unit tests failed. For example, TestHashAgg failed.

java.lang.IllegalArgumentException: resource /com/dremio/exec/expr/fn/impl/gaggr/SumFunctions.java relative to com.dremio.exec.expr.fn.impl.gaggr.SumFunctions$NullableFloat4Sum not found.

	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:383)
	at com.google.common.io.Resources.getResource(Resources.java:209)
	at com.dremio.exec.expr.fn.FunctionInitializer.getSourceURL(FunctionInitializer.java:138)
	at com.dremio.exec.expr.fn.FunctionInitializer.get(FunctionInitializer.java:142)
	at com.dremio.exec.expr.fn.FunctionInitializer.checkInit(FunctionInitializer.java:98)
	at com.dremio.exec.expr.fn.FunctionInitializer.getMethod(FunctionInitializer.java:79)
	at com.dremio.exec.expr.fn.BaseFunctionHolder.meth(BaseFunctionHolder.java:111)
	at com.dremio.exec.expr.fn.BaseFunctionHolder.meth(BaseFunctionHolder.java:107)
	at com.dremio.exec.expr.fn.AggrFunctionHolder.setup(AggrFunctionHolder.java:45)
	at com.dremio.exec.expr.fn.AggrFunctionHolder.renderStart(AggrFunctionHolder.java:110)
	at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitFunctionHolderExpression(EvaluationVisitor.java:209)
	at com.dremio.exec.expr.EvaluationVisitor$InnerMethodNester.visitFunctionHolderExpression(EvaluationVisitor.java:1057)
	at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1160)
	at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:810)
	at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:790)
	at com.dremio.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:43)
	at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitValueVectorWriteExpression(EvaluationVisitor.java:428)
	at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitUnknown(EvaluationVisitor.java:354)
	at com.dremio.exec.expr.EvaluationVisitor$InnerMethodNester.visitUnknown(EvaluationVisitor.java:1107)
	at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitUnknown(EvaluationVisitor.java:1376)
	at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitUnknown(EvaluationVisitor.java:991)
	at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitUnknown(EvaluationVisitor.java:790)
	at com.dremio.exec.expr.ValueVectorWriteExpression.accept(ValueVectorWriteExpression.java:58)
	at com.dremio.exec.expr.EvaluationVisitor.addExpr(EvaluationVisitor.java:106)
	at com.dremio.exec.expr.ClassGenerator.addExpr(ClassGenerator.java:307)
	at com.dremio.exec.expr.ClassGenerator.addExpr(ClassGenerator.java:295)
	at com.dremio.sabot.op.aggregate.hash.HashAggOperator.setupUpdateAggrValues(HashAggOperator.java:244)
	at com.dremio.sabot.op.aggregate.hash.HashAggOperator.createAggregatorInternal(HashAggOperator.java:208)
	at com.dremio.sabot.op.aggregate.hash.HashAggOperator.setup(HashAggOperator.java:146)
	at com.dremio.sabot.BaseTestOperator.validateSingle(BaseTestOperator.java:519)
	at com.dremio.sabot.BaseTestOperator.validateSingle(BaseTestOperator.java:508)
	at com.dremio.sabot.BaseTestOperator.validateSingle(BaseTestOperator.java:500)
	at com.dremio.sabot.aggregate.hash.TestHashAgg.validateAgg(TestHashAgg.java:53)
	at com.dremio.sabot.aggregate.hash.TestHashAgg.floatWork(TestHashAgg.java:302)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

‘SumFunctions.java’ is in ‘dremio/sabot/kernel/target/generated-sources/fmpp/com/dremio/exec/expr/fn/impl/gaggr’ and directory ‘fmpp’ is set as source folder in IntelliJ. I think that this source expects in both source folder and resource folder because Resources.getResource failed.

Please tell me the instruction of setting up the development environment with IntelliJ.

thanks.

0 Likes

#2

I’m new here, so bear with me. I’ve got dremio-oss (3.1.9) loaded into intellij 2019.1. Doing mvn (Apache Maven 3.6.0) builds without error. (Java Open Jdk Version: 1.8.0_171)

I’ve done builds from command line (mvn clean install -DskipTests). I run build through intellij with maven with same result.

The only reoccurring problem I’ve had is diagnosing dremio exceptions related to instructions:
mvn compile exec:exec -pl dac/daemon

But it sounds like you are hitting a compile time issue… I guess I would suggest removing intellij from the equation and getting a compile working from mvn directly before introducing intellij into the mix.

0 Likes

#3

Hi Bruce,

Thanks for your reply!! I will try again while referring to your results.

0 Likes

#4

I’ll write up some experiences building this thing on my site and reference them. Looks like there might be some interest looking through the uncategorized questions…

1 Like

#5

Here is a quick write-up of my build procedure for dremio-oss.

http://www.wafermovement.com/2019/04/dremio-oss-build-gpc/

0 Likes