How to set up development environment for dremio-oss with IntelliJ

Hi,

I tried to set up development environments for dremio-oss with IntelliJ.

  1. mvn clean install -DskipTests
  2. Import source code with pom.xml to IntelliJ

But some unit tests failed. For example, TestHashAgg failed.

java.lang.IllegalArgumentException: resource /com/dremio/exec/expr/fn/impl/gaggr/SumFunctions.java relative to com.dremio.exec.expr.fn.impl.gaggr.SumFunctions$NullableFloat4Sum not found.

	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:383)
	at com.google.common.io.Resources.getResource(Resources.java:209)
	at com.dremio.exec.expr.fn.FunctionInitializer.getSourceURL(FunctionInitializer.java:138)
	at com.dremio.exec.expr.fn.FunctionInitializer.get(FunctionInitializer.java:142)
	at com.dremio.exec.expr.fn.FunctionInitializer.checkInit(FunctionInitializer.java:98)
	at com.dremio.exec.expr.fn.FunctionInitializer.getMethod(FunctionInitializer.java:79)
	at com.dremio.exec.expr.fn.BaseFunctionHolder.meth(BaseFunctionHolder.java:111)
	at com.dremio.exec.expr.fn.BaseFunctionHolder.meth(BaseFunctionHolder.java:107)
	at com.dremio.exec.expr.fn.AggrFunctionHolder.setup(AggrFunctionHolder.java:45)
	at com.dremio.exec.expr.fn.AggrFunctionHolder.renderStart(AggrFunctionHolder.java:110)
	at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitFunctionHolderExpression(EvaluationVisitor.java:209)
	at com.dremio.exec.expr.EvaluationVisitor$InnerMethodNester.visitFunctionHolderExpression(EvaluationVisitor.java:1057)
	at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1160)
	at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:810)
	at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:790)
	at com.dremio.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:43)
	at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitValueVectorWriteExpression(EvaluationVisitor.java:428)
	at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitUnknown(EvaluationVisitor.java:354)
	at com.dremio.exec.expr.EvaluationVisitor$InnerMethodNester.visitUnknown(EvaluationVisitor.java:1107)
	at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitUnknown(EvaluationVisitor.java:1376)
	at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitUnknown(EvaluationVisitor.java:991)
	at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitUnknown(EvaluationVisitor.java:790)
	at com.dremio.exec.expr.ValueVectorWriteExpression.accept(ValueVectorWriteExpression.java:58)
	at com.dremio.exec.expr.EvaluationVisitor.addExpr(EvaluationVisitor.java:106)
	at com.dremio.exec.expr.ClassGenerator.addExpr(ClassGenerator.java:307)
	at com.dremio.exec.expr.ClassGenerator.addExpr(ClassGenerator.java:295)
	at com.dremio.sabot.op.aggregate.hash.HashAggOperator.setupUpdateAggrValues(HashAggOperator.java:244)
	at com.dremio.sabot.op.aggregate.hash.HashAggOperator.createAggregatorInternal(HashAggOperator.java:208)
	at com.dremio.sabot.op.aggregate.hash.HashAggOperator.setup(HashAggOperator.java:146)
	at com.dremio.sabot.BaseTestOperator.validateSingle(BaseTestOperator.java:519)
	at com.dremio.sabot.BaseTestOperator.validateSingle(BaseTestOperator.java:508)
	at com.dremio.sabot.BaseTestOperator.validateSingle(BaseTestOperator.java:500)
	at com.dremio.sabot.aggregate.hash.TestHashAgg.validateAgg(TestHashAgg.java:53)
	at com.dremio.sabot.aggregate.hash.TestHashAgg.floatWork(TestHashAgg.java:302)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

‘SumFunctions.java’ is in ‘dremio/sabot/kernel/target/generated-sources/fmpp/com/dremio/exec/expr/fn/impl/gaggr’ and directory ‘fmpp’ is set as source folder in IntelliJ. I think that this source expects in both source folder and resource folder because Resources.getResource failed.

Please tell me the instruction of setting up the development environment with IntelliJ.

thanks.

I’m new here, so bear with me. I’ve got dremio-oss (3.1.9) loaded into intellij 2019.1. Doing mvn (Apache Maven 3.6.0) builds without error. (Java Open Jdk Version: 1.8.0_171)

I’ve done builds from command line (mvn clean install -DskipTests). I run build through intellij with maven with same result.

The only reoccurring problem I’ve had is diagnosing dremio exceptions related to instructions:
mvn compile exec:exec -pl dac/daemon

But it sounds like you are hitting a compile time issue… I guess I would suggest removing intellij from the equation and getting a compile working from mvn directly before introducing intellij into the mix.

Hi Bruce,

Thanks for your reply!! I will try again while referring to your results.

I’ll write up some experiences building this thing on my site and reference them. Looks like there might be some interest looking through the uncategorized questions…

1 Like

Here is a quick write-up of my build procedure for dremio-oss.

http://www.wafermovement.com/2019/04/dremio-oss-build-gpc/

Hi Bruce,

Could you please elaborate on how you got dremio-oss loaded into intellij? When I try to load all of it in, intellij creates iml files within each subproject. Consequently the project fails to build.

Can you share the error or problem that you are seeing with the build? As far as I remember, I ran through the intellij help information on maven: https://www.jetbrains.com/help/idea/maven-support.html

And intellij loaded the project and then started downloading dependent packages in the background (there are tons ~2.2G in my case). Perhaps there is some dependency not getting downloaded?

Sorry you’re having difficulty with this. I have .iml files all over as well but I don’t think it conflicts with an intellij maven build. Also, I can see that the .gitignore lists “*.iml” to ignore so I’m pretty sure that others use intellij to work with Dremio code.

I know I wouldn’t try to work with this code much outside an IDE.

Hi Bruce, thanks for the response. I have attached a screenshot of the error that I see when I try to import this project into intellij.

  1. When the time came to select the content root I went for just the client submodule and selected it. I left others unmarked. I double checked this after having imported the project from Project setting --> modules

  2. When I tried to build, I got lots of errors of the form "Can’t find symbol ". I googled a bit and deleted the .iml files present in the submodule which houses the file that manifested that error.

I must be doing something wrong, so your guidance is really appreciated.

I can’t help you much from what you’ve shared. From what I see, you haven’t built all the sub-modules that are needed to compile the code you are trying to build. This is a very large project with many sub-modules.

Don’t bother with intellij until you get something close to a full build with maven (mvn) from the command line. Review README.md and from a command line do build in the root directory.

mvn clean install -DskipTests

Hi Bruce,

I solved the problem myself. I hadn’t imported the project as a maven project. Recreating with that option helped.

Thanks for the help though.

1 Like