User-written and third party functions

1

greetings friends –

I’m trying to interface our library with Dremio and found the sample internal function on GitHub. The example looks super easy. But the server crashes when scanning the added JAR.

I dropped my own code and fell back to just the original function. Still fails wen scanning the reduced JAR.

Has anyone else tried this example?

https://github.com/dremio-hub/dremio-internal-function-example/

Can any of you make sense of this error?

Dremio is exiting. Failure while starting services.
java.lang.RuntimeException: Unexpected class load failure while attempting to load Function Registry (io.netty.buffer.ArrowBuf)
        at com.dremio.common.scanner.persistence.TypeDescriptor$ClassTypeDescriptor.getType(TypeDescriptor.java:96)
    ...
        at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:146)
        Suppressed: java.lang.RuntimeException: Unexpected class load failure while attempting to load Function Registry (io.netty.buffer.ArrowBuf)
                at com.dremio.common.scanner.persistence.TypeDescriptor$ClassTypeDescriptor.getType(TypeDescriptor.java:96)
            ...
                at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:149)
                Suppressed: java.lang.NullPointerException
                        at com.dremio.provision.service.ProvisioningServiceImpl.close(ProvisioningServiceImpl.java:490)
                        ... 9 more
        Caused by: java.lang.ClassNotFoundException: io.netty.buffer.ArrowBuf
                at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
                at java.lang.Class.forName0(Native Method)
                at java.lang.Class.forName(Class.java:264)
                at com.dremio.common.scanner.persistence.TypeDescriptor$ClassTypeDescriptor.getType(TypeDescriptor.java:93)
                ... 24 more
Caused by: java.lang.ClassNotFoundException: io.netty.buffer.ArrowBuf
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at com.dremio.common.scanner.persistence.TypeDescriptor$ClassTypeDescriptor.getType(TypeDescriptor.java:93)
        ... 19 more
2

I have the same issue. Any chance that you solved this?

3

@trothr @jkeen Have you tried to recompile the code after you dropped the jar?

4

No, the readme just says you need to put the jar in place, it didn’t mention a need to recompile after putting the jar in place. At a high level, how should I do this? Will I need to clone the dremio github repo? We are using an older version of dremio (14.0) so that will be an obstacle, correct?

5

@jkeen Try below 2 steps

  • Import org.apache.arrow.memory.ArrowBuf instead of io.netty.buffer.ArrowBuf
  • Recompile your UDF
  • Retry
7

I get an error when trying to recompile.

dremio-internal-function-example/src/main/java/com/dremio/udf/ConcatOp.java:[28,31] cannot find symbol
[ERROR]   symbol:   class ArrowBuf
[ERROR]   location: package org.apache.arrow.memory

It seems like maven is using an older set of java packages based on the UDF example project. I am new to maven and haven’t figured out how to fix that.