No applicable constructor/method found Error

1

Hi,

When I run the following query,

select * from (select * from “@hidekoji”.“test”) as limited_result limit 1000

I got following error:

Query execution error. Details:[ \nSYSTEM ERROR: CompileException: Line 64, Column 30: No applicable constructor/method found for actual parameters “org.apache.arrow.vector.holders.UnionHolder”; candidates are: "public void com.dremio.exec.vector.complex.fn.JsonWriter…

Is there some missing setup? or is this known issue?

Below is the details found in server.log

2018-07-19 00:06:04,728 [e0 - 24aff1d4-451b-81b7-7d3b-3a9e952b4800:frag:0:0] ERROR com.dremio.sabot.driver.SmartOp - CompileException: Line 64, Column 30: No applicable constructor/method found for actual parameters “org.apache.arrow.vector.holders.UnionHolder”; candidates are: “public void com.dremio.exec.vector.complex.fn.JsonWriter.write(org.apache.arrow.vector.complex.reader.FieldReader) throws com.fasterxml.jackson.core.JsonGenerationException, java.io.IOException”
com.dremio.common.exceptions.UserException: CompileException: Line 64, Column 30: No applicable constructor/method found for actual parameters “org.apache.arrow.vector.holders.UnionHolder”; candidates are: “public void com.dremio.exec.vector.complex.fn.JsonWriter.write(org.apache.arrow.vector.complex.reader.FieldReader) throws com.fasterxml.jackson.core.JsonGenerationException, java.io.IOException”
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.driver.SmartOp.contextualize(SmartOp.java:137) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.driver.SmartOp$SmartSingleInput.setup(SmartOp.java:256) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput(Pipe.java:73) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput(Pipe.java:63) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.driver.SmartOp$SmartSingleInput.accept(SmartOp.java:197) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.driver.StraightPipe.setup(StraightPipe.java:103) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.driver.StraightPipe.setup(StraightPipe.java:102) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.driver.Pipeline.setup(Pipeline.java:58) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution(FragmentExecutor.java:334) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.exec.fragment.FragmentExecutor.run(FragmentExecutor.java:228) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.exec.fragment.FragmentExecutor.access$800(FragmentExecutor.java:84) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run(FragmentExecutor.java:580) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.task.AsyncTaskWrapper.run(AsyncTaskWrapper.java:107) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:71) [dremio-extra-sabot-scheduler-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.dremio.exec.exception.ClassTransformationException: Failure generating transformation classes for value:

package com.dremio.s;

import com.dremio.exec.exception.SchemaChangeException;
import com.dremio.exec.expr.VectorResolver;
import com.dremio.exec.expr.fn.FunctionErrorContext;
import com.dremio.exec.record.VectorAccessible;
import com.dremio.sabot.exec.context.FunctionContext;
import com.dremio.sabot.op.project.ProjectorTemplate;
import io.netty.buffer.ArrowBuf;
import org.apache.arrow.vector.NullableVarCharVector;
import org.apache.arrow.vector.complex.UnionVector;
import org.apache.arrow.vector.holders.NullableBigIntHolder;
import org.apache.arrow.vector.holders.NullableVarBinaryHolder;
import org.apache.arrow.vector.holders.NullableVarCharHolder;
import org.apache.arrow.vector.holders.UnionHolder;

public class ProjectorGen9
extends ProjectorTemplate
{

FunctionErrorContext work0;
ArrowBuf work1;
FunctionErrorContext work2;
UnionVector vv3;
NullableBigIntHolder constant7;
NullableVarCharVector vv9;

public ProjectorGen9() {
    __INIT__();
}

public void doEval(int inIndex, int outIndex)
    throws SchemaChangeException
{
    {
        UnionHolder out4 = new UnionHolder();
        {
            out4 .isSet = vv3 .isSet((inIndex));
            if (out4 .isSet == 1) {
                vv3 .get((inIndex), out4);
            }
        }
        //---- start of eval portion of convert_toJSON function. ----//
        NullableVarBinaryHolder out5 = new NullableVarBinaryHolder();
        {
            if (out4 .isSet == 0) {
                out5 .isSet = 0;
            } else {
                final NullableVarBinaryHolder out = new NullableVarBinaryHolder();
                UnionHolder input = out4;
                ArrowBuf buffer = work1;
                FunctionErrorContext errCtx = work2;

JsonConvertTo$ConvertToJson_eval: {
if (input.isSet()) {
out.start = 0;

    java.io.ByteArrayOutputStream stream = new java.io.ByteArrayOutputStream();

    try {
        com.dremio.exec.vector.complex.fn.JsonWriter jsonWriter = new com.dremio.exec.vector.complex.fn.JsonWriter(stream, true, false);

        jsonWriter.write(input);
    } catch (Exception e) {
        throw errCtx.error().message("%s", e).build();
    }

    byte[] bytea = stream.toByteArray();

    out.buffer = buffer = buffer.reallocIfNeeded(bytea.length);
    out.buffer.setBytes(0, bytea);
    out.end = bytea.length;
}

}

                work1 = buffer;
                work2 = errCtx;
                out.isSet = 1;
                out5 = out;
            }
        }
        //---- end of eval portion of convert_toJSON function. ----//
        //---- start of eval portion of castVARCHAR function. ----//
        NullableVarCharHolder out8 = new NullableVarCharHolder();
        {
            if ((out5 .isSet*constant7 .isSet) == 0) {
                out8 .isSet = 0;
            } else {
                final NullableVarCharHolder out = new NullableVarCharHolder();
                NullableVarBinaryHolder in = out5;
                NullableBigIntHolder length = constant7;
                FunctionErrorContext errCtx = work0;

CastVarBinaryVarChar_eval: {
int charCount = com.dremio.exec.expr.fn.impl.StringFunctionUtil.getUTF8CharLength(in.buffer, in.start, in.end, errCtx);

out.buffer = in.buffer;
out.start = in.start;
if (charCount <= length.value || length.value == 0) {
    out.end = in.end;
} else
{
    out.end = com.dremio.exec.expr.fn.impl.StringFunctionUtil.getUTF8CharPosition(
        in.buffer,
        in.start,
        in.end,
        (int) length.value,
        errCtx
    );
}

}

                work0 = errCtx;
                out.isSet = 1;
                out8 = out;
            }
        }
        //---- end of eval portion of castVARCHAR function. ----//
        if (!(out8 .isSet == 0)) {
            vv9 .setSafe((outIndex), out8 .isSet, out8 .start, out8 .end, out8 .buffer);
        }
    }
}

public void doSetup(FunctionContext context, VectorAccessible incoming, VectorAccessible outgoing, com.dremio.sabot.op.project.Projector.ComplexWriterCreator writerCreator)
    throws SchemaChangeException
{
    {
        work0 = (context).getFunctionErrorContext(0);
        work1 = (context).getManagedBuffer();
        work2 = (context).getFunctionErrorContext(1);
        vv3 = ((UnionVector) VectorResolver.simple((incoming), UnionVector.class, 1));
        /** start SETUP for function convert_toJSON **/ 
        {
            ArrowBuf buffer = work1;
            FunctionErrorContext errCtx = work2;
             {}
            work1 = buffer;
            work2 = errCtx;
        }
        /** end SETUP for function convert_toJSON **/ 
        NullableBigIntHolder out6 = new NullableBigIntHolder();
        out6 .value = 65536L;
        out6 .isSet = 1;
        constant7 = out6;
        /** start SETUP for function castVARCHAR **/ 
        {
            NullableBigIntHolder length = constant7;
            FunctionErrorContext errCtx = work0;
             {}
            work0 = errCtx;
        }
        /** end SETUP for function castVARCHAR **/ 
        vv9 = ((NullableVarCharVector) VectorResolver.simple((outgoing), NullableVarCharVector.class, 1));
    }
}

public void __INIT__()
    throws SchemaChangeException
{
}

}

at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
at com.dremio.exec.compile.CodeCompiler.getImplementationClass(CodeCompiler.java:58) ~[dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.compile.CodeCompiler.getImplementationClass(CodeCompiler.java:51) ~[dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.expr.CodeGenerator.getImplementationClass(CodeGenerator.java:168) ~[dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.op.project.ProjectOperator.setup(ProjectOperator.java:162) ~[dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.sabot.driver.SmartOp$SmartSingleInput.setup(SmartOp.java:252) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
... 12 common frames omitted

Caused by: com.dremio.exec.exception.ClassTransformationException: Failure generating transformation classes for value:

package com.dremio.s;

import com.dremio.exec.exception.SchemaChangeException;
import com.dremio.exec.expr.VectorResolver;
import com.dremio.exec.expr.fn.FunctionErrorContext;
import com.dremio.exec.record.VectorAccessible;
import com.dremio.sabot.exec.context.FunctionContext;
import com.dremio.sabot.op.project.ProjectorTemplate;
import io.netty.buffer.ArrowBuf;
import org.apache.arrow.vector.NullableVarCharVector;
import org.apache.arrow.vector.complex.UnionVector;
import org.apache.arrow.vector.holders.NullableBigIntHolder;
import org.apache.arrow.vector.holders.NullableVarBinaryHolder;
import org.apache.arrow.vector.holders.NullableVarCharHolder;
import org.apache.arrow.vector.holders.UnionHolder;

public class ProjectorGen9
extends ProjectorTemplate
{

FunctionErrorContext work0;
ArrowBuf work1;
FunctionErrorContext work2;
UnionVector vv3;
NullableBigIntHolder constant7;
NullableVarCharVector vv9;

public ProjectorGen9() {
    __INIT__();
}

public void doEval(int inIndex, int outIndex)
    throws SchemaChangeException
{
    {
        UnionHolder out4 = new UnionHolder();
        {
            out4 .isSet = vv3 .isSet((inIndex));
            if (out4 .isSet == 1) {
                vv3 .get((inIndex), out4);
            }
        }
        //---- start of eval portion of convert_toJSON function. ----//
        NullableVarBinaryHolder out5 = new NullableVarBinaryHolder();
        {
            if (out4 .isSet == 0) {
                out5 .isSet = 0;
            } else {
                final NullableVarBinaryHolder out = new NullableVarBinaryHolder();
                UnionHolder input = out4;
                ArrowBuf buffer = work1;
                FunctionErrorContext errCtx = work2;

JsonConvertTo$ConvertToJson_eval: {
if (input.isSet()) {
out.start = 0;

    java.io.ByteArrayOutputStream stream = new java.io.ByteArrayOutputStream();

    try {
        com.dremio.exec.vector.complex.fn.JsonWriter jsonWriter = new com.dremio.exec.vector.complex.fn.JsonWriter(stream, true, false);

        jsonWriter.write(input);
    } catch (Exception e) {
        throw errCtx.error().message("%s", e).build();
    }

    byte[] bytea = stream.toByteArray();

    out.buffer = buffer = buffer.reallocIfNeeded(bytea.length);
    out.buffer.setBytes(0, bytea);
    out.end = bytea.length;
}

}

                work1 = buffer;
                work2 = errCtx;
                out.isSet = 1;
                out5 = out;
            }
        }
        //---- end of eval portion of convert_toJSON function. ----//
        //---- start of eval portion of castVARCHAR function. ----//
        NullableVarCharHolder out8 = new NullableVarCharHolder();
        {
            if ((out5 .isSet*constant7 .isSet) == 0) {
                out8 .isSet = 0;
            } else {
                final NullableVarCharHolder out = new NullableVarCharHolder();
                NullableVarBinaryHolder in = out5;
                NullableBigIntHolder length = constant7;
                FunctionErrorContext errCtx = work0;

CastVarBinaryVarChar_eval: {
int charCount = com.dremio.exec.expr.fn.impl.StringFunctionUtil.getUTF8CharLength(in.buffer, in.start, in.end, errCtx);

out.buffer = in.buffer;
out.start = in.start;
if (charCount <= length.value || length.value == 0) {
    out.end = in.end;
} else
{
    out.end = com.dremio.exec.expr.fn.impl.StringFunctionUtil.getUTF8CharPosition(
        in.buffer,
        in.start,
        in.end,
        (int) length.value,
        errCtx
    );
}

}

                work0 = errCtx;
                out.isSet = 1;
                out8 = out;
            }
        }
        //---- end of eval portion of castVARCHAR function. ----//
        if (!(out8 .isSet == 0)) {
            vv9 .setSafe((outIndex), out8 .isSet, out8 .start, out8 .end, out8 .buffer);
        }
    }
}

public void doSetup(FunctionContext context, VectorAccessible incoming, VectorAccessible outgoing, com.dremio.sabot.op.project.Projector.ComplexWriterCreator writerCreator)
    throws SchemaChangeException
{
    {
        work0 = (context).getFunctionErrorContext(0);
        work1 = (context).getManagedBuffer();
        work2 = (context).getFunctionErrorContext(1);
        vv3 = ((UnionVector) VectorResolver.simple((incoming), UnionVector.class, 1));
        /** start SETUP for function convert_toJSON **/ 
        {
            ArrowBuf buffer = work1;
            FunctionErrorContext errCtx = work2;
             {}
            work1 = buffer;
            work2 = errCtx;
        }
        /** end SETUP for function convert_toJSON **/ 
        NullableBigIntHolder out6 = new NullableBigIntHolder();
        out6 .value = 65536L;
        out6 .isSet = 1;
        constant7 = out6;
        /** start SETUP for function castVARCHAR **/ 
        {
            NullableBigIntHolder length = constant7;
            FunctionErrorContext errCtx = work0;
             {}
            work0 = errCtx;
        }
        /** end SETUP for function castVARCHAR **/ 
        vv9 = ((NullableVarCharVector) VectorResolver.simple((outgoing), NullableVarCharVector.class, 1));
    }
}

public void __INIT__()
    throws SchemaChangeException
{
}

}

at com.dremio.exec.compile.ClassTransformer.getExtendedImplementationClass(ClassTransformer.java:197) ~[dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.compile.ClassTransformer.getImplementationClass(ClassTransformer.java:169) ~[dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:73) ~[dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.exec.compile.CodeCompiler$Loader.load(CodeCompiler.java:69) ~[dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
... 20 common frames omitted
2

Any conclusion on this?

3

@tonio, this usually means there is some kind of mixed type in your dataset. This can often happen with, for example, a set of JSON documents that have the same name for a field in each, but different datatypes.

What source are you querying?

1 Like
4

I’m actually querying a parquet from S3, but that parquet was the result of a DMS migration from mongodb to s3.

5

@ben I’ve selected a single column (which has no mixed datatypes) from the table and it worked. So, what you pointed out seems to be the case. Do you recommend creating a VDS and converting the datatypes, or is there a more direct workaround to this problem?

6

@tonio, it would be best to have the data cleaned before reaching Dremio, but you should be able use a VDS to curate the data in this case as well.