Reflections on S3 - no results when hitting "run"

almirb · October 27, 2022, 5:38pm

Confirmed the v.23.0.1 give the same error:
01:13 WARN: [kryo] Unable to load class org.apache.calcite.sql.type.OperandTypes$$Lambda$287/0x0000000840486040 with kryo's ClassLoader. Retrying with current..

Maybe I’ll need to stick with ProtoRelSerializerFactory from the EE version.

Benny_Chow · February 15, 2023, 8:23pm

@almirb fyi, v24 now includes the ProtoRelSerializerFactory serializer. v23 did not.

github.com

dremio/dremio-oss/blob/master/sabot/serializer/src/main/java/com/dremio/exec/planner/serializer/ProtoRelSerializerFactory.java

/*
 * Copyright (C) 2017-2019 Dremio Corporation
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.dremio.exec.planner.serializer;

import java.util.Set;
import java.util.stream.Collectors;

This file has been truncated. show original

almirb · February 16, 2023, 12:05am

Hi @Benny_Chow !
I was just looking for this on the new version!
Thanks for remembering this thread!

I’ll try the new version and see how It goes.
One more question: would I still need to create the sabot-module.conf setting the serializer as com.dremio.exec.planner.serializer.ProtoRelSerializerFactory?

Benny_Chow · February 16, 2023, 4:47am

Hi @almirb

You don’t need to create sabot-module.conf because it is already included inside the dremio-sabot-serializer JAR which is included in the CE distribution now:

github.com

dremio/dremio-oss/blob/master/sabot/serializer/src/main/resources/sabot-module.conf

#
# Copyright (C) 2017-2019 Dremio Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

dremio.planning.serializer = com.dremio.exec.planner.serializer.ProtoRelSerializerFactory
dremio.planning.legacy.serializer = com.dremio.exec.planner.serialization.kryo.KryoRelSerializerFactory
dremio.profile.serializer = com.dremio.exec.planner.serializer.ProtoRelSerializerFactory
dremio.classpath.scanning.base.classes += com.dremio.exec.planner.serializer.RelNodeSerde

This file has been truncated. show original

almirb · February 24, 2023, 6:51pm

Hi @Benny_Chow !

I upgraded Dremio on our servers to v24, and now the reflections of flatten datasets are built successfully (without needing to replace the default serializer)!

Thanks!

Muneer · April 14, 2023, 11:22am

Hi @Benny_Chow

We are on v24 AWS dremio edition. We suddenly started getting the below errors for our sybase databases where we have loaded in the sybase connector from

SELECT * FROM account

No Results

Add ColumnGroup ByJoin

0 Columns

Unable to create dataset. NoSuchMethodError: org.apache.calcite.sql.SqlSelectOperator.createCall(Lorg/apache/calcite/sql/SqlNodeList;Lorg/apache/calcite/sql/SqlNodeList;Lorg/apache/calcite/sql/SqlNode;Lorg/apache/calcite/sql/SqlNode;Lorg/apache/calcite/sql/SqlNodeList;Lorg/apache/calcite/sql/SqlNode;Lorg/apache/calcite/sql/SqlNodeList;Lorg/apache/calcite/sql/SqlNodeList;Lorg/apache/calcite/sql/SqlNode;Lorg/apache/calcite/sql/SqlNode;Lorg/apache/calcite/sql/SqlNodeList;Lorg/apache/calcite/sql/parser/SqlParserPos;)Lorg/apache/calcite/sql/SqlSelect;.

Any ideas/suggestions on what could be causing this and any remedies?

balaji.ramaswamy · April 17, 2023, 5:58am

@Muneer Let me check with the architect who wrote the connector and get back to you

Muneer · April 18, 2023, 8:58am

Thanks @balaji.ramaswamy !

balaji.ramaswamy · April 21, 2023, 5:01am

@Muneer

I’d imagine it’s because the ARP connector needs to be built against a higher Dremio OSS version. There was probably some change in the framework.

Was this working in previous versions?

Muneer · April 26, 2023, 1:13pm

@balaji.ramaswamy yes this was working in as recently as the V23 version and is also working on v20.1

The connections to the sybase DB all get created ok. Its only when we try to query, these calcite errors show up.

balaji.ramaswamy · April 26, 2023, 10:48pm

@Muneer

Thanks for the feedback, we will review and see what is going on

Muneer · May 15, 2023, 9:22am

@balaji.ramaswamy …Any updates on this? We are hoping to productionize our environment and have only this as a blocker.

balaji.ramaswamy · May 15, 2023, 6:20pm

@Muneer It looks like the custom connector to Sybase is no longer working (in V24), Someone needs to take up the connector and fix the bug. This was a custom connector and not part of the regular Dremio GA connectors

balaji.ramaswamy · May 22, 2023, 4:48pm

@Muneer a new version has been pushed now, can you please check?

Muneer · May 23, 2023, 10:52am

Thanks very much @balaji.ramaswamy We will check now and report back. Many thanks to @Jonny for making the change! Thanks again Jonny.

Topic		Replies	Views
Unable to connect S3 source: Dremio Community Edition	2	303	December 21, 2023
Dremio 23 not match any reflections with MINIOS3	31	2296	February 16, 2023
Error configuring Storage s3	2	2003	March 29, 2020
Can not read data from reflection [S3 compatibility]	1	1866	May 20, 2020
Storing reflection data in S3	3	1389	October 13, 2017

Reflections on S3 - no results when hitting "run"

Related Topics