Dremio ansi sql dialect - is there option to change?

1

Hi - Dremio use ansi sql dialect but our users have many queries executed in spark using HiveQL dialect. The changes are very small but due size and number of existing query we look for possibility to change dialect to be able to use below statment in dremio query:

select CAST (‘1.0’ AS INT)
RETURN GandivaException: Failed to cast the string 1.0 to int32_t

select cast(from_unixtime(‘1196440219’, ‘yyyy-MM-dd’)as varchar)
RETURN FROM_UNIXTIME does not support operand types (VARCHAR,VARCHAR)

image

So is there option to change query dialect in Dremio?

2

@Pawel Dremio currently does not support HQL, so if there is a specific Hive function not in the Dremio SQL library then you would have to replace with an equivalent function

3

In dremio community edition I can see option to add YARN engine - Can we use Apache Hadoop Engine for our spark dialect query?
Shouldn’t this work as spark specific query could be executed on spark cluster?

The only issue is how to select engine for query in Dremio community edition.

4

@Pawel Yarn engine is to just deploy Dremio executors as containers on the Hadoop data nodes. It has nothing to do with supporting something that is not available in the Dremio SQL library. What is the Spark SQL you need to run?

5

Hi Balaji
Anyway is there option to use Yarn engine in Community Edition - I was able to add this engine but couldn’t find way to use it?

6

@Pawel

Click on settings-engines-add engine then you have to fill out your Yarn resource manager, Hadoop Name node details