Hi - Dremio use ansi sql dialect but our users have many queries executed in spark using HiveQL dialect. The changes are very small but due size and number of existing query we look for possibility to change dialect to be able to use below statment in dremio query:
select CAST (‘1.0’ AS INT)
RETURN GandivaException: Failed to cast the string 1.0 to int32_t
select cast(from_unixtime(‘1196440219’, ‘yyyy-MM-dd’)as varchar)
RETURN FROM_UNIXTIME does not support operand types (VARCHAR,VARCHAR)
So is there option to change query dialect in Dremio?
@Pawel Dremio currently does not support HQL, so if there is a specific Hive function not in the Dremio SQL library then you would have to replace with an equivalent function
In dremio community edition I can see option to add YARN engine - Can we use Apache Hadoop Engine for our spark dialect query?
Shouldn’t this work as spark specific query could be executed on spark cluster?
The only issue is how to select engine for query in Dremio community edition.
@Pawel Yarn engine is to just deploy Dremio executors as containers on the Hadoop data nodes. It has nothing to do with supporting something that is not available in the Dremio SQL library. What is the Spark SQL you need to run?