Dremio error when run seelct count(*)

1

Hi guys, I’ve defined a VDS (not reflections) that is a join between another 10 VDS (with reflections) this run in approx 1 minute (with 20 seconds time planing) without problems generate 1 million rows.

But when I run a simple query over it:
select count(*) from path.my.vds;

dremio crashes.

I’ve atach a Profile of it.

e5391b64-0905-43a6-8326-759d7e247bb9-query count.zip (17,4 KB)

1 Like
2

I have the same problem

3

@dacopan

Does your Dremio actually crash as in your coordinator goes down and you have to start it again? The query error says only the query got cancelled see message below

RESOURCE ERROR: Job was cancelled because the query went beyond system capacity during query planning. Please simplify the query.

What is the heap memory allocated on the coordinator, send us output of the below command from the coordinator. Also share the query profile of the query that runs in 1 minute (20s planning)

ps -ef | grep dremio

Thanks,
Bali

4

Sorry by my ewrong expression, dremio not crashes, only query fail.

this is the profile of the query of the VDS (called “stg_oficiales”.“CLIENTES_TOTAL_EXTENDIDO”) that slow planing but run sucess

large-planing.zip (167,2 KB)

once saved the VDS I want to run select count(*) from stg_oficiales"."CLIENTES_TOTAL_EXTENDIDO
large-planing-count.zip (19,8 KB)

if you need this is the original query of the VDS
sql_clientes_total_extendido.zip (2,0 KB)

5

@dacopan

It does look like a very complex query can you please answer the second question on what is the heap memory setting on the coordinator? Send me the dremio-env file from the conf folder and/or output of the command given above

6

@balaji.ramaswamy Hi, if the problem is memory I don’t undrestand why run sucess when use select x.aaa, y.cccc … but fail only when select count(*) … of the VDS

/opt/jdk8//bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/data/dremio-community/lib -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/data/dremio-community/log/server.gc -Ddremio.log.path=/data/dremio-community/log -Ddremio.plugins.path=/data/dremio-community/plugins -Xmx4096m -XX:MaxDirectMemorySize=8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/dremio-community/log -Dio.netty.maxDirectMemory=0 -DMAPR_IMPALA_RA_THROTTLE -DMAPR_MAX_RA_STREAMS=400 -cp /data/dremio-community/conf:/data/dremio-community/jars/:/data/dremio-community/jars/ext/:/data/dremio-community/jars/3rdparty/*:/opt/jdk8//lib/tools.jar com.dremio.dac.daemon.DremioDaemon