As soon as i have updated my dremio, i am feeling the performance has been decreased…Even the jobs page is taking time load,The previews are taking 10 minutes to get loaded which was instant before. Do i have to do some changes in the setting.
Initially,I increased my split limit to 100,000 when i was using version 3.2 but after the upgrade i did the reset.
Best
Abhi
That sounds like accessing the Dremio database is slow - can you turn off the coordinator node and make sure there is no zombie Dremio process running?
I am sorry for asking this but can you please share how i can do that. I am new to drmeio.
Which operating system are you running Dremio on?
I am running it on centos.
Shut down Dremio and use the ps command to check if there are any other Dremio processes running.
its running on one master node and 5 other nodes… i should do it on all of it??
I did that but its still the same.
On the master node, can you run the compact and reindex commands documented at https://docs.dremio.com/advanced-administration/metadata-cleanup.html?
I will do that but i also got the below error and i guess i have to increase the size :-
Dremio Daemon Started as master
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /var/log/dremio/java_pid25240.hprof ...
Heap dump file created [4404436386 bytes in 23.571 secs]
Dremio is exiting. Node lost its master status.When you upgraded, did you by chance overwrite any Dremio configuration files that added more heap? That would explain why Dremio is performing poorly.
No, I copied the config file and and re copied it and didnt overwrite it. I also have increased my heap size and its nto performing good. I am using a machien with a huge computing power.
If you want i can share my config here incase i am missing something.
Also, Below is the snapshot of memory utilization of the nodes and the master node:-
I dont know why the cpu and memory utilization of master node is always zero and for other its quite high.
The master cpu/memory doesn’t work correctly in the node activity screen, you should look at the usage on the command line on the master node. Most likely your heap setting on the master is too low (see dremio-env in the conf directory).
this is the output of myps -ef | grep dremio command
root 6217 1 0 Aug27 ? 00:00:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--ens5.lease -pf /var/run/dhclient-ens5.pid -H dremio-master ens5
dremio 6703 1 3 Aug27 ? 00:33:20 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/opt/dremio/lib -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/dremio/server.gc -Ddremio.log.path=/var/log/dremio -Xmx20480m -XX:MaxDirectMemorySize=30720m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dremio -Dio.netty.maxDirectMemory=0 -DMAPR_IMPALA_RA_THROTTLE -DMAPR_MAX_RA_STREAMS=400 -cp /etc/dremio:/opt/dremio/jars/*:/opt/dremio/jars/ext/*:/opt/dremio/jars/3rdparty/* com.dremio.dac.daemon.DremioDaemon
root 11564 11549 0 10:31 pts/0 00:00:00 grep --color=auto dremioAlso, here are my environment settings :-
# Directory where Dremio logs are written
# Default to $DREMIO_HOME/log
#
DREMIO_LOG_DIR=/var/log/dremio
#
# Send logs to console and not to log files. The DREMIO_LOG_DIR is ignored if set.
#
#DREMIO_LOG_TO_CONSOLE=1
#
# Directory where Dremio pidfiles are written
# Default to $DREMIO_HOME/run
#
DREMIO_PID_DIR=/var/run/dremio
#
# Max total memory size (in MB) for the Dremio process
#
# If not set, default to using max heap and max direct.
#
# If both max heap and max direct are set, this is not used
# If one is set, the other is calculated as difference
# of max memory and the one that is set.
#
#DREMIO_MAX_MEMORY_SIZE_MB=
#
# Max heap memory size (in MB) for the Dremio process
#
# Default to 4096 for server
#
DREMIO_MAX_HEAP_MEMORY_SIZE_MB=20480
#
# Max direct memory size (in MB) for the Dremio process
#
# Default to 8192 for server
#
DREMIO_MAX_DIRECT_MEMORY_SIZE_MB=30720
#
# Max permanent generation memory size (in MB) for the Dremio process
# (Only used for Java 7)
#
# Default to 512 for server
#
#DREMIO_MAX_PERMGEN_MEMORY_SIZE_MB=512
#
# Garbage collection logging is enabled by default. Set the following
# parameter to "no" to disable garbage collection logging.
#
#DREMIO_GC_LOGS_ENABLED="yes"
#
# The scheduling priority for the server
#
# Default to 0
#
# DREMIO_NICENESS=0
#
#
# Number of seconds after which the server is killed forcibly it it hasn't stopped
#
# Default to 120
#
#DREMIO_STOP_TIMEOUT=120
# Extra Java options
#
#DREMIO_JAVA_EXTRA_OPTS=any changes i should make in these to perform them well and fix my master node
You could move more memory from direct to heap. What kind of queries are you running and how many are you running at a time?
I am running max 2 or 3 queries at a time but mostly one.
My data is very huge i am collecting 600 gigs of data everyday and its growing day by day.
How much you recommend to move more to heap
as of now its
DREMIO_MAX_HEAP_MEMORY_SIZE_MB=20480
Why i am getting delays in loading even the preview or even the job page. Initially it was fine before the update?
Is it possible you are running a coordinator and an executor on the same node? Make sure on your master node that in dremio.conf, services.executor.enabled should be false.
Hey @doron that is set to false. They are all on different machines
services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: false
}
Also I should not increase the heap memory more than 32 as its java based right?