Cannot start dremio executor

1

Hello,

My dremio master node works fine, but I cannot open dremio executor because

 Could not bind to port 45678

I tried to kill the process using port 45678 in master node or executor node. But the issue still unsolved.

Can I change the port in config file? what is the dremio-env to change the port and should i make change on master or executor?

Thank you.

3

Here is the documentation on dremio.conf content: http://docs.dremio.com/advanced-administration/dremio-conf.html
Look for fabric: { port: 45678

4

Even I changed the port, it gives me the same error log.

Catastrophic failure occurred. Exiting. Information follows: Failed to start services, daemon exiting.

com.dremio.common.exceptions.UserException: Could not bind to port 56789.
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746)
at com.dremio.exec.rpc.BasicServer.bind(BasicServer.java:220)
at com.dremio.services.fabric.FabricServer.bind(FabricServer.java:72)
at com.dremio.services.fabric.FabricServiceImpl.start(FabricServiceImpl.java:92)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.dac.daemon.NonMasterSingletonRegistry.start(NonMasterSingletonRegistry.java:54)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:177)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:183)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:112)
Caused by: java.net.BindException: Address already in use

5

Do you have a firewall preventing port binding? Or do you have another process running on those specific ports? You can check via netstat -tulpn for example

6

To add here - are you trying by any chance to start executor on the same node as master? That combination is not really working.

7

helllo @yufeldman,

No, my master node and executor node are two different node.

@anthony, netstat -tulpn | grep 56789 on master gives
tcp 0 0 0.0.0.0:56789 0.0.0.0:* LISTEN 121054/java

netstat -tulpn | grep 56789 on executor gives
tcp 0 0 0.0.0.0:56789 0.0.0.0:* LISTEN 117107/java

8

And so what are those processes?
Especially 117107 ?
you can find it out by running ps -aef | grep 117107

9

@yufeldman
Here is the output

pz50apl+ 117107 1 1 13:09 ? 00:00:37 /usr/bin/java/jdk1.8.0_181/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/opt/dremio/lib -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/opt/dremio/log/server.gc -Ddremio.log.path=/opt/dremio/log -Xmx4096m -XX:MaxDirectMemorySize=40960m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/dremio/log -Dio.netty.maxDirectMemory=0 -DMAPR_IMPALA_RA_THROTTLE -DMAPR_MAX_RA_STREAMS=400 -cp /opt/dremio/conf:/opt/dremio/jars/:/opt/dremio/jars/ext/:/opt/dremio/jars/3rdparty/* com.dremio.dac.daemon.DremioDaemon dremio start
pz50apl+ 128410 121897 0 13:48 pts/0 00:00:00 grep --color 117107

10

So Dremio is running? Is it that somehow happening that you have two processes starting and second one has an issue, as address is already in use?

11

I don’t understand how can this happen.
Because when I do dremio status, it gives me dremio is not running.

12

Can you manually kill all those processes and then restart Dremio with coordinator first then executors?

13

I killed all processes.

But still get the same results.
Master was able to run, but not executor.

14

Hi @carol

To change the fabric port use below document

https://docs.dremio.com/advanced-administration/dremio-conf.html?h=45678

Thanks
@balaji.ramaswamy