Dremio Distributed Cluster Issues

1

Hello there,

I’m trying to create an on-prem Dremio cluster.

I’m using Ubuntu 22.04, and running a MinIO S3 instance for storage. Dremio is running direct on the servers, I’m not using Docker etc.

I intend to configure a 3-node cluster, with a single master-coordinator and a couple of executors. I don’t require HA, just a setup that scales performance-wise.

It took some work, but I have successfully configured my master-coordinator instance, and have the distributed file store working complete with S3 buckets created by Dremio (e.g. ‘accelerator’, ‘metadata’ etc.)

To setup dremio.conf, I used the template here: Configuring via dremio.conf | Dremio Documentation

For the master-coordinator, I set coordinator enabled: true, master enabled: true and executor enabled: false.

I’m running embedded zookeeper on the master, on port 2181.

After starting the Dremio service, the web portal is available (I had to use a different port to 8080 due to other software running on this instance), and it shows the master-coordinator in the node listing screen. Zookeeper is listening on port 2181.

All good. I’ve not yet tested setting up a data source connection etc. I will get to this testing later. However, for the minute, it looks like the master-coordinator is active and online.

The issue arises when adding my first executor node.

I copied core-site.xml, dremio.conf and dremio-env to my second node.

Then, in dremio.conf, I amended the following (this is an excerpt from the conf file, as it follows the exact dremio.conf template as outlined via the link above and is quite lengthy):

services: {
coordinator: {
enabled: false,
# Auto-upgrade Dremio at startup if needed
auto-upgrade: false,
master: {
enabled: false,
# configure an embedded ZooKeeper server on the same node as master
embedded-zookeeper: {
enabled: false,
port: 2181,
path: ${paths.local}/zk
}
},
executor: {
enabled: true
},

I also double checked the zookeeper settings - everything is correct in terms of the hostname and the port. I have used the ‘nc -zv’ command to check that node 2 can communicate on port 2181 to the master-coordinator node - everything tested successfully.

I also set web enabled: false as, for the minute, I want to force all admin via the web interface on the master-coordinator.

Upon trying to start the Dremio service, I receive the following error:

com.dremio.service.namespace.RemoteNamespaceException: dataset listing failed: com.dremio.common.exceptions.UserRemoteException: SYSTEM ERROR: UnsupportedOperationException: non-master coordinators or executors do not support dataset listing

Just out of curiosity, I then set the following for a coordinator-only role:

services: {
coordinator: {
enabled: true,
# Auto-upgrade Dremio at startup if needed
auto-upgrade: false,
master: {
enabled: false,
# configure an embedded ZooKeeper server on the same node as master
embedded-zookeeper: {
enabled: false,
port: 2181,
path: ${paths.local}/zk
}
},
executor: {
enabled: false
},

Upon starting the service I get the following error:

com.google.common.cache.CacheLoader$InvalidCacheLoadException: CacheLoader returned null for key class com.dremio.exec.server.options.SystemOptionManager$OptionStoreCreator

I’ve poured over the logs and the online documentation, but I must be missing something.

I’ve also tried removing large chunks of the dremio.conf template to get me back to something that looks a little more like the examples in this documentation (to simplify the config more than anything else):

Configuring Dremio Services | Dremio Documentation

To no avail.

I’ve been going round in circles for a couple of days now. I’m hoping someone can assist me with this, else I may have to just stick with the single node setup, which would be less than ideal.

Thanks in advance!

2

@shaunbetts Welcome to the Dremio Community!

Does your Executor know where to connect to for Zookeeper? Since you’re using embedded Zookeeper, in your dremio.conf of the Executor; see if you have something like this:

zookeeper: "<your_coordinator_ip>:2181"

Also, another thing to look at is if your nodes know each others hostnames (as well as their own hostname) via /etc/hosts. If not, one quick way out is to publish the IPs by adding the following in dremio.conf. Do this for all of your Coordinators and Executors:

registration.publish-host: "<current_node's_ip>"
3

Hi!

Thanks for your reply.

Yep, sorry - per my (rather long description of the issue) - the executor has been configured to connect to the embedded zookeeper instance on the master-coordinator. Apologies, I did write that, but you probably had to read an awful lot to get to that bit! Hopefully I didn’t waste too much of your time reading everything.

Regards registration.publish-host, I couldn’t find much documentation on this setting - and thought I’d seen something on another community posting about this having been deprecated?

I think I did try this setting - however, not in the way that I think you describe…

Just to be clear - on the executor node, the registration.publish-host should be set to the executors IP. I think when I did this previously I set this setting to the master-coordinator IP on the executor node’s configuration.

I will also check /etc/hosts - just in case!

4

@lenoyjacob following your advice, I checked the hosts file on both the master and the executor - everything seems fine IMO (unless there’s specific entries that you recommend that I have?)

I have also re-configured the dremio.conf on the executor to use the registration.publish-host to no avail.

I set it to the executor IP, then to it’s hostname, then to the IP of the master-coordinator and finally to the hostname of the master-coordinator, with no luck.

Other than 2181, are there any other ports required to get the cluster to work? As far as I’m aware, the servers can communicate freely between each other, but if there’s anything that I might be able to investigate I will gladly do so.

5

My apologies, I may have missed that. Thank you for clarifying.

Yes. Do the same for the coordinator as well. i.e on the coordinator node, the registration.publish-host should be set to the coordinators IP. This should ideally be an IP address.

6

@lenoyjacob thank you for the clarification.

That worked! :open_mouth:

I now have a single master-coordinator and a single executor in my cluster.

I’ll try for the third node now, hopefully that’ll work too!

Do you happen to know where registration.publish-host is documented? I’m surprised I missed such a important setting…

On startup of the service, I do get some general errors such as:

Please increase the maximum number of file descriptors to at least 65536.

Dremio utilizes a large number of file descriptors.
The maximum number of file descriptors, currently set at 1024, is not sufficient.

I was about to ask if it was an issue with this!

7

Great to hear that it worked!

Looks like not. I’ll raise a doc ticket to add more description in the Dremio config reference page.

Yes, you should fix that. Broadly, this limits the number of files that the Dremio process can open. You should be able to find guides to do that on Linux. It’s documented here. Also make note of configuring DREMIO_MAX_DIRECT_MEMORY_SIZE_MB in dremio-env if you haven’t already.

8

@lenoyjacob - thanks so much for your assistance. I can confirm that I now have a x3 node cluster comprising a single master-coordinator + x2 executors. I think the doc ticket would be really useful, if that setting is core to successful cluster configuration, thanks for raising it and hopefully that’ll help others in the future.

Incidentally, is it acceptable, under the ‘Node Activity’ of the web interface, for the executor nodes to be showing IP Address = ‘127.0.0.1’ and Port = ‘N/A’? The master-coordinator role shows it’s primary IP on port 31010.

I’m pretty sure the IP address information on this screen is being influenced by the /etc/hosts file on each node, I just wanted to check if you knew whether or not this was an indicator of cluster health i.e.: the executors being shown as operating on 127.0.0.1:N/A won’t cause issues in cluster communications.

I did read the memory configuration sections of the documentation, and I opted to simply use the DREMIO_MAX_MEMORY_SIZE_MB setting, leaving the MAX_HEAP_MEMORY / MAX_DIRECT_MEMORY commented out (as I understood from the documentation that this would be a valid configuration.)

On my executors, this is currently set to 131072, or 128GB. Literally:

DREMIO_MAX_MEMORY_SIZE_MB=131072

I still get the message though (I hadn’t realised that the message was related to the memory configuration.)

I’m probably asking too many questions in a single thread now. I’d be happy to raise a new thread if that’s useful.

9

I am inclined to think that there is an issue in cluster configuration, based on some initial testing.

Essentially, I created a new S3 data source and limited my configuration to a single bucket. This works well, I can see everything in the bucket.

When I create a new data set, from a single CSV contained within the bucket, everything works well - I can query the dataset (SELECT * etc.)

However, within my bucket, I also have a sub-folder that contains ~80 small CSVs - all of the same schema configuration. I can create a single data set for the folder, containing all 80 CSVs. Again, this all works well.

However, when I try to query this (folder) data set I get connection issues.

If I remove one of the executors (so that I have x1 master-coordinator + x1 executor), the cluster runs the query correctly.

Examination of the log files for the failed job shows the following:

SYSTEM ERROR: UnknownHostException: node2

Fragment 1:0

[Error Id: 2fab0eaa-62eb-4420-a185-c64acdbe5011 on node3:0]

  • (com.dremio.exec.rpc.ConnectionFailedException) Command failed while establishing connection. Failure type CONNECTION.*

It looks like there’s some inter-node communications happening in order to execute the query.

I will see if I can manipulate the /etc/hosts files to get ‘Node Activity’ to show the actual node IP addresses, rather than 127.0.0.1; I assume this (at this stage) to be a good candidate for the root of my issues.

10

I can confirm that updating the /etc/hosts file to remove any extraneous 127.0.0.1 entries did the trick.

‘Node Activity’ now shows the correct IP addressing, and the queries execute with both executor nodes in the cluster.

…still get the weird memory warning on service startup though(!)

11

Great to hear all executors are working.

Could you go into details on the memory error you are getting?

Or are you talking about the file descriptors warning?

12

@lenoyjacob yes, apologies, the file descriptor error on startup.

I’ve set DREMIO_MAX_MEMORY_SIZE to the following:

Master-Coordinator 64GB
Executor 1 128GB
Executor 2 128GB

Does the warning show by default, or do I have an issue that I need to deal with?

Thanks in advance for your insight on this.

13

Yes, IMO, you have to fix it. It is the number of files that the Dremio process can open. This is a Linux level setting. So please google for a guide that helps you change that for your OS (for example, here’s one).

You have to do this for all nodes.

14

Thanks @lenoyjacob I will take a look into this and let you know how I get on.