Zookeeper ACL configuration in Dremio OSS

1

Hi,

I’m trying to enable ACL in the zookeeper used in my dremio OSS cluster.
I’m testing the solution in local: the configuration is an external zookeeper with 2 dremio nodes (1 coordinator and 1 executor) just to figure out where/how configure the ACL.

I’m struggling to understand how I can achieve it: I don’t see any documentation on properties that I can force in the zookeeper section of the dremio.conf file or some env variables to set/pass (I’m using docker in my setup with dremio-oss:24.3 image).

Is there any reference on how to do this ? From my understanding at the moment dremio is ignoring any conf/variable and creating the /dremio node on zookeeper with public access. Is it possible to configuring this in Dremio somehow?

Thanks,
Ivan

2

@ancelot182

When you say ACL, I assume three things

  • ZK address, usually a IP address
  • ZK node
  • ZK port

Do you have something like this in your dremio.conf?

zookeeper: "<host1>:2181,<host2>:2181,<host3>:2181/dremioprod/dremio"
3

HI Balaji,

thanks for the reply.
Yes I’ve the the zookeeper property

zookeeper: "zk:2181"

and I’ve tried to specify the node as well

zookeeper: "zk:2181/dremio"

What I’m missing is the ability to enable authentication on the zookeeper node and have Dremio use it.
For example my zookeeper use a digest mechanism for the authentication on the node:
addauth digest ${USER}:${PASS}

My Dremio should log to the node onluy using a technical user with username/password but I can’t find documentation on how to pass the information to the container.

For example I imagined something like:

environment:
      DREMIO_ZOOKEEPER_QUORUM: zk:2181
      DREMIO_ZOOKEEPER_AUTH_SCHEME: digest
      DREMIO_ZOOKEEPER_AUTH_USERNAME: dremio_user
      DREMIO_ZOOKEEPER_AUTH_PASSWORD: dremio_password

Are there any configuration Dremio OSS can use it for this ?

Thanks,
Ivan

4

@balaji.ramaswamy any ideas ?

5

@Ivan

I have not seen this before, let me see if I can find answers and get back to you

Thanks
Bali

6

Ok, let me know if you get any ideas otherwise I must think of something specifically for this.

7

@ancelot182

Can you check the Zookeepr section of values.yaml as I see a way to specify a different username but do not see password

8

I don’t get which file are you referring to, sorry.

9

where have you defined your Zoo Keeper settings?

10

Ah ok.

In the dremio.conf I’ve setted the external zookeeper like this:
zookeeper: "zk:2181"

In the docker-compose that I launch to simultate the cluster I’ve the zookeeper configuration:

ZOO_CFG_EXTRA: |
        requireClientAuthScheme=digest
        authProvider.1=org.apache.zookeeper.server.auth.DigestAuthenticationProvider

to enable the digest mode.
I’ve tried to pass as environment variable to the coordinator container this values:

environment:
      DREMIO_ZOOKEEPER_QUORUM: zk:2181
      DREMIO_ZOOKEEPER_AUTH_SCHEME: digest
      DREMIO_ZOOKEEPER_AUTH_USERNAME: dremio_user
      DREMIO_ZOOKEEPER_AUTH_PASSWORD: dremio_password

But the dremio coordinator is ignoring them. I cannot find in the documentation other ways to pass the zookeeper configurations. Maybe there’s a comfiguration file that I can override as volume to pass more configuration or something like that ?

Thanks

11

@ancelot182 Maybe this is a Zoo keeper setting, have you tried via zoo.cfg ? But the setting name would not have the keyword DREMIO in it

12

Are the setting defined in the zoo.cfg passed to the Dremio Zk client ? cause I guess the problem is that the client should have user/password to connect to the zk