Unable to Connect to NAS, Pt. 2

1

(This is the same issue as a previous post. Here it is presented more cleanly, with more info.)

Running Dremio 3.0.6 community locally on CentOS 7.6. Trying to connect to a simple directory of sample data via the NAS connector. On the UI, I get error Failure while configuring source , and the log reports Caused by: java.lang.Exception: Unavailable: file /home/user1/Shared/sample_data does not exist . (But it does.)

UI:

06%20AM
Screen Shot 2019-01-25 at 11.50.06 AM.png843×433 18.8 KB

server.log:

2019-01-25 11:49:28,599 [qtp1175829504-86] INFO  c.d.exec.catalog.CatalogServiceImpl - User Error Occurred [ErrorId: e215ec80-65dc-4e77-91e5-0ca717824dc1]
com.dremio.common.exceptions.UserException: Failure while configuring source [Local_Data]
	at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:634) [dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:378) [dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.catalog.CatalogServiceImpl.access$600(CatalogServiceImpl.java:100) [dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:954) [dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:507) [dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:189) [dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:157) [dremio-dac-backend-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:148) [dremio-dac-backend-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:144) [dremio-dac-backend-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:80) [dremio-dac-backend-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
Caused by: java.util.concurrent.ExecutionException: java.lang.Exception: Unavailable: File /home/user1/Shared/sample_data does not exist
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500) ~[guava-20.0.jar:na]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:401) ~[guava-20.0.jar:na]
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) ~[guava-20.0.jar:na]
	at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:68) ~[guava-20.0.jar:na]
	at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:104) ~[guava-20.0.jar:na]
	at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:622) [dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	... 55 common frames omitted
Caused by: java.lang.Exception: Unavailable: File /home/user1/Shared/sample_data does not exist
	at com.dremio.exec.catalog.ManagedStoragePlugin$1.run(ManagedStoragePlugin.java:252) ~[dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.concurrent.RenamingRunnable.run(RenamingRunnable.java:36) ~[dremio-common-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.concurrent.SingletonRunnable.run(SingletonRunnable.java:41) ~[dremio-common-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.concurrent.SafeRunnable.run(SafeRunnable.java:40) ~[dremio-common-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.concurrent.Runnables$1.run(Runnables.java:45) ~[dremio-common-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]

Additional information:

[user1@localhost /]$ ps -ef | grep dremio
dremio   15096     1  1 11:11 ?        00:00:54 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/opt/dremio/lib -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/opt/dremio/log/server.gc -Ddremio.log.path=/opt/dremio/log -Xmx4096m -XX:MaxDirectMemorySize=8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/dremio/log -Dio.netty.maxDirectMemory=0 -DMAPR_IMPALA_RA_THROTTLE -DMAPR_MAX_RA_STREAMS=400 -cp /opt/dremio/conf:/opt/dremio/jars/*:/opt/dremio/jars/ext/*:/opt/dremio/jars/3rdparty/* com.dremio.dac.daemon.DremioDaemon
user1    16623 14868  0 11:58 pts/0    00:00:00 grep --color=auto dremio
[user1@localhost /]$ sudo su -
Last login: Fri Jan 25 11:54:56 EST 2019 on pts/0
[root@localhost ~]# su - dremio -c "ls -ltrh /home/user1/Shared/sample_data"
This account is currently not available.
[root@localhost ~]# su - dremio
Last login: Fri Jan 25 11:59:15 EST 2019 on pts/0
This account is currently not available.
[root@localhost ~]# ls -ltrh /home/user1/Shared/sample_data/
total 8.0K
-rwxrwx---. 1 root vboxsf 83 Jan 17 16:06 tax_payments.csv
-rwxrwx---. 1 root vboxsf 27 Jan 17 16:06 postal_code_lookup.csv

Installation notes: I get the exact same results whether I’ve installed Dremio as a .rpm, or as a tarball as was previously suggested. And I do mean identical, down to the character, aside from timestamps and pid’s. In both cases I did a fresh install of Dremio as user1.

2

Hi @can1

Your files are owned by root and Dremio does not have access to read them, try this

su -
cd /home/user1/Shared/
chmod -R 777 sample_data

Now try to add the NAS path /home/user1/Shared/

Thanks
@balaji.ramaswamy

3

Hi @balaji.ramaswamy. Thanks for the reply. This sounds like a very good obvious idea, but unfortunately it doesn’t work. I don’t seem to be able to permission dremio user. I tried the code you suggested, as well as adding dremio user to the owning group, but neither seems to work.

[root@localhost Shared]# su -
Last login: Fri Jan 25 14:38:18 EST 2019 on pts/0
[root@localhost ~]# ls -ltrh /home/user1/Shared/sample_data/
total 8.0K
-rwxrwx---. 1 root vboxsf 83 Jan 17 16:06 tax_payments.csv
-rwxrwx---. 1 root vboxsf 27 Jan 17 16:06 postal_code_lookup.csv
[root@localhost ~]# chmod -R 777 /home/user1/Shared/sample_data/
[root@localhost ~]# ls -ltrh /home/user1/Shared/sample_data/
total 8.0K
-rwxrwx---. 1 root vboxsf 83 Jan 17 16:06 tax_payments.csv
-rwxrwx---. 1 root vboxsf 27 Jan 17 16:06 postal_code_lookup.csv
[root@localhost ~]# usermod -aG dremio vboxsf
[root@localhost ~]# ls -ltrh /home/user1/Shared/sample_data/
total 8.0K
-rwxrwx---. 1 root vboxsf 83 Jan 17 16:06 tax_payments.csv
-rwxrwx---. 1 root vboxsf 27 Jan 17 16:06 postal_code_lookup.csv

The UI and server.log error messages remain the same.

4

Further update: I am made a copy of the sample_data directory. I was able to modify the permissions of the copy, but, Dremio still fails to connect the source with the same UI and server.log messages.

ETA: commands used:

[root@localhost ~]# su -
[root@localhost ~]# cp -R /home/user1/Shared/sample_data/ /home/user1/Desktop/sample_data
[root@localhost ~]# chmod -R -f 777 /home/user1/Desktop/sample_data/
[root@localhost ~]# ls -ltrh /home/user1/Desktop/sample_data/
total 8.0K
-rwxrwxrwx. 1 root root 83 Jan 25 14:53 tax_payments.csv
-rwxrwxrwx. 1 root root 27 Jan 25 14:53 postal_code_lookup.csv
5

Further update: also happens with just-released Dremio 3.1

6

Can you add any other NAS source? Is there any directory on your localhost that Dremio can access?

7

Hi @ben, @balaji.ramaswamy. On Ben’s suggested I tried other directories. I was able to set up access to /. From there I navigated to /home, but it failed to navigate to /home/user1. When I explicitly permission the dremio user on /home/user1, I was able to navigate all the way to /home/user1/Desktop/sample_data and /home/user1/Shared/sample_data.

I had previously permissioned the dremio user on those latter two directories. But perhaps it doesn’t work if dremio doesn’t have access all the way down the tree?

8

Hi @can1,

When you configure the NAS source for your local filesystem, you have to specify the absolute path. The user configure to use run the Dremio service/deamon needs to have permissions to at least read every element of that path and the datasets in it. So /home/user1/Shared/sample_data should be readable as a dataset if /home, user1,Shared and sample_data are readable; if Dremio can’t read som component in the path here, you’ll get an error.

9

Hi @ben. Thanks for your response. That is very useful information. I might suggest that be added to the documentation, since I don’t believe it is there. (https://docs.dremio.com/data-sources/nas.html)