I’m still confused how Dremio works with HDFS data source
As I can see, I just fill out information about name node and port, such as 50.12.50.28, 8020 then I can access to HDFS
But when I tried to query data from this data source, it could not execute the query to retrieve data
Can you please help
Thank you so much
Query Type:
UI (preview)
Duration:
12m:20s
Start Time:
04/23/2018 15:25:25
End Time:
04/23/2018 15:37:45
User:
angnd
Job ID:
25226789-b3d4-5c1d-0894-2e9cd3333000
Failure while retrieving metadata for table “HDFS - Test”.“user”.admin.test.“data-1486786524601.csv”.
Can you share the job’s query profile along with any server logs (server.log/server.err) please?
And here is the last 200 lines in server.log
2018-04-24 09:58:19,691 [qtp385635961-180] WARN c.d.d.s.admin.profile.ProfileWrapper - Failed to deserialize acceleration details
java.lang.NullPointerException: null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:210) ~[guava-18.0.jar:na]
at com.google.common.collect.FluentIterable.(FluentIterable.java:84) ~[guava-18.0.jar:na]
at com.google.common.collect.FluentIterable$1.(FluentIterable.java:93) ~[guava-18.0.jar:na]
at com.google.common.collect.FluentIterable.from(FluentIterable.java:92) ~[guava-18.0.jar:na]
at com.dremio.dac.server.admin.profile.AccelerationWrapper.computeRelationships(AccelerationWrapper.java:41) ~[dremio-dac-backend-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.dac.server.admin.profile.AccelerationWrapper.(AccelerationWrapper.java:37) ~[dremio-dac-backend-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.dac.server.admin.profile.ProfileWrapper.(ProfileWrapper.java:105) ~[dremio-dac-backend-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.dac.admin.ProfileResource.renderProfile(ProfileResource.java:129) [dremio-dac-backend-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.dac.admin.ProfileResource.getProfile(ProfileResource.java:118) [dremio-dac-backend-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:365) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]
2018-04-24 10:11:12,512 [2521002a-3ff9-4699-180e-699b44047200:foreman] INFO c.d.service.namespace.QuietAccessor - User Error Occurred [ErrorId: c7794777-ac53-4970-8956-92e38ed795a6]
com.dremio.common.exceptions.UserException: Failure while attempting to retrieve metadata information for table “HDFS - Test”.“user”.admin.test.“data.csv”.
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:648) ~[dremio-common-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.service.namespace.QuietAccessor.getDataset(QuietAccessor.java:44) [dremio-services-namespace-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.MaterializedDatasetTable.getRowType(MaterializedDatasetTable.java:69) [dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at org.apache.calcite.sql.validate.ProcedureNamespace.validateImpl(ProcedureNamespace.java:68) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2859) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2844) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3077) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:208) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:866) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:577) [calcite-core-1.12.0-201802162217200854-87dbbf1.jar:1.12.0-201802162217200854-87dbbf1]
at com.dremio.exec.planner.sql.SqlConverter.validate(SqlConverter.java:188) [dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateNode(PrelTransformer.java:165) [dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateAndConvert(PrelTransformer.java:153) [dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan(NormalHandler.java:43) [dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:66) [dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:293) [dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: com.dremio.common.exceptions.ExecutionSetupException: Failure while setting up text reader for file hdfs://52.37.52.40:8020/user/admin/test/data.csv
at com.dremio.exec.store.dfs.FileSystemDatasetAccessor.getDatasetInternal(FileSystemDatasetAccessor.java:177) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.easy.EasyFormatDatasetAccessor.buildDataset(EasyFormatDatasetAccessor.java:108) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.dfs.FileSystemDatasetAccessor.getDataset(FileSystemDatasetAccessor.java:119) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.service.namespace.QuietAccessor.getDataset(QuietAccessor.java:42) [dremio-services-namespace-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
… 24 common frames omitted
Caused by: java.lang.RuntimeException: com.dremio.common.exceptions.ExecutionSetupException: Failure while setting up text reader for file hdfs://52.37.52.40:8020/user/admin/test/data.csv
at com.dremio.exec.store.easy.EasyFormatDatasetAccessor.getBatchSchema(EasyFormatDatasetAccessor.java:148) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.dfs.FileSystemDatasetAccessor$1.run(FileSystemDatasetAccessor.java:151) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.dfs.FileSystemDatasetAccessor$1.run(FileSystemDatasetAccessor.java:146) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_162]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_162]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807) ~[hadoop-common-2.8.0.jar:na]
at com.dremio.exec.store.dfs.FileSystemDatasetAccessor.getDatasetInternal(FileSystemDatasetAccessor.java:145) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
… 27 common frames omitted
Caused by: com.dremio.common.exceptions.ExecutionSetupException: Failure while setting up text reader for file hdfs://52.37.52.40:8020/user/admin/test/data.csv
at com.dremio.exec.store.easy.text.compliant.CompliantTextRecordReader.setup(CompliantTextRecordReader.java:142) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.dfs.implicit.AdditionalColumnsRecordReader.setup(AdditionalColumnsRecordReader.java:62) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.easy.EasyFormatDatasetAccessor.getBatchSchema(EasyFormatDatasetAccessor.java:137) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
… 33 common frames omitted
Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-941339591-10.0.177.89-1524018173390:blk_1073741841_1017 file=/user/admin/test/data.csv
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1019) ~[hadoop-hdfs-client-2.8.0.jar:na]
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:641) ~[hadoop-hdfs-client-2.8.0.jar:na]
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:918) ~[hadoop-hdfs-client-2.8.0.jar:na]
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:983) ~[hadoop-hdfs-client-2.8.0.jar:na]
at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:147) ~[hadoop-common-2.8.0.jar:na]
at com.dremio.exec.store.dfs.FSDataInputStreamWrapper.read(FSDataInputStreamWrapper.java:112) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.easy.text.compliant.TextInput.read(TextInput.java:211) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.easy.text.compliant.TextInput.updateBuffer(TextInput.java:242) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.easy.text.compliant.TextInput.start(TextInput.java:160) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.easy.text.compliant.TextReader.start(TextReader.java:346) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.easy.text.compliant.CompliantTextRecordReader.readFirstLineForColumnNames(CompliantTextRecordReader.java:164) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.easy.text.compliant.CompliantTextRecordReader.generateColumnNames(CompliantTextRecordReader.java:241) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.store.easy.text.compliant.CompliantTextRecordReader.setup(CompliantTextRecordReader.java:123) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
… 35 common frames omitted
2018-04-24 10:11:12,512 [2521002a-3ff9-4699-180e-699b44047200:foreman] INFO query.logger - {“queryId”:“2521002a-3ff9-4699-180e-699b44047200”,“schema”:"",“queryText”:“select * from table(“HDFS - Test”.“user”.admin.test.“data.csv” (type => ‘text’, fieldDelimiter => ‘,’, comment => ‘#’, “escape” => '”’, quote => ‘"’, lineDelimiter => ‘\r\n’, extractHeader => false, skipFirstLine => false, autoGenerateColumnNames => true, trimHeader => true))",“start”:1524563925085,“finish”:1524564672512,“outcome”:“FAILED”,“username”:“quangnd”,“commandDescription”:""}
2018-04-24 15:08:24,533 [qtp385635961-344] WARN c.d.d.s.admin.profile.ProfileWrapper - Failed to deserialize acceleration details
java.lang.NullPointerException: null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:210) ~[guava-18.0.jar:na]
at com.google.common.collect.FluentIterable.(FluentIterable.java:84) ~[guava-18.0.jar:na]
at com.google.common.collect.FluentIterable$1.(FluentIterable.java:93) ~[guava-18.0.jar:na]
at com.google.common.collect.FluentIterable.from(FluentIterable.java:92) ~[guava-18.0.jar:na]
at com.dremio.dac.server.admin.profile.AccelerationWrapper.computeRelationships(AccelerationWrapper.java:41) ~[dremio-dac-backend-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.dac.server.admin.profile.AccelerationWrapper.(AccelerationWrapper.java:37) ~[dremio-dac-backend-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.dac.server.admin.profile.ProfileWrapper.(ProfileWrapper.java:105) ~[dremio-dac-backend-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.dac.admin.ProfileResource.renderProfile(ProfileResource.java:129) [dremio-dac-backend-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.dac.admin.ProfileResource.getProfile(ProfileResource.java:118) [dremio-dac-backend-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:365) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]
2018-04-24 15:10:37,121 [2520b712-06e3-5099-f52a-ba780ca3e000:foreman] INFO c.d.s.s.LocalSchedulerService - Cancelling task com.dremio.exec.store.CatalogServiceImpl$PerSourceNamespaceUpdateTask@6b4a6720
2018-04-24 15:10:37,274 [2520b712-06e3-5099-f52a-ba780ca3e000:foreman] INFO c.d.exec.planner.sql.SchemaUtilities - User Error Occurred [ErrorId: 1c7aa20a-a0b7-4e64-b0a8-244e42e1144a]
com.dremio.common.exceptions.UserException: Unable to create or drop tables/views. Schema [__support] is immutable for this user.
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:648) ~[dremio-common-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.planner.sql.SchemaUtilities.resolveToMutableSchemaInstance(SchemaUtilities.java:195) [dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.planner.sql.handlers.query.CreateTableHandler.getPlan(CreateTableHandler.java:87) [dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:66) [dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:293) [dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]
2018-04-24 15:10:37,274 [2520b712-06e3-5099-f52a-ba780ca3e000:foreman] INFO query.logger - {“queryId”:“2520b712-06e3-5099-f52a-ba780ca3e000”,“schema”:"",“queryText”:“CREATE TABLE “__support”.“8ef60aaf-7645-4505-b3d9-3f8b37cf98ae” \n STORE AS (type => ‘json’, prettyPrint => false) \n WITH SINGLE WRITER\n AS\nSELECT * \nFROM \n json\nWHERE\n (\n cast(\n substr(“timestamp”,\n 0,\n length(“timestamp”) - 4\n ) as timestamp\n ) between timestamp’2018-04-23 09:55:14.660’ and timestamp’2018-04-23 10:07:54.281’\n AND \n levelValue >= 30000\n )\n OR\n thread like ‘%25225278-4d55-6a3c-c686-e597cc9d9800%’\n”,“start”:1524582637100,“finish”:1524582637274,“outcome”:“FAILED”,“username”:“quangnd”,“commandDescription”:""}
2018-04-24 15:10:37,276 [qtp385635961-344] WARN c.dremio.dac.support.SupportService - Failure while attempting to query log files for support submission.
com.dremio.common.exceptions.UserException: Unable to create or drop tables/views. Schema [__support] is immutable for this user.
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:648) ~[dremio-common-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.planner.sql.SchemaUtilities.resolveToMutableSchemaInstance(SchemaUtilities.java:195) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.planner.sql.handlers.query.CreateTableHandler.getPlan(CreateTableHandler.java:87) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:66) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:293) ~[dremio-sabot-kernel-1.4.9-201802191836310213_7195059.jar:1.4.9-201802191836310213_7195059]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_162]
This error looks similar to the one posted on Unable to retrieve metadata info from HDFS using Dremio. Can you detail which Hadoop distribution you are using?
I use hortonworks distribution, I’m not in the office now so cannot check the version
Is there any clue why it doesnt work.
May I ask how Dremio read data from HDFS
As far as I know, in order to read data from HDFS, credential must be provided
However, I don’t see any option to configure in Dremio, so I cannot figure out how Dremio is able to read data in HDFS
Can you please advise
Thank you so much
https://docs.dremio.com/data-sources/hdfs.html#dremio-and-hdfs
Per our documentation, by default we use the credentials/user of the Dremio daemon service account. If you turn impersonation on, read access “will have their access mediated by the HDFS privileges associated with their Dremio login name” instead.