Parquet Logical Type Support

mitchell.davis · 2019-01-10 15:48:57 UTC

Running into this error:

java.lang.UnsupportedOperationException: SMALLINT is not supported

I understand that INT8 and INT16 are only logical data types in parquet and that the format converts them to INT32 anyway, but is there any way or plan to have dremio support these Logical Data Types?

github.com

apache/parquet-format/blob/master/LogicalTypes.md#numeric-types

<!--
  - Licensed to the Apache Software Foundation (ASF) under one
  - or more contributor license agreements.  See the NOTICE file
  - distributed with this work for additional information
  - regarding copyright ownership.  The ASF licenses this file
  - to you under the Apache License, Version 2.0 (the
  - "License"); you may not use this file except in compliance
  - with the License.  You may obtain a copy of the License at
  -
  -   http://www.apache.org/licenses/LICENSE-2.0
  -
  - Unless required by applicable law or agreed to in writing,
  - software distributed under the License is distributed on an
  - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  - KIND, either express or implied.  See the License for the
  - specific language governing permissions and limitations
  - under the License.
  -->

Parquet Logical Type Definitions

This file has been truncated. show original

balaji.ramaswamy · 2019-01-10 17:06:17 UTC

Hi @mitchell.davis,

Wondering if you are querying the Parquet file via Hive or directly on the file system? Any chance you can send me the parquet-tool meta for this Parquet file?

Thanks,
@balaji.ramaswamy

mitchell.davis · 2019-01-10 21:16:41 UTC

metadata.zip (13.0 KB)

My Bad @balaji.ramaswamy, I uploaded the wrong metadata, I’ve corrected it in this one.

We’re querying directly from AWS S3.

balaji.ramaswamy · 2019-01-10 22:08:04 UTC

@mitchell.davis

Thanks. Let me look into this and get back to you. Meanwhile if the Parquet files or one of the Parquet Files is relatively small and does not have sensitive data, kindly attach

Thanks
@balaji.ramaswamy

mitchell.davis · 2019-01-10 22:27:49 UTC

It’s not small and it is kinda sensitive. Sorry @balaji.ramaswamy. Any insight you can give me would be great.

ben · 2019-01-14 15:51:29 UTC

Hi @mitchell.davis,

Can you share any information about how this parquet file that gives the exception was generated?

Thanks,
@ben

mitchell.davis · 2019-01-14 16:28:16 UTC

It was built with Spark @ben. I’m currently working on generating a proof of concept that I’ll attach here.

mitchell.davis · 2019-01-14 18:57:58 UTC

@ben @balaji.ramaswamy: I’ve created a test parquet file that is producing the error in our Dremio Implementation.

part-00000-77734830-7ea8-4e3d-9ea7-250d7b04661d.snappy.parquet.zip (687 Bytes)

Specifically, here is the error that is coming up. For some reason it came up under jobs this time. I usually have to navigate the server logs.

DATA_READ ERROR: Failure while attempting to retrieve metadata information for table uploads.<REDACTED>."part-00000-77734830-7ea8-4e3d-9ea7-250d7b04661d.snappy_parquet-2d705144-d72a-4631-b24b-7730db0b0f9e".

SQL Query select * from table("__home".<REDACTED>."part-00000-77734830-7ea8-4e3d-9ea7-250d7b04661d.snappy_parquet-2d705144-d72a-4631-b24b-7730db0b0f9e" (type => 'parquet')) limit 500


  (java.lang.UnsupportedOperationException) SMALLINT is not supported
    com.dremio.exec.store.dfs.MetadataUtils.toPartitionValue():183
    com.dremio.exec.store.parquet.ParquetFormatDatasetAccessor.getSplits():260
    com.dremio.exec.store.parquet.ParquetFormatDatasetAccessor.buildAll():194
    com.dremio.exec.store.parquet.ParquetFormatDatasetAccessor.buildDataset():175
    com.dremio.exec.store.dfs.FileSystemDatasetAccessor.getDataset():112
    com.dremio.service.namespace.QuietAccessor.getDataset():42
    com.dremio.exec.store.MaterializedDatasetTable.getRowType():73
    org.apache.calcite.sql.validate.ProcedureNamespace.validateImpl():69
    org.apache.calcite.sql.validate.AbstractNamespace.validate():84
    org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():943
    org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():924
    org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2971
    org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom():2956
    org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect():3197
    org.apache.calcite.sql.validate.SelectNamespace.validateImpl():60
    org.apache.calcite.sql.validate.AbstractNamespace.validate():84
    org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace():943
    org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery():924
    org.apache.calcite.sql.SqlSelect.validate():226
    org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression():899
    org.apache.calcite.sql.validate.SqlValidatorImpl.validate():609
    com.dremio.exec.planner.sql.SqlConverter.validate():180
    com.dremio.exec.planner.sql.handlers.PrelTransformer.validateNode():174
    com.dremio.exec.planner.sql.handlers.PrelTransformer.validateAndConvert():163
    com.dremio.exec.planner.sql.handlers.PrelTransformer.validateAndConvert():159
    com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan():43
    com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan():69
    com.dremio.exec.work.foreman.AttemptManager.run():292
    java.util.concurrent.ThreadPoolExecutor.runWorker():1149
    java.util.concurrent.ThreadPoolExecutor$Worker.run():624
    java.lang.Thread.run():748

balaji.ramaswamy · 2019-01-14 20:35:54 UTC

Hi @mitchell.davis

The attached Parquet (full of nulls) queries fine from uploads, NAS and S3,

What version of Dremio are you using?
Can we have the job profile that got created last time?
If you copy the file to your computer, are you able to upload to Dremio via the upload button?

Thanks
@balaji.ramaswamy

mitchell.davis · 2019-01-14 21:00:22 UTC

Here is the profile:

784df5d4-3f6c-40d6-904d-baee81e19836.zip (4.3 KB)

That was gathered from downloading it on my machine and uploading it via the upload button.

Our version of dremio is: 3.0.0

balaji.ramaswamy · 2019-01-14 22:00:51 UTC

Hi @mitchell.davis

I was able to reproduce the issue on an older version of Dremio. Not happening on a later version.

Caused by: java.lang.UnsupportedOperationException: SMALLINT is not supported
at com.dremio.exec.store.dfs.MetadataUtils.toPartitionValue(MetadataUtils.java:183) ~[dremio-sabot-kernel-2.1.8-201810192316430922-9b6e669.jar:2.1.8-201810192316430922-9b6e669]

Kindly download 3.0.6 from our website and give it a shot

Thanks
@balaji.ramaswamy

mitchell.davis · 2019-01-15 01:46:20 UTC

Sounds good @balaji.ramaswamy. I’ll upgrade and report back.

mitchell.davis · 2019-01-15 03:48:09 UTC

Well, @balaji.ramaswamy I upgraded. I tested on the parquet file I attached and everything seems to work. However, when I point at the original data, it still fails. The worst part is that I’m not getting any errors in the logs now.

The screenshot above is from the error we get in the UI after I setup the DataSet in the UI. I’m scouring the logs and nothing is coming through that hints anything is wrong. Are there settings I’m missing that would push those kinds of errors to the logs?

Also, the documentation says we can only restore from the same version of the software. Does that mean any version or only major releases? I lost all my metadata forgetting that I couldn’t update.

balaji.ramaswamy · 2019-01-15 07:28:35 UTC

Hi @mitchell.davis

Can you please click help-about Dremio and see if the version shows 3.0.6?

Thanks
@balaji.ramaswamy

mitchell.davis · 2019-01-15 14:37:39 UTC

@balaji.ramaswamy

mitchell.davis · 2019-01-15 15:45:34 UTC

@balaji.ramaswamy I was able to find the new error in the logs: (I cut off the bottom of the stack trace, did you need that?)

2019-01-15 03:42:31,695 [qtp160830059-109] ERROR c.d.d.server.GenericExceptionMapper - Unexpected exception when processing POST http://dremio.veriship.net/apiv2/datasets/new_untitled?parentDataset=<REDACTED>&newVersion=0003598160851519&limit=150 : java.lang.NullPointerException
java.lang.NullPointerException: null
at com.dremio.dac.explore.DatasetsResource.getDatasetSummary(DatasetsResource.java:268) ~[dremio-dac-backend-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
at com.dremio.dac.explore.DatasetsResource.newUntitled(DatasetsResource.java:141) ~[dremio-dac-backend-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
at com.dremio.dac.explore.DatasetsResource.newUntitledFromParent(DatasetsResource.java:208) ~[dremio-dac-backend-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]

balaji.ramaswamy · 2019-01-15 16:20:11 UTC

Hi @mitchell.davis

This is a different error. Can you try adding it as a local NAS as opposed to S3 or direct upload?

Thanks
@balaji.ramaswamy

mitchell.davis · 2019-01-15 16:40:38 UTC

@balaji.ramaswamy, I did just that and the same thing happened. Here is the error that popped up in the logs:

2019-01-15 16:36:29,220 [qtp160830059-150] WARN  com.dremio.exec.catalog.DatasetSaver - Failure while retrieving and saving dataset temp.problemfiles.
java.lang.UnsupportedOperationException: SMALLINT is not supported
	at com.dremio.exec.store.dfs.MetadataUtils.toPartitionValue(MetadataUtils.java:183) ~[dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.store.parquet.ParquetFormatDatasetAccessor.getSplits(ParquetFormatDatasetAccessor.java:267) ~[dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.store.parquet.ParquetFormatDatasetAccessor.buildAll(ParquetFormatDatasetAccessor.java:201) ~[dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.store.parquet.ParquetFormatDatasetAccessor.buildDataset(ParquetFormatDatasetAccessor.java:182) ~[dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.store.dfs.FileSystemDatasetAccessor.getDataset(FileSystemDatasetAccessor.java:112) ~[dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.catalog.DatasetSaver.completeSave(DatasetSaver.java:65) ~[dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.catalog.DatasetManager.createOrUpdateDataset(DatasetManager.java:380) [dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.catalog.CatalogImpl.createOrUpdateDataset(CatalogImpl.java:497) [dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.exec.catalog.DelegatingCatalog.createOrUpdateDataset(DelegatingCatalog.java:179) [dremio-sabot-kernel-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.dac.service.source.SourceService.createPhysicalDataset(SourceService.java:535) [dremio-dac-backend-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.dac.resource.SourceResource.saveFolderFormat(SourceResource.java:375) [dremio-dac-backend-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
2019-01-15 16:36:29,680 [qtp160830059-20402] ERROR c.d.d.server.GenericExceptionMapper - Unexpected exception when processing POST http://dremio.veriship.net/apiv2/datasets/new_untitled?parentDataset=temp.problemfiles&newVersion=0009912186041139&limit=150 : java.lang.NullPointerException
java.lang.NullPointerException: null
	at com.dremio.dac.explore.DatasetsResource.getDatasetSummary(DatasetsResource.java:268) ~[dremio-dac-backend-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.dac.explore.DatasetsResource.newUntitled(DatasetsResource.java:141) ~[dremio-dac-backend-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at com.dremio.dac.explore.DatasetsResource.newUntitledFromParent(DatasetsResource.java:208) ~[dremio-dac-backend-3.0.6-201812082352540436-1f684f9.jar:3.0.6-201812082352540436-1f684f9]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_191]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_191]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]

@balaji.ramaswamy, according to the stack trace, it looks like the split logic is choosing a SMALLINT column as a partition split which it doesn’t support: https://github.com/dremio/dremio-oss/blob/16b26a201d241b496cb9b03f7d301ff2b139eea9/sabot/kernel/src/main/java/com/dremio/exec/store/dfs/MetadataUtils.java#L183

I can create a pull request to update that logic but I have NO idea what that will break.

balaji.ramaswamy · 2019-01-15 17:05:29 UTC

Hi @mitchell.davis

I just downloaded 3.0.6 Community Edition from the Dremio website and just followed the exact same steps on your Parquet and works fine. I have a screen recording and wanted to see if you can find anything I missed or you missed

I am not able to attach the screen recording as it is ~ 23 MB

Do you have an email I share the recording?

Thanks
@balaji.ramaswamy

mitchell.davis · 2019-01-15 19:32:02 UTC

@balaji.ramaswamy, the parquet file you’re using works. However, that was cleansed data. I pulled out as much sensitive data as I could. The problem popped back up when I pointed to the original data that you don’t have. I’m working to figure out if I can cleanse it a different way that will break 3.0.6.