Error "Definition of this dataset is out of date." when query VDS

1

Guys,

When I’m trying to query some VDS that I just created, I’m getting this error:

Definition of this dataset is out of date. There were schema changes in "dataset"

When I change the query to use de Full path (PDS), it works. I’m appending no more data to those tables and I’m consuming Parquet from S3.

Running Dremio 3.1.0 here. Full stack trace below.

com.dremio.dac.service.errors.InvalidQueryException: Definition of this dataset is out of date. There were schema changes in lims_data."result".

	at com.dremio.dac.explore.DatasetTool.toInvalidQueryException(DatasetTool.java:354)
	at com.dremio.dac.explore.DatasetTool.newUntitled(DatasetTool.java:399)
	at com.dremio.dac.explore.DatasetsResource.newUntitled(DatasetsResource.java:149)
	at com.dremio.dac.explore.DatasetsResource.newUnitledSql(DatasetsResource.java:172)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.dremio.common.exceptions.UserException: Definition of this dataset is out of date. There were schema changes in lims_data."result".

	at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746)
	at com.dremio.exec.planner.sql.ExtendedToRelContext.checkRowTypeConsistency(ExtendedToRelContext.java:106)
	at com.dremio.exec.planner.sql.ExtendedToRelContext.expandView(ExtendedToRelContext.java:80)
	at com.dremio.exec.planner.logical.ViewTable.toRel(ViewTable.java:116)
	at com.dremio.exec.catalog.DremioPrepareTable.toRel(DremioPrepareTable.java:89)
	at org.apache.calcite.sql2rel.RelStructuredTypeFlattener.rewriteRel(RelStructuredTypeFlattener.java:657)
	at sun.reflect.GeneratedMethodAccessor247.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.calcite.util.ReflectUtil.invokeVisitorInternal(ReflectUtil.java:257)
	at org.apache.calcite.util.ReflectUtil.invokeVisitor(ReflectUtil.java:214)
	at org.apache.calcite.util.ReflectUtil$1.invokeVisitor(ReflectUtil.java:464)
	at org.apache.calcite.sql2rel.RelStructuredTypeFlattener$RewriteRelVisitor.visit(RelStructuredTypeFlattener.java:721)
	at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:46)
	at org.apache.calcite.rel.RelVisitor.visit(RelVisitor.java:44)
	at org.apache.calcite.sql2rel.RelStructuredTypeFlattener$RewriteRelVisitor.visit(RelStructuredTypeFlattener.java:716)
	at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:45)
	at org.apache.calcite.rel.RelVisitor.visit(RelVisitor.java:44)
	at org.apache.calcite.sql2rel.RelStructuredTypeFlattener$RewriteRelVisitor.visit(RelStructuredTypeFlattener.java:716)
	at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:45)
	at org.apache.calcite.rel.RelVisitor.visit(RelVisitor.java:44)
	at org.apache.calcite.sql2rel.RelStructuredTypeFlattener$RewriteRelVisitor.visit(RelStructuredTypeFlattener.java:716)
	at org.apache.calcite.rel.BiRel.childrenAccept(BiRel.java:45)
	at org.apache.calcite.rel.RelVisitor.visit(RelVisitor.java:44)
	at org.apache.calcite.sql2rel.RelStructuredTypeFlattener$RewriteRelVisitor.visit(RelStructuredTypeFlattener.java:716)
	at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
	at org.apache.calcite.rel.RelVisitor.visit(RelVisitor.java:44)
	at org.apache.calcite.sql2rel.RelStructuredTypeFlattener$RewriteRelVisitor.visit(RelStructuredTypeFlattener.java:716)
	at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
	at org.apache.calcite.rel.RelVisitor.visit(RelVisitor.java:44)
	at org.apache.calcite.sql2rel.RelStructuredTypeFlattener$RewriteRelVisitor.visit(RelStructuredTypeFlattener.java:716)
	at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72)
	at org.apache.calcite.rel.RelVisitor.visit(RelVisitor.java:44)
	at org.apache.calcite.sql2rel.RelStructuredTypeFlattener$RewriteRelVisitor.visit(RelStructuredTypeFlattener.java:716)
	at org.apache.calcite.sql2rel.RelStructuredTypeFlattener.rewrite(RelStructuredTypeFlattener.java:177)
	at com.dremio.exec.planner.sql.DremioSqlToRelConverter.flattenTypes(DremioSqlToRelConverter.java:69)
	at com.dremio.exec.planner.sql.SqlConverter.toConvertibleRelRoot(SqlConverter.java:286)
	at com.dremio.exec.planner.sql.handlers.PrelTransformer.toConvertibleRelRoot(PrelTransformer.java:703)
	at com.dremio.exec.planner.sql.handlers.PrelTransformer.convertToRelRootAndJdbc(PrelTransformer.java:713)
	at com.dremio.exec.planner.sql.handlers.PrelTransformer.convertToRel(PrelTransformer.java:779)
	at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateAndConvert(PrelTransformer.java:168)
	at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateAndConvert(PrelTransformer.java:162)
	at com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan(NormalHandler.java:43)
	at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:69)
	at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:288)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
1 Like
2

I am also facing above issue in my case deleting VDS and re-create with same name works for me

3

Sometimes this can also be solved by opening the VDS, adding a space (or some other non-consequential whitespace character), rerunning the query and saving the results.

4

@Vikash_Singh, @allan.sene,

We recently changed the logic or how the schema’s associated with each VDS are handled. If there is a change in the type definition of some column in the PDS which a VDS references, formerly, Dremio would simply throw an error about type mismatch and you’d have to initiate the action I described in my last comment (add some white space, rerun). Now, Dremio goes ahead and updates the VDS automatically when you query it, so long as the user has “Can Edit” permissions on the VDS. Other users that do not have this, will get the error you saw and will have to ask the VDS owner to run a query against it.

You might be seeing this because of our new support for Decimal type. Columns that used to be cast to Double are now their true Decimal types, so VDS have to be updated in this way.

1 Like