I see that dremio-oss can read delta.io tables.
But it can’t read tables created by delta engine, which is actually databricks version of delta inside Databrick Runtime.
The only reason is Dremio connector can’t read the delta log where the checkpoint is multiple parquet files, not one.
While spark connector for delta-io can read such table, out of the box.
Does this issue exist only in OSS or also in Managed Dremio and Dremio Cloud?
I was able to find that from version 19.0, Dremio supports multi-parquet file checkpoints. But for some reason, Dremio checks some metadata in every parquet file independently, while it must build checkpoint completely and then check any metadata fields.
Specifically, it’s related to protocol.minRequiredReaderVersion which in my case exists in only one of the parquet file partitions, because I never change Protocol.
From delta protocol documentation
Since it is possible that a writer will fail while writing out one or more parts of a multi-part checkpoint,
readers must only use a complete checkpoint, wherein all fragments are present. For performance
reasons, readers should search for the most recent earlier checkpoint that is complete.
A given snapshot of the table can be computed by replaying the events committed to the table in
ascending order by commit version. A given snapshot of a Delta table consists of:
* A single protocol action
...
@GrigorievNick This issue seems to have been fixed in 19.0, what version did you check?
@balaji.ramaswamy I use 19.1 default for dremio chart_v2.
Unexpected exception when processing PUT http://internal-k8s-eksinternaldldrem.elb.amazonaws.com/apiv2/source/datalake-mhr-credentials/folder_format/organic : java.lang.IllegalStateException: Commit Json for version 0 not found while reading metadata
java.lang.IllegalStateException: Commit Json for version 0 not found while reading metadata
at com.google.common.base.Preconditions.checkState(Preconditions.java:508)
at com.dremio.exec.store.deltalake.DeltaSnapshotListProcessor.findValidSnapshots(DeltaSnapshotListProcessor.java:79)
at com.dremio.exec.store.deltalake.DeltaLakeTable.getListOfSnapshot(DeltaLakeTable.java:193)
at com.dremio.exec.store.deltalake.DeltaLakeTable.getAllSplits(DeltaLakeTable.java:99)
at com.dremio.exec.store.deltalake.DeltaLakeFormatDatasetAccessor.listPartitionChunks(DeltaLakeFormatDatasetAccessor.java:171)
at com.dremio.exec.store.dfs.FileSystemPlugin.listPartitionChunks(FileSystemPlugin.java:1541)
at com.dremio.exec.catalog.DatasetSaverImpl.saveUsingV1Flow(DatasetSaverImpl.java:245)
at com.dremio.exec.catalog.DatasetSaverImpl.save(DatasetSaverImpl.java:121)
at com.dremio.exec.catalog.DatasetSaverImpl.save(DatasetSaverImpl.java:143)
at com.dremio.exec.catalog.DatasetManager.createOrUpdateDataset(DatasetManager.java:494)
at com.dremio.exec.catalog.CatalogImpl.createOrUpdateDataset(CatalogImpl.java:1036)
at com.dremio.exec.catalog.SourceAccessChecker.createOrUpdateDataset(SourceAccessChecker.java:285)
at com.dremio.exec.catalog.DelegatingCatalog.createOrUpdateDataset(DelegatingCatalog.java:257)
at com.dremio.dac.service.source.SourceService.createPhysicalDataset(SourceService.java:545)
at com.dremio.dac.resource.SourceResource.saveFolderFormat(SourceResource.java:389)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at com.dremio.dac.server.GenericResponseHeadersFilter.doFilter(GenericResponseHeadersFilter.java:46)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.dremio.dac.server.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:66)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:713)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.lang.Thread.run(Thread.java:748)
I also check a unit test TestDeltaLogCheckpointParquetReader.testMultiPartCheckpointParquet in prepared data both checkpoint file has separate row with minReaderVersion. And the value of minReaderVersion is the same in both files. So look like that’s what dremio expects, but as I write in first message delta protocol require to build all checkpoint completely from all multy part files.