Unable to create reflection due to failed to read metadata

1

Hello,
I’m facing error when create reflection. here’s the error log :

8/16/2019 4:58:18 PM2019-08-16 07:58:18,687 [22a99de5-baf2-f849-9c5c-c7d77dc60200:foreman] 
INFO  c.dremio.exec.catalog.DatasetManager - User Error Occurred [ErrorId: 3369741a-4c12-4398-9f26-0149e264bda7]

8/16/2019 4:58:18 PMcom.dremio.common.exceptions.UserException: Failure while attempting to read metadata for table "__accelerator"."6fd10ba2-7008-4ec0-85ae-a41f16c39e10"."10c609d9-bffe-4b3a-a0cb-58c5858ea854" from source.

8/16/2019 4:58:18 PM	at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-3.1.8-201903290151120189-36bb2bf.jar:3.1.8-201903290151120189-36bb2bf]

8/16/2019 4:58:18 PM	at com.dremio.exec.catalog.DatasetManager.createDataset(DatasetManager.java:424) [dremio-sabot-kernel-3.1.8-201903290151120189-36bb2bf.jar:3.1.8-201903290151120189-36bb2bf]

8/16/2019 4:58:18 PM	at com.dremio.exec.catalog.CatalogImpl.createDataset(CatalogImpl.java:457) [dremio-sabot-kernel-3.1.8-201903290151120189-36bb2bf.jar:3.1.8-201903290151120189-36bb2bf]

8/16/2019 4:58:18 PM	at com.dremio.exec.catalog.DelegatingCatalog.createDataset(DelegatingCatalog.java:159) [dremio-sabot-kernel-3.1.8-201903290151120189-36bb2bf.jar:3.1.8-201903290151120189-36bb2bf]

8/16/2019 4:58:18 PM	at com.dremio.service.reflection.load.LoadMaterializationHandler.refreshMetadata(LoadMaterializationHandler.java:148) [dremio-services-accelerator-3.1.8-201903290151120189-36bb2bf.jar:3.1.8-201903290151120189-36bb2bf]

8/16/2019 4:58:18 PM	at com.dremio.service.reflection.load.LoadMaterializationHandler.toResult(LoadMaterializationHandler.java:112) [dremio-services-accelerator-3.1.8-201903290151120189-36bb2bf.jar:3.1.8-201903290151120189-36bb2bf]

8/16/2019 4:58:18 PM	at com.dremio.exec.planner.sql.handlers.commands.DirectWriterCommand.plan(DirectWriterCommand.java:96) [dremio-sabot-kernel-3.1.8-201903290151120189-36bb2bf.jar:3.1.8-201903290151120189-36bb2bf]

8/16/2019 4:58:18 PM	at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:295) [dremio-sabot-kernel-3.1.8-201903290151120189-36bb2bf.jar:3.1.8-201903290151120189-36bb2bf]

8/16/2019 4:58:18 PM	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]

8/16/2019 4:58:18 PM	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]

8/16/2019 4:58:18 PM	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

8/16/2019 4:58:18 PMCaused by: java.lang.NullPointerException: null

8/16/2019 4:58:18 PM	at com.dremio.service.reflection.materialization.AccelerationStoragePlugin$1.apply(AccelerationStoragePlugin.java:171) ~[dremio-services-accelerator-3.1.8-201903290151120189-36bb2bf.jar:3.1.8-201903290151120189-36bb2bf]

8/16/2019 4:58:18 PM	at com.dremio.service.reflection.materialization.AccelerationStoragePlugin$1.apply(AccelerationStoragePlugin.java:165) ~[dremio-services-accelerator-3.1.8-201903290151120189-36bb2bf.jar:3.1.8-201903290151120189-36bb2bf]

8/16/2019 4:58:18 PM	at com.google.common.collect.Iterators$7.transform(Iterators.java:750) ~[guava-20.0.jar:na]

8/16/2019 4:58:18 PM	at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47) ~[guava-20.0.jar:na]

8/16/2019 4:58:18 PM	at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47) ~[guava-20.0.jar:na]

8/16/2019 4:58:18 PM	at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:52) ~[guava-20.0.jar:na]

8/16/2019 4:58:18 PM	at com.google.common.collect.MultitransformedIterator.hasNext(MultitransformedIterator.java:50) ~[guava-20.0.jar:na]

8/16/2019 4:58:18 PM	at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:249) ~[guava-20.0.jar:na]

8/16/2019 4:58:18 PM	at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:209) ~[guava-20.0.jar:na]

8/16/2019 4:58:18 PM	at com.google.common.collect.FluentIterable.toList(FluentIterable.java:614) ~[guava-20.0.jar:na]

8/16/2019 4:58:18 PM	at com.dremio.service.reflection.materialization.AccelerationStoragePlugin.getDataset(AccelerationStoragePlugin.java:177) ~[dremio-services-accelerator-3.1.8-201903290151120189-36bb2bf.jar:3.1.8-201903290151120189-36bb2bf]

8/16/2019 4:58:18 PM	at com.dremio.exec.catalog.ManagedStoragePlugin.getTable(ManagedStoragePlugin.java:436) ~[dremio-sabot-kernel-3.1.8-201903290151120189-36bb2bf.jar:3.1.8-201903290151120189-36bb2bf]

8/16/2019 4:58:18 PM	at com.dremio.exec.catalog.DatasetManager.createDataset(DatasetManager.java:422) [dremio-sabot-kernel-3.1.8-201903290151120189-36bb2bf.jar:3.1.8-201903290151120189-36bb2bf]

8/16/2019 4:58:18 PM	... 9 common frames omitted

Is there any workaround for this problem ?
Thanks for any help!

2

@Eugene_Gwon

Are you using executors on a separate node with the reflections stored on the local file system?

3

Hello,
It’s single node on docker, version is dremio-oss:3.1.8, and I mounted host directory to /opt/dremio/data.

source files are stored in local filesystem(mounted AWS EFS).
there was about 30k of small json files in local filesystem.
It works very well few days ago, but suddenly failed to create reflection. I guess that It happens after I delete/add bunch of files in directory, which is configured as PDS.

I re-create docker container and try to create reflection by using API, but It shows an error message ‘too many files open’; at that moment, I found that dremio process was actually holds more than 60k of files, which is twice of source files(I check it using this command : lsof -p PID_OF_DREMIO|wc -l).

After I delete/recreate PDS and VDS, It works without any problem… :thinking: