Reflection refresh is failing with NullPointerException in Dremio 24.2.2

1

Reflection refresh is failing with NullPointerException in Dremio 24.2.2

SQL Query REFRESH REFLECTION ‘2351ad47-9327-46fa-b378-aa57a8f4d824’ AS ‘2df705e6-cc92-481e-9bfe-c7466a65d36e’

(java.lang.NullPointerException) null
com.dremio.service.reflection.materialization.AccelerationStoragePlugin.createNewTable():470
com.dremio.exec.catalog.CatalogImpl.createNewTable():1199
com.dremio.exec.catalog.DelegatingCatalog.createNewTable():206
com.dremio.service.reflection.refresh.RefreshHandler.getPlan():264
com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan():59
com.dremio.exec.work.foreman.AttemptManager.plan():565
com.dremio.exec.work.foreman.AttemptManager.lambda$run$4():462
com.dremio.service.commandpool.ReleasableBoundCommandPool.lambda$getWrappedCommand$3():140
com.dremio.service.commandpool.CommandWrapper.run():70
com.dremio.context.RequestContext.run():109
com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$4():226
com.dremio.common.concurrent.ContextMigratingExecutorService$ComparableRunnable.run():206
java.util.concurrent.Executors$RunnableAdapter.call():511
java.util.concurrent.FutureTask.run():266
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():750

image
image1893×883 90.9 KB

2

Hi @Swetha,

It looks like Dremio has problems getting the right physical path to write data to. We added a null check to that function in the upcoming 24.3 release. However, it would be great if you can help us with additional details, so we understand what pattern leads to this NPE. Please answer the following questions:

  1. What type of Dremio deployment is this?
  2. Is it possible to upload the profile of the failed reflection?
  3. Can you upload dremio.conf, dremio.env and core-site.xml as well?
  4. I’m also interested in the type of storage you have (is it distributed storage)?
  5. Do you use any storage proxy?
  6. Where there any support keys enabled/disabled recently?
  7. Is this a new reflection, or did it use to work in the past? What else can you tell us about it?
  8. Are other reflections failing with the same NPE, or are they successful?

Thanks, Bogdan

3

Hi @bogdan.coman,

  1. What type of Dremio deployment is this?

Have deployed Dremio image as a container.

  1. Is it possible to upload the profile of the failed reflection?

PFA f97dda67-31e6-4bb4-9bd9-dfcba4e586a5.zip

  1. Can you upload dremio.conf, dremio.env and core-site.xml as well?

PFA

  1. I’m also interested in the type of storage you have (is it distributed storage)?

It is not a distributed storage.

  1. Do you use any storage proxy?

No

  1. Where there any support keys enabled/disabled recently?

No new support key were enabled/disabled. We just have these configured -dremio.iceberg.enabled, limits.single_field_size_bytes.

  1. Is this a new reflection, or did it use to work in the past? What else can you tell us about it?

No we had same reflections before as well in lower versions of Dremio 20.X and 22.x. The reflections were working in the older versions.

  1. Are other reflections failing with the same NPE, or are they successful?

Yes, all the reflections are failing with NPE.

Thanks,
Swetha
f97dda67-31e6-4bb4-9bd9-dfcba4e586a5.zip (11.2 KB)[dremio_conf.zip|attachment](upload://cTll2o5oOUwQe0XAsdBxr9OP8dG.zip) (1.9 KB)

4

Hi @Swetha,

Thank you for answering my questions. You shouldn’t disable Iceberg, in this release (and going forward) it should be enabled by default. I can’t say for sure it will fix the NPE, but I strongly suggest to enable it and try again. Build a new reflection afterwards and see if that works.

You are using the Enterprise Edition of Dremio, this support channel is for the Community Edition. I suggest you contact us through the designated support channel, where SLAs are in place.

Thanks, Bogdan

5

Hi @bogdan.coman ,

If we enable iceberg, we get into the below issue.

image
image1893×898 104 KB

Also, we are using Community Edition of Dremio.

Thanks,
Swetha

6

Hi Swetha,

You’re correct, you are using CE, sorry about that, so the org name and I just assumed :slight_smile:

The error in the last screenshot also points to a storage issue. Your dremio.conf says you don’t configure the distributed storage, but I wonder if the dremio.conf file is the same on all the nodes? If it is, then can you add paths.dist in dremio.conf to point to a local directory: file:/// on all the nodes?

Then start Dremio up and let me know if it works.

Also, were there any changes in regards to the distributed storage? Did you use something in the past, which now you removed?
How many nodes are in this cluster?

Thanks, Bogdan

7

Hi @bogdan.coman,

Looks like the NPE issue is no more reproducible with 24.3.0 image. Can you please confirm if this issue is fixed as part of 24.3.0?

Also, we observed an issue with one query. The same query used to work in older versions.

Query: SELECT ActivityLogID, Content, Encoding FROM ActivityLog_Content;

Not Working: SELECT ActivityLogID, Content, Encoding FROM ActivityLog_Content;

image
image1327×561 21.4 KB

Working: SELECT ActivityLogID, Content, “Encoding” FROM ActivityLog_Content;

Expecting double quotes around Encoding. Any idea why is this happening?

Thanks,
Swetha

8

Please find the screenshot for working query

Working_Query
Working_Query1318×481 41.2 KB

Thanks,
Swetha

9

Looks like the NPE issue is no more reproducible with 24.3.0 image. Can you please confirm if this issue is fixed as part of 24.3.0?

Thanks for testing with 24.3, as requested. Looks like the implementation of the NPE check helped your usecase.

Expecting double quotes around Encoding. Any idea why is this happening?

It shouldn’t, if that’s the exact name of the column. If there would be any special characters, then yes, double quotes would be expected around the column name. I suggest you start a new thread for each new unrelated question you have.

Thanks, Bogdan