IllegalStateException: Selected partition ... does not have associated splits

Hi,

since i’m running Dremio version 17.0 CE, I often encounters IllegalStateException : Selected partition … does not have associated splits… during reflections refresh or query that try to use such reflections.

It seems that some reflection materializations are inconsistent, and I have to manually remove and recreate reflections so that it works again.

I’m attaching a query profile b3aaa326-51ae-4f06-a5de-e9c74f571cf5.zip (182.6 KB)

Reflections are stored on s3 and I did not remind this kind of issue in previous Dremio versions.

Any insight of the origin of the issue.
Is there any way to know which materialization is causing the issue from the error message ?

Thanks in advance

@dfleckinger This is a known issue where this code path will not exist in 18.x, as a workaround in 17.x, of this happens, please drop and recreate the reflection

Ok thanks @balaji.ramaswamy . is there any way to find which reflection is causing the issue, given the partition split id displayed in the error ?

thanks

@dfleckinger It is the reflection on Reporting.helpers.daily_ndv_iid_s_id_by_wtbid_date_country, reflection ID # 517942fc-b020-4c30-95fc-4768eb975cf2, so you can run the below query

Select * from sys.reflections where reflection_id=‘517942fc-b020-4c30-95fc-4768eb975cf2’

@balaji.ramaswamyI was speaking about the id provided in the error message :
“IllegalStateException: Selected partition 0ccd6464-f657-46ee-ba35-39d8faf4169e_1632362750288_0 does not have associated splits. Please report to administrator.”

The reflection refresh query execution often involves many different reflections.
From the id above (id of a partition), I would like to know how to identify which reflection is causing the issue, so I can drop and recreate it.
Is there any sys table providing the list of partitions for reflection materializations ?

You say that this is a known issue. Is it documented somewhere, I don’t find anything related to it in Dremio 18 release notes (Dremio)

Thanks

@dfleckinger It is a bug that will not reproduce in 18.0 as the code path has changed. The profile of the failed job should have the reflection ID, which you can query sys.reflections to ind out the dataset name

Hi @balaji.ramaswamy ,

We faced exact same issue in dremio version 19.
Although the issue got resolved after dropping and recreating the reflection and has not been reported since then, but I wanted to know if there is something we can do to avoid this kind of problem in the first place.

-Preeti

@Preeti Have you enabled iceberg and unlimited splits? and refreshed metadata once for all PDS?

https://docs.dremio.com/advanced-administration/metadata-caching/#improved-metadata-refreshes-preview