Limit List field exceeded the maximum number of elements 128

Hi,

I’ve encountered since I am on Community Edition v14.0 the error
List field exceeded the maximum number of elements 128, as some of my parquet files have list with more than 128 items.

I did not have this error message in the past (with v4.9)
Do you confirm it’s a new limit ? I do not see it in documentation about limits : Dremio
Is there any way to change this limit ?

Thanks

Here is the error log :

2021-03-29 09:50:56,145 [grpc-default-executor-16139] INFO c.d.service.jobs.JobResultsStore - User Error Occurred [ErrorId: 1e156f0f-2185-4028-b241-1e8b016b8327]
com.dremio.common.exceptions.UserException: List field ‘data_cart_items’ exceeded the maximum number of elements 128
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:804)
at com.dremio.service.jobs.JobResultsStore.loadJobData(JobResultsStore.java:145)
at com.dremio.service.jobs.JobResultsStore$LateJobLoader.load(JobResultsStore.java:294)
at com.dremio.service.jobs.JobDataImpl.range(JobDataImpl.java:46)
at com.dremio.service.jobs.LocalJobsService.getJobData(LocalJobsService.java:906)
at com.dremio.service.jobs.JobsFlightProducer.getStream(JobsFlightProducer.java:76)
at org.apache.arrow.flight.FlightService.doGetCustom(FlightService.java:111)
at org.apache.arrow.flight.FlightBindingService$DoGetMethod.invoke(FlightBindingService.java:144)
at org.apache.arrow.flight.FlightBindingService$DoGetMethod.invoke(FlightBindingService.java:134)
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172)
at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
at io.grpc.util.TransmitStatusRuntimeExceptionInterceptor$1.onHalfClose(TransmitStatusRuntimeExceptionInterceptor.java:74)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:820)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

@dfleckinger

Dremio did not support complex type resolution during planning in v4.9 and were treating the datatype is a mixed type. In the current version, the planner detects the right data type as ARRAY and hit the maximum elements per LIST limit

How many such datasets are affected?

Thanks
Bali

I think only one dataset is affected, but not the least important unfortunately.
More than 128 cart items is frequent in FMCG baskets.

If there is no way to use this dataset anymore, I will need to process all the data beforehand by flattening it.

@dfleckinger Here you go “store.parquet.list_items.threshold”

Thanks @balaji.ramaswamy , works great ! That fixed my issue