How to handle Dremio Limit 32000 bytes of a field

Dremio University
1

While accessing a collection (XYZ) in a MongoDB, Dremio is showing an error “Field ‘description’ exceeds the size limit of 32000 bytes”. As per Dremio documentation, this is the limit in Dremio.

So, created a copy of the collection (XYZ_Copy) and dropped the “description” field, it is confirmed that the “description” field is deleted from the collection XYZ_Copy. When accessed XYZ_Copy collection from Dremio, it is still showing the same error “Field ‘description’ exceeds the size limit of 32000 bytes” even after the field is dropped.

2

Hello @Vijay

What do you mean by copy?

You created a VDS on top of XYZ without “description” and still hitting the issue?

Thanks,
Rakesh

3

No, a copy of the collection in MongoDB itself without the “description” field.

4

@Vijay

Is it possible a different column also has a field > 32K

5

No, there is no other field of size > 32K. Dremio is showing error only on this field.

6

Deleting a field does not literally delete the field in MongoDB, but UNSET the field. So, internally the field exists, so the meta data. As Dremio accesses metadata, it is finding that field which was deleted (by using UNSET).

After troubleshooting, We are creating a new collection with the needed fields only, instead of copying the whole collection and UNSET a field.

New collection with only the needed is now accessible from Dremio. Working on it . Will keep posted the updates.

7

Dear Dremio Community,

Like to share a solution to handle 32000 bytes limit in Dremio.

We had an issue accessing mongoDB Collection havng a value of size 32000 bytes in a field even that field is not selected in Dremio.

Solution to fix this is, Create a view in mongoDB with the fields you need without that 32000 bytes field causing the error. Then, access the view from Dremio, It is Working well.

9

You can use for current session in jdbc connection:

alter session set limits.single_field_size_bytes = 99000;

or system wide (affect to webUI too):

alter system set limits.single_field_size_bytes = 99000;
1 Like
10

For me after running

alter session set limits.single_field_size_bytes = 99000;(from the UI)

The error changed to new limit size wierdly lesser than 99000

Field with index 7 exceeds the size limit of 65536 bytes

Is 65536 bytes the max field size set by you folks ?

11

@vyz101

Yes 65K is the other limit, are you able to truncate your source column to 32K at source (probably on a new column) and bring that into Dremio?

Thanks
Bali

1 Like