What is the reason for the 800 column max?

I’m aware of the maximum number of columns being defined as 800:

What is the reason for this limit - it seems very arbitrary so I wondered that the technical limitations are? We’re currently using ~ 790 but have more we would like to add. We have found the ui to become sluggish when viewing this many columns, so is it just to limit slowdown on the ui?

I’ve noticed in the code a METADATA_LEAF_COLUMN_MAX defined:

Am I correct in thinking this is where the limit is defined, and can you see any problems with us trialing a build which has this increased?



With the current architecture and heap memory Dremio uses to load column/split metadata into heap, 800 is an optimal limit. Now you may ask, “Can I increase heap and increase the column limit”, increasing heap can cause longer Full GC cycles and hence make the coordinator go unresponsive