What is the reason for the 800 column max?

Saltxwater · April 22, 2021, 10:42am

Hi!
I’m aware of the maximum number of columns being defined as 800:
Dremio?

What is the reason for this limit - it seems very arbitrary so I wondered that the technical limitations are? We’re currently using ~ 790 but have more we would like to add. We have found the ui to become sluggish when viewing this many columns, so is it just to limit slowdown on the ui?

I’ve noticed in the code a METADATA_LEAF_COLUMN_MAX defined:

github.com

dremio/dremio-oss/blob/398ead8918dc1dd7c2c1e8c4d4e4ba53d78f3fed/sabot/kernel/src/main/java/com/dremio/exec/catalog/CatalogOptions.java#L44


public static final LongValidator STORAGE_PLUGIN_CREATE_MAX = new PositiveLongValidator("store.plugin.wait_millis", TimeUnit.HOURS.toMillis(4), TimeUnit.SECONDS.toMillis(120));


// Maximum time to wait when starting/creating storage plugin during startup before failing.
public static final LongValidator STARTUP_WAIT_MAX = new PositiveLongValidator("store.start.wait_millis", TimeUnit.HOURS.toMillis(4), TimeUnit.SECONDS.toMillis(120));


// When metadata impacting configuration parameter is changed, if old metadata should be kept
public static final BooleanValidator STORAGE_PLUGIN_KEEP_METADATA_ON_REPLACE =
    new BooleanValidator("store.plugin.keep_metadata_on_replace", false);


// Maximum number of leaf columns allowed for metadata
public static final LongValidator METADATA_LEAF_COLUMN_MAX = new PositiveLongValidator("store.plugin.max_metadata_leaf_columns", Integer.MAX_VALUE, 800);


// Maximum nested levels allowed for a column
public static final LongValidator MAX_NESTED_LEVELS = new PositiveLongValidator("store.plugin.max_nested_levels", 64, 16);


// ORC ACID table factor for leaf columns
public static final LongValidator ORC_DELTA_LEAF_COLUMN_FACTOR = new PositiveLongValidator("store.hive.orc_delta_leaf_column_factor", Integer.MAX_VALUE, 5);


// Maximum number of single split partitions allowed to be saved together
public static final LongValidator SINGLE_SPLIT_PARTITION_MAX = new PositiveLongValidator("store.plugin.max_single_split_partitions", Long.MAX_VALUE, 500);

Am I correct in thinking this is where the limit is defined, and can you see any problems with us trialing a build which has this increased?

Thanks

balaji.ramaswamy · April 23, 2021, 5:23am

@Saltxwater

With the current architecture and heap memory Dremio uses to load column/split metadata into heap, 800 is an optimal limit. Now you may ask, “Can I increase heap and increase the column limit”, increasing heap can cause longer Full GC cycles and hence make the coordinator go unresponsive

Thanks
Bali

Topic		Replies	Views
Dremio 800 column per table hard limit work around	6	2091	November 7, 2021
Change max columns from 800 to custom max value	1	1575	February 27, 2020
32k Limit for Parquet columns	6	3968	July 22, 2020
Dremio Query issues	4	1153	May 31, 2021
Dremio Limits on number of columns in Tables, way to increase it Dremio University	4	2047	May 22, 2020

What is the reason for the 800 column max?

Related topics