Dremio configuration Constants / Documentation

Site Feedback
1

Hi all,

I had a question about where I could manage the size of each block of parquet file. I found this key store.parquet.block-size and also this blog post https://www.dremio.com/tuning-parquet/.

So Iget new questions “where is the documentation of each key that help to configure and tune Dremio?”. In the file ExecConstants.java (https://github.com/dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/ExecConstants.java) we have many entries ! But there is no documentation where all that keys are referenced.

So could we have a documentation page where all Dremio cofiguration key are listed, detailed and described ?

Thanks

2

Changing any of those values may result in unexpected behavior, which is why most of them are undocumented.

3

Hi !

Finally it appears that we can modify that configuration options with the command

ALTER SYSTEM SET `<<option>>` = <<value>>;

So, it’s why I think some documentation would be needed.

Cheers

4

Apologies to reopen this thread - is it possible to run this on the community version? Trying to complete the dremio university lesson “Handling Nested Fields” - but opening the lessons restaurant_reviews.parquet file returns this response.
List field ‘friends’ exceeded the maximum number of elements 128.Show more

Attempting to follow the instructions in this page Tuning Parquet file performance | Dremio -
ALTER SYSTEM SET store.parquet.block-size = 1073741824;
My community installation responds :
Failure parsing the query. with a red underline on the whitespace between set and `store

Is there another method for applying this setting at all?

5

Apologies to reopen this thread - is it possible to run this on the community version? Trying to complete the dremio university lesson “Handling Nested Fields” - but opening the lessons restaurant_reviews.parquet file returns this response.
List field ‘friends’ exceeded the maximum number of elements 128.Show more

Attempting to follow the instructions in this page Tuning Parquet file performance | Dremio -

finally got the syntax to work - ALTER SYSTEM SET “store.parquet.block-size”

The lesson parquet file is only 200K - have reset this feature to 8GB and still is not happy

Any ideas please?
List field ‘friends’ exceeded the maximum number of elements 128.Show more