While I was aware of this 800 column hard limit my understanding was that the SQL query had to refer all of those 800+ columns and only then it threw an error which is highly unlikely as very few monstrous queries may refer 800+ columns.
My altered understanding is that the dataset does not even get loaded and throws the error ‘Number of fields in dataset exceeded the maximum number of fields of 800’, if the SQL query has a table with 800+ columns even if the actual SQL query uses just 1-2 columns from that particular table.
In other words below query will fail in dremio
Select ColumnA from table ABC
if ABC is a table with 800+ columns
I get it that you guys need to load it and you have put a hard limit there but now is there any workaround this issue. Is there a way we can increase the hard limit to say 2000 instead of 800. If not please consider to make this configurable.