Thanks for your interest on this issue.
I am not able to recover the post
It would be good to know why that is. I spent hours typing up the original posts. The message from Akismet suggested that the posts were simply “temporarily hidden”. Is Akismet hiding things from the human staff at dremio? Is AI taking over?!
but I read through your question
I’m confused. So is the situation that you can see the original post contents but can’t unhide it? If so, it would be good if you could send me the original post contents so that I can make it part of a new post. Hours of my life went into those posts. It would be good if I didn’t have to do it all over again.
In versions 18.0 and above, Dremio supports unlimited splits for PARQUET/ORC and AVRO formats
Again, thanks for your help on this issue. We experienced this issue with the latest 21.2.0 release. The data seems to be in csv format that is then gzipped up. I go into more details in my original post but we are trying to query aws load balancer access logs in s3.
It would be good if these limits are either configurable or can simply be removed. We don’t mind if unreasonable queries took unreasonable amounts of time to run. If that’s the case, we would simply cancel those queries and find optimisations ourselves through the query conditions.
We don’t even mind if unreasonable queries made the server fall over. That would at least give us an opportunity to resolve the issue ourselves by adding more resources.
What we do have issues with is dremio precluding us of having any access to our data after making claims of being able to handle data lakes with petabytes of data. We also have issues with dremio suggesting that we change our data structure after previously expressing an understanding and respect for divergent data formats of your customers.
In this case, it’s not that we are resisting changes to our data structure, it’s just that we have no control over the data structure that aws is producing for their load balancer access logs.
All this discussion might be very confusing for anyone reading this thread without the context from the original post. So if you have access to the original post contents, please send it to me somehow and I will repost the original post contents and we can continue the discussion there.
Thanks again for your interest.