My team and I would sometimes encounter this error message when posting queries using the REST API:
We guessed that the cause of this error may be due to the fact that the physical datasets that we are trying to query have nested json fields (Thus, multiple query attempts might be needed for schema learning).
We spotted that, at times, queries that require schema learning would be (automatically) re-attempted by Dremio (as seen in the screenshot below), while at other times, no (automatic) re-attempts will be performed and the error will immediately be thrown out (as seen in the previous screenshot), which we find this behaviour quite peculiar.
We are currently using Dremio version 4.7.2 (Community Edition). We have read in the release notes for Dremio version 4.5.0 that this issue was resolved by only pushing down projections that are simple column references.
Hence, we would like to check if this issue we faced is a bug and if there is any way to allow limited auto re-attempts of the query for schema learning before Dremio would throw back the error message.
Here are the job profiles of the two screenshots to aid in the debugging of this issue:
Job profile of the failed query:
389394d2-c16b-4219-b6da-afb3ac59ce26.zip (15.4 KB)
Job profile of the successful query after automatic schema learning re-attempt(s):
9aec65a0-c715-423f-8e04-7e214253537b.zip (31.8 KB)