Dremio Rest API not retrieving 500 records with offset and limit


#1

We are using Dremio server 3.1.6 on Linux. Dremio Rest API not retrieving 500 records with offset and limit. Is this 500 configurable?


#2

The 500 limit per call is not configurable. Are you saying that using limit/offset to retrieve job results is not working correctly?

REST is not really designed to get large amounts of data which is why we have to limit each call.


#3

Thanks for the reply Doron.

One observation when using the API ’ select distinct col from table ***" , dremio job shows 2000+ records
Where as API retrieves first 500 and later calls with offset and limit randomly gets less than 500. So instead of 2000+ we get 650+ < 800.

it’s only to do with that distinct query. The other queries with 20000 records all work fine.

Work around is create a data set and use , data set also we have to use limit in where clause.
or if the original select query has limit, then it works fine.

It has some thing to do with distinct query I guess. Any inputs that you can suggest ?