Rest api on sql and job

1

How can I pull back more than 500 rows on the rest api post and get? I did:

POST /api/v3/sql

with a sql that returns over 10000 rows, but after getting that job id and do a get

GET /api/v3/job/{id}/results?offset={offset}&limit={limit}

I only have a result set of 500

2

The limit can’t be set higher then 500. However, you can repeatedly call the results endpoint by increasing the offset by the limit for each call until you’ve read all the data.

3

We on purpose limit to 500 at a time as asking for a lot of rows at once can lead to performance issues. If you need to fetch large result sets we recommend using jdbc/ODBC as those are much more performant for that.

4

If we want to get all rows from the data set using below,
GET /api/v3/job/{id}/results?offset={offset}&limit={limit}

This API supports to download only 500 records.
https://docs.dremio.com/software/rest-api/jobs/get-job/#syntax

It would be best if you used the pagination to fetch in all rows
Example:
To get the first 500 rows,

curl 'http://localhost:9047/api/v3/job/1d74d555-f5eb-eb33-ec67-a0c06d885000/results?offset=0&limit=500' \

To get the next set of 500 rows,

curl 'http://localhost:9047/api/v3/job/1d74d555-f5eb-eb33-ec67-a0c06d885000/results?offset=500&limit=500' \

likewise, you need to increase offset value.

offset=1000&limit=500
offset=1500&limit=500
..
..

curl 'http://localhost:9047/api/v3/job/1d74d555-f5eb-eb33-ec67-a0c06d885000/results?offset={StartOffset}&limit=500' \

Hope the above steps are helpful.