REST API SQL BUG with "offset"

1

I ran a SQL statement using the REST API and my Dremio server says it found 1567 rows, but the REST API only returned back 1058

I ran the following debug statements which should show overlapping results, but that isn’t the case

>>> r = requests.get(dremio_server + "/api/v3/job/" + job_id + "/results?limit=500&offset=1000", headers=headers)
>>> len(r.json()['rows'])
58
>>> r = requests.get(dremio_server + "/api/v3/job/" + job_id + "/results?limit=500&offset=1056", headers=headers)
>>> len(r.json()['rows'])
2
>>> r = requests.get(dremio_server + "/api/v3/job/" + job_id + "/results?limit=500&offset=1057", headers=headers)
>>> len(r.json()['rows'])
1
>>> r = requests.get(dremio_server + "/api/v3/job/" + job_id + "/results?limit=500&offset=1058", headers=headers)
>>>
>>> len(r.json()['rows'])
500
>>> r = requests.get(dremio_server + "/api/v3/job/" + job_id + "/results?limit=500&offset=1500", headers=headers)
>>> len(r.json()['rows'])
67

The last statement shows there are 1567 rows, but a bunch of rows got lost somehow, but can still be picked up using different offsets.

2

Looks like this bug has already been reported in Pagination query does not return as requested records

I tried the suggested fix which was to delete the metadata and refresh it, but it didn’t help…

Basically the Dremio UI returns back 1567 rows, but the REST API SQL calls returns back 1125.

Seems there is some bug that is truncating the rows between 1000 and 1500.

3

The pagination issue was fixed in Dremio 4.5.0.