Arrow Flight query dremio size limit

Dremio University
1

Dears
I add arrow flight to query dremio.
When query sql is ‘select field_a, field_b from table_a limit 5000’. rowCount is 3900. It is less than 5000.
When query sql is ‘select * from table_a limit 5000’. rowCount is 560. It is less than 3900 and 5000.
All in all. I think there exist size limit. When query result exceed the size. The exceeded part can’t displayed. Anyone else could help me. Thank you in advance.

Below info is my code.

FlightClient client = FlightClient.builder(allocator, Location.forGrpcInsecure(HOST, PORT)).build();
client.authenticateBasic(USERNAME, PASSWORD);
String query = “select field_a, field_b from table_a limit 5000”;
final FlightInfo flightInfo = client.getInfo(FlightDescriptor.command(query.getBytes(StandardCharsets.UTF_8)));
final FlightStream stream = client.getStream(flightInfo.getEndpoints().get(0).getTicket());
VectorSchemaRoot root = stream.getRoot();
final long rowCount = root.getRowCount();

2

@mccann.ren

You should have profiles created on the Dremio jobs UI, can you please send them for both the scenarios?

3

@balaji.ramaswamy
Thanks a lot. It seems that there is no appropriate profiles.
As I want to query 5000 data. There are only 3986 data according to allow flight. But There are 5000 data according to JDBC.
It’s obvious that allow flight query data is less than that we want to query. Especially when we query huge data.