Encounter pyarrow Invalid Parameter Error When Creating View Using Dremio Flight Endpoint

I’m utilizing Dremio Flight Endpoint in Python to execute TPCH SQL queries and create a view in Dremio. However, each attempt results in the following error message:

from dremio.arguments.parse import get_config
from dremio.flight.endpoint import DremioFlightEndpoint
import os

if __name__ == "__main__":
    # Parse the config file.
    args = get_config()
    sql_query = read_sql_from_file(sql_file)
    # Set the query in args
    args['query'] = sql_query
    # print(args)
    # Instantiate DremioFlightEndpoint object with updated query
    dremio_flight_endpoint = DremioFlightEndpoint(args)

     flight_client = dremio_flight_endpoint.connect()
     # print(dremio_flight_endpoint)
     # Get reader
     reader = dremio_flight_endpoint.get_reader(flight_client)

     # Print out the data as a dataframe
     df = reader.read_pandas()
     print(f"Data from {sql_file}:")


create view  "poc".tpch.revenue0 as
                l_suppkey as supplier_no ,
                sum(l_extendedprice * (1 - l_discount)) as total_revenue
                l_shipdate >= date '1995-02-01'
                and l_shipdate < date '1995-02-01' + interval '3' month
        group by
pyarrow.lib.ArrowInvalid: Flight returned invalid argument error, with message: Validation of view sql failed. null

I’ve ensured that the submitted SQL query statements comply with Dremio’s syntax requirements, yet the view creation still fails. I’ve also inspected communication with the Dremio server, but haven’t identified any apparent issues.

UI surface :

It’s worth noting that the same SQL queries executed successfully in the Dremio UI interface, and the view was created without any problems.

I’m reaching out here to see if anyone has encountered a similar issue or if anyone can provide further guidance to resolve this problem. Are there any additional details or solutions regarding this error? Thank you!

@Han-lai What version of Dremio are you using?


The issue I encountered is indeed identical to the error code mentioned in your post. I encountered the problem while using the DremioFlightEndpoint interface. Thank you for bringing this to my attention. Could you please clarify if there are any limitations regarding JDBC or ODBC interfaces, such as any restrictions on creating views when using DremioFlightEndpoint?"

and the same error in odbc like this : Error when create view using odbc Flight - Dremio

This is the ADBC interface I created based on the documentation I referred to. : arrow-flight-client-examples/python at main · dremio-hub/arrow-flight-client-examples (github.com)

This is a known issue with 24.3 that is being addressed in the next version.