Apache Arrow flight C# implementation code to connect to Dremio

1

We are trying to connect to dremio service using Apache Arrow Flight C# client. We couldn’t find any working code sample in the following links:

We tried the following c# code to connect to dremio from arrow flight client which dint worked for us:

String encoded = System.Convert.ToBase64String(System.Text.Encoding.GetEncoding(“ISO-8859-1”).GetBytes(“user_name” + “:” + “password”));
var credentials = CallCredentials.FromInterceptor((context, metadata) =>
{
metadata.Add(“Authorization”, "Basic " + encoded);
return Task.CompletedTask;
});
GrpcChannel channel = GrpcChannel.ForAddress(“dremio_url”, new GrpcChannelOptions
{
Credentials = ChannelCredentials.Create(new SslCredentials(), credentials)
});
FlightClient client = new FlightClient(channel);
client.ListActions();
while (await actions.ResponseStream.MoveNext(default))
{
Console.WriteLine(actions.ResponseStream.Current);
}

We are getting the following exception while running the above code:

[2021-10-15T06:14:56.704Z] System.Private.CoreLib: Exception while executing function: ExtractData. Grpc.Net.Client: Status(StatusCode=“Internal”, Detail=“Error starting gRPC call. HttpRequestException: The SSL connection could not be established, see inner exception. IOException: The handshake failed due to an unexpected packet format.”, DebugException="System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. [2021-10-15T06:14:56.721Z] —> System.IO.IOException: The handshake failed due to an unexpected packet format. [2021-10-15T06:14:56.734Z] at System.Net.Security.SslStream.StartReadFrame(Byte buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) [2021-10-15T06:14:56.772Z] at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)

Any help to resolve the issue in this code sample is greatly appreciated.

2

@Gandhirajan Have asked internally for someone from engineering to respond, it seems this is the same as Basic Authentication via C# - #5 by balaji.ramaswamy

3

Thanks for your response. It would be great if get some pointers to sort this out.

4

Hi @Gandhirajan ,

First I would suggest validating your environment using the Java or Python applications.
What is the URL you are using? Are you using the correct port for Flight? The default is 32010, which is different from the usual HTTP port (9047) and ODBC/JDBC port (31010).

The code looks be setting the basic header correctly. However instead of sending basic credentials every request, it’s recommended that you use interceptors to read the bearer token response header after the first request, and use it on all subsequent requests instead of the username/password.

5

Hi @jduong The code works fine with Java and Python client. And have tried with 32010 as well as 9047 with no luck. We are not even able to establish first successful connection with the dremio service we are trying to access. We are able to access the dremio service URL from browser as well using the respective credentials.

6

Hi @Gandhirajan
Can you confirm what URL you are using in the call to GrpcChannel.ForAddress()?

7

Hi @jduong in our case dremio service is hosted in Azure. Have tried the URL “http://<host_ip>:9047/” as well as “http://<host_ip>:30210/”.

8

9047 definitely won’t work. That’s for the REST interface.

I assume in the second URL, you mean you used http://<host_ip>:32010 rather than 30210. You may need to use https://<host_ip>:32010 instead to use TLS.

9

HI @jduong Have tried https URL with SSL credentials as well http URL with insecure credentials option with no luck.

10

There is a working “HTTP” example here:

This uses the httpClient in the GrpcChannelOptions to add the header.

            var httpClient = new HttpClient();
            httpClient.DefaultRequestHeaders.Add("Authorization", "Basic " + encoded);

            var address = $"http://{host}:{port}";
            var channel = GrpcChannel.ForAddress(address, new GrpcChannelOptions
            {
                HttpClient = httpClient
            });

Were you able to get the HTTPS to work?