Bug when using a proxy with async S3

1

There’s a bug in these classes com.dremio.plugins.s3.store.ApacheHttpConnectionUtil and com.dremio.plugins.util.awsauth.ApacheHttpConnectionUtil (which contains almost the same code).

It’s not possible to use: URI.create(host + ":" + port) without a scheme.

For example the code:

URI uri = URI.create("myhost:3128");
System.out.println("URI=" + uri);
System.out.println("Host=" + uri.getHost());
System.out.println("Port=" + uri.getPort());

output:

URI=myhost:3128
Host=null
Port=-1

When used with a scheme:

URI uri = URI.create("http://myhost:3128");
System.out.println("URI=" + uri);
System.out.println("Host=" + uri.getHost());
System.out.println("Port=" + uri.getPort());

output:

URI=http://myhost:3128
Host=myhost
Port=3128

So the URI created in ApacheHttpConnectionUtil classes is later used by ProxyConfiguration in its constructor (called by ProxyConfiguration.DefaultClientProxyConfigurationBuilder#build method) in resolveHost and resolvePort functions. If the endpoint (the URI created) is defined, it uses endpoint.getHost() and endpoint.getPort() which work exactly like the proxy config isn’t set by returning null and -1!

Another issue (I could be wrong!)

In both classes ApacheHttpConnectionUtil you are using conf.getBoolean(Constants.SECURE_CONNECTIONS, Constants.DEFAULT_SECURE_CONNECTIONS) to check if the proxy must use a secure connexion or not.
This is not correct, this setting is for the S3 endpoint, not for the proxy.

So in my fix below I used a new property fs.s3a.proxy.secure (default false because it was my usecase) to know if the scheme http or https.

Proposed fixes (I’m using)

My fix for com.dremio.plugins.s3.store.ApacheHttpConnectionUtil, in function initProxySupport:

    final String proxyHost = conf.getTrimmed(Constants.PROXY_HOST, "");
    int proxyPort = conf.getInt(Constants.PROXY_PORT, -1);
    final String scheme;
    if (proxyPort < 0) {
      // WARN NO!!! This is for the s3, not the proxy!
      if (conf.getBoolean(Constants.SECURE_CONNECTIONS, Constants.DEFAULT_SECURE_CONNECTIONS)) {
        proxyPort = 443;
        scheme = "https://";
      } else {
        proxyPort = 80;
        scheme = "http://";
      }
    } else if (conf.getBoolean("fs.s3a.proxy.secure", false)) {
      // TODO Right now we have no way (except this new parameter) to know
      //      if the proxy use a secure connection or not!
      scheme = "https://";
    } else {
      scheme = "http://";
    }

    builder.endpoint(URI.create(scheme + proxyHost + ":" + proxyPort));

My fix for com.dremio.plugins.util.awsauth.ApacheHttpConnectionUtil in function initProxySupport:

    final String proxyHost = conf.getTrimmed(Constants.PROXY_HOST, "");
    int proxyPort = conf.getInt(Constants.PROXY_PORT, -1);
    if (!proxyHost.isEmpty()) {
      final String scheme;
      if (proxyPort < 0) {
        // WARN NO!!! This is for the s3, not the proxy!
        if (conf.getBoolean(Constants.SECURE_CONNECTIONS, Constants.DEFAULT_SECURE_CONNECTIONS)) {
          proxyPort = 443;
          scheme = "https://";
        } else {
          proxyPort = 80;
          scheme = "http://";
        }
      } else if (conf.getBoolean("fs.s3a.proxy.secure", false)) {
        // TODO Right now we have no way (except this new parameter) to know
        //      if the proxy use a secure connection or not!
        scheme = "https://";
      } else {
        scheme = "http://";
      }

      builder.endpoint(URI.create(scheme + proxyHost + ":" + proxyPort));

      ...

best regards,
fred

2

Hello @fred ,

Thank you for reporting this!

Would you mind confirming which version of Dremio you are using?

Best,
Francisco

3

This issue is there since 18.0.0 (when these classes were inserted or refactored), up to the latest master.

So it’s on all versions!

Hope that help :slight_smile:

image

4

Thanks, @fred!

We created a backlog ticket for this. We will update the thread once we have a clearer idea of when a fix will come out.

Best,
Francisco

5

Hello @fred,

Thank you once again for reporting this.
We’ve reviewed this internally and can confirm that this issue has been fixed on our side. The correction is expected to be included starting with Dremio version 26.1.3.

Best regards,
Pedro

1 Like