Bug when using a proxy with async S3

1

There’s a bug in these classes com.dremio.plugins.s3.store.ApacheHttpConnectionUtil and com.dremio.plugins.util.awsauth.ApacheHttpConnectionUtil (which contains almost the same code).

It’s not possible to use: URI.create(host + ":" + port) without a scheme.

For example the code:

URI uri = URI.create("myhost:3128");
System.out.println("URI=" + uri);
System.out.println("Host=" + uri.getHost());
System.out.println("Port=" + uri.getPort());

output:

URI=myhost:3128
Host=null
Port=-1

When used with a scheme:

URI uri = URI.create("http://myhost:3128");
System.out.println("URI=" + uri);
System.out.println("Host=" + uri.getHost());
System.out.println("Port=" + uri.getPort());

output:

URI=http://myhost:3128
Host=myhost
Port=3128

So the URI created in ApacheHttpConnectionUtil classes is later used by ProxyConfiguration in its constructor (called by ProxyConfiguration.DefaultClientProxyConfigurationBuilder#build method) in resolveHost and resolvePort functions. If the endpoint (the URI created) is defined, it uses endpoint.getHost() and endpoint.getPort() which work exactly like the proxy config isn’t set by returning null and -1!

Another issue (I could be wrong!)

In both classes ApacheHttpConnectionUtil you are using conf.getBoolean(Constants.SECURE_CONNECTIONS, Constants.DEFAULT_SECURE_CONNECTIONS) to check if the proxy must use a secure connexion or not.
This is not correct, this setting is for the S3 endpoint, not for the proxy.

So in my fix below I used a new property fs.s3a.proxy.secure (default false because it was my usecase) to know if the scheme http or https.

Proposed fixes (I’m using)

My fix for com.dremio.plugins.s3.store.ApacheHttpConnectionUtil, in function initProxySupport:

    final String proxyHost = conf.getTrimmed(Constants.PROXY_HOST, "");
    int proxyPort = conf.getInt(Constants.PROXY_PORT, -1);
    final String scheme;
    if (proxyPort < 0) {
      // WARN NO!!! This is for the s3, not the proxy!
      if (conf.getBoolean(Constants.SECURE_CONNECTIONS, Constants.DEFAULT_SECURE_CONNECTIONS)) {
        proxyPort = 443;
        scheme = "https://";
      } else {
        proxyPort = 80;
        scheme = "http://";
      }
    } else if (conf.getBoolean("fs.s3a.proxy.secure", false)) {
      // TODO Right now we have no way (except this new parameter) to know
      //      if the proxy use a secure connection or not!
      scheme = "https://";
    } else {
      scheme = "http://";
    }

    builder.endpoint(URI.create(scheme + proxyHost + ":" + proxyPort));

My fix for com.dremio.plugins.util.awsauth.ApacheHttpConnectionUtil in function initProxySupport:

    final String proxyHost = conf.getTrimmed(Constants.PROXY_HOST, "");
    int proxyPort = conf.getInt(Constants.PROXY_PORT, -1);
    if (!proxyHost.isEmpty()) {
      final String scheme;
      if (proxyPort < 0) {
        // WARN NO!!! This is for the s3, not the proxy!
        if (conf.getBoolean(Constants.SECURE_CONNECTIONS, Constants.DEFAULT_SECURE_CONNECTIONS)) {
          proxyPort = 443;
          scheme = "https://";
        } else {
          proxyPort = 80;
          scheme = "http://";
        }
      } else if (conf.getBoolean("fs.s3a.proxy.secure", false)) {
        // TODO Right now we have no way (except this new parameter) to know
        //      if the proxy use a secure connection or not!
        scheme = "https://";
      } else {
        scheme = "http://";
      }

      builder.endpoint(URI.create(scheme + proxyHost + ":" + proxyPort));

      ...

best regards,
fred