Using applications with similar API's

Is it possible to use an application such as minio (which contains the same APIs as S3) with dremio?
I am having trouble setting up my minio bucket in dremio using the S3 source type even though they are virtually the same.

What kind(s) of errors do you get?

“Failed to create workspace for buckets owned by the account” I am assuming I should be able to use my minio credentials, but I also tried without using any credentials because my minio bucket is public

Could you share/look at your server.log (should be under dremio/log directory) for more logging about the error? I would also assume you may need “minio” jar file(s) to be included on the classpath

org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.23.2.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.23.2.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.23.2.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.23.2.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.23.2.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.23.2.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.23.2.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.23.2.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.23.2.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.23.2.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.23.2.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.23.2.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.15.v20160210.jar:9.2.15.v20160210]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential. (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: C405EF1B050D6A90)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4162) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:914) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.amazonaws.services.s3.AmazonS3Client.listBuckets(AmazonS3Client.java:920) ~[aws-java-sdk-s3-1.11.156.jar:na]
at com.dremio.plugins.s3.store.S3FileSystem.initialize(S3FileSystem.java:99) ~[dremio-s3-plugin-1.0.8-201707190805180330-27f36e1.jar:1.0.8-201707190805180330-27f36e1]

Tried not using credentials because it was public and tred throwing the uri in

You can try to list your bucket in “External buckets”, but I don’t think full URI will work, just the name

From an exception I can see header is malformed which is probably expected. I am not really familiar with minio.
I’ll try to research a bit and get back to you whether quick around is feasible.

1 Like

Is your minio bucket on AWS or with some other cloud storage provider?

FYI, to clear up some obvious confusion in this thread:

@William_Thornton: it looks like they’re hard-coding the AWS URI and thus expecting an “actual” S3 bucket. Nothing obvious I can see to point it to an alternate location to look for S3 (could be hacked up with some local networking rules, but the utility of that is up to you).

@yufeldman Minio is a distributed storage application that acts as a drop-in replacement for AWS S3 (all of S3) rather than a particular bucket or account. It looks like Dremio really wants to connect to an actual AWS URI, which is the cause of the issue. If Dremio allowed one to specify the URI of a public bucket (even if it thinks it can already calculate that because it follows a predictable pattern) everything should “just work”; Minio is fully compatible with S3 at an API and protocol level.

Hope that helps!

@jeffknupp
Thank you for the info on minio. I also did some reading :).
Yes - at this point it tries to connect to s3a://bucket/ where fs.s3a.endpoint is set to bucket’s region endpoint

@jeffknupp I’m slightly confused, so there is a way to set up minio via dremio? What would you suggest doing to my network rules to allow this?
Thanks for the help @yufeldman @jeffknupp

Well, if you changed your /etc/hosts file to redirect the s3 HTTP endpoint to your minio HTTP endpoint, hypothetically it should work, but I don’t want to send you down a rabbit hole.

Basically, if you can trick dremio into thinking it’s hitting the “real” S3 but instead have it redirected to your minio installation, everything _should_work.

Can you help us understand by when (date?), the ability to provide connection properties like fs.s3a.endpoint will be supported?

I would like to use the S3 connector to connect to Oracle Object Storage.
https://docs.cloud.oracle.com/iaas/Content/Object/Tasks/s3compatibleapi.htm

Supporting S3 compatible products is something on our roadmap, but timing is still TBD.