Hello Everyone,
Hope you are good and doing great. I am trying to connect Dremio cluster with Azure Data Lake Storage Gen1. Which is default source from Dremio Cluster. However, when I Provide with the required details I am getting the error saying, “Error starting new source: Test java.lang.NullPointerException: null”.
However, I have attached Screenshots of both, Source and Error logs I am getting. Just to make it more clear, I am using all valid credentials and trying to access Azure using localhost machine. I am running docker under debug mode, due to which, I figured out the real error is with Authentication. i.e. It is not able to authenticate me as it says, Error Fetching Access token. which is highlighted in screenshot of logs. I tried same credentials with python code and it worked fine. Which states that credentials are correct. But still Dremio is not able to fetch the tokens. Can you please me solve this issue? Any kind of help is always welcomed and Thank you in advance.
@ben,
Thank you for extending your helping hand. Just to answer your queries, I have followed all the steps from the docs you have mentioned above. For login.microsoft.com I am having token endpoint URL for OAuth2.0. I am using same URL programatically for the accessing by passing that URL under OAuth 2.0 Token Endpoint text field of ADLS Gen1 new source page.
However, as mentioned above, all these links are working, as I have written Python script for the same purpose and it worked fine. Also, I hit the URL for OAuth 2.0 Token endpoint with Curl command and it was working either.
@Nahush, can you send me scrubbed images of your ADLS gen 1 configuration window and a text file with the error message you are seeing? email = ben@dremio.com
Hi @ben,
I have mailed you the required images and text file of error. However, I am also uploading it here. Zip file contains both the things you mentioned. Please let me know if you want anything more.
@balaji.ramaswamy I have gone through the link. I think the issue is with our Azure Directory configurations. Will post on the forum once I solve the issue.
Thank you.
@balaji.ramaswamy
I am able to access the data lake storage using REST API.
But not able to access it using Dremio or using a simple Java class using azure-data-lake-store-sdk.
The exception thrown is the same in both case 2020-07-22 10:13:58,092 [start-adls] WARN c.d.e.catalog.ManagedStoragePlugin - Error starting new source: adls java.lang.Exception: Unavailable: Error enumerating directory / Error fetching access token Operation null failed with exception java.io.IOException : Server returned HTTP response code: 400 for URL: https://login.microsoftonline.com/<tenand-id>/oauth2/v2.0/token Last encountered exception thrown after 5 tries. [java.io.IOException,java.io.IOException,java.io.IOException,java.io.IOException,java.io.IOException] [ServerRequestId:null] at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:501) at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:563) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-07-22 10:13:58,095 [start-adls] INFO c.d.s.s.LocalSchedulerService - Cancelling task metadata-refresh-wakeup-adls 172.17.0.1 - - [22/Jul/2020:10:13:58 +0000] "PUT /apiv2/source/adls/?nocache=1595412735267 HTTP/1.1" 400 555 "http://localhost:9047/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36"
