Does Dremio already support Azure Data Lake Storage Gen 2?
If Yes: Does Dremio benefit from the ADLS Gen2 improvements in regards of performance, e.g. on scans over many small (parquet) files?
If No: Is there an idea when this will be available? More a month, a quarter or a year from now?
This is not supported today mainly because Gen2 is still in preview mode and not GA. Once Gen2 becomes GA, we will look into officially supporting it soon after.
Looks like ADL Gen2 is available
ADLS Gen2 Now Available!
Hello – Thank you for your interest in testing Azure Data Lake Storage Gen2. ADLS Gen2 is a highly scalable and cost-effective data lake solution for big data analytics. It combines the power of a high-performance file system with massive scale and economy to help you speed your time to insight. ADLS Gen2 extends Azure Blob Storage capabilities and is optimized for analytics workloads. You can store data once and access it via exiting Blob Storage and HDFS-compliant file system interfaces with no programming changes or data copying. Data Lake Storage Gen2 is the most comprehensive data lake available – and now it’s available to you in our full public preview.
Yes, it looks like it’s now in “public preview” (not to be confused with General Availability) so it’s fully open to all customers of Azure vs. “limited public preview” which was introduced back in June.
FYI: ADLS Gen 2 GA will 1st of Feb. As it is a ring0 implementation it will be most likely directly available for all regions.
We expect great performance improvements and also tangible cost reduction.
Hi, Thomas!
Thanks for the information! Has the GA date (Feb, 1st) been offiicially announced? The official Gen2 docs on Azure don’t mention it yet.
Thanks, Tim
Inofficially. But from the right guys. You know what I mean…
any update on when we have adl v2 connector for dremio
No specific timeframe we can share at this point, but we do know it’s important and as such is prioritized high. Thanks.
Looks like Azure Data Lake Gen2 Relase 4 days back
Is there any timeframe to update Dremio ADL connector to ADLGen2 ,i am quite intersting in connecting ADLGen2 with dremio
Yep, we’re aware ADLS Gen2 just went GA and we have a plan to deliver it. No specific ETA right now, but hopefully soon.
Do we have refrance for custome connector
Do you mean a way to build your own? No, not yet, but stay tuned.
Please advise ETA for ADLS Gen2 and HDI 4.0
An ADLS Gen2 support is in plan and we’re working on it. We’re currently targeting a May/June timeframe.