Dremio on ADLS Gen2


#1

Does Dremio already support Azure Data Lake Storage Gen 2?

If Yes: Does Dremio benefit from the ADLS Gen2 improvements in regards of performance, e.g. on scans over many small (parquet) files?

If No: Is there an idea when this will be available? More a month, a quarter or a year from now?


#2

This is not supported today mainly because Gen2 is still in preview mode and not GA. Once Gen2 becomes GA, we will look into officially supporting it soon after.


#3

Looks like ADL Gen2 is available

ADLS Gen2 Now Available!
Hello – Thank you for your interest in testing Azure Data Lake Storage Gen2. ADLS Gen2 is a highly scalable and cost-effective data lake solution for big data analytics. It combines the power of a high-performance file system with massive scale and economy to help you speed your time to insight. ADLS Gen2 extends Azure Blob Storage capabilities and is optimized for analytics workloads. You can store data once and access it via exiting Blob Storage and HDFS-compliant file system interfaces with no programming changes or data copying. Data Lake Storage Gen2 is the most comprehensive data lake available – and now it’s available to you in our full public preview.


#4

Yes, it looks like it’s now in “public preview” (not to be confused with General Availability) so it’s fully open to all customers of Azure vs. “limited public preview” which was introduced back in June.


#5

FYI: ADLS Gen 2 GA will 1st of Feb. As it is a ring0 implementation it will be most likely directly available for all regions.

We expect great performance improvements and also tangible cost reduction.


#6

Hi, Thomas!

Thanks for the information! Has the GA date (Feb, 1st) been offiicially announced? The official Gen2 docs on Azure don’t mention it yet.

Thanks, Tim


#7

Inofficially. But from the right guys. You know what I mean…