Hi Team, I have been playing around with the community edition on Azure. This was setup via the marketplace therefore it’s NOT using AKS.
I wanted to inquire about some of the features that I have found in the docs but can’t see it in my Dremio instance. I can’t tell if these are a limitation of the edition or cloud provider, its unfortunately not clear in the Dremio software docs.
Querying Files and Directories
Docs here indicates there is a feature to turn on partition column inference, but there is no ‘Enable partition column inference’ check box when I look at the settings of the source. There is the ability to add a property which I am not sure is the same thing? Or if there is a specific property to add.
This is advertised here as the ability to auto stop executors on Azure. But if you navigate to elastic engines in Dremio it seems to only allow you to setup what seems to be a Hadoop engine. You have no control through Dremio of the initial VM cluster setup. Is this an enterprise feature only and therefore would the approach be to manually control the scaling of your VM cluster through Azure if you using the community edition?
Querying Files and Directories - I have checked internally and will get back to you
Elastic Engines -Even in Enterprise, engine creation via UI is only for Yarn and AWSE deployments, for K8’s we have to do it via configs
Thanks @balaji.ramaswamy we are evaluating Dremio for our business, and we are predominantly on an Azure stack hence the use of Dremio Software at this stage. Do you have any idea when Dremio Cloud will be coming to Azure I know the Pricing Page says ‘Coming Soon’.
@nikhil.makan What is the source you are trying to use the partition column inference feature on? Will get back on the Dremio cloud on Azure ETA
Source is Azure Data Lake Gen 2.
@balaji.ramaswamy I deployed Dremio this morning on Azure AKS, and the advanced option for the source to enable partition column inference is available now.
I am not sure why it was not available when deploying via the Azure market place, perhaps a different version gets installed using ARM? For AKS I am using the latest images available i.e. 22.1.
@nikhil.makan Sounds like it,
@balaji.ramaswamy is there any reason why I can’t see the data lineage button for a virtual dataset.
I can see this on Dremio University
But I only see this on my end
Software docs seems to say it should be there.
I can see a version of this graph in the jobs. But that an odd place to go look up the lineage of a virtual dataset.
@nikhil.makan Could you confirm you’re using dremio-ee? It won’t be there in dremio-oss if that’s what you’re using…
@lenoyjacob thanks, that will be it. I assume ee means enterprise edition? I am using the community edition so the dremio-oss image. Bit of a weird feature to keep only in enterprise considering you can see a simplified version of it in the jobs section in CE, is that intended or still coming to CE?