Hi ,
I have updated dremio from 4.1 to 4.6 now,as i see some of the issues with
I have deployed dremio on aks with 1 co-ordinator ,1 executor on seprate node which having 16 cpu 128 GB ram,but still some of the Query took 25 min to execute ,i also see cpu usage normal pleaase find job profile as well as cpu details
here is dremio env file:
DREMIO_MAX_HEAP_MEMORY_SIZE_MB=13312
DREMIO_MAX_DIRECT_MEMORY_SIZE_MB=50384
dd34d21a-240a-4c35-b67a-232bfc2d6cb3.zip (195.6 KB)
i also able to reproduced issue when Query having more joins like this
From my experience, this typically happens when query planner thinks that reflection exists. At runtime (after query planning), dremio follows the query plan to get the reflection, and if can’t obtain the reflection (either due to reflection parquet file corruption, metadata corruption, or illogical where-clause), it resorts to running the whole query without reflections, that is, it defaults to original data file(s) which contributed to the reflection. This takes a long time. What is misleading is that it still shows Acceleration used icon. I think this issue will go away if you rebuild all of your reflections. Let the community know.
The profile shows that most of the time is being spent in reading data from JDBC ([tourmaline].[dbo].[FDCTurbineRecords]).
It looks like the push down does not have a FILTER, Are you joining HDFS with RDBMS?
SELECT [FDCTurbineRecords].[ProductionDay], [FDCTurbineRecords].[EntityId] COLLATE LATIN1_GENERAL_BIN2 AS [EntityId], [FDCTurbineRecords].[EntityType] COLLATE LATIN1_GENERAL_BIN2 AS [EntityType], [FDCTurbineRecords].[EntityName] COLLATE LATIN1_GENERAL_BIN2 AS [EntityName], [FDCTurbineRecords].[LocationType] COLLATE LATIN1_GENERAL_BIN2 AS [LocationType], [FDCTurbineRecords].[LocationId] COLLATE LATIN1_GENERAL_BIN2 AS [LocationId], [FDCTurbineRecords].[LocationName] COLLATE LATIN1_GENERAL_BIN2 AS [LocationName], [FDCTurbineRecords].[District] COLLATE LATIN1_GENERAL_BIN2 AS [District], [FDCTurbineRecords].[DistrictId] COLLATE LATIN1_GENERAL_BIN2 AS [DistrictId], [FDCTurbineRecords].[Area] COLLATE LATIN1_GENERAL_BIN2 AS [Area], [FDCTurbineRecords].[AreaId] COLLATE LATIN1_GENERAL_BIN2 AS [AreaId], [FDCTurbineRecords].[Field] COLLATE LATIN1_GENERAL_BIN2 AS [Field], [FDCTurbineRecords].[FieldId] COLLATE LATIN1_GENERAL_BIN2 AS [FieldId], [FDCTurbineRecords].[ProductionDate], [FDCTurbineRecords].[DownStreamLocationType] COLLATE LATIN1_GENERAL_BIN2 AS [DownStreamLocationType], [FDCTurbineRecords].[DownStreamLocationId] COLLATE LATIN1_GENERAL_BIN2 AS [DownStreamLocationId], [FDCTurbineRecords].[DownStreamLocationName] COLLATE LATIN1_GENERAL_BIN2 AS [DownStreamLocationName], [FDCTurbineRecords].[CreatedBy] COLLATE LATIN1_GENERAL_BIN2 AS [CreatedBy], [FDCTurbineRecords].[ModifiedBy] COLLATE LATIN1_GENERAL_BIN2 AS [ModifiedBy], [FDCTurbineRecords].[FacilityCode] COLLATE LATIN1_GENERAL_BIN2 AS [FacilityCode], [FDCTurbineRecords].[TenantName] COLLATE LATIN1_GENERAL_BIN2 AS [TenantName], [FDCTurbineRecords].[TenantId] COLLATE LATIN1_GENERAL_BIN2 AS [TenantId], [FDCTurbineRecords].[CorrectionFactor], [FDCTurbineRecords].[MeterPurpose] COLLATE LATIN1_GENERAL_BIN2 AS [MeterPurpose], [FDCTurbineRecords].[CreatedDateTime], [FDCTurbineRecords].[ModifiedDateTime], [FDCTurbineRecords].[HcType] COLLATE LATIN1_GENERAL_BIN2 AS [HcType], [FDCTurbineRecords].[LiquidUnitsOfMeasurements] COLLATE LATIN1_GENERAL_BIN2 AS [LiquidUnitsOfMeasurements], [FDCTurbineRecords].[Tag] COLLATE LATIN1_GENERAL_BIN2 AS [Tag], [FDCTurbineRecords].[TodayReading], [FDCTurbineRecords].[YesterdayReading], [FDCTurbineRecords].[TodayVolume], [FDCTurbineRecords].[YesterdayVolume], [FDCTurbineRecords].[Change], [FDCTurbineRecords].[BSW], [FDCTurbineRecords].[HcVolume], [FDCTurbineRecords].[WaterVolume], [FDCTurbineRecords].[SandVol], [FDCTurbineRecords].[UpStreamLocationName] COLLATE LATIN1_GENERAL_BIN2 AS [UpStreamLocationName], [FDCTurbineRecords].[UpStreamLocationId] COLLATE LATIN1_GENERAL_BIN2 AS [UpStreamLocationId], [FDCTurbineRecords].[UpStreamLocationType] COLLATE LATIN1_GENERAL_BIN2 AS [UpStreamLocationType], [FDCTurbineRecords].[Tanks] COLLATE LATIN1_GENERAL_BIN2 AS [Tanks], [FDCTurbineRecords].[OriginalVolume], [FDCTurbineRecords].[Reason] COLLATE LATIN1_GENERAL_BIN2 AS [Reason], [FDCTurbineRecords].[ValidFrom], [FDCTurbineRecords].[ValidTo], [FDCTurbineRecords].[NGLAnalysis] COLLATE LATIN1_GENERAL_BIN2 AS [NGLAnalysis], [FDCTurbineRecords].[CondyAnalysis] COLLATE LATIN1_GENERAL_BIN2 AS [CondyAnalysis], [FDCTurbineRecords].[DownStreamMeterLink] COLLATE LATIN1_GENERAL_BIN2 AS [DownStreamMeterLink], [FDCTurbineRecords].[DownStreamMeterPurpose] COLLATE LATIN1_GENERAL_BIN2 AS [DownStreamMeterPurpose], [FDCTurbineRecords].[NGLType] COLLATE LATIN1_GENERAL_BIN2 AS [NGLType], [FDCTurbineRecords].[IsDeleted], [FDCTurbineRecords].[ProducingTo], [FDCTurbineRecords].[ReadingType], [FDCTurbineRecords].[DispositionPt], [FDCTurbineRecords].[DownStreamMeterLinkType] COLLATE LATIN1_GENERAL_BIN2 AS [DownStreamMeterLinkType], [FDCTurbineRecords].[DownStreamFacilityCode] COLLATE LATIN1_GENERAL_BIN2 AS [DownStreamFacilityCode]
FROM [tourmaline].[dbo].[FDCTurbineRecords@balaji.ramaswamy, yes i have many join query on hdfs with sql rdbms.
is any drawback we cant use hdfs with RDBMS,please advice alternative
I need to check if the join caused the FILTER not to be pushed down to the JDBC SCAN
Will check and get back to you
Thanks
Bali