Dremio AutoScale Not Scaling

1

Hello
i have enabled Dremio 24.3.7
and followed steps to enable Autoscale
Have autoscale enabled at AKS level
below is what i have setup in helm,
the default node count after helm upgrade is at 7 in Target Nodes and Ready Nodes

Its been 24 hrs and there is no autoscaling activity , node count didnt go to 1. so wondering why its not happening

 nodeLifecycleService:
    enabled: true
    maxEngines: 10
    metricsPort: 9010
    terminationGracePeriodSeconds: 60
    scalingMetrics:
      default:
        enabled: true
        cpuAverageUtilization: 50
        memoryAverageUtilization: 50
      userDefinedMetrics:
        - pods:
          metric:
            name: threads_waiting_count
          target:
            averageValue: "20"
            type: AverageValue
          type: Pods
    scalingBehavior:
      scaleDown:
        defaultPolicy:
          enabled: true
          value: 1
          periodSeconds: 30
        userDefinedPolicies:
          - type: Pods
            value: 2
            periodSeconds: 10
      scaleUp:
        defaultPolicy:
          enabled: true
          value: 900
          periodSeconds: 60
        userDefinedPolicies:
          - type: Percent
            value: 30
            periodSeconds: 30
2

@ptatl

Can they verify whether Dremio’s metrics are being published correctly by running the following?
kubectl exec -it -n <DREMIO_NAMESPACE> dremio-executor-0 -c dremio-executor -- curl localhost:9010/metrics Also, it would be interesting to see the output of the horizontalPodAutoscaler and podMonitor that was defined by their helm charts:
kubectl describe -n <DREMIO_NAMESPACE> hpa
kubectl describe -n <DREMIO_NAMESPACE> podmonitor