Disable generating query profile and displaying in ui

1

Hi All

I want to run a small query in dremio with high tps. Each query is generating query profile and it’s stats and storing & displaying in ui. I don’t want these stats to be generated. How i can disable this .

Thanks & Regards
Rahul

2

@Rahul

The jobs and profiles are stored in the RocksDB, the internal KV store for Dremio’s metadata, by default we keep 30 days, you can clean up at a higher frequency buy using the parameter “jobs.max.age_in_days”.

Note: This will clean up both the job and the profile information, there is no way currently where we can only clean up the profile

Thanks
Bali

3

Dear Balaji

Thanks for the information regarding job and the profile information clean up.

I am running a small mysql query through dremio and it’s executing in <1s. In normal mysql it’s giving about 1000 tps (milli seconds latency) . I am suspecting dremio job information writing is reducing the tps. So i want to disable the job information writing. You given configuration for cleanup.

Is there any way to disable this job information writing.?
Above I have explained my test case. Will dremio support queries to be executed in milli second latency (oltp applications) ?

Thanks & Regards
Rahul

4

@Rahul

Want to make sure the issue is with results writing. If you open the job profile, are you seeing the time spent on ARROW WRITER, if yes then you can do 2 things

  1. Run via JDBC client like DBeaver that does not write results
  2. In case you are writing results to a distributed storage, maybe write it to a local SSD
    paths: {
    accelerator: “hdfs://<NAMENODE_HOST>:8020/path”
    downloads: “/data/dremio/downloads”
    uploads: “hdfs://<NAMENODE_HOST>:8020/path”
    results: “/data/dremio/results”
    scratch: “/data/dremio/scratch”
    }