Hi!
We’re using Dremio AWS build 25.0.7-202407181421320869-2632b04f and this is what happens sometimes:
Someone opens a VDS for editing but just because (for example) to do some experimental modification on it, they are not saving the modification before closing the editor.
That modified query can get “stuck” somewhere in the caches of Dremio and later if someone opens the VDS for editing it will show the modified version by someone else.
The modification is saved in the VDS definition, the VDS runs without the modifications and if you query sys.views the definition there will be the correct one, but if you open an editor there is a different (usually much older) version edited by someone else in the organization.
Is there a fix for this in later versions or some way to eliminate these stuck editing versions?
@Forgi Engineering is looking at fixing the issue ASAP. In the meantime, to mitigate the issue on your side, we recommend setting store.dataset.versions.limit to a higher value, such as 10,000 (the default is 50). This change will prevent the dataset trimmer from deleting older dataset versions, ensuring the correct view definition is retained.
Please note that this change will cause the RocksDB database to grow. You may need to monitor the database to ensure it doesn’t exceed manageable size limits.
Run the below API to check the store size (Can be run when Dremio is online). This will generate the zip file you are redirecting to, in the example kvstore_summary.zip ,
curl --location --request GET 'localhost:9047/apiv2/kvstore/report?store=none' \--header 'Authorization: _dremioirr8hj6qnpc3tfr3omiqvev51c' > kvstore_summary.zip
extract the zip file and if you open the kvstores_stats.log file, you will see an entry like below
datasetVersions
basic rocks store stats
* Estimated Number of Keys: 68295
* Estimated Live Data Size: 565258829
* Total SST files size: 601784965
* Pending Compaction Bytes: 0
* Estimated Blob Count: 0
* Estimated Blob Bytes: 0
You can run this weekly once to see if there is increase in Total SST files size and Estimated Number of Keys
Seems simply executing ANY sql query in the edit widnow of VDS overrides the tip version of query window history showing not the latest SAVED source sql of vds BUT last sql executed from it (edit window). So somehow edit window starts acting like simple sql query window.