I want to be able to store “descriptive comments on each column and table”. As I understand it, Dremio does not support this.
I imagine the AI plain-language-to-sql would be able to readily leverage these comments about columns so it could generate more accurate sql statements.
Having the comments would also support better data discovery. Tools like DataHub already look for the comment metadata in Snowflake, and that’s been really useful.
Some source systems already have descriptions of each column, and if I could extract these and include them in either datasets or views I maintain in Dremio, I think it would contribute to both data discovery and improved AI-based queries.
I fully expect a relatively small % of customers would actually leverage this level of detail in the metadata, but I suspect it will grow in importance with the advent of the AI-aided sql generation.
The wiki helps, yes. And your AI-driven feature to auto-generate wiki content is really cool. However, having per-column metadata we can add would be the general solution, much like how Snowflake supports comments on columns, tables, views, etc. The column-level comments could flow through to global data catalogs like DataHub (that spans multiple different data catalogs and sources). And I expec column-level comments could be leveraged by AI to make data queries even more accessible to end users.
Since your AI-driven metadata system is pretty good, I think it would be really useful to have the AI generate/regenerate per-column “comment” metadata. It’s already doing it, though it’s only in the wiki.
Hi @capnjosh,
My name is Imran and I’m from the Dremio Product Management team.
A description field for columns is on our roadmap and it should be coming in H1 2025 as part of our Glossary feature in Dremio.
As @balaji.ramaswamy pointed out, for tables the wiki is the right place for table-level descriptions. Hope this helps.
