Dremio/Liquibase integration - Read/Write area

1

In order to use Liquibase to automate datasets and reflections management,
one would need a writable area (the right to send INSERT/UPDATE/DELETE statements on the two liquibase changelog tables).
Calcite has DML support. Could it be possible to accept DML statements on a datasource ?

Best regards,

2

Sorry, I am not familiar with Liquidbase, but let’s see what we can figure out. Could you say a little more about how you would like to use Liquidbase with Dremio?

Dremio doesn’t support updates to underlying sources - data access is read only. However, Dremio does support DDL for creating sources and virtual datasets. You can read more about this here:

https://docs.dremio.com/sql-reference/sql-commands/

3

Hello Kelly,

Thank you for your fast reply !

I know about Dremio DDL features, it is a powerful way to automate Dremio objects management.

Liquibase is a kind of ‘source control’ for database schemas I have been using for years,

But I just realized there is a ‘offline’ possibility, using a CSV file to persist database applied changelogs.

Thank you for your great tool.

https://www.liquibase.org/

Best

1 Like
4

We deploy Dremio views in multiple environments and manage the schemas in source control. We’ve messed with flyway and liquibase for relational databases (postgres, etc.), but they didn’t work out for tools like Hive and Dremio - we use external tables to point to prepared marts in Hive, so it’s a lot like Dremio in our world.

Since we deploy these views/tables in multiple environments, we’ve made templates out of our DDL statements and checked those into source control in a structure that looks suspiciously like the flyway migrate sql structure. Rather than having an actual table with the version, we populate a file to indicate what schema we’re on, and when it’s time to upgrade we perform a very similar set of operations:

  1. check to see what version we think we’re on
  2. make sure the deployed schema looks like that version (see REST apis… this was less robust in dremio before those became available)
  3. perform migrations in order by populating templates and running the ddl
  4. update the version file

It’s not perfect, but it supports the familiar flow and works well enough for what we do. Recreating those steps is pretty simple in whatever scripting/programming language you like to use.

3 Likes