ANALYZE TABLE on tables with columns using reserved words in the name fails

1

Here’s a simplified example:

ANALYZE TABLE datalake.iceberg.mytable FOR COLUMNS ("from", ts) COMPUTE STATISTICS

translates to a job with the following query without quotes around the from column

SELECT 'datalake.iceberg.mytable' as TABLE_PATH, ndv("ORIGINAL_from") as "NDV_from" , ndv("ORIGINAL_ts") as "NDV_ts" , count(*) as "RCOUNT_null" , count("ORIGINAL_from") as "COLRCOUNT_from" , count("ORIGINAL_ts") as "COLRCOUNT_ts" , tdigest("ORIGINAL_ts", true) as "TDIGEST_ts" FROM (Select from as "ORIGINAL_from", ts as "ORIGINAL_ts"from datalake.iceberg.mytable)

The job fails because of the missing quotes in the inner SELECT

Tested on both 24.1.0 and 24.2.2

2

Doing the same on a VDS where the column has been renamed results in a NPE:

         SYSTEM ERROR: NullPointerException



  (com.dremio.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: java.lang.NullPointerException
    com.dremio.exec.work.foreman.AttemptManager.run():514
    com.dremio.common.concurrent.ContextMigratingExecutorService$1.run():69
    com.dremio.context.RequestContext.run():96
    com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$4():226
    java.util.concurrent.ThreadPoolExecutor.runWorker():1128
    java.util.concurrent.ThreadPoolExecutor$Worker.run():628
    java.lang.Thread.run():829
  Caused By (java.util.concurrent.ExecutionException) java.lang.NullPointerException
    java.util.concurrent.CompletableFuture.reportGet():395
    java.util.concurrent.CompletableFuture.get():1999
    com.dremio.exec.work.foreman.AttemptManager.run():467
    com.dremio.common.concurrent.ContextMigratingExecutorService$1.run():69
    com.dremio.context.RequestContext.run():96
    com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$4():226
    java.util.concurrent.ThreadPoolExecutor.runWorker():1128
    java.util.concurrent.ThreadPoolExecutor$Worker.run():628
    java.lang.Thread.run():829
  Caused By (java.lang.NullPointerException) null
    com.dremio.exec.planner.sql.handlers.direct.AnalyzeTableStatisticsHandler.toResult():70
    com.dremio.exec.planner.sql.handlers.commands.DirectWriterCommand.plan():100
    com.dremio.exec.work.foreman.AttemptManager.plan():571
    com.dremio.exec.work.foreman.AttemptManager.lambda$run$4():462
    com.dremio.service.commandpool.ReleasableBoundCommandPool.lambda$getWrappedCommand$3():140
    com.dremio.service.commandpool.CommandWrapper.run():70
    com.dremio.context.RequestContext.run():96
    com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$4():226
    com.dremio.common.concurrent.ContextMigratingExecutorService$ComparableRunnable.run():206
    java.util.concurrent.Executors$RunnableAdapter.call():515
    java.util.concurrent.FutureTask.run():264
    java.util.concurrent.ThreadPoolExecutor.runWorker():1128
    java.util.concurrent.ThreadPoolExecutor$Worker.run():628
    java.lang.Thread.run():829
3

@dotjdk We will look into this, in the meantime, what is the reason you are running ANALYZE, for row count estimates?

4

I was doing a pretty selective query on a table in our dev env. The table has about 6.5 billion rows with a VDS that joins it to a few dimension tables. I waited a maximum of 20 minutes for the query to finish before giving up. (The query returns less than 100K rows)

After the ANALYZE TABLE it was down to 4-5 seconds

5

@dotjdk

Can you please send us the before and after profiles, want to see if the analyze fixed row count estimates and thus the plan

6

Unfortunately I am unable to share the profiles as they contain confidential information. I can verify the row count estimates when I am back at the office.