Creating virtual tables via API is slow


#1

When we create virtual tables using the REST API, it can be quite slow. Sometimes the API will take up to 30-45s (or longer) before it returns. We have about ~125 virtual queries (and growing) and its really slow. Is Dremio running the query as part of the API call? Is there a way to speed this up?


#2

Hi @jhaynie

Can you please pass on one of the profiles for the 30-45s job?

Share a Profile

Thanks
@balaji.ramaswamy


#3

There are no jobs (that I can find) when we create them via REST API.


#4

@jhaynie

Is there similar performance when run via a jdbc client like Dbeaver?


#5

I think I found by changing in the job UI to “internal” and they now show up.

Here’s one that is ~11s. It’s sometimes slow, sometimes faster.

e8708773-fe2b-4a8b-9159-4e4a9c0f69af.zip (79.5 KB)

I can see from this screen that it is actually running these queries to create a prepared statement. I wish there was a way that we could do these async since we have currently 128 virtual tables and we are likely to create another 50-75 in the next few weeks. Its making it really super slow to start up a container to run our software.


#6

In case others need to find how to get jobs like this


#7

Hi @jhaynie

We should not be doing prepare statements for REST API calls. Did you mean ODBC client. If you this is really REST calls what is the client you are using?

Thanks
@balaji.ramaswamy


#8

We do some work when generating a VDS (like figuring out the schema) and your query is quite complex. You can run multiple API calls in parallel to speed things up for now to help speed things up.


#9

we are just hitting the API from Go. Nothing special.