# Dremio University **URL:** https://community.dremio.com/c/dremio-university/6.md?page=2 [Latest](https://community.dremio.com/latest.md) · [Categories](https://community.dremio.com/categories.md) **Page:** 3 --- ## [Incremental reflections on a view](https://community.dremio.com/t/incremental-reflections-on-a-view/11055)
**Author:** [@kyleahn](https://community.dremio.com/u/kyleahn)\ **Replies:** 2\ **Last updated:** [October 16, 2023, 6:28pm UTC](https://community.dremio.com/t/incremental-reflections-on-a-view/11055 "2023-10-16T18:28:32Z")
Hi, I have 3 APPEND-ONLY whose reflections are set to be incrementally refreshed on a partitioned column named log\_timestamp manually by our application logic every 15 minutes or so. A view is created off these 3 Ic… --- ## [IOException: .../\_delta\_log/0000000000000000010.checkpoint.parquet is not a Parquet file. expected magic number \[80, 65, 82, 49\] at tail, but found \[55, 118, -20, -120\]](https://community.dremio.com/t/ioexception-delta-log-0000000000000000010-checkpoint-parquet-is-not-a-parquet-file-expected-magic-number-80-65-82-49-at-tail-but-found-55-118-20-120/11051)
**Author:** [@mehdi.hala.ext](https://community.dremio.com/u/mehdi.hala.ext)\ **Replies:** 2\ **Last updated:** [October 9, 2023, 7:33am UTC](https://community.dremio.com/t/ioexception-delta-log-0000000000000000010-checkpoint-parquet-is-not-a-parquet-file-expected-magic-number-80-65-82-49-at-tail-but-found-55-118-20-120/11051 "2023-10-09T07:33:28Z")
Hello Dremio Team, I’m facing an issue when I’m trying to read a delta table on Dremio once it gets to the point where it has checkpoints. All the tables are read just fine apart from the one that has a checkpoint in t… --- ## [Dremio web sockets issue on Azure AKS](https://community.dremio.com/t/dremio-web-sockets-issue-on-azure-aks/10915)
**Author:** [@Nikola2](https://community.dremio.com/u/Nikola2)\ **Replies:** 3\ **Last updated:** [October 4, 2023, 5:53am UTC](https://community.dremio.com/t/dremio-web-sockets-issue-on-azure-aks/10915 "2023-10-04T05:53:15Z")
Hello Dremio team, I’m trying to deploy Dremio CE latest(v24)Test environment on Azure AKS using helm. After deploying when i login on WebUI I’m receiving error Your Internet connection may be offline or WebSockets to … --- ## [Using Iceberg external reflection and raw reflection together](https://community.dremio.com/t/using-iceberg-external-reflection-and-raw-reflection-together/10906)
**Author:** [@kyleahn](https://community.dremio.com/u/kyleahn)\ **Replies:** 8\ **Last updated:** [September 28, 2023, 7:32am UTC](https://community.dremio.com/t/using-iceberg-external-reflection-and-raw-reflection-together/10906 "2023-09-28T07:32:03Z")
My use case involves providing near-real time updates of data with very low latency requirements. I figured out that careful configuration of raw and aggregate reflections could meet the requirements. However, they are n… --- ## [Error accessing parquet table in MinIO](https://community.dremio.com/t/error-accessing-parquet-table-in-minio/10972)
**Author:** [@moy8011](https://community.dremio.com/u/moy8011)\ **Replies:** 9\ **Last updated:** [September 11, 2023, 5:30pm UTC](https://community.dremio.com/t/error-accessing-parquet-table-in-minio/10972 "2023-09-11T17:30:15Z")
I have an error when running a query to any table pointing to parquet files in MinIO. I get the error “IllegalStateException: Invalid AWSCredentialsProvider provided:”. I have other tables pointing to CSV files in the s… --- ## [\[Hive\]\[Delta lake\] Can not query with with = operator in Dremio 24.1.0](https://community.dremio.com/t/hive-delta-lake-can-not-query-with-with-operator-in-dremio-24-1-0/10748)
**Author:** [@ducvo](https://community.dremio.com/u/ducvo)\ **Replies:** 22\ **Last updated:** [September 11, 2023, 11:46am UTC](https://community.dremio.com/t/hive-delta-lake-can-not-query-with-with-operator-in-dremio-24-1-0/10748 "2023-09-11T11:46:45Z")
I connect source hive metastore hive 3 store delta lake tables. and I have table users with id and name data id name 1 Alex 2 Su But when I excuse query SELECT \* FROM datalakelocal.ducvo.users WHERE id = 1; … --- ## [Encountered exception during row count estimation,Memory was leaked by query](https://community.dremio.com/t/encountered-exception-during-row-count-estimation-memory-was-leaked-by-query/10931)
**Author:** [@fiona](https://community.dremio.com/u/fiona)\ **Replies:** 4\ **Last updated:** [September 7, 2023, 4:58am UTC](https://community.dremio.com/t/encountered-exception-during-row-count-estimation-memory-was-leaked-by-query/10931 "2023-09-07T04:58:58Z")
I have 300k records in my dataset, they are partitioned into 30 date folder. when I query any data for one date, below error error. I think the key words are “RecordPruner - Encountered exception during row count estim… --- ## [Load balancing REST API requests across master and coordinator nodes](https://community.dremio.com/t/load-balancing-rest-api-requests-across-master-and-coordinator-nodes/10900)
**Author:** [@kyleahn](https://community.dremio.com/u/kyleahn)\ **Replies:** 3\ **Last updated:** [August 24, 2023, 5:17am UTC](https://community.dremio.com/t/load-balancing-rest-api-requests-across-master-and-coordinator-nodes/10900 "2023-08-24T05:17:16Z")
Hi, team Dremio. Based on dremio-coordinator.yaml helm chart, I am seeing that web port 9047 isn’t in the container port. This implies that the load balancer does not balance REST API requests across master and coordina… --- ## [Unable to accelerate with reflections created off iceberg tables](https://community.dremio.com/t/unable-to-accelerate-with-reflections-created-off-iceberg-tables/10903)
**Author:** [@kyleahn](https://community.dremio.com/u/kyleahn)\ **Replies:** 4\ **Last updated:** [August 22, 2023, 4:11am UTC](https://community.dremio.com/t/unable-to-accelerate-with-reflections-created-off-iceberg-tables/10903 "2023-08-22T04:11:38Z")
Hi! TL;DR - looking to get raw and aggregate reflections working. My use case requires creating a raw and aggregate reflection (count) of a view consisting of 3 iceberg tables. These three tables are LEFT OUTER joined,… --- ## [Querying Specific User's Authorized Data using Dremio REST API](https://community.dremio.com/t/querying-specific-users-authorized-data-using-dremio-rest-api/10907)
**Author:** [@ssol](https://community.dremio.com/u/ssol)\ **Replies:** 1\ **Last updated:** [August 22, 2023, 3:37am UTC](https://community.dremio.com/t/querying-specific-users-authorized-data-using-dremio-rest-api/10907 "2023-08-22T03:37:53Z")
I have admin privileges and can access all SPACEs and DATASETs. However, I’m trying to retrieve only the data they are authorized to access using a specific user ID. To put it simply, I’m seeking a way to preview the re… --- ## [Connect Dremio to Iceberg Hive Catalog located at S3](https://community.dremio.com/t/connect-dremio-to-iceberg-hive-catalog-located-at-s3/10893)
**Author:** [@khassan](https://community.dremio.com/u/khassan)\ **Replies:** 2\ **Last updated:** [August 15, 2023, 3:57pm UTC](https://community.dremio.com/t/connect-dremio-to-iceberg-hive-catalog-located-at-s3/10893 "2023-08-15T15:57:02Z")
The Environment setup as below: Dremio v 2.4.1 Hive standalone metastore as Iceberg catalog. Hive Version is 2.x Catalog and data stored at on-prem s3 Storage. No hadoop or HDFS is installed . The table is successfull… --- ## [Iceberg query performance with many parquet files](https://community.dremio.com/t/iceberg-query-performance-with-many-parquet-files/10764)
**Author:** [@kyleahn](https://community.dremio.com/u/kyleahn)\ **Replies:** 12\ **Last updated:** [July 22, 2023, 7:00am UTC](https://community.dremio.com/t/iceberg-query-performance-with-many-parquet-files/10764 "2023-07-22T07:00:05Z")
Hi Dremio community, I have an Iceberg table with Glue catalog. This Iceberg table has about 60 records per file (hourly partitioned), and the number of files have grown to about 300 files over the last couple of weeks.… --- ## [Which JDBC version of .jar file is needed to connect Dremio with Dataiku](https://community.dremio.com/t/which-jdbc-version-of-jar-file-is-needed-to-connect-dremio-with-dataiku/10793)
**Author:** [@ShubhamHengane](https://community.dremio.com/u/ShubhamHengane)\ **Replies:** 4\ **Last updated:** [July 20, 2023, 9:52am UTC](https://community.dremio.com/t/which-jdbc-version-of-jar-file-is-needed-to-connect-dremio-with-dataiku/10793 "2023-07-20T09:52:13Z")
Hello, Trying to connect Dremio with Dataiku, for that using below .jar file to connect: 3.0.6 3.3.1 4.6.0 but any of this jar file is not working, Can you please point me to use the correct version of JDBC .jar fil… --- ## [Dremio helm chart contribution](https://community.dremio.com/t/dremio-helm-chart-contribution/9228)
**Author:** [@GrigorievNick](https://community.dremio.com/u/GrigorievNick)\ **Replies:** 6\ **Last updated:** [July 4, 2023, 1:42pm UTC](https://community.dremio.com/t/dremio-helm-chart-contribution/9228 "2023-07-04T13:42:03Z")
Hi, I have question regarding dremio-cloud-tools/charts/dremio\_v2 at master · dremio/dremio-cloud-tools · GitHub. Does it open for contribution? Or recommended scenario to use is every one fork and maintain it’s own c… --- ## [RocksDBException occuered when execute com.dremio.dac.cmd.Clean#go method](https://community.dremio.com/t/rocksdbexception-occuered-when-execute-com-dremio-dac-cmd-clean-go-method/10716)
**Author:** [@ChickenSoup76](https://community.dremio.com/u/ChickenSoup76)\ **Replies:** 3\ **Last updated:** [June 15, 2023, 2:53pm UTC](https://community.dremio.com/t/rocksdbexception-occuered-when-execute-com-dremio-dac-cmd-clean-go-method/10716 "2023-06-15T14:53:53Z")
I use a tool to execute com.dremio.dac.cmd.Clean#go , try to clean up the job profile in my local dremio,but this exception occurred.It seems like some other proccess block RocksDB.But I dont know how to fix It.What shou… --- ## [All classes under org.apache.twill.api were not found when the test was started](https://community.dremio.com/t/all-classes-under-org-apache-twill-api-were-not-found-when-the-test-was-started/10656)
**Author:** [@ChickenSoup76](https://community.dremio.com/u/ChickenSoup76)\ **Replies:** 6\ **Last updated:** [June 14, 2023, 3:07am UTC](https://community.dremio.com/t/all-classes-under-org-apache-twill-api-were-not-found-when-the-test-was-started/10656 "2023-06-14T03:07:08Z")
But these classes are all here --- ## [Security concerns about Dremio storing data source credentials](https://community.dremio.com/t/security-concerns-about-dremio-storing-data-source-credentials/10672)
**Author:** [@ajaybamel](https://community.dremio.com/u/ajaybamel)\ **Replies:** 0\ **Last updated:** [May 30, 2023, 12:38pm UTC](https://community.dremio.com/t/security-concerns-about-dremio-storing-data-source-credentials/10672 "2023-05-30T12:38:18Z")
Hi everyone! Hope you are all doing good. I have some queries related to Dremio’s security, concerns more towards how Dremio stores the data source credentials. From what I got to know while going through the below to… --- ## [About regexp symbols](https://community.dremio.com/t/about-regexp-symbols/10635)
**Author:** [@Vijay1](https://community.dremio.com/u/Vijay1)\ **Replies:** 1\ **Last updated:** [May 27, 2023, 1:34pm UTC](https://community.dremio.com/t/about-regexp-symbols/10635 "2023-05-27T13:34:35Z")
Can we use re expressions in Dremio like, ~, |, \*, $ these symbols? --- ## [Dremio iceberg fail to read parquet generated by itself](https://community.dremio.com/t/dremio-iceberg-fail-to-read-parquet-generated-by-itself/10650)
**Author:** [@donaldtse118](https://community.dremio.com/u/donaldtse118)\ **Replies:** 1\ **Last updated:** [May 26, 2023, 7:19pm UTC](https://community.dremio.com/t/dremio-iceberg-fail-to-read-parquet-generated-by-itself/10650 "2023-05-26T19:19:44Z")
Hi there, I got a question when using Dremio with Iceberg… I’m running to my job to read multiple simple parquets. using MERGE INTO to continue updating an Iceberg table with the defined primary key. Then I hit this err… --- ## [Reflections on S3 - no results when hitting "run"](https://community.dremio.com/t/reflections-on-s3-no-results-when-hitting-run/9469)
**Author:** [@almirb](https://community.dremio.com/u/almirb)\ **Replies:** 34\ **Last updated:** [May 23, 2023, 10:52am UTC](https://community.dremio.com/t/reflections-on-s3-no-results-when-hitting-run/9469 "2023-05-23T10:52:03Z")
Hi! I installed Dremio CE (v 22.0.0-202206221430090603-1fa4049f ) on an EC2 instance. Configured IAM Role profiles giving permissions to a S3 bucket for reading and writing (as said on Dremio docs). Configured dremio… --- ## [Name of 'pbids' file becomes corrupted (if it is korean)](https://community.dremio.com/t/name-of-pbids-file-becomes-corrupted-if-it-is-korean/10538)
**Author:** [@mgjung02](https://community.dremio.com/u/mgjung02)\ **Replies:** 2\ **Last updated:** [May 9, 2023, 5:16am UTC](https://community.dremio.com/t/name-of-pbids-file-becomes-corrupted-if-it-is-korean/10538 "2023-05-09T05:16:22Z")
If the file name of the “pbids” file downloaded from Dremio is in English, it is fine, but if it is in Korean, it becomes corrupted. for example, our VDS name is “국토교통부\_키스콘 건설업체정보 서비스\_건설업체등록\_IR” and if we download ‘pb… --- ## [How to make {"mode": "Import"} default when creating 'pbids' file?](https://community.dremio.com/t/how-to-make-mode-import-default-when-creating-pbids-file/10558)
**Author:** [@ssol](https://community.dremio.com/u/ssol)\ **Replies:** 4\ **Last updated:** [May 7, 2023, 9:37pm UTC](https://community.dremio.com/t/how-to-make-mode-import-default-when-creating-pbids-file/10558 "2023-05-07T21:37:32Z")
When I create pbids file from Dremio by clicking the Power BI icon, the default setting for data connectivity mode is “DirectQuery”. In the case of data connection with “Import” mode, the data is imported into the Power… --- ## [How can i create a java udf or python udf](https://community.dremio.com/t/how-can-i-create-a-java-udf-or-python-udf/10553)
**Author:** [@vigneshpathapati](https://community.dremio.com/u/vigneshpathapati)\ **Replies:** 3\ **Last updated:** [April 27, 2023, 8:22pm UTC](https://community.dremio.com/t/how-can-i-create-a-java-udf-or-python-udf/10553 "2023-04-27T20:22:46Z")
i wanted to know how can i create a java udf or python udf, can some explain in detail step by step procedure --- ## [How to create a array in dremio with ctas and how to do a select query on it](https://community.dremio.com/t/how-to-create-a-array-in-dremio-with-ctas-and-how-to-do-a-select-query-on-it/10433)
**Author:** [@vigneshpathapati](https://community.dremio.com/u/vigneshpathapati)\ **Replies:** 9\ **Last updated:** [April 26, 2023, 10:44pm UTC](https://community.dremio.com/t/how-to-create-a-array-in-dremio-with-ctas-and-how-to-do-a-select-query-on-it/10433 "2023-04-26T22:44:23Z")
CREATE TABLE sample\_data.posts\_array AS select id, title, body, CONVERT\_FROM(tags, ‘json’) from DBShift.dbshift.sample\_data.posts\_val when running select it running but with CTAS it is throughing error \[Code: 0, SQ… --- ## [Can we create python udfs in dremio by any chance?](https://community.dremio.com/t/can-we-create-python-udfs-in-dremio-by-any-chance/10546)
**Author:** [@Vijay1](https://community.dremio.com/u/Vijay1)\ **Replies:** 1\ **Last updated:** [April 24, 2023, 6:35pm UTC](https://community.dremio.com/t/can-we-create-python-udfs-in-dremio-by-any-chance/10546 "2023-04-24T18:35:30Z")
want to create some python udfs in dremio, how to do that? --- ## [Incremental data reflection with Iceberg](https://community.dremio.com/t/incremental-data-reflection-with-iceberg/10388)
**Author:** [@kyleahn](https://community.dremio.com/u/kyleahn)\ **Replies:** 26\ **Last updated:** [April 17, 2023, 7:40pm UTC](https://community.dremio.com/t/incremental-data-reflection-with-iceberg/10388 "2023-04-17T19:40:16Z")
Hi, DremIO iceberg doc calls out that incremental reflection is not supported for Iceberg tables. Is there a roadmap that this is going to be supported in the future? Dremio. Thanks, Kyle --- ## [“Enable partition column inference” option doesn’t work](https://community.dremio.com/t/enable-partition-column-inference-option-doesn-t-work/10262)
**Author:** [@Pawel](https://community.dremio.com/u/Pawel)\ **Replies:** 13\ **Last updated:** [April 11, 2023, 3:58pm UTC](https://community.dremio.com/t/enable-partition-column-inference-option-doesn-t-work/10262 "2023-04-11T15:58:02Z")
We would like to start using option “Enable partition column inference” to have well formatted columns for partitioned parquet files but looks like this option doesn’t work like is described on documentation paged No ne… --- ## [Failed Zookeeper with k8s](https://community.dremio.com/t/failed-zookeeper-with-k8s/10477)
**Author:** [@emilaineborato](https://community.dremio.com/u/emilaineborato)\ **Replies:** 3\ **Last updated:** [April 11, 2023, 3:57pm UTC](https://community.dremio.com/t/failed-zookeeper-with-k8s/10477 "2023-04-11T15:57:26Z")
Hi, I’m trying to deploy dremio on a kubernetes cluster, but I’m having problems with zookeeper and I’m not able to solve it. Send error message here ZOO\_MY\_ID=3 ZOO\_SERVERS=server.1=zk-0.zk-hs.dremio.svc.cluster.loca… --- ## [Unable to connect Flight Server](https://community.dremio.com/t/unable-to-connect-flight-server/10499)
**Author:** [@Arnie](https://community.dremio.com/u/Arnie)\ **Replies:** 0\ **Last updated:** [April 11, 2023, 7:44am UTC](https://community.dremio.com/t/unable-to-connect-flight-server/10499 "2023-04-11T07:44:38Z")
Hello I just tried to query Dremio EE 20.4.2 through Arrow Flight using sample code (examply/py) in documentation : Dremio. I always get the same error My authentication does not pass … Where did I fail ? --- ## [Distributed Storage Authentication in EKS](https://community.dremio.com/t/distributed-storage-authentication-in-eks/10392)
**Author:** [@kyleahn](https://community.dremio.com/u/kyleahn)\ **Replies:** 1\ **Last updated:** [April 9, 2023, 8:29pm UTC](https://community.dremio.com/t/distributed-storage-authentication-in-eks/10392 "2023-04-09T20:29:18Z")
In the Dremio\_v2 helm chart’s “Values-Reference.md”, I see the following: distStorage.aws.authentication Type: String The valid values for distStorage.aws.authentication are: metadata (default) - Dremio will attempt … [Previous page](https://community.dremio.com/c/dremio-university/6.md?page=1) [Next page](https://community.dremio.com/c/dremio-university/6.md?page=3)