I added a hive data source and configured the hive metastore. The data of this hive is stored on s3. What other configurations do I need to add to create tables and perform queries in dremio? The current error is as follows: java.nio.file.AccessDeniedException: s3a://iris/: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
@fuliu Where have you declared the access_key, secret_key variables as the error says it is not able to load these variables
@fuliu See general tab
@fuliu My bad, you are having Hive external tables where the PARQUET files are on S3 and the S3 bucket has access and secret key, yes then it should go via advanced option, but it should get the same name as defined in core-site.xml or you can copy core-site.xml, see doc below
When I start the source code through idea, the hive data source is not scanned, so I cannot see the hive data source on the UI. Can you guide me on how to configure it?
@fuliu Have you validared that the core-site.xml has the correct parameter name and value for the access key and secret key? Can you try passing the settings through the advanced option tab under the Hive source in Dremio. I assume you are able to query this table via Hive shell?
Basically Dremio needs permission to read the file on S3
Some of the parameters here should be in your core-site.xml or if the core-site.xml has conflicting values for the same parameter names then add to source advanced option
@fuliu you need to define the parameters as they would be configured in the core-site.xml, ie:
After I added these parameters, the data in csv format can be queried, but other formats, such as parquet, json, and iceberg, fail to query
ae94bd7b-f159-4de0-ac58-c0899d0d4306.zip (20.1 KB)
59ab728c-e4fb-43bb-a869-5a294bd720bb.zip (16.9 KB)
@fuliu Can you see if you this thread helps?