Query failures due to inconsistent metadata and S3 Versioning Support

shragj · February 4, 2021, 1:07am

Hello,

While running queries on mutating S3 data we encounter FileNotFound exceptions, which we have traced to a conflict between Dremio’s metadata and the S3 object’s (current) timestamp.

github.com

dremio/dremio-oss/blob/master/plugins/s3/src/main/java/com/dremio/plugins/s3/store/S3AsyncByteReaderUsingSyncClient.java#L140


public void run() {
  // S3 Async reader depends on S3 libraries available from application class loader context
  // Thread that runs this runnable might be created from Hive readers from a different
  // class loader context. So, always changing the context to application class loader.
  try (Closeable swapper =
         ContextClassLoaderSwapper.swapClassLoader(S3FileSystem.class)) {
    final GetObjectRequest.Builder requestBuilder = GetObjectRequest.builder()
      .bucket(bucket)
      .key(path)
      .range(S3AsyncByteReader.range(offset, len))
      .ifUnmodifiedSince(instant);
    if (requesterPays) {
      requestBuilder.requestPayer(REQUESTER_PAYS);
    }
    final GetObjectRequest request = requestBuilder.build();
    final Stopwatch watch = Stopwatch.createStarted();
    try {
      final ResponseBytes<GetObjectResponse> responseBytes = invoker.invoke(() -> s3.getObjectAsBytes(request));
      byteBuf.setBytes(dstOffset, responseBytes.asInputStream(), len);
      logger.debug("[{}] Completed request for bucket {}, path {} for {}, took {} ms", threadName, bucket, path, request.range(),

The S3AsyncByteReader does not seem to have the same behavior, btw.

github.com

dremio/dremio-oss/blob/master/plugins/s3/src/main/java/com/dremio/plugins/s3/store/S3AsyncByteReader.java#L63-L67


GetObjectRequest.builder()
  .range(range(offset, len))
  .bucket(bucket)
  .key(path)
  .build(),

We have enabled asynchronous access when possible, so we were wondering why the sync client is used, and also why there is a difference in behavior of the sync vs async based readers.

Also, using S3 versioned objects (and storing versions in Dremio metadata instead of timestamps) would be a way to guarantee metadata is never inconsistent (but can be stale) and queries don’t fail. Does Dremio support using version metadata?

balaji.ramaswamy · February 4, 2021, 8:54am

@shragj

It seems like after Dremio learnt about a certain file, it was deleted. We have an open internal ticket to address this. Dremio currently does this when files are deleted on Azure storage by collecting metadata on the fly. Profile will have 2 attempts, but not on S3

shragj · February 4, 2021, 2:40pm

Thanks for explaining.
Our files (S3 objects) are never deleted, just modified (replaced).

balaji.ramaswamy · February 5, 2021, 8:44am

@shragj

If file names change, you will run into this issue

Topic		Replies	Views
Failure mode for Dremio metadata refresh for asynchronously mutating data sets	4	1565	January 26, 2022
Exception on 90% of queries	4	1877	August 24, 2020
S3 (minio) errors while getting data from reflection	3	1770	April 8, 2020
Reading S3 directory file not found error	6	1551	April 17, 2018
The sourceis currently unavailable. Metadata is not accessible	1	25	October 27, 2024

Query failures due to inconsistent metadata and S3 Versioning Support

Related topics