How to use alluxio as storage("distributed path") instead of hdfs

1

Hey all,
I used HDFS as the storage of Dremio before, Now we try to connect Dremio to Alluxio cluster (HDFS as Alluxio’s under storage system), but we found that there is no support for “alluxio://” protocol (in dremio.conf) in the existing documents. How can I do this?

2

@meiosischen Has to be HDFS, does Alluxio provide a way to map it as a HDFS file scheme?

3

you can use “Alluxio S3 API” as if s3 storage

4

@meiosischen and @balaji.ramaswamy,

It may be possible to use the Alluxio “Transparent URI” feature of the enterprise version of Alluxio. This will allow all hdfs:// references to be redirected to the Alluxio shim filesystem (alluxio://).

See: Transparent URI - Alluxio v2.8.0 (stable) Documentation.

Also, I have tested Dremio with Alluxio using Alluxio’s S3 API. In Dremio, I configured a new data source using the “Amazon S3 Source” data source type, then I selected the “Enable compatibility mode” checkbox. Finally, I specified several user supplied connection properties in Dremio, like this:

 fs.s3a.path.style.access=true 
 fs.s3a.endpoint=http://<ALLUXIO_MASTER_NODE_IP>:39999/api/v1/s3

Hope this helps,

Greg