I created a dremio 3.2 cluster today, but I cant create table from a s3 sales folder.
example path
s3://bucket-name/sales/year=2019/month=03/day=01/files.parquet
This option is possible in community version ?
1 Like
HI Noctorua,
yes dremio can create table from folder, I am using dremio mac community version.
probably you could try copy the s3 folder to your cluster local dir, and create table from local dir , see if it works.
Yes this is supported in Dremio CE.
Cab you explain how you tried to set this up?
I tried with some files in the local folder and it worked.
But I realize that it can not do anything s3 with many folders and files
bucket-name/sales/
250000 objects - 40.3 GB
logs
server.log
2019-05-17 12:40:56,747 [catalog-source-synchronization] WARN c.d.exec.catalog.CatalogServiceImpl - Failure while synchronizing sources.
server.gc
2019-05-17T12:40:07.446+0000: 69622.586: [GC (Allocation Failure) [PSYoungGen: 763584K->12181K(766976K)] 2840695K->2090084K(3513344K), 0.0090466 secs] [Times: user=0.06 sys=0.00, real=0.01 secs]
Hello @noctorua,
The Alllocation Failure message is normal Java Garbage Collection logging, so it can be ignored in this case.
Can you describe how you reach this error?
It seems like you’ve successfully added an S3 source in Dremio but you are having difficulties promoting a file path in that S3 source to a physical dataset (PDS). If this is the case, please try reproducing the error and attach the dremio server.log from the time you did so, and we can take a look.
1 Like
Hi Ben,
I`m describe the steps:
1 - S3 Configuration in Dremio, OK
2 - Select a Bucket-name/sale1 (parquet format and contains 5 files only), show me the preview data, e save successfull. OK until now
3 - Select a Bucket-name/sales2 (parquet format and contains 250000 objects and 40.3 GB), this case not show me the previw data,
I select parquet format and click in save, but occurs this error:
Something went wrong
HTTP 405 Method Not Allowed
A attach the server.log file but a cant see errors about this in file.
server.log.zip (4,1 KB)
1 Like
Hi guys,
I try install from cloudformation in my aws account, but the problem persistent
Build
3.2.4-201906051751050278-1bcce62
Edition
Community Edition
Build Time
05/06/2019 14:57:05
Can someone help me?