Error creating datasoure S3 bucket through Python3 requests

Dremio University
1

Hi all,

I m trying to add our datasource S3 bucket in Dremio with python requests, here’s my code:

payload={
  "entityType": "source",
  "config": {
    "accessKey": $myaccessKey$, 
    "accessSecret":$myaccessSecret$, 
    "propertyList": [
        {"name": "fs.s3a.endpoint", 
         "value": $myendpoint$,
        }, 
        {"name": "fs.s3a.path.style.access", 
         "value": "true"},
        {"name": "fs.s3a.proxy.host", 
         "value": $host$},
        {"name": "fs.s3a.proxy.port", 
         "value": $port$},
        {"name": "fs.s3a.proxy.username", 
         "value": $myusername$},
        {"name": "fs.s3a.proxy.password", 
         "value": $mypassword$},
    ],
      "rootPath": "/", 
      "credentialType": "ACCESS_KEY", 
      "enableAsync": True, "compatibilityMode": True, 
      "isCachingEnabled": True, 
      "maxCacheSpacePct": 95, 
      "requesterPays": False, 
      "enableFileStatusCheck": True
  },
  "type": "S3",
  "name": "S3_outscale_test",
  "metadataPolicy": {
    "authTTLMs": 86400000,
    "namesRefreshMs": 60000,
    "datasetRefreshAfterMs": 3600000,
    "datasetExpireAfterMs": 7200000,
    "datasetUpdateMode": "PREFETCH_QUERIED",
    "deleteUnavailableDatasets": True,
    "autoPromoteDatasets": False
  },
  "accelerationGracePeriodMs": 14400000,
  "accelerationRefreshPeriodMs": 14400000,
  "accelerationNeverExpire": True,
  "accelerationNeverRefresh": True,
  "allowCrossSourceSelection": False,
}

dreRequest.apiPost("catalog",payload)

{'errorMessage': 'Could not connect to S3 source. Check your S3 data source settings and credentials.',

dreRequest.apiPost is my function (requests.post) , I m sure the header and token are correct like explained in the doc (Doc dremio post catalog). However the error message is always “Could not connect to S3 source…”.

I already added S3 manually in dremio, it workd. Also I tried multiple times in the environment with or without proxy, the credentials are correct for S3 connection but still failed.

Is there anything wrong with the payload config ?

Thanks for help