Implement secret key credentials for s3:// AWS protocol

Description

Secret key ID and secret key should be placeable into URI. E.g. `s3://AKIAJPD44VO5HMS7NCBQ:7k3eVISkvFwShOPEOGYP4ThmNKBqKM9zO2QuZK84@pavel-h2o-test/iris.csv` (not real keys).

This approach effectively enables H2O users to reach different buckets accessible under different identities.

  • Revisit current client-creation (one static client is no longer enough)

  • Re-work key cache

  • Re-work bucket cache

  • Keep current version of AWS client (bound to Hadoop client version).

  • How are clients bound to buckets going to be cleaned ? Will those ever deprecate ?

  • If possible, enable global bucket accessibility in the "old API" we're currently using, as common use-case is to access different buckets around the world, which may not necessarily be tied to the endpoint configured in core-site.xml.

  • Fix locking mechanism when client is obtained (if still applicable).

Assignee

Pavel Pscheidl

Fix versions

Reporter

Pavel Pscheidl

Support ticket URL

None

Labels

None

Affected Spark version

None

Customer Request Type

None

Task progress

None

ReleaseNotesHidden

None

CustomerVisible

No

Priority

Major
Configure