Allow to specify AWS credentials at runtime

Description

This improvement is motivated by - the user is using URL-encoded AWS credentials in order to access S3 object using different AWS credentials. Using URL-encoded solution is more of a workaround than a proper solution because of security risks associated with putting credentials in URL (they might end up in logs, be visible in Flow UI...).

Instead of that, we should allow users to define the credentials at runtime - provide a function that will define a SecurityContext with the user provided credentials. PersistS3 can be instantiated for the given SecurityContext. We can then think about having just a single security context (at a given time) or allowing multiple contexts to coexist at the same time to enable the original use case.

Assignee

Pavel Pscheidl

Fix versions

Reporter

Michal Kurka

Support ticket URL

None

Labels

None

Affected Spark version

None

Customer Request Type

None

Task progress

None

ReleaseNotesHidden

None

CustomerVisible

No

Priority

Major
Configure