Hello Team
Can someone help me to use kubernetes Secrets within kedro pipelines to connect with S3 or any other external service.
If you're on K8s, best practice is to use IRSA https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
can you please explain this in one liner how IRSA will help me to fetch and use secrets ?
Currently I have stored the credentials in env variables and using resolver to pass the credentials
training_data_insert_records: type: pandas.CSVDataset filepath: <a target="_blank" rel="noopener noreferrer" href="s3://ml-datawarehouse/warehouse/extraction/doc_table_insert.csv">s3://ml-datawarehouse/warehouse/extraction/doc_table_insert.csv</a> save_args: index: False versioned: True credentials: client_kwargs: aws_access_key_id : ${oc.env:AWS_ACCESS_KEY_ID} aws_secret_access_key : ${oc.env:AWS_SECRET_ACCESS_KEY} aws_session_token : ${oc.env:AWS_SESSION_TOKEN}