Using kubernetes secrets in kedro pipelines for s3 and ...

At a glance

The community member is seeking help to use Kubernetes Secrets within Kedro pipelines to connect with S3 or other external services. Another community member suggests using IRSA (IAM Roles for Service Accounts) as the best practice if running on Kubernetes. The second community member asks for a one-liner explanation of how IRSA will help fetch and use secrets, as they are currently storing the credentials in environment variables and using a resolver to pass them. They want to know if there will be any changes to the way they are using datasets.

Useful resources

VVishal Pandey

Hello Team

Can someone help me to use kubernetes Secrets within kedro pipelines to connect with S3 or any other external service.

2 comments

mmarrrcin

If you're on K8s, best practice is to use IRSA https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

VVishal Pandey

can you please explain this in one liner how IRSA will help me to fetch and use secrets ?

Currently I have stored the credentials in env variables and using resolver to pass the credentials

training_data_insert_records:
  type: pandas.CSVDataset
  filepath: <a target="_blank" rel="noopener noreferrer" href="s3://ml-datawarehouse/warehouse/extraction/doc_table_insert.csv">s3://ml-datawarehouse/warehouse/extraction/doc_table_insert.csv</a>
  save_args:
    index: False   
  versioned: True
  credentials:
    client_kwargs:
      aws_access_key_id : ${oc.env:AWS_ACCESS_KEY_ID}
      aws_secret_access_key : ${oc.env:AWS_SECRET_ACCESS_KEY}
      aws_session_token : ${oc.env:AWS_SESSION_TOKEN}

I want to know will there be any change the way i am using datasets

Add a reply

Join the Kedro community

Using kubernetes secrets in kedro pipelines for s3 and external service connections