Join the Kedro community

Updated 2 months ago

Using kubernetes secrets in kedro pipelines for s3 and external service connections

Hello Team

Can someone help me to use kubernetes Secrets within kedro pipelines to connect with S3 or any other external service.

m
V
2 comments

can you please explain this in one liner how IRSA will help me to fetch and use secrets ?

Currently I have stored the credentials in env variables and using resolver to pass the credentials

training_data_insert_records:
  type: pandas.CSVDataset
  filepath: <a target="_blank" rel="noopener noreferrer" href="s3://ml-datawarehouse/warehouse/extraction/doc_table_insert.csv">s3://ml-datawarehouse/warehouse/extraction/doc_table_insert.csv</a>
  save_args:
    index: False   
  versioned: True
  credentials:
    client_kwargs:
      aws_access_key_id : ${oc.env:AWS_ACCESS_KEY_ID}
      aws_secret_access_key : ${oc.env:AWS_SECRET_ACCESS_KEY}
      aws_session_token : ${oc.env:AWS_SESSION_TOKEN}

I want to know will there be any change the way i am using datasets

Add a reply
Sign up and join the conversation on Slack