Join the Kedro community

Updated 4 months ago

Using env variable in kedro pipeline node

At a glance

👋 Hello, team!
I want to use env variable in a node of a kedro pipeline
I have set up the credentials.yml as this, however it doesn't populate it with the env var inside the node (works for fetching kedro SnowParktableDataset though)

snowflake_creds:
    user: ${oc.env:snowflake_svc_user}

Any other ways to do this?

3 comments

MMerel Theisen

Do you want to use any environment variable or specifically credentials? Credentials are purposefully abstracted in Kedro for security reasons, so I would advice using any sensitive data directly in a node if you can prevent it.

AAkshata Patel

Specifically credentials
I found a way using OmegaConfigLoader like this:

config_loader = OmegaConfigLoader(conf_source=".")
credentials = config_loader["credentials"]

openai_client_creds_dict = None
if "azure_openai_client_creds" in credentials:
   openai_client_creds_dict = credentials["azure_openai_client_creds"]

But open to better suggestions 😄

MMerel Theisen

We have an experimental dataset for openai with langchain: https://github.com/kedro-org/kedro-plugins/blob/main/kedro-datasets/kedro_datasets_experimental/langchain/_openai.py this might give some inspiration on how to handle this.

Add a reply