Using a Parameter in the Catalog.yml with Omegaconfigloader

You can use the oc.env omega conf revolver here I think

wdym? to use a parameter in the catalog.yml? oc.env reads from environment variables

Do you want to override a Kedro parameter or read a parameter defines in catalog.yml?

How did you select a parameter in workflow?

i want to:

1. do a run with a parameter (prinscreen 1)
2. (then this is passed as a task run in Parameters - "--run_folder","{{job.parameters.run_folder}}")
3. now i want this "run_folder" param to also be used in my catalog - e.g.:

mytable:
  catalog: mycatalog
  write_mode: overwrite
  database: gold_dm_${params:run_folder}
  table: mytable

You're likely looking for the runtime_params resolver, identical to globals but specifically for overriding params at runtime

In your above code snippet, just replace params with runtime_params:
mytable:
catalog: mycatalog
write_mode: overwrite
database: gold_dm_${runtime_params:run_folder}
table: mytableace

Ah nice, I didn't know about runtime_params. Very useful, especially for changing output folder. But does it also try to find it in globals, or do we have to provide it everytime we run?

Although I am using a custom loader based on TemplatedConfigLoader in 1 project.

Another simple solution could be the following:

In your main entrypoint script, dump the global parameters needed to be overriden in conf/local/globals.yml

main.py

import os
import yaml

os.makedirs("conf/local", exist_ok=True)

with open("conf/local/globals.yml", "w") as f:
    yaml.dump(global_params_to_override)

# Now run your kedro pipeline (assure that local env is default else specify it)

Ah this is great, didn't know about runtime_params either. That solves for my use case entirely! Thank you!

Great! how could we have laid out the docs so it was easier to for you to find?

well, to be fair, the section on runtime_params was right above the one i've linked to in my question, even though I could swear I had read the entire parametrization docs 😅 so i was helpless, the docs are great!

It's complex stuff and there's lots of content there, any feedback is still helpful!

well maybe the configuration docs could be organized by use cases / usage patterns - or at least provided a summary of use cases and solutions at the end, something like:

• I don't need to use parameters in catalog.yml -> regular parameters.yml and globals.yml with no custom resolvers
• I need to use parameters in catalog.yml but will not override them at runtime -> use globals and resolve them in catalog.yml (and parameters.yml, if needed)
• I need to use parameters in catalog.yml AND will override them at runtime -> use runtime_params and resolve them in catalog.yml (and parameters.yml, if needed)

it's still helpful, we could maybe structure a tutorial around that flow