Join the Kedro community

Updated 2 weeks ago

Running A Kedro Pipeline With Dynamic Parameters

Hey team, I am working with a codease that uses kedro 0.18.3 and cannot change it. I need to run a pipeline through a python code iterating over some parameters. However, running the way I researched does not updtate the params. Is there something in my code I might be missing:
```bootstrap_project(PROJECT_ROOT)


class TargetShare(TypedDict):
min: float
max: float


@supress_logs
def run_kvi_engine_with_inputs(
kvi_target: TargetShare, foreground_target: TargetShare
) -> pd.DataFrame:
with KedroSession.create(project_path=PROJECT_ROOT) as session:
context: KedroContext = session.load_context()
catalog: DataCatalog = context.catalog

updated_params = context.params | {
“kvi_engine.product_group_target_boundaries”: {
“kvi”: kvi_target,
“foreground”: foreground_target,
}
}

catalog.add(“parameters”, MemoryDataSet(updated_params), replace=True)
context._extra_params = updated_params
session.run(pipeline_name=“kvi_engine”)
rank_output = catalog.load(“kvi_engine.kvi_rank_output”)
#rank_output is not changing
return rank_output
```

R
D
8 comments

Awesome. With this I can add the parameters in data catalog, however when I do session.run(pipe) in the sequence it does not receive the updated catalog. Is there a way to link it

Interesting, ideally the same kedro session should give you the updated catalog. Let me confirm. Thank you

I tested within a KedroSession and the catalog does change. I haven't passed any extraparams or updated context

is it version 0.18?

sorry it is 0.19.10 but let me give a try with 0.18.

It is working the same way in 0.18. Tested with 0.18.8

Must be a problem in my end. Ended up writing a script to rewrite the yml 😆

Add a reply
Sign up and join the conversation on Slack