What is the cleanest way to run the entire pipeline multiple times?
I have a parameter, observed_date = '2024-10-01', defined in parameters.yml, that I use to run the pipeline. At the end of the pipeline, the output is saved to or replaced in a SQL table.
Now, I want to loop over this pipeline for every 5 day from Jan 2022 till October 2024.
Manually, this would require updating the parameters.yml file each time I want to change the date and rerun the pipeline (kedro run).
I don't want to introduce a loop directly into the pipeline, as it’s cleaner when observed_date is treated as a single date rather than a list of dates.
However, I’d like to find a clean way to loop over different dates, running kedro run for each date.
You could loop in a script and pass the observed_date as a parameter: kedro run --params observed date=<observed>
Thank you! I also just read about the parameters at run time capability. I should be able to perform a loop for the observed date I want.
https://docs.kedro.org/en/stable/configuration/parameters.html
