Guys, could someone help with using KedroContext properly?
I want to add a --only-missing CLI parameter to kedro run so that it runs pipelines using the run_only_missing method. From what I understand, adding this parameter to the default CLI was rejected because it can be implemented via KedroContext customization.
However, I’m not sure how to do this correctly. Or maybe I am missing something 😔
Could someone share an example or a code snippet because I don't see the usage of this class in the docs (e.g. here or here)?
Hey, @qwerty
You can inject extra parameters through the session store which are further passed to context. For that, you need to modify SESSION_STORE_CLASS in the settings.py
But I don’t think it can help with injecting --only-missing CLI parameter to kedro run . In session.run default runner.run method is used and what you need is to call runner.run_only_missing instead.
The easiest way to do it is by calling it from runner directly: https://docs.kedro.org/en/stable/nodes_and_pipelines/run_a_pipeline.html#run-pipelines-with-io
Thanks for the response @Elena Khaustova !
Is it possible to somehow override the default session.run? I mean I would like to have other parameters processed as-is and don't think it is a good idea to copy-paste the code from the default implementation but if there is some simple way to extend the default logic with something like
if config.only_missing: SequentialRunner().run_only_missing(...) else: # do usual stuff...it would be great
Hmm, unfortunately, we don’t provide any specific API to do this, I mean override session.run . You could do this by modifying source code though and building kedro from source.
To do it externally you’ll have to go from runner. Maybe you can extend your project with a command that calls kedro run if --only_missing is not provided and SequentialRunner().run_only_missing(...) otherwise.
