Join the Kedro community

Home
Members
Ben Shaughnessy
B
Ben Shaughnessy
Offline, last seen 3 days ago
Joined January 16, 2025

Can anyone suggest a good way of dynamically changing a catalog entries path? For example, by default I want to use local paths for my intermediate datasets, but when I deploy to production I don't want anything to be saved locally. Duplicating the catalog.yml in the conf/production/ folder is not ideal, as I will have to maintain two sets of each catalog entry.

31 comments
B
d
E
M

Can anyone suggest the best way to access:

  1. Catalog definition
  2. Pipeline definition

Before the pipeline runs, and ideally outside the normal kedro run life cycle?

Im trying to accomplish two very different things with this

  1. is trying to implicitly figure out which nodes depend on each other via memory datasets, to support using memory datasets in a distributed argo pipeline running a kedro pipeline
  2. generate documentation via a mermaid diagram that I can store in a readme file. Similar to kedro viz (but with some subtle key features)

4 comments
J
N

Hello, I've worked on a lot of Kedro pipelines the past ~year and am a big fan, but there's one detail that I've seen cause some very confusing problems that id like help with.

Whenever there's an error loading a pipeline, whether it be a syntax error, missing import, etc... instead of the Kedro process erroring out, it will just not use that pipeline and continue on without that pipeline. This is not only confusing, but can lead to some pretty big problems in a model without any errors occurring.

I was wondering how I disable this, forcing Kedro to raise errors when loading pipelines? I tried googling but couldn't find anything.

Thanks!

15 comments
D
B
N