Kedro
Log in

Join the Kedro community

Updated 7 days ago

Kedro project structure considerations

GGalen Seilis
·

This is an open question to anyone who has experience using Kedro. How do you think about what should go into a single Kedro project vs what should be split into multiple Kedro projects.

For some point of comparison, here has been my thinking. These are not universal rules, but are rather tendencies I have noticed.

  • A Kedro project may be coupled to a business project. In which case, all artifacts pertaining to that project are defined and produced within it.
  • For hobby projects I tend to make them as small as possible; targeting a small set of questions to answer (like half-a-dozen or less).

How do you think about the scope of a Kedro project?

E
G
Y
5 comments
EElena Khaustova

Ultimately, the decision hinges on balancing the need for modularity with the complexity of managing multiple projects, though this is not only the case for Kedro projects.

Here are a few considerations:

  • Align the Kedro project scope with the business initiative. It should encapsulate all artifacts, data pipelines, and logic relevant to that specific goal, enabling easier collaboration and deployment.
  • Keep the scope small and target a specific problem or question. This makes the project manageable and easier to iterate on.
  • For large or cross-functional initiatives, consider splitting into multiple projects if:
  • Different teams or workflows are involved.
  • There are distinct, loosely coupled components.
  • Reusability of pipelines or code is a priority but it’s hard to manage dependencies within the same project.

Edited

GGalen Seilis

Thank you for sharing your take on this. I appreciate it. That all sounds good in the abstract.

YYury Fedotov

could you elaborate on the last point?

Consider splitting if reusability of pipelines or code is a priority

Isn’t it the inverse? The more you’re interested in reusing things, the more you want a single large Kedro project?

EElena Khaustova

, I forgot to mention the last bit about dependencies in the sentence - thanks for pointing

YYury Fedotov

Oh, I see. Thanks!

Add a reply
Sign up and join the conversation on Slack