Best Practices for Kedro Usage

Hi Олег Литвинов, If you haven't seen this already, for complex projects kedro recommends you to use namespaces and modular pipelines as a good practice. Check modular pipelines and namespace docs for further information.

I will check this out, thank you! Would be happy to hear various options and experiences 🙂

An here is kedro-mlflow and its tutorial which is specifically design to address these issues: https://github.com/Galileo-Galilei/kedro-mlflow-tutorial

Think about it as "scikit learn like pipeline but for any arbitrary kedro pipeline"

It requires mlflow though

Also see this section in the documentation : https://kedro-mlflow.readthedocs.io/en/stable/source/05_pipeline_serving/index.html

Thank you very much, colleagues! I appreciate your ideas! Please, let me know if there are any other options to consider

Dear , thank you for sharing the docs. I see how this helps to establish good separation of preprocessing and modelling as well as training of two different models (via namespaces). However I still don't have any good idea of how to reuse, for example, the preprocessing logic\nodes\parameters during the model inference. Do you, probably, have some examples of such?

Dear , thank you for the tutorial. This makes a lot of sense and addresses my original question. Looks like the core idea here is to use tags, right? I see this example haven't been updated for a while. Is there any particular reason for that? Is this approach still considered as the best practice?

Hi, some answers:

• A namespace is a way to tag all the nodes of a pipeline, so both suggestions are very correlated
• Yes the key idea is to use tags because (using sklearn vocab) some steps are "fit" (e.g. create something from data to reuse at inference time - in mlflow vocab this is called an "artifact") and other steps are "transform" (e.g. apply a fitted object on data). You never want to do "fit_transform" because you need to separate the steps: you "fit" only at training time and you "transform" both at training time and at inference time
• Unfortunately the example has not been updated because the starter changed between 0.18 and 0.19 and I have to update all examples and screenshots and I never took the time, but it perfectly works in 0.19. There were no breaking change on pipeline and nodes between the two major versions.
• I don't know if it's "best practices" but given the number of related issues in this channel, in the kedro-mlflow repo issues and discussions (look for "pipeline_ml_factory" keyword to see them), and the numerous projects I've seen in production using it, I am quite confident this is considered a good approach (likely the best available at the time)

Thank you very much for a follow up! This sounds great.

In the meantime, I found a very similar issue mentioned here: https://github.com/kedro-org/kedro/issues/464. It looks like this issue\thread started somewhere there. This is a very useful discussion helping to frame some understanding. According to it, I see that model serving was outside of the interests of Kedro. But this was 4 years ago. So look like now it's pretty well covered and addresses the main inference goals. Thank you again!

Yes, the original poster contributed directly to the kedro mlflow code base back then

If you want deep control over pipeline serving, check out kedro-boot and the fastapi mapping

https://github.com/takikadiri/kedro-boot

After a couple days of investigating realised how similar tags and namespaces are. Is there any preferred way of using one or the other?

Hi ,

Both tags and namespaces help you group nodes in your kedro project and structure a complex project.

Inclusive Grouping : By this I mean nodes are part of more than one group, For example a kedro node can have multiple tags

• Tagging is inclusive
• Tagging cannot provide modularity
• Not suitable for deployment because of non-exclusivity
• Tagging is good when you want nodes belonging to more than one group

• Namespaces provide hierarchy. Hierarchy is exclusive
• Namespace provide modularity
• Most suitable for visualization and deployment
• Namespace is good when you want nodes belonging to one group