Kedro
Log in

Join the Kedro community

Updated 2 months ago

Are there any current efforts to create a plug-in that more consistently integrates DVC into Kedro?

At a glance

The community member is interested in exploring a plugin to better integrate Kedro and DVC, as the current workflow has limitations such as DVC not being able to use Kedro's catalog aliases or handle environments. The community member suggests a "brute force" approach of generating the dvc.yaml file for all environments, but is curious if there are more elegant solutions. Another community member responds that they have been exploring Kedro + DVC integration as part of research on Kedro's versioning, and thinks a plugin is a good idea that will be considered. They offer to help if the original poster wants to build something. There is also a link provided to a related discussion on the Kedro GitHub repository.

Useful resources
DDylan F
·

Watched the Kedro+DVC video. Was great :) I am curious if there is a current effort around a plug-in to marry DVC into Kedro more consistently? The main drawback of the current workflow is DVC can't use the catalog alias' and also therefore can't handle environments. The brute force way would be just writing a script that generated the dvc.yaml for all environments with the literal paths which is doable. But wanted to see if anyone has thought of other more elegant ways to merge the two.

A
2 comments
AAnkita Katiyar

Hey Dylan, I was exploring the integration of Kedro + DVC as part of our research around improving Kedro’s versioning. The presentation was to show how the integration works as of now. I think a plugin is a good idea and something we’ll consider properly in the discussions around versioning but if you wanted to build something, that would also be great! I can help out 🙂

AAnkita Katiyar

Some discussion here - https://github.com/kedro-org/kedro/issues/4239

Add a reply
Sign up and join the conversation on Slack