Join the Kedro community
A
M
M
M
D
Hello Team!
So it's been a few months since we started using kedro and it's time to deploy some of the pipelines we have created.
We need to choose an orchestrator but this is not our field of expertise, so I wanted to ask for some help. We would like something simple to setup and use collaboratively. Also my company requires it is free (at least for now), our cloud provider is AWS and we already use mlflow. Here are the alternatives we found:
hello !
disclaimer: I'm not an expert in any of these tools (alternative way of saying "it depends" 🙂)
when you talk about the limitations of the various free tiers, I guess those apply to the corresponding Cloud/Hosted option, right? taking the example of Prefect, to the best of my knowledge Prefect OSS doesn't have any limitations. the free tier of Prefect Cloud does, though (max of 5 000 runs/day). I guess something like that applies to Flyte vs Union, or Airflow vs Amazon MWAA (yes, AWS offers a managed Airflow service).
if you intend to <i>operate</i> the orchestrator yourself, then you're free to choose from the different OSS options. what do you want out of an orchestrator? given that your business logic (hence CPU-bound tasks) will live in the Kedro pipelines themselves, probably you'll want to pick a simple orchestrator that dispatches tasks, centralizes logs, displays execution status (and in my personal opinion, you don't need Kubernetes for that).
tl;dr: think carefully whether you want to operate your orchestrator yourself, or use some managed service.
then, you need to think <i>where</i> your pipelines will run. considering that your cloud provider is AWS, you'll be looking at Amazon EC2, Amazon ECS, etc. definitely <i>not</i> the same hardware where your orchestrator lives, otherwise you risk taking it down accidentally!
again taking Prefect as an example, looks like prefect-aws allows you to deploy your flows on ECS.
the most tried and tested orchestrator out there is Airflow, and there's an official Kedro plugin for it. but building Kedro translators isn't really a terribly difficult task, just see the code snippet in our docs that translates pipelines to Prefect for example.
I'll let others comment on their specific experiences 👂
Hi and thanks for your reply!
to the best of my knowledge Prefect OSS doesn't have any limitations. the free tier of Prefect Cloud does, though (max of 5 000 runs/day)I missed that, thanks for pointing it out!
if you intend to <i>operate</i> the orchestrator yourself, then you're free to choose from the different OSS options. what do you want out of an orchestrator? given that your business logic (hence CPU-bound tasks) will live in the Kedro pipelines themselves, probably you'll want to pick a simple orchestrator that dispatches tasks, centralizes logs, displays execution status (and in my personal opinion, you don't need Kubernetes for that).Indeed, I think this is what I need. Also, I wonder if it could allow me to:
prefect and airflow documentations, thanks! :)