Hi gang,
Is there any reasonable way to make asyncio and kedro work together? I have some external IO client that provides only async interface and I can't make it work from sync context, since there is already a running and managed asyncio loop somewhere in kedro. Do coroutines as kedro nodes make any sense?
I don't think there is an async loop in Kedro
--async in your runner CLI argsDo coroutines as kedro nodes make any sense?Not going to be very helpful here, but I randomly had somebody bring this up a few months back, that why couldn't Kedro just go all-in on
asyncio and coroutines for execution. But I don't know that anybody has actually tried it. If you do, let us know? 🙂yes @Deepyaman Datta I've also wondered that... It would make scheduling nodes executions and parallelism of data loading easy and out of the box
multiprocessing is really unreliable with speed ups and suitable only for some specific tasks that are parallel & indepdendent by nature, while threading has GIL problem
I am not super familiar with asyncio and am not aware if they managed to get around GIL via syncing with the await contracts - concurrency topic in python is very convoluted because of GIL