Join the Kedro community

Updated 12 hours ago

Another Question From My Side

Another question from my side.

I have a node which outputs a dictionary called train_test_dts which I am saving as a pickle with the backend joblib.
When I then try to run my pipeline with the parallel-runner like this:

kedro run --pipeline feature_engineering --params env=dev,inference_dt=2025-01-05 --runner ParallelRunner
Then I am getting the following error:
AttributeError: The following datasets cannot be used with multiprocessing: ['train_test_dts']
In order to utilize multiprocessing you need to make sure all datasets are serialisable, i.e. datasets should not make use of lambda functions, nested functions, closures etc.
If you are using custom decorators ensure they are correctly decorated using functools.wraps().

Any idea why that happens and what I could do to fix that?

P
1 comment

Solved. Apparently one cannot use input from GCS for these steps. Changed the path away from GCS to local folder, and it worked.

Add a reply
Sign up and join the conversation on Slack