Another question from my side.
I have a node which outputs a dictionary called train_test_dts
which I am saving as a pickle with the backend joblib
.
When I then try to run my pipeline with the parallel-runner like this:
kedro run --pipeline feature_engineering --params env=dev,inference_dt=2025-01-05 --runner ParallelRunnerThen I am getting the following error:
AttributeError: The following datasets cannot be used with multiprocessing: ['train_test_dts'] In order to utilize multiprocessing you need to make sure all datasets are serialisable, i.e. datasets should not make use of lambda functions, nested functions, closures etc. If you are using custom decorators ensure they are correctly decorated using functools.wraps().
Solved. Apparently one cannot use input from GCS for these steps. Changed the path away from GCS to local folder, and it worked.