Parallelrunner and dataset error

Question

Hey team I am facing an issue that when using ParallelRunner. Basically I am trying to have model-training for many different use-cases (tabular data) parallelized through ParallelRunner, though the problem that I am facing is

DatasetError: Data for MemoryDataset has not been saved yet.

Is that something that you have seen before?

Rashida Kanchwala · Answer

it looks like a node requires a MemoryDataset as input, and that MemoryDataset hasn't been created yet.

Paul Mora · Answer

Yes haha - that I also understood from there

Paul Mora · Answer

But it is working when using SequentialRunner

Paul Mora · Answer

And ThreadRunner

Paul Mora · Answer

But this error comes when using ParalleRunner, suggesting that its running nodes in the wrong order

Huong Nguyen · Answer

hey  @Paul Mora  i guess this possibly happened in ParalleRunner if dependencies are not managed/defined in the right way. Could you share how you define your pipeline and perhaps data catalog as well please?

Paul Mora · Answer

@Melvin Kok  agreed kindly to help with providing this Huong 🙂

Huong Nguyen · Answer

perfect thank you,  @Melvin Kok  when you have the time, feel free to DM me 😄

Join the Kedro community

Parallelrunner and dataset error