Testing pipeline with spark object output and memory dataset configuration

So you can use Kedro this way, but it's not actually the way we recommend unless you have a specific reason to do so.

I would really recommend that you follow the Spaceflights tutorial since it covers the key concepts and abstracts some of this complexity

we also have a full training course on YouTube :youtube:
https://www.youtube.com/playlist?list=PL-JJgymPjK5LddZXbIzp9LWurkLGgB-nY

But this is for integration tests for pipelines

ah gotcha!

that falls into a good reason

This is the recommended way in the Kedro documentation to write pipeline tests: https://docs.kedro.org/en/stable/tutorial/test_a_project.html

give me sec

No worries take your time I'd appreciate any help I can get

You might be able to get some inspiration from the Kedro code base tests!

Let me see if I can find a good example

Oh that's a great idea actually, I'm having Friday brain!

This is a little old - but relevant
https://github.com/kedro-org/kedro/discussions/1068

I've been desperate to get a kedro-test micro-framework off the ground but it's been hard to prioritise

If we end up using Kedro we might be interested in doing some OSS contributions with it so could maybe help

Maybe this one helps? https://github.com/kedro-org/kedro/blob/main/tests/pipeline/test_pipeline_integration.py

hmmmm mine is similar but I'm having the issue that I don't know how to specify that the output of 1 pipeline should use copy_mode "assign"

I guess something like:

dataset = MemoryDataset({"data": 42}, copy_mode="assign")
DataCatalog().add_feed_dict({"dataset":dataset})

Yes indeed, it's:

catalog = DataCatalog(
datasets={
"data_utility": MemoryDataset(copy_mode="assign"),
"extract_model_features": MemoryDataset(copy_mode="assign"),
},
)

I got it working now thanks!