Hi team, are there any best practices for optimizing spark code within Kedro pipelines? I have a large pipeline that executes at the last node due to lazy eval. I would like to look at execution plans, etc.
Any suggestions? I suppose this would apply to Polars/Ibis/other similar frameworks.
This is up to the execution engine - i.e. Polars / Spark is gonna to have completely different execution plan. Ibis is different in that catagory
