Hey there, I'm testing Kedro capabilities to work with DeltaLake. I have a Delta table that is going to be updated every day with new data, and some pipelines that need to recompute models daily. The table is pretty small now but the total data should be increase and might not fit in memory (load all the table and then filter it).
I'm currently using the pandas deltalake dataset.
what are my options in the future? beside pyspark