The Mysterious Kedro-mlflow Behavior

Hey @Martin Gonzalez, would be helpful if you could share some error logs/screenshots or a repo with which I could reproduce this?

@Ankita Katiyar

@Ankita Katiyar It seems to be a problem with the parameters given to the node through the pipeline configuration file. When the node does not receive as input the parameters it does not generate two runs

Could you explain a bit more about the parameters? They’re not coming from the conf/base/parameters.yml but somewhere else?

node(
    func=fit_xgboost,
    name='fit_xgboost',
    inputs=['X_train', 'y_train', 'params:xgboost_conf'],
    outputs='XGBRegressor',
)

So if there’s a params:something in the pipeline inputs, it’s creating a spurious run but not otherwise?

yes

Okay, I’ll try this out and get back to you. CCing @Yolan Honoré-Rougé incase you have any insights

Hi can you give your kedro /mlflow /kedro mlflow versions?

Can you try downgrading each one by one ?

i have the same issue

kedro==0.19.11
kedro-mlflow==0.14.0
mlflow==2.20.1

kedro==0.19.10

downgrading kedro to 0.19.10 solves the issue

Wow very surprising, this type of error usually comes from mlflow or kedro-mlflow, not kedro.

Thanks for the info

The bug lies somewhere here : https://github.com/kedro-org/kedro/compare/0.19.10...0.19.11, but I don't see anything major so this is likely unexpected. It seems to affect the hooks workflow

Three related yet different bugs in the last 2 days on kedro-mlflow: https://github.com/Galileo-Galilei/kedro-mlflow/issues/624, https://github.com/Galileo-Galilei/kedro-mlflow/issues/623, https://github.com/Galileo-Galilei/kedro-mlflow/issues/622

@Ankita Katiyar I strongly suspect that the runner exectuor messes up pluggy https://github.com/kedro-org/kedro/compare/0.19.10...0.19.11#diff-6cfc8b43afcb8bfb74e8fe5f8ce1403285a191edc6abf63d763e000c572d8c5f

For those having the issue, can you confirm you use the standard runner, and not ThreadRunner or ParallelRunner ?

https://github.com/kedro-org/kedro/blob/a064c469623d88734bc90ed7c889690d10535f15/kedro/runner/sequential_runner.py#L50-L53

For the record, the following tests are failing :

The pipeline does not seem to be properly closed on error, and metrics create new runs.

@Yolan Honoré-Rougé yes, I was using the sequential runner

Thanks for looking into this @Yolan Honoré-Rougé, i’ll try to figure out what’s going on on Kedro side today!

@Martin Gonzalez Can you try with pip install kedro-mlflow==0.14.3 ? It should be fixed