Join the Kedro community

M
M
M
D
M
Members
Thomas d'Hooghe
T
Thomas d'Hooghe
Offline, last seen 22 hours ago
Joined September 25, 2024

Hi all,

When running uv run kedro run, the node in blue gets run before running the nodes upstream, while these are input for the blue node (it basicually unions two datasets back together). I would not expect this behavior, as I thought the entire pipeline should be executed as a DAG? Am I wrong in this assumption here? I have the following pipelines: ingestion, data_prep, feature, model_input, modeling and reporting .

7 comments
L
T

Hi all!

I am working with a clustering pipeline that I regularly want to rerun to monitor cluster migrations. I am using SnowflakeTableDatasets to save data directly to the data warehouse. Now, since it is not possible to have the same input and output dataset in Kedro, I was wondering what would be best practice to rerun clustering and store to the same SnowparkTableDataset when storing on a different timestamp for example. Would appreciate your help here!

5 comments
R
T
m

Hi all!

I was wondering if it is possible to get the name of a node while running inside a node. Use case I want to use it for writing a decorator around nodes that stores statistics on data processing, and I want to pass the name of the node to that as well, such that I can create a nice dataset 🙂

Cheers,
Thomas

10 comments
1
J
T
V
m
N