Hello, is there a way to get Kedro to create folders in paths if they do not exist? For instance if my data structure is
data:
outputs:
And I have the catalog entrydata@pandas: type: pandas.CSVDataset filepath: data/outputs/csv_files/data.csvIt would be nice for Kedro to automatically create
csv_files inside data and store data.csv afterwards.{'non_existent_subfolder/file_name': data} for being saved in the catalog entrydata@PartitionedDataset:
type: partitions.PartitionedDataset
path: data/outputs
dataset:
type: pandas.ExcelDataset
save_args:
sheet_name: Sheet1
load_args:
sheet_name: Sheet1
filename_suffix: ".xlsx"It would be nice for Kedro to create 'non_existen_subfolder' automatically inside data/outputs . I already tryed it and Kedro does not creates folders when they don't exist. Is there a way of changing this default behaviour?It would be nice for Kedro to automatically createPretty sure this is the case for many years, did you get some error?csv_filesinside data and storedata.csvafterwards.
@Nok yes! The error I get is that Kedro could not find the path data/outputs/non_existen_subfolder
@Nok specifically i get FileNotFoundError: [Errno 2] No such file or directory: my node is a generator and the data catalog output is a partitioned dataset:
def generator_node(df):
for idx, row in df.iterows():
....
yield {f'non_existen_subfolder/{file_name}':data}And my catalog entry goes like thispartitioned_data:
type: partitions.PartitionedDataset
path: data/outputs
dataset:
type: kedro_pamflow.datasets.custom_dataset.CustomDataset {file_name:dada} instead of {f'non_existen_subfolder/{file_name}':data}@Nok I already solved it managing it handling it in the _save method in my custom class 😄 thx