Join the Kedro community

Updated 2 months ago

Can same top-level keys exist in catalog.yml files in base and local directories

The omegaconfigloader scans the conf/ for subdirectories like base/ and local/ . Suppose , we have a catalog.yml file in both base/ and local/ .

Can we have the same top level keys in catalog.yml for base/ and local/ ??

D
M
V
7 comments

Yes, local (a special environment) will overwrite the common keys from base.

Hi I've seen you ask many questions this week, did you know about the Kedro Award Winning Docs? https://docs.kedro.org/en/stable/index.html They are super high quality and actually explain a lot of what you have been asking.

The functionality of config environments is described here: https://docs.kedro.org/en/stable/configuration/configuration_basics.html#configuration-environments and https://docs.kedro.org/en/stable/configuration/configuration_basics.html#configuration-loading

For some reason the below code throws error

import logging.config
from kedro.config import OmegaConfigLoader
from kedro.framework.project import settings
project_path = "/home/vishal/Documents/workspace/mlops/data-pipelines/"
conf_path = str(project_path + settings.CONF_SOURCE)
conf_loader = OmegaConfigLoader(conf_source=conf_path, env="local")
conf_catalog = conf_loader["catalog"]
conf_params = conf_loader["parameters"]

ValueError: Duplicate keys found in /home/vishal/Documents/workspace/mlops/data-pipelines/conf/base/catalog.yml and
/home/vishal/Documents/workspace/mlops/data-pipelines/conf/local/catalog.yml: case_data_insert_records,
case_data_update_records, case_master, doc_table, dummy_csv_resolver, dumm...

yes i am following the documentation carefully.

"However, when you use the OmegaConfigLoader directly, it assumes <i>no</i> environment."

Add a reply
Sign up and join the conversation on Slack