Hey team, hope you are doing well. I have the following question (I already tried to see whether any previous question answers it, with no luck)
I have primary data paths such as:
"prm_customer_base":
table_name: primary_${_environment}.prm_customer_base
<<: *_conn--env that I am running. So, I created a file under conf/dev/catalog_dev.yml which contains_environment: dev
conf/dev/globals.yml (+ removing the underscore then ofc). That seems to work, though I am not sure how to feel having a globals.yml file for each environment, since I was thinking having multiple globals is defeating the point.AFAIK catalog parameters must be defined in the same file where they are used or in globals. So using one globals.yml per env is the correct and best way for your problem (assuming "prm_customer_base" is defined in the base catalog).
The problem that I am facing now though is that the globals are entirely overwriting themselves.
Hence for example I have in conf/dev/globals.yml
environment: dev
conf/base/globals.ymltest_size: 0.8
--env dev I am losing the test_size parameter, which is quite a shame..Further the docs make me think that globals is not the only way to achieve what we are trying to do: https://docs.kedro.org/en/stable/configuration/advanced_configuration.html
Decided in the end against the entire globals thing and went with runtime_params which seems to make more sense for environment introduced path changes
Yes, that I know. But what happened if I have two globals (in different environments) then they are not merging, but when taking --env=dev for example, then the globals of base it not used anymore.
In the end I used of runtime-params, which seems like the more straight forward approach 🙂
Apologies as I view this on phone and the ${_environment} spill into a newline I thought it s a key.
I think using runtime_params is a good approach here. Though I don't understand why it gives you an interpolation error. It should be a valid config.
Though the interpolation does not work, and I get the error that the interpolation key is not found.