Hi,
let's say I have this simple SQLQueryDataset
table_output:
type: pandas.SQLQueryDataset
sql: >
select * from dev.outputparameters.ymlmode: 'ABC'
table_output to filter itself based on parameter modetable_output will be filtered based on the 'ABC' paramstable_output:
type: pandas.SQLQueryDataset
sql: >
select * from dev.output
where mode = ?
load_args:
params:
- 'ABC'parameters.yml as part of the table_output definition?table_output:
type: pandas.SQLQueryDataset
sql: >
select * from dev.output
where mode = ?
load_args:
params:
- ${parameters.mode} # to reference parameters in parameters.yml?Hey, you can use globals - https://docs.kedro.org/en/stable/configuration/advanced_configuration.html#how-to-use-global-variables-with-the-omegaconfigloader
@Ankita Katiyar thanks! globals.yml seems to be what I need. I'm just curious as to why it's not used as the default place to hold parameters definition, given that it's accessible across all config files, which I think very useful in many cases.
The only thing now is that I have to redefine my parameters in parameters.yml so that it resolves from globals.yml, which seems to be like a double definition.
It's mostly for separation. parameters.yml are meant to be node argument, something that get passed into your node function. Where the parameters.mode here are acting more like a template variable.
You could also move this mode to catalog.yml
table_output:
type: pandas.SQLQueryDataset
sql: >
select * from dev.output
where mode = ${_mode}
load_args:
params:
- ${_mode} # to reference parameters in parameters.yml?
_mode: "some_mode"globals (the anti-pattern here is use globals.yml for everything)is it possible to do something similar with a namespaced dataset?{mycountry}.table_output: type: pandas.SQLQueryDataset sql: > select * from mytable.output where country = ${mycountry} load_args: params: - ${{mycountry}.name}
This can be done with factories, though I think it is built for a slightly different purpose.
Yes you are right. You can use the namedspace values e.g. {mycountry} directly in the params or filepath etc. but you can't use it as a key value.
For example if in params you had France: short_name: FR