Join the Kedro community

Home
Members
Paul Weiss
P
Paul Weiss
Offline, last seen 2 hours ago
Joined September 12, 2024

Has anyone successfully implemented a custom expectation for the use with kedro-expectations? When I copy an example of a custom exception (https://github.com/great-expectations/great_expectations/blob/develop/contrib/expe[…]erimental/expectations/expect_multicolumn_values_to_be_equal.py) to gx/plugins/expectations, gx is not able to find it and throws an exception.

2 comments
N
P

Hi everyone! Has anyone implemented a customer log handler and successfully configured it in logging.yml? I'm getting a "No module named 'myproject.handlers'" error. I guess the logging is instantiated at a point where the project hasn't been loaded yet. Any idea how to get a custom logger running?

6 comments
D
P

What is the right place to validate command line parameters to kedro run? Let's say I want to check if parameter A has been passed (kedro run --params=a=1) if "env" is local and parameter C exists when env is "prod". It would be great if the process just stops with an error message and not a stack trace.

1 comment
d

Hi all! The Kedro documentation has a nice example of how to validate data with great expectations. But it only looks at one dataset at a time. But what would I do if I need to validate the data of a node that merges two datasets? Let's say one table is a lookup table and the other table may only contain entries that exists in the lookup table? Has anyone every checked multiple datasets at a time? Do you have an example for that?

5 comments
N
d

Hi all! I have an issue with credentials not being inserted into the dataset configuration:

credentials.yml
my_connection_string: "<connection>"<br /><br />catalog.yml<br />table1:<br /> type: pandas.CSVDataSet<br /> filepath: <a target="_blank" rel="noopener noreferrer">abfs://my_container/table1.csv</a><br /> credentials:<br /> connection_string: my_connection_string<br /><br />I get "unable to connect to account for Connection string is either blank or malformed.." errors. I guess "my_connection_string" is used as the connection string not "<connection>".

3 comments
P
N

Hi all, when providing an abfs:// path (Azure Data Lake Storage) in the catalog, what credentials exactly do I have to provide? An example would be really helpful.