Join the Kedro community

Home
Members
Shu-Chun Wu
S
Shu-Chun Wu
Offline, last seen 2 days ago
Joined October 29, 2024

Is there a way to use ImageDataGenerator and flow_from_directory functions in kedro? I would like to save the dataset in memory and then use it later for model training, but I got the error message: DatasetError: Failed while saving data to data set MemoryDataset().

3 comments
S
J
Y

Hi folks,
We have our own MLFlow server on internal S3.
Below are the setting I used locally:

os.environ["MLFLOW_TRACKING_URI"] = "<a target="_blank" rel="noopener noreferrer" href="https://xxx.com/mlflow/">https://xxx.com/mlflow/</a>"
os.environ["MLFLOW_S3_ENDPOINT_URL"] = "<a target="_blank" rel="noopener noreferrer" href="http://s3xxx.com">http://s3xxx.com</a>"
os.environ["S3_BUCKET_PATH"] = "<a target="_blank" rel="noopener noreferrer" href="s3://xxx/mlflow">s3://xxx/mlflow</a>"
os.environ["AWS_ACCESS_KEY_ID"] = "xxx"
os.environ["AWS_SECRET_ACCESS_KEY"] = "xxx"
os.environ['MLFLOW_TRACKING_USERNAME'] = 'xxx'
os.environ['MLFLOW_TRACKING_PASSWORD'] = 'xxx'
os.environ["MLFLOW_TRACKING_SERVER_CERT_PATH"] = "C:\\xxx\\ca-bundle.crt"
EXPERIMENT_NAME = "ZeMC012"
In order to use in Kedro framework, I create a mlflow.yml file in conf/local folder and the content like this:
server: 
  mlflow_tracking_uri: <a target="_blank" rel="noopener noreferrer" href="https://xxx.com/mlflow/">https://xxx.com/mlflow/</a>
  MLFLOW_S3_ENDPOINT_URL: <a target="_blank" rel="noopener noreferrer" href="http://s3xxx.com">http://s3xxx.com</a>
  S3_BUCKET_PATH: <a target="_blank" rel="noopener noreferrer" href="s3://xxx/mlflow">s3://xxx/mlflow</a>
  AWS_ACCESS_KEY_ID: xxx
  AWS_SECRET_ACCESS_KEY: xxx
  MLFLOW_TRACKING_USERNAME: xxx
  MLFLOW_TRACKING_PASSWORD: xxx
  MLFLOW_EXPERIMENT_NAME: ZeMC012
  MLFLOW_TRACKING_SERVER_CERT_PATH: C:/xxx/ca-bundle.crt
But I got error ValidationError: 8 validation errors for KedroMlflowConfig
How should I modify it?

6 comments
D
S

Hello!!!
I currently have an images classification use case. I have 7 classes and save images for each class separately (one class one folder). Not I setup the catalog.yml like this:
"{class_name}_data":
type: partitions.PartitionedDataset
filepath: ../data/01_raw/B4CD/{class_name}
dataset:
type: pillow.ImageDataset
But when I use catalog.load('XXXX')
What should I write for 'XXXX'? {class_name}_data or I have to load each folder/class separately?

3 comments
A
S

Hi, I just want to run "kedro catalog list", then I got this error below:

NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher
Which library with which version should I install?

3 comments
S
L

I use vs code with my kedro project. When I open it, I got error message like this:
Traceback (most recent call last):
File "c:\Users\wushuchu\.vscode\extensions\kedro.kedro-0.2.2\bundled\tool\lsp_server.py", line 75, in <module><br /> import yaml<br />ModuleNotFoundError: No module named 'yaml'<br /><br />2024-10-30 14:51:13.157 [info] [<strong>Error</strong> - 2:51:13 PM] Server initialization failed.<br />2024-10-30 14:51:13.157 [info] Message: Pending response rejected since connection got disposed<br /> Code: -32097<br />2024-10-30 14:51:13.157 [info] [Info - 2:51:13 PM] Connection to server got closed. Server will restart.<br />2024-10-30 14:51:13.157 [info] true<br />2024-10-30 14:51:13.158 [info] [<strong>Error</strong> - 2:51:13 PM] Kedro client: couldn't create connection to server.<br /><br />But I have installed PyYAML==6.0.2 and also kedro extension in vs code.<br />Can anyone help here?

5 comments
d
N
S

Hi all, I tried to run this code "config_loader = OmegaConfigLoader(conf_source="conf/")", but I got this error message.
Any idea how could I solve it? 😟

6 comments
L
S