Visualizing Kedro Pipeline with Kedro Viz

would u be able to provide the longer traceback.

sure,

Starting Kedro Viz ...
[01/14/25 17:46:18] INFO     Using 'conf/logging.yml' as logging configuration. You can change this by setting the          __init__.py:270
                             KEDRO_LOGGING_CONFIG environment variable accordingly.                                                        
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro/io/catalog_config_resolver.py", line 98, in _fetch_credentials
    return credentials[credentials_name]
KeyError: 'adls_creds'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/anaconda/envs/gpy/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/anaconda/envs/gpy/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro_viz/server.py", line 121, in run_server
    load_and_populate_data(
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro_viz/server.py", line 59, in load_and_populate_data
    catalog, pipelines, session_store, stats_dict = kedro_data_loader.load_data(
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro_viz/integrations/kedro/data_loader.py", line 177, in load_data
    return _load_data_helper(
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro_viz/integrations/kedro/data_loader.py", line 106, in _load_data_helper
    catalog = context.catalog
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro/framework/context/context.py", line 190, in catalog
    return self._get_catalog()
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro/framework/context/context.py", line 234, in _get_catalog
    catalog: DataCatalog = settings.DATA_CATALOG_CLASS.from_config(
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro/io/data_catalog.py", line 285, in from_config
    config_resolver = CatalogConfigResolver(catalog, credentials)
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro/io/catalog_config_resolver.py", line 33, in __init__
    self._resolved_configs = self._resolve_config_credentials(config, credentials)
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro/io/catalog_config_resolver.py", line 258, in _resolve_config_credentials
    resolved_configs[ds_name] = self._resolve_credentials(
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro/io/catalog_config_resolver.py", line 130, in _resolve_credentials
    return {k: _resolve_value(k, v) for k, v in config.items()}
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro/io/catalog_config_resolver.py", line 130, in <dictcomp>
    return {k: _resolve_value(k, v) for k, v in config.items()}
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro/io/catalog_config_resolver.py", line 125, in _resolve_value
    return cls._fetch_credentials(value, credentials)
  File "/anaconda/envs/gpy/lib/python3.10/site-packages/kedro/io/catalog_config_resolver.py", line 100, in _fetch_credentials
    raise KeyError(
KeyError: "Unable to find credentials 'adls_creds': check your data catalog and credentials configuration. See <a target="_blank" rel="noopener noreferrer" href="https://kedro.readthedocs.io/en/stable/kedro.io.DataCatalog.html">https://kedro.readthedocs.io/en/stable/kedro.io.DataCatalog.html</a> for an example."

ipython seems to work normally as well, only kedro viz gives me this error

does kedro viz look for values in conf/credentials.yml? I did not define anything there

yes u r right, it looks for it there! https://docs.kedro.org/en/stable/configuration/credentials.html

any way to use the hooks instead?

from kedro.framework.hooks import hook_impl
from pyspark import SparkConf
from pyspark.sql import SparkSession
from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential

class AzureSecretsHook:
    def __init__(self):
        self.credentials = None

    @hook_impl
    def after_context_created(self, context) -> None:
        credentials = DefaultAzureCredential(exclude_managed_identity_credential=True)
        vault_url = 'xxx'
        client = SecretClient(vault_url=vault_url, credential=credentials)

        secrets = {
            "client_secret": "client-secret",
            "client_id": "client-id",
            "account_name": "storage-account-bld",
            "tenant_id": "tenant-id",
        }
        adls_creds = {
            "adls_creds":{
                cred_name: client.get_secret(secret_name).value
                for cred_name, secret_name in secrets.items()
            }
        }

        context.config_loader["credentials"] = {
            **context.config_loader["credentials"],
            **adls_creds,
        }
        self.credentials = context.config_loader["credentials"]

Yes you can use hooks to register secrets - https://docs.kedro.org/en/stable/hooks/common_use_cases.html#use-hooks-to-load-external-credentials

i think you will still need to have an empty credentials.yml file

i do have an empty credentials.yml file in locals/

@Gauthier Pierard, let me get back to you on this one. it seems like an issue with our CatalogConfigResolver. we will need to investigate this and revert

ok @Rashida Kanchwala, thanks

if there's a need to open an issue let me know

kedro viz run --include-hooks

thanks Ankita!! 🙂

i confirm. cheers