Hello everyone
I'm using kedro inside a notebook on Databricks (unfortunately I need to use notebooks and I don't have authorization to use the new kedro-databricks plugin yet). It turns out that I'm having a very bad experience with the logs. The notebook cell is constantly blinking and freezing, so I can't view the logs. Below is my log configuration file. Could anyone suggest a better log configuration for Databricks notebooks?
version: 1
disable_existing_loggers: False
formatters:
simple:
format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
handlers:
console:
class: logging.StreamHandler
level: INFO
formatter: simple
stream: <a target="_blank" rel="noopener noreferrer" href="ext://sys.stdout">ext://sys.stdout</a>
info_file_handler:
class: logging.handlers.RotatingFileHandler
level: INFO
formatter: simple
filename: info.log
maxBytes: 10485760 # 10MB
backupCount: 20
encoding: utf8
delay: True
rich:
class: kedro.logging.RichHandler
rich_tracebacks: True
loggers:
kedro:
level: INFO
grp_global:
level: INFO
root:
handlers: [console]Does changing the logging levels to WARNING improve the blinking/freezing issue? I'm thinking maybe the logging messages being sent too fast after each other might be contributing to this problem.
I'm thinking if there's any way to stagger the logging messages without having to write a custom handler. Does disabling rich logging improve anything?
https://docs.kedro.org/en/stable/logging/index.html#how-to-use-logging-without-the-rich-library
I have just found out that the KEDRO_LOGGING_CONFIG variable was not defined for my project. I will try to define it and evaluate again.
Thanks, Laura.
That worked. Now this is actually using the console handler, which no longer freezes the notebook.
But it's a bit weird. Do you know how to remove the [dark_orange] flag?
Eg:
INFO 2024-10-29 17:35:12,027: Loading data from [dark_orange]params:HMC[/dark_orange] (MemoryDataset)... - kedro.io.data_catalog