Python logging with the standard library.
Let's condense the basics of the logging Python module.
tldr
- If you don't configure any logger, only the log levels
WARNand up will show up. - In your program (not a library), set up the root logger as one of the first things you do.
Calling
logging.basicConfigdoes that for example. - In each module, have a global
logger = loging.getLogger(__name__)variable. - If you write a library, never add a handler to your loggers, because this is the consumer's job.
Either don't add any handlers, or use a
NullHandler- see here
Logging module basics
In a nutshell:
-
A logger can have:
- handlers: a log handler is responible for the output of a log record.
A logger can have several handlers: e.g., it can log to console and a file.
Each handler can be configured differently, e.g. regarding the log level.
If you define a custom handler, you need to implement the
emit(record)method that outputs a log record. - formatters: a handler can have a formatter that defines how a log record should be formatted.
- filters: a handler (or a logger) can filter messages and only emits log records that pass the filter. You can simply add a filter function that takes a record (string) and returns zero or one.
- handlers: a log handler is responible for the output of a log record.
A logger can have several handlers: e.g., it can log to console and a file.
Each handler can be configured differently, e.g. regarding the log level.
If you define a custom handler, you need to implement the
-
Loggers follow a hierachy. When a child logger receives a log record it first passes it to its own handlers. Per default, log records are then passed to any parent loggers. This is important for configuring the loggers. The hierarchy can be established in two ways:
- By configuring the root logger.
This is done by e.g. calling
logging.basicConfig. Any logger that you create withlogging.getLogger('my_name')will be a child of the root logger. Note: callinglogging.basicConfigif the root logger is already configured will have no effect. - By establishing a logger hierarchy via dot notation:
main.child, where themainlogger would be the parent of thechildlogger. This way, we could define a main logger in our main module and configure it there:main_logger = logging.getLogger("main")And in each submodule we explicitly create a child logger:library_logger = logging.getLogger("main.library"). Now, in tutorials and the Python docs you always see the recommendation to uselogging.getLogger(__name__)in all sub modules. This makes a lot of sense, if you have a package structure, because then the logging hierarchy is automatically configured:If we use a module as an entry point (i.e., we call it as something likemy_package my_package/__init__.py -> configure the root logger here my_package/some_module.py -> Use `logger = logging.getLogger(__name__)`python my_package/start.py, set the logger manually using the file name so that it is not just logged as__main__:logger = logging.getLogger("my_package.start").
- By configuring the root logger.
This is done by e.g. calling
- Use a logger per module, not one per class.
If you need a logger in a class, initialize it automatically as
self.logger = logging.getLogger(f"{__name__}.{self.__class__.__name__}")or similar. Don't pass loggers around as parameters.
Tidbits
- You can pass additional information to logging calls.
See
LoggerAdaptersand theextrakeyword that allows to use additional parameters in the format string. - In larger projects, configure loggers via text configuration.
See the
logging.configdocumentation. Either uselogging.config.fileConfig(which uses theconfigparsermodule) with ini-style config files, or use a library to de-serialize a text format to a dictionary and then initialize the logger withlogging.config.dictConfig.