.. _lofarpipe-logging:

*******
Logging
*******

One of the major changes made by the LOFAR pipeline system to the
:mod:`~lofarpipe.cuisine` system was the introduction of logging using the
standard `Python logging module
<http://docs.python.org/library/logging.html>`_. All instances of recipes
derived from :class:`~lofarpipe.cuisine.WSRTrecipe` (in other words, every
recipe developed using the framework) has an associated logger (in fact, an
instance of :class:`~lofarpipe.support.pipelinelogging.SearchingLogger`
available as the attribute ``self.logger``), which supports the standard
logging methods: see the Python documentation for details. The logging system
is also available in much the same way on remote hosts using the pipeline's
:ref:`distribution system <lofarpipe-remotecommand>`.

Note that by default only messages of level :const:`logging.WARNING` and
higher are logged. Use of the ``-v`` or ``--verbose`` flag the command line
will log messages at level :const:`logging.INFO`; ``-d`` or ``--debug`` at
level :const:`logging.DEBUG`.

Logs are output to standard output and also to a file. By default, the file is
located in the ``job_directory``, but this, together with the format used for
logging, may be configured through the :ref:`configuration file <config-file>`
if required.

The :mod:`lofarpipe.support` module provides a number of helper functions for
working with logs, which are documented here.

Searching logs
==============

Sometimes, it is convenient to be able to keep track of messages sent to a
logger. For example, pipeline tools may send metadata to the log rather than
output it in any other, more useful, fashion.

As mentioned above, all recipes have an associated instance of
:class:`~lofarpipe.support.pipelinelogging.SearchingLogger`. This can have any
number of regular expression patterns defined, and it will then store for
later reference any log entries which match those patterns. For example, a
recipe could include the code:

.. code-block:: python

   self.logger.searchpatterns["new_pattern"] = "A log entry"

This would record all log entries matching "A log entry". Later, a list of all
those entries can be retrieved:

.. code-block:: python

   matches = self.logger.searchpatterns["new_pattern"].results
   self.logger.searchpatterns.clear()

Note that messages generated by all subsidiary loggers -- including those on
remote hosts -- are included.  The call to
:meth:`~lofarpipe.support.pipelinelogging.SearchPatterns.clear` simply
instructs the logger to stop searching for that pattern, to avoid incurring
overhead in future.

.. autoclass:: lofarpipe.support.pipelinelogging.SearchingLogger
   :show-inheritance:

.. autoclass:: lofarpipe.support.pipelinelogging.SearchPattern

.. autoclass:: lofarpipe.support.pipelinelogging.SearchPatterns
   :show-inheritance:

.. autofunction:: lofarpipe.support.pipelinelogging.getSearchingLogger

Logging process output
======================

Many pipeline recipes run external executables. These tools may produce useful
logging output, either by writing to ``stdout`` (or ``stderr``), or by using a
library such as `log4cplus <http://log4cplus.sourceforge.net/>`_ or `log4cxx
<http://logging.apache.org/log4cxx/>`_. The framework makes it possible to
ingest that output and re-route it through the standard pipeline logging
system.

Standard output/error
---------------------

.. autofunction:: lofarpipe.support.pipelinelogging.log_process_output

Logging libraries
-----------------

The output from ``log4cplus`` or ``log4cxx`` is currently intercepted by
simply redirecting it to a file and then logging the contents of that file as
it is updated via the
:func:`~lofarpipe.support.pipelinelogging.log_file` function.

.. autoclass:: lofarpipe.support.pipelinelogging.LogCatcher

.. autoclass:: lofarpipe.support.pipelinelogging.CatchLog4CPlus
   :show-inheritance:

.. autoclass:: lofarpipe.support.pipelinelogging.CatchLog4CXX
   :show-inheritance:

.. autofunction:: lofarpipe.support.pipelinelogging.log_file

Logging resource usage
======================

This is a decorator which makes it easy to log the amount of time (wall and
CPU) used by parts of a pipeline.

.. autofunction:: lofarpipe.support.pipelinelogging.log_time