Simon Krueger

Logging django apps to syslog

I spent most of my day filling in the holes of my django/python logging knowledge. I've never done logging in python or a django app before. And thus our code base, had zero logging in it. A huge problem when it comes time to debug issues in production. I read a few articles that I found at Full Stack Python and also went through the Python doc logging tutorials. Taking Some Pain out of Python Logging was an eye opener when Hynek explained that syslog can capture and record the python log records. Syslog was exactly what I set out to do (I didn't know it was called syslog. I just knew about /var/logs, so today was a good learning experience). I am not doing exactly what Hynek does. But it accomplishes my goal of getting our logs into syslog.

Here is the final LOGGING dictionary in settings.py I went with.

# settings.py
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse',
        },
        'require_debug_true': {
            '()': 'django.utils.log.RequireDebugTrue',
        },
    },
    'formatters': {
        'verbose': {
            'format': '%(process)-5d %(thread)d %(name)-50s %(levelname)-8s %(message)s'
        },
        'simple': {
            'format': '[%(asctime)s] %(name)s %(levelname)s %(message)s',
            'datefmt': '%d/%b/%Y %H:%M:%S'
        },
    },
    'handlers': {
        'console': {
            'level': 'DEBUG',
            'filters': ['require_debug_true'],
            'class': 'logging.StreamHandler',
            'formatter': 'simple'
        },
        'syslog': {
         'level': 'DEBUG',
         'class': 'logging.handlers.SysLogHandler',
         'facility': 'local7',
         'address': '/dev/log',
         'formatter': 'verbose'
       },
    },
    'loggers': {
        # root logger
        '':{
            'handlers': ['console', 'syslog'],
            'level': 'INFO',
            'disabled': False
        },
        'thingsforwork': {
            'handlers': ['console', 'syslog'],
            'level': 'DEBUG',
            'propagate': False,
        },
    },
}

Ubuntu 14.04 uses the rocket-fast system for log processing (rsyslog) and we need to tell it about our local7 facility. I got this one from serverfault.

# /etc/rsyslog.d/30-thingsforwork.conf
local7.*                                             /var/log/thingsforwork.log
& ~  # This stops local7.* from going anywhere else.

Do a sudo service rsyslog restart so rsyslog.d can pickup the new config. And ta-da!

# /var/log/thingsforwork.log
May 28 03:19:57 vagrant 11874 140156620121856 thingsforwork.dashboard.views.home                 DEBUG    we are in the home page, this is a debug
May 28 03:19:57 vagrant 11874 140156620121856 thingsforwork.dashboard.views.home                 INFO     We are logging django! e01be3ae-e9d1-49d7-97ac-00c5454f3026
May 28 03:20:02 vagrant 11874 140156620121856 requests.packages.urllib3.connectionpool           INFO     Starting new HTTP connection (1): dashboard.reviewpush.com
May 28 03:20:02 vagrant 11874 140156605413120 requests.packages.urllib3.connectionpool           INFO     Starting new HTTPS connection (1): tartan.plaid.com
May 28 03:20:02 vagrant 11874 140156392699648 googleapiclient.discovery                          INFO     URL being requested: GET https://www.googleapis.com/discovery/v1/apis/analytics/v3/rest
May 28 03:20:02 vagrant 11874 140156375914240 requests.packages.urllib3.connectionpool           INFO     Starting new HTTPS connection (1): quickbooks.api.intuit.com