Log (logging) in Python

Logging is always an important topic when you try to develop a "real-world" app. I once wrote my own logging system, but apparently the best practice is to use something provided by Python Standard Library, not to reinvent the wheel.

Here, I have no intent to write a complete tutorial of teaching the python's logging framework. I extract the most useful/simple part(s) of logging from my viewpoints, which I believe it solves 90% cases I(you) need.


Firstly, you should try to build a customized logger. It is impractical to use the logging module directly via changing basicConfig() because you cannot tell where the log comes from if you have multiple modules in your app. This is one of the reason why I write this article.

To get a "customized" logger, firstly you should assign a name to it.

mlogging = logging.getLogger("test_log") #Get a logging for this module

If you would like to have some beautiful format on logging records, logging.Formatter is something you need to set it up. Sure you should do it because the default one seems to be a little clumsy. To change the date format (that is, %(asctime)s), please refer to things used in time.strftime(). And to create a format for a record, the related variables can be found in attributes of LogRecord.

# -- Set up Formatter --
log_format = '%(asctime)s(%(name)s--%(levelname)s):%(message)s'
log_date_format = '[%m/%d/%y %H:%M:%S]'
uniform_fmt = logging.Formatter(fmt=log_format, datefmt=log_date_format)

Mostly, we store logs in a file, so you need a file handler.

# -- Log to File --
hdlr = logging.FileHandler('OnlyOneFile.log')   # or "mlogger.name + 'log' if you would like to separate logs
hdlr.setFormatter(uniform_fmt)

If you would like to see log in your screen, say, for debugging, you have to link it to STDOUT.

# -- Log to Console --
hdlrConsole = logging.StreamHandler(sys.__stdout__)
hdlrConsole.setFormatter(uniform_fmt)

Do not forget to link your handler with your logger.

mlogging.addHandler(hdlr)
mlogging.addHandler(hdlrConsole)

Finally, you have to assign a log level so that you can get all the details. Otherwise, The default level is WARNING. (All default levels)

mlogging.setLevel(logging.DEBUG) #this is the most detailed level

Putting all together, here is an example:

import sys
import logging

mlogging = logging.getLogger("test_log") #Get a logging for this module
"""
logging.basicConfig(
    #filename=mlogging.name + '.log', #Use the name of logger as log file name
    filename='onefile.log', #Use the name of logger as log file name
    level=logging.DEBUG,
    format='%(asctime)s(%(name)s--%(levelname)s):%(message)s',
    datefmt='[%m/%d/%y %H:%M:%S]'
"""

# -- Set up Formatter --
log_format = '%(asctime)s(%(name)s--%(levelname)s):%(message)s'
log_date_format = '[%m/%d/%y %H:%M:%S]'
uniform_fmt = logging.Formatter(fmt=log_format, datefmt=log_date_format)

# -- Log to File --
hdlr = logging.FileHandler('OnlyOneFile.log')
hdlr.setFormatter(uniform_fmt)

# -- Log to Console --
hdlrConsole = logging.StreamHandler(sys.__stdout__)
hdlrConsole.setFormatter(uniform_fmt)

mlogging.addHandler(hdlr)
mlogging.addHandler(hdlrConsole)
mlogging.setLevel(logging.DEBUG)

mlogging.debug('This is debug msg')
mlogging.info('Ok~ info now')
mlogging.warning('Warning here')

try:
    x = 4/0
except Exception as e:
    mlogging.exception(e)

Before we leave the discussion, you should pay attention to mlogging.exception(e). This is the way we log an exception, including its trace-back information.

Intall Python 3 on Ubuntu

0. Install related packages:
sudo apt-get install build-essential libncursesw5-dev libreadline5-dev libssl-dev libgdbm-dev libbz2-dev libc6-dev libsqlite3-dev tk-dev g++ gcc

1. Download the file to /tmp:
cd /tmp
wget http://www.python.org/ftp/python/3.1.1/Python-3.1.1.tgz

2. Unzip the tar zip file:
tar xzvf Python-3.1.1.tgz

3. Run configuration command ./configure
* In case you encounter some problem, install build-essential

4. make (optional)

5. Install the program to system:
sudo make install

Appending to Your Python Path

Question:

How do you append directories to your Python path?

Answer:

Your path (i.e. the list of directories Python goes through to search for modules and files) is stored in the path attribute of the sys module. Since path is a list, you can use the append method to add new directories to the path.

For instance, to add the directory /home/me/mypy to the path, just do:

    import sys
    sys.path.append("/home/me/mypy") 

src: Here

Free Python 3.x Tutorial

This article shall be constantly updated to collect useful resource for learning python 3.x .

1. Dive into Python 3
   
   I didn't buy this book because while I start this article, it hasn't be published yet, but definitely this is a very very good book. It's much better than the book I bought from amazon, Programming in Python 3 by Mark Summerfield. The book I bought is too difficult to understand, and the only reason I bought it is that it's one of the earliest book published for python 3.x. I'm still waiting for the publication of the Dive into Python 3, and I believe this book is better than the one I owned now.
   


2. Official Python v3.1 documentation
   
   I believe that you can find all the update information from here. Anyway, it's something that you would always check for answers.
   

Collection Types in Python 3

I would like to document some useful snippet for handling collections in Python 3.x.
set:

To add item to a set:

s.add(x)

To check whether item is in a set:

x in s #return True if set s contains item x

To remove item from a set:

s.remove(x)

Randomly picks n lines in a text file

I try to google, but I cannot find the easy way to do this in command. Often times, I would like to randomly sample (without duplicate) lines in a text file. Here is my python code.

#!/bin/python3
# The goal of this script is to randomly choose n lines from a text files by line without duplicate

import sys, os, pdb, random

#The prefix will add to the front of each line
prefix = ""

if len(sys.argv) != 3 and len(sys.argv) != 4:
    print("syntax: randomlines.py <input_file> <num_of_sample>")
    sys.exit()
else:
    input_file = sys.argv[1]
    num_of_sample = int(sys.argv[2])

if len(sys.argv) == 4:
    prefix = sys.argv[3]

#pdb.set_trace()
lines = open(input_file, "r").read().splitlines()
if len(lines) < num_of_sample:
    print("The number of random lines you asked for is larger than the lines in the target file.\n" +
        "Shrink it to lines of target file automatically.\n" +
        "-----------------------------------------------\n")
    num_of_sample = len(lines)

chosen_lines_num = random.sample(range(0,len(lines)), num_of_sample)   #range(0,3) only generate 0,1,2
for i in chosen_lines_num:
    print(prefix + lines[i])

Sorting Python Dictionary by Value

This is a slower version:

sorted(adict.items(), key=lambda (k,v): v)

This is a faster version:

from operator import itemgetter
sorted(d.items(), key=itemgetter(1))

source: here

However, what if value is a list, and we would like to sort by the items inside the list?

For example, we have a dictionary variable x:

>>> x
{'a': [1, 2, 3], 'b': [0, 3, 1]}

We can sort them by the following command

>>> sorted(x.items(), key = (lambda k: k[0])) #sort by key
[('a', [1, 2, 3]), ('b', [0, 3, 1])]
>>> sorted(x.items(), key = (lambda k: k[1])) #sort by value
[('b', [0, 3, 1]), ('a', [1, 2, 3])]
>>> sorted(x.items(), key = (lambda k: k[1][0])) #sort by first item in value(a list)
[('b', [0, 3, 1]), ('a', [1, 2, 3])]
>>> sorted(x.items(), key = (lambda k: k[1][1])) #sort by second item in value(a list)
[('a', [1, 2, 3]), ('b', [0, 3, 1])]