Python module for processing a file line-by-line

Note: Since writing this post, I’ve learned about the fileinput module, which turns most of the following into a oneliner:

import fileinput
for line in fileinput.input():
    process(line)

It works both with stuff you pipe into the program, or if you use a filename as argument.

Read on for my old post

In this post I’ll show how you could process a file, line-by-line with some “plugable” Python code (see code listings below):

cat some_file | line_by_line.py   ...

In the example above, a “line proc module” is any python module that contains a function called proc_line that takes a string, and returns another string. Such as the following:

# Example of proc_line
def proc_line(line):
    return line # Hint: Do something to the line before returning

Example: Uppercasing

In this example the user has written a “line proc module” called “uppercase” (see code listings below):

$ echo "bla bla bla" | ./line_by_line.py uppercase
BLA BLA BLA

Example: Chaining

Here the user has written two “line proc modules”, and chained them together (see code listings below):

echo "bla bla bla" | ./line_by_line.py uppercase leet
BL4 BL4 BL4

First “uppercase” is applied, then “leet” is applied, to each line.

Code listings

uppercase.py:

def proc_line(line):
    return line.upper()

leet.py:

def proc_line(line):
    return line.replace("A", "4").replace("a", "4")

line_by_line.py:

#!/usr/bin/python

import sys

def main(argv):
    
    usermodules = []
    if len(argv) < 2:
        print "Usage: line_by_line.py  ..."
        print "module most contain a function proc_line(line)"
        return 1
    try:
        for i in range(1, len(argv)):
            usermodules.append(__import__( argv[i] ))
    except:
        print 'Failed to import module "%s"' % (argv[1])
        return 1
    line = sys.stdin.readline()
    try:
        while line:
            # do something to the line and print the result
            for mod in usermodules:
                line = mod.proc_line(line)
            
            print line

            # fetch new line
            line = sys.stdin.readline()
    except EOFError:
        return None
    return 0

if __name__ == "__main__":
    main(sys.argv)