So, say you have a big set of local variables which need to go into the string you are building, say for a YAPWF(Yet Another Python Web Framework).

Thats means you have this messy looking code:

s="%d%f" % (car, calc)

Its messy, because you’ve separated contextual information from the actual place that it matters, and if the string is very long and confusing, it gets messy, and opens you up to errors down the road. In addition, you are tied to having those variables exist locally.

However, there is a much, much nicer way to do this.

d={"number":100, "float":4.2, "status":"okay", "ph":8665782}
s="%(number)d%(float)f%(status)s%(ph)d" % d
print s
1004.2okay8665782

What this means is now you can throw in each of the relevant possible variables into the key that matters into a single dictionary object, and pass that around easily. The only thing you’re tied to now are the keys of the dictionary, giving you a bit of Adapter action.

Just one note, the format goes like this:

"%({key name}){format modifier}" % dictionary

As well, you can name the keys relevant names, like status, phone, etc, and your string formatting becomes a lot easier to read.

First, lets take a look at a couple of classes:

class B(object):
    def foo(self):
        return '10'
 
class A(object):
    def __init__(self):
        self.a='a'
    def foo(self):
        return '1'

The important part here is the (object) subclassing. This trick only works if you do that. Lets take alook at it in operation:

>>> a=A()
>>> a
<changetype.A object at 0x7f0f18dbf9d0>
>>> a.__class__
<class 'A'>
>>> a.a
'a'
>>> a.foo
<bound method A.foo of <changetype.A object at 0x7f0f18dbf9d0>>
>>> a.foo()
'1'
>>> a.__class__=B
>>> a.a
'a'
>>> a.foo
<bound method B.foo of <changetype.B object at 0x7f0f18dbf9d0>>
>>> a.foo()
'10'
>>> isinstance(a, B)
True

As you can see, I was able to override the foo() method with the one in the Beta class. As well, this only works on instanced objects. Now, what use is this you may ask? It is basically meta-programming. For 90% of programming tasks, meta-programming is not needed at all. But for that remaining 10%, you can do some seriously cool stuff, like changing the operation of a class during runtime, without modifying the class variables.

As a real-life example, I just finished writing a simplistic query parser. I realized I had forgetten to take into account price queries, so, I created a PriceToken class, that on instantiation, changes its own class to that of a TermToken. Otherwise, I would have had to rewrite about 100 lines of code to allow the use of a PriceToken. Pretty much useful because I’m lazy. But, yes, in Python, you can change the type of an object on the fly.

A few more lessons I’ve learned:

• sometimes theres a faster way to do stuff.
• make sure you aren’t putting repetitive data into a database
• verify the data format before you download 2-3 GB of it

Okay, so, faster way to do stuff. Python has both a performance advantage and disadvantage. The advantage is that it only takes one or two hours to code up something to process data. The disadvantage is that in an effort to be clear and simple, one can end up coding a task that takes 20-100x longer than it should. Like I did.

I had a lot of nested loops, python speed loops, involving lots of duplicate handling, inserting to a database, etc. It took five hours to go through only 14 sets of queries to clear out. I had a lot of duplicate rows that needed to be erased. For a query looking for rows with an id of 15, there are 25,607 records. Removing the duplicates yields… 807. Brilliant.

Today, I decided to take most of the loops, and replace them with their functional counterparts, map, lambda, and Google’s functional tools in python, goopy. In only two hours, it has gone through now, 160 ids. In less than half the time. Pretty amazing, eh?

Here’s a sample:

for row in all:
     curs=curs.execute('insert into elements2 values(?, ?, ?, ?, ?)', (row[0], row[1], str(row[2]), str(row[3]), ''))

replaced with:

map(lambda row: <code>conn.execute('insert into elements2 values(?, ?, ?, ?, ?)', (row[0], row[1], str(row[2]), str(row[3]), ''))</code>, all)

A little bit more difficult to read, yes, but, the speed improvements are well over 50 times faster…

All this work to clear out duplicate entries, and how intensive it can be to fix, is a perfect example of why the “Look Before You leap” programming paradigm works better than, “Ask for Forgiveness Not Permission”.

Now, once the processing is done, I’ll be able to go through all the data… and add a piece of information I forgot to process in the first place to each entry. Yay me!

Memory handling schemes go awry

Well, not quite. More like, a leaky abstraction, compounded by my own idiot mistakes in the code. When memory is allocated in a loop, it is normally a smart optimization to hold onto that memory. What this means is that there are some cool things that you can do, like, refer to variables that were allocated inside a loop, and they will be in the local scope. They will just have the latest value thats been assigned to that variable.

However, in my case it backfired. The memory was held onto… and never reused. Essentially, I had a memory leak.

My own stupid mistakes

What happened, is that I was loading in a variable from a pickled file. However, what I had done was:

elements = pickle.load(open('file', 'rb'))

Normally, this would be okay, however, due to the size of the files, and the fact I was doing this in a loop, over different files, is that there was a now opened multi-megabyte file floating around, that has not been closed or reallocated. Actually, several. I came home after two hours to find that almost all of my RAM and 70% of my swap space had been used. Thats 70% of 3.8 GB. Yeah, great job Zeroth.

The lesson here is, that what works well for a small loop, or small files, does not neccesarilly work well for large files or large loops. Always make sure you close every file you open, and if worst comes to worst, allocate memory by using the del keyword in Python. I’m using this, and there is very moderate memory growth, as compared to the quite massive almost 6 GB of memory the code was taking up before.

So the proper way to do this is:

f=open('file', 'rb')
elements = pickle.load(f)
f.close()
del f

I do need to make this clear. This fault was not due to Python, or any problems or bugs in it, but rather with my own sloppy code. Even the best of us make mistakes, and I’m definitely not the best of us.

A new installment in my ever-popular series.

Todays installment will concern one of the best performance enhancing tricks in python: list comprehensions.

Most beginning pythonistas will produce code like this:

lst=[]
for i in range(0, 10):
     lst.append(i**2)

Now, thats pretty okay. It runs… okay. It is the kind of code I would expect from a newcomer to Python, however, one of Python’s weaknesses is its slower running speed. This, by the way, is a weakness for pretty much any interpreted language, like Java or Ruby.

Now, imagine you have a much larger data set to work with. On the order of thousands, hundreds of thousands. Processing all of it at Python speed will drag your program down.

But, you are in luck: there is a faster way.

Unfortunately, this faster method has its own set of issues, and the first and most obvious one, is legibility. Done wrong this technique can and will obscure the meaning of your program, and make it harder to maintain.

This technique is list comprehensions. The upper piece of code would be, in a list comprehension,

lst=[i**2 for i in range(0, 10)]

This line of code runs in C speed, not Python speed. For a large dataset, this can be an improvement of 10-100 times faster!

Now, lets break it down: there are three parts to a list comprehension. The first portion is an expression. The second portion is a for clause. And the third portion is one or more for or if clauses.

The first for clause in the list comprehension declares a variable, and the expression must do something with that variable.

Or, for those language lawyers:

test ::= or_test | lambda_form

testlist ::= test ( “,” test )* [ "," ]

list_display ::= “[" [listmaker] “]”

listmaker ::= expression ( list_for | ( “,” expression )* [","] )

list_iter ::= list_for | list_if

list_for ::= “for” expression_list “in” testlist [list_iter]

list_if ::= “if” test [list_iter]

Basically, a list comprehension consists of:

[expression using 'x' variable for clause declaring 'x' (optionally: if 'x' meets some condition)]

Again, one of the cardinal rules of optimization are that you only optimize when you know where the bottlenecks are. So, when you first code a for loop or list initialization, use the regular for loop. Then, after the program works, you can optimize it out to a list comprehension, and test the resulting speed.

This will, in most cases, make your program faster, and make you a Python guru. So, till next time, keep on hissing!

Due to its nature, Python is very flexible and very versatile, and over the years, I’ve learned some very very cool and useful python idioms and tricks.

What I will show today is a Design Pattern that fits the ideology of Python well, and delivers a great amount of flexibility and usefulness. Its called the Borg Design Pattern.

I’m sure we’re all familiar with the Singleton design pattern. The Borg DP is almost precisely the opposite. Instead of one instantiated object, and just one, the Borg DP allows you to instantiate any number of the objects, and have them all share the exact same traits, methods, variables.
(assume there is proper formatting for this function)

1| class Borg:
2| _share_state={}
3| def __init__(self, *args, **kwargs):
4| self.__dict__=self._shared_state


Here’s the beauty of the Borg DP: on any of the instantiated objects, if you modify a variable, all of the instantiated objects reflect that modification. Every single one of them. Hence the name, Borg Design Pattern.

The important line here is line two: “__dict__={}” before you declare any methods of variables. When you do this in Python, all instantiated classes have a reference to that single variable and object. And __dict__ is an internal dict of the class, that stores all methods and all variables!

The power of this trick is such that, you can use it for many of the same places you once used a Singleton, but without needing to worry about references, thread safe access, even checking how many objects have been instantiated. It cleans up all associated code, and gives you the confidence to know exactly what is going on with the object.

And thats part one! Stay tuned for more Cool Python Tricks(tm), hopefully one a week!

EDIT: Fixed borg design pattern, small bug.