This week was a bit of varied work.

I did this week:

• Cleaned up, refactored, and commented code. This took a day
• Hunting down a memory leak. This took two days. It ended up being leaving opening a video stream without closing it before looping.
• Worked on getting loops to work properly. This took the better part of a day, as there’s various things that need to be manipulated in between loops for proper playback.
• Did various experiments with sound. My current attempt has been disastrous.

What I’m trying to do right now, is work with SDL_mixer. This goes on behind another file, lowering the dependency between SDL_mixer and my module, to make it easier and more portable to other sound systems in the future. One method I tried is to give the sdl_mixer adapter code a reference to a Condition variable, and to send a signal through the Condition variable when it needs more samples to play, whereupon my audio thread would pass it an already allocated buffer, which would then be added into a Mix_Chunk struct, and played via the SDL_mixer api. After doing this, my audio_thread would work on allocating another bunch of samples to pass it when necessary. This… didn’t work. Mainly because the sdl_mixer adapter code would send a signal back to the audio thread, but the audio thread wouldn’t catch the signal(I was using SDL_cond and associated functions to do this) for some reason. Add in as well the issues of deadlocking, which were quite literally textbook, where thread A grabs the GIL and tries to grab a mutex, while thread B grabs the mutex, and tries to grab the GIL. I fixed that.

I’ve also tried building a queue of buffers, which the adapter code will pull out and play when the current buffer is finished. Whenever the audio thread calls playBuffer, it copies that buffer to a newly allocated buffer of the same length, and then that new buffer is stored on the queue.  This isn’t really quite debugged as apparently I need to brush up on pointer manipulation and memory allocation. I think I’m on the right path, by building a queue of buffers, I just need to do it right.

If anyone has any suggestions on how to do this, I will gladly welcome any help!

At the moment, the module will load pretty much any type of movie that ffmpeg supports, play it(infinite amount of times, or a specified amount of times), with pause, and stop working. Seeking does not work yet. Only video works. I’ve also added a few basic parameters to get, like the width and height of the movie itself. Next week, I will work on adding the ability to specify the height and width of the video file. As well, I think I could probably alter the repr a bit to get the current time-stamp. In fact I’ll do that right now. I should also add docs like the other pygame modules.

Okay, so its near the end of the first week of development. Its been an exciting week, learning the ins and outs of python module development.

First, I have code that compiles, links, and can be successfully imported. However, no movie objects are created, there’s a bug somewhere that causes it to return a None object to me. All the code is in svn.

To successfully compile the code, one needs the ffmpeg-0.5 package of source code. In there, follow this process:

1. ./configure –enable-shared –prefix=”/usr”
2. make
3. make install
4. cd libswscale/
5. make
6. make install

This will install the libraries and headers into the /usr directory if you are on Linux.

Then(assuming you have all the other dependencies, see here):

1. download the branch of source-code, at svn://seul.org/svn/pygame/branches/tylerthemovie
2. ./configure
3. python setup.py build
4. python setup.py install

And thats it! Of course, like I said, it is not yet really operational yet. I will be fixing this bug.

In addition, I will be spending some time over the weekend and up to Wednesday helping out with the local AI/CG/CRV conference hosted by UBC-Okanagan.

I ran across this a long time ago, never really used it till more recently, but it really shows the power and capability of Python.

So, say you have a big set of local variables which need to go into the string you are building, say for a YAPWF(Yet Another Python Web Framework).

Thats means you have this messy looking code:

s="%d%f" % (car, calc)

Its messy, because you’ve separated contextual information from the actual place that it matters, and if the string is very long and confusing, it gets messy, and opens you up to errors down the road. In addition, you are tied to having those variables exist locally.

However, there is a much, much nicer way to do this.

d={"number":100, "float":4.2, "status":"okay", "ph":8665782}
s="%(number)d%(float)f%(status)s%(ph)d" % d
print s
1004.2okay8665782

What this means is now you can throw in each of the relevant possible variables into the key that matters into a single dictionary object, and pass that around easily. The only thing you’re tied to now are the keys of the dictionary, giving you a bit of Adapter action.

Just one note, the format goes like this:

"%({key name}){format modifier}" % dictionary

As well, you can name the keys relevant names, like status, phone, etc, and your string formatting becomes a lot easier to read.

Okay, so its another entry in my long-abandoned python tricks series.

First, lets take a look at a couple of classes:

class B(object):
    def foo(self):
        return '10'
 
class A(object):
    def __init__(self):
        self.a='a'
    def foo(self):
        return '1'

The important part here is the (object) subclassing. This trick only works if you do that. Lets take alook at it in operation:

>>> a=A()
>>> a
<changetype.A object at 0x7f0f18dbf9d0>
>>> a.__class__
<class 'A'>
>>> a.a
'a'
>>> a.foo
<bound method A.foo of <changetype.A object at 0x7f0f18dbf9d0>>
>>> a.foo()
'1'
>>> a.__class__=B
>>> a.a
'a'
>>> a.foo
<bound method B.foo of <changetype.B object at 0x7f0f18dbf9d0>>
>>> a.foo()
'10'
>>> isinstance(a, B)
True

As you can see, I was able to override the foo() method with the one in the Beta class. As well, this only works on instanced objects. Now, what use is this you may ask? It is basically meta-programming. For 90% of programming tasks, meta-programming is not needed at all. But for that remaining 10%, you can do some seriously cool stuff, like changing the operation of a class during runtime, without modifying the class variables.

As a real-life example, I just finished writing a simplistic query parser. I realized I had forgetten to take into account price queries, so, I created a PriceToken class, that on instantiation, changes its own class to that of a TermToken. Otherwise, I would have had to rewrite about 100 lines of code to allow the use of a PriceToken. Pretty much useful because I’m lazy. But, yes, in Python, you can change the type of an object on the fly.

As part of my new job, as a research student, I have to process a lot of data. On the order of several hundreds of thousands of records. So, I turned to my favourite language, Python. Its what caused my earlier issues with memory management.

A few more lessons I’ve learned:

• sometimes theres a faster way to do stuff.
• make sure you aren’t putting repetitive data into a database
• verify the data format before you download 2-3 GB of it

Okay, so, faster way to do stuff. Python has both a performance advantage and disadvantage. The advantage is that it only takes one or two hours to code up something to process data. The disadvantage is that in an effort to be clear and simple, one can end up coding a task that takes 20-100x longer than it should. Like I did.

I had a lot of nested loops, python speed loops, involving lots of duplicate handling, inserting to a database, etc. It took five hours to go through only 14 sets of queries to clear out. I had a lot of duplicate rows that needed to be erased. For a query looking for rows with an id of 15, there are 25,607 records. Removing the duplicates yields… 807. Brilliant.

Today, I decided to take most of the loops, and replace them with their functional counterparts, map, lambda, and Google’s functional tools in python, goopy. In only two hours, it has gone through now, 160 ids. In less than half the time. Pretty amazing, eh?

Here’s a sample:

for row in all:
     curs=curs.execute('insert into elements2 values(?, ?, ?, ?, ?)', (row[0], row[1], str(row[2]), str(row[3]), ''))

replaced with:

map(lambda row: <code>conn.execute('insert into elements2 values(?, ?, ?, ?, ?)', (row[0], row[1], str(row[2]), str(row[3]), ''))</code>, all)

A little bit more difficult to read, yes, but, the speed improvements are well over 50 times faster…

All this work to clear out duplicate entries, and how intensive it can be to fix, is a perfect example of why the “Look Before You leap” programming paradigm works better than, “Ask for Forgiveness Not Permission”.

Now, once the processing is done, I’ll be able to go through all the data… and add a piece of information I forgot to process in the first place to each entry. Yay me!