Thats right, the dreaded two m’s. If you use C or C++, its quite dreaded. However, for about the first time ever, I ran into memory management issues in Python! Well, partially due to my own mistakes, and partially due to the way Python handles memory.

Memory handling schemes go awry

Well, not quite. More like, a leaky abstraction, compounded by my own idiot mistakes in the code. When memory is allocated in a loop, it is normally a smart optimization to hold onto that memory. What this means is that there are some cool things that you can do, like, refer to variables that were allocated inside a loop, and they will be in the local scope. They will just have the latest value thats been assigned to that variable.

However, in my case it backfired. The memory was held onto… and never reused. Essentially, I had a memory leak.

My own stupid mistakes

What happened, is that I was loading in a variable from a pickled file. However, what I had done was:

elements = pickle.load(open('file', 'rb'))

Normally, this would be okay, however, due to the size of the files, and the fact I was doing this in a loop, over different files, is that there was a now opened multi-megabyte file floating around, that has not been closed or reallocated. Actually, several. I came home after two hours to find that almost all of my RAM and 70% of my swap space had been used. Thats 70% of 3.8 GB. Yeah, great job Zeroth.

The lesson here is, that what works well for a small loop, or small files, does not neccesarilly work well for large files or large loops. Always make sure you close every file you open, and if worst comes to worst, allocate memory by using the del keyword in Python. I’m using this, and there is very moderate memory growth, as compared to the quite massive almost 6 GB of memory the code was taking up before.

So the proper way to do this is:

f=open('file', 'rb')
elements = pickle.load(f)
f.close()
del f

I do need to make this clear. This fault was not due to Python, or any problems or bugs in it, but rather with my own sloppy code. Even the best of us make mistakes, and I’m definitely not the best of us.

So, I’ve spent the past five days moving everything from OddCo over to a new server, at webfaction.

The reason I decided to go with them is that they provide the ability to run django, RoR, etc, within a shared account. No compiling needed, no .htaccess tricks, no worries about long running threads. Its all taken care of, and hosted at one of the world’s best datacentres(despite the big explosion a while back!)

If you have a feedburner rss feed, then the change has been transparent, and if you have a non-feedburner feed, then well, use feedburner. However, I have set it up so that any old links will automatically go to the right place, including the same format!

The beauty of Apache.

Okay, so I managed to fix the problems I was having with jQuery and ajax, no thanks to any of the relevant communities.

But here is my solution, if you were wondering. I’ll be linking to code files, instead of posting code snippets in the blog.

My first problem was getting galleria to see the new image I had just loaded via ajax. At first, I reprocessed the entire list with galleria, but that ended up with some hilarious results. I ended up with images running down the page, multiple times, with the index number added multiple times to the title, so it ended up going 7:7:7:7, for the image title.

Not quite what I wanted!

I then delved into the jcarousel.galleria.js file I got from Alex Wilson’s site, altering a few lines of the code.(Line 163-166)

I also did a bit of code in my on-page javascript, which would remove a few classes and stuff before being processed by galleria. But re-running galleria on the entire list again and again, for each image loaded, is well… tedious. Its slow, and was quite clunky.

So, I then found the most important parts of the galleria code, (Line 140 onwards), and extracted that out into a new function, to process a single image. A bit of editing, and this is the final form: my on-page javascript code.

This works the best, I’ve found, with very minimal jerky movements.

Anyone that is aware of the net neutrality debate has an inkling of what this post’s title means. But its the most important concept of this decade, if not this century.

Prologue to progress

Technology, not for the first time in human history, has enabled communication on a formerly impossible scale. One of Lawrence Lessig’s talks, at TED, talked about how each advance of technology, from the printing press to the radio, at first, enabled wider communications for all. But then, they became commercialized, and what was once a two-way line of communication became one way; companies creating content, and consumers consuming content.

This is the way companies like it. To them, technology for the masses, means more money, more consumers for their content. Radio, reached millions of people without needing wires, yet, to reach them all, it required a large radio tower- a hefty investment. For this reason, radio quickly became, in the mainstream at least, less about conversations and sharing between two or more people, but about popular consumption.

We are facing the same problem, yet again, where a gigantic advance in communication has spurred a rash, no, a plague of creation like none before. Technology enables, always. It enables advance, progress, profit, and enlightenment.

Changes to communication

This current advance, the internet, has even spurred changes in what were formerly one-way communications technologies. Unable to afford to publish newspapers, we published blogs and podcasts. Unable to use radio to communicate, we began to use bbs(later known as forums), chat rooms, and instant messaging.

You can tell when a new technology or derivative thereof is reaching the mainstream when a store like Radioshack(aka The Source by Circuit City in Canada), sells home products meant for such technology/services. Radioshack/The Source sells a home stereo system, that can tune into most of your favourite internet radio stations!

Thats pretty mainstream, if you ask me. But that example is only one-way, broadcast communication, which is not the internet. Merely an old form, redone with the internet. Radio has become cheaper to do, thanks to the internet.

Enabling encouragement

That however, is not my point. Technology enables people. So long as it is two-way, so long as people can participate, it can inspire and further society and art. So long as there is open and neutral access for all, can technology enable societal advance.

We’ve seen, time after time, how groups can either self-destruct, or achieve great things. And thats the best part, the potential for that to happen. So long as there is the potential to do great things, be they code(Linux), music(OCRemix), etc, then it will be achieved. By someone, by a group somewhere.

This, though, galls the corporate overlords. They have no control over these productions. Why, god damn it, they’re not even making any money off of this! Thats just not right. To them at least. They want us to consume, to suckle at the teat of consumption. Then they can make money. Then they can control things. Creativity that is not shackled to corporate goals scares them. It genuinely scares them.

Let them be scared. Write, compose, direct, create. Make them scared. Challenge them on net neutrality, and take hold of your progress. Don’t be spoon-fed content, never be content with that. Always do, and create, instead of consume.

Moxie Motto’s!

If you do create, and realize that people want what you can create, never fall into the trap of the corporate overlords. The value of what you create, is the potential. Encourage participation, work, insight and criticism. Become better, enrich others. Participate.

That is the motto of this new generation, of this generation of poets and writers and programmers and artists.

Our motto:

I create, therefore I am human.

A friend of mine recently started playing around with Python, and he asked me about a module I hadn’t heard of. I checked it out, and its pretty awesome. Its a new module, called subprocess, which was new in Python 2.4

Its pretty awesome, in that you can redirect the outputs in any which way, to another server, through a daemon program, to a file, anything.

Here’s a sample snippet to show how it works:

>>> import subprocess
>>> p=subprocess.Popen('ls -l', bufsize=10, shell=True)
>>> total 20
drwxr-xr-x 2 tyler tyler 4096 2008-03-25 20:31 admin
drwxr-xr-x 2 tyler tyler 4096 2008-03-25 19:42 admin_doc
drwxr-xr-x 3 tyler tyler 4096 2008-06-01 10:22 comics
drwxr-xr-x 2 tyler tyler 4096 2008-06-01 10:23 feeds
drwxr-xr-x 2 tyler tyler 4096 2008-04-18 14:43 panels

Anyway, if you make anything cool using this, let me know!