I just recently finished reading a book, no, rather, a comic, by Scott McCloud, called Understanding Comics: The Invisible Art. It was a fantastic read, very illuminating in many aspects. A lot of things I didn’t understand about art in general are finally beginning to be less murky.

One of the most striking concepts was the six stages of art. And it resonated. I realized that those six stages correspond as well to the development and progress of a programmer.

In his comic, the six stages are as follows: Idea/Purpose, Form, Idiom, Structure, Craft, Surface.

The Six Steps of Art

Let’s start at the top: Surface.

Surface applies to the most basic of programming. Simple statements, flow control, basic functions. Every programmer begins their growth at this stage. They learn how basic math works, properties, etc. They learn how to order statements, if conditionals, and loop guards.

Next is Craft.

Craft is where they take the basics, the Surface, and begin constructing bigger things. Functions, classes, basic data structures. Here, they learn how to construct a complex piece of smaller simpler pieces, but they are only just barely beginning to learn why and how it works that way. Any data structures they build are based off of basic data structures, and they don’t yet understand deeply the fundamental ideas of efficiency, logic, and computability. The good programmer also begins to learn about source control, using their tools, building, and deploying their work.

Structure is next in their growth path.

Here, they begin to learn how to organize their complex pieces into a working piece. How to access and parse text files, basic database interactions. They’ll do things the slow, wrong way, but its necessary to reach the stage of development. Here, they begin to worry about efficiency, the right way of doing things. Object Oriented programming starts to make sense as a conceptual model(which is all it really is). The now competent programmer is capable of learning other basic conceptual models.

Idiom is the biggest part of the growth.

Here, they begin to learn about Design Patterns, and begin to apply them. They apply them at every stage, from the basic Surface, to the overall Structure of the program. They also begin to learn about the ideas and concepts behind data protocols, to use them where needed. This is when programmers become cynical about Silver Bullets. This is when they finally realize there is no substitute for skill, experience, and effort. Many programmers do extremely well staying at this stage. However, to become a top-notch programmer, one needs to see and explore the limits of programming.

That is where Form comes in. Programmers at this stage explore the efficacy of Design Patterns, breaking them where and when needed, even developing their own versions. They begin to develop solid ideas about how API’s should work, about what is computable, about efficiency, and about Logic. These are the programmers that take already established tools and concepts, and bring us exciting new ones. Notable examples are: Bittorrent, Linux, and the Mouse. They looked at what was being done, and brought something new, and world-changing to the world of computing.

The final, highest level of development is Ideas/Purpose. This is where programming Gods like Knuth and Stallman rest. This is where Computer Scientists do their heady, ivory tower research that is oh-so-important to the world at large. This is where concepts of algorithms, data structures, efficiency, and logic are explored in their abstract. This is where Computer Science is rooted, and where the birth of the computer industry started. This is where Boole developed his ideas of Boolean logic, where Cray designed his supercomputers, and the very basis of all digital electronics.

Now, this is the normal path most programmers take in their growth. Even though, in College and University, we’re taught the ideas and purpose of programming from almost the start… when we don’t have the basic building blocks of understanding of logic and program construction.

As my friend Khumba just said: “You can’t play Protoss until you’ve played a few games with Terran.” You need to understand the basic building blocks of logic, statements and flow-control and the experience with those to understand the higher level concepts of functions, routines, threads, classes, modules, executables, algorithms.

The thing is though, we can start at any stage we like, from Ideas/Purpose, to the Idioms of programming. But the greatest, best programmers amongst us will explore all of these stages in depth, learning what they need from it to improve themselves and their art. Because thats what Programming is. It is an art, and we are the artists, working with the pure stuffs of thought. We translate the abstract to binary logic, to 1′s and 0′s, and for that… we are artists.

If you disagree, or agree, let me know. Theres a handy comment box right down there.

Thats right, the dreaded two m’s. If you use C or C++, its quite dreaded. However, for about the first time ever, I ran into memory management issues in Python! Well, partially due to my own mistakes, and partially due to the way Python handles memory.

Memory handling schemes go awry

Well, not quite. More like, a leaky abstraction, compounded by my own idiot mistakes in the code. When memory is allocated in a loop, it is normally a smart optimization to hold onto that memory. What this means is that there are some cool things that you can do, like, refer to variables that were allocated inside a loop, and they will be in the local scope. They will just have the latest value thats been assigned to that variable.

However, in my case it backfired. The memory was held onto… and never reused. Essentially, I had a memory leak.

My own stupid mistakes

What happened, is that I was loading in a variable from a pickled file. However, what I had done was:

elements = pickle.load(open('file', 'rb'))

Normally, this would be okay, however, due to the size of the files, and the fact I was doing this in a loop, over different files, is that there was a now opened multi-megabyte file floating around, that has not been closed or reallocated. Actually, several. I came home after two hours to find that almost all of my RAM and 70% of my swap space had been used. Thats 70% of 3.8 GB. Yeah, great job Zeroth.

The lesson here is, that what works well for a small loop, or small files, does not neccesarilly work well for large files or large loops. Always make sure you close every file you open, and if worst comes to worst, allocate memory by using the del keyword in Python. I’m using this, and there is very moderate memory growth, as compared to the quite massive almost 6 GB of memory the code was taking up before.

So the proper way to do this is:

f=open('file', 'rb')
elements = pickle.load(f)
f.close()
del f

I do need to make this clear. This fault was not due to Python, or any problems or bugs in it, but rather with my own sloppy code. Even the best of us make mistakes, and I’m definitely not the best of us.

I’ve spent almost five years programming as a hobby, and my first language was Python. Its still one of my favourites for many reasons; simplicity, power, rapid development, and its fun.

Due to its nature, Python is very flexible and very versatile, and over the years, I’ve learned some very very cool and useful python idioms and tricks.

What I will show today is a Design Pattern that fits the ideology of Python well, and delivers a great amount of flexibility and usefulness. Its called the Borg Design Pattern. Read more