## Graham C10M talk at Shmoocon 2013

This video was mentioned on highscalability.com, so I thought I’d have a look. Knowning this stuff is useful when you’re in the business of delivering large amounts of geographical data to a large amount of clients.

C10M = 10 million concurrent requests.

## Using the Python debugger

A few days ago I found out that using the Python debugger is so easy, I can’t believe I haven’t used it before.

Import the module:

`import pdb`

Set a breakpoint somewhere in your code:

```def some_function(self, x, y, z): pdb.set_trace() ...```

Run your program. Now every time ‘some_function’ is called, the Python interpreter will break. At this point you could:

• type `x` to inspect the argument passed to the `x` parameter
• hit the ‘n’ button to skip over the next line of code
• hit the ‘c’ button to resume the program
• hit the ‘h’ button to get help

Easy enough?

## Sequential writes leveldb versus system_x

OK, calling it a benchmark is a bit of an overstatement. It’s taking two different database libraries for a quick spin, and seeing how fast they can write a bunch of integers to disk. A second benchmark checks how fast we can read them.

In this mini-test, I’m running leveldb against a new embedded database library, let’s call it system_x. The purpose is really just so that I can remember some rough numbers regarding these useful database libraries.

I used the `time` command to gather results, which shows real, user and sys time spent.

## Python: inverse of zip

So you have a list of tuples, created with the zip built-in function in Python. Like this:

`z = [(1, 'a'), (2, 'b'), (3, 'c')]`

And you want to reverse zip, to get these two lists:

```x = [1, 2, 3] y = ['a', 'b', 'c']```

## How to convert numpy matrix with different floats to matrix with zeros and ones

Given a numpy matrix (mixed_float_matrix) with a variety of float values, how do you convert it into a matrix, with zeros and ones (ones in place of non-zero values in the original matrix)?

## Plotting data on maps with matplotlib

I’m learning about matplotlib, and actually just bought the book Matplotlib for Python Developers.

## Geographical plots

Browsing stackoverflow, the matplotlib homepage, and other resources, I eventually came by this stackoverflow post, which mentions BaseMap. Since the data that I’m plotting is inherently geographical, it makes sense to show the data on a map.

There are several nice examples on the basemap Github page.

## Heatmaps

Often I want to create heatmaps of the data, using matplotlib.

On stackoverflow there are several posts on this topic:

There are different colormaps available for matplotlib, if you want to try different colorschemes.

## Making your Python code installable with PIP

I like to install 3rd party Python libraries using pip. Pip and easy_install can automatically download and install Python code from PyPi (also known as The Cheese Shop). This is how to publish your own Python code on PyPi, so people can do this:

`pip install yourawesomeproject`

## Install SQLAlchemy

This is easy (using pip):

`pip install sqlalchemy`

## Learn SQLAlchemy

http://docs.sqlalchemy.org/en/latest/core/tutorial.html

This tutorial uses an in memory SQLite database, which is cool in itself.

Various topics:

## How to install gfortran on Mac OS X

Why did I install gfortran? Well, not to write Fortran programs. I tried installing SciPy using pip install scipy, and I got a message that a Fortran compiler was needed.

## Install

This is how I installed gfortran on my Mac:

Visit hpc.sourceforge.net, and select a binary distribution for your version of Mac OS X, e.g. gfortran-snwleo-intel-bin.tar.gz for Snow Leopard.