skipperkongen.dk

Plotting data on maps with matplotlib

kostas — Thu, 10 May 2012 15:52:20 +0000

I’m learning about matplotlib, and actually just bought the book Matplotlib for Python Developers.

Geographical plots

Browsing stackoverflow, the matplotlib homepage, and other resources, I eventually came by this stackoverflow post, which mentions BaseMap. Since the data that I’m plotting is inherently geographical, it makes sense to show the data on a map.

There are several nice examples on the basemap Github page.

Heatmaps

Often I want to create heatmaps of the data, using matplotlib.

On stackoverflow there are several posts on this topic:

There are different colormaps available for matplotlib, if you want to try different colorschemes.

Making your Python code installable with PIP

kostas — Wed, 02 May 2012 14:21:28 +0000

I like to install 3rd party Python libraries using pip. Pip and easy_install can automatically download and install Python code from PyPi (also known as The Cheese Shop). This is how to publish your own Python code on PyPi, so people can do this:

pip install yourawesomeproject

The packaging chapter in dive into python 3 is a good starting point, also for Python 2, and this guide is a shorter rehash of that chapter. Just enough to get you started with publishing code on PyPi.

Step 1: Structure your code

Follow the tutorial, and you should have the right structure. Let’s assume that you have the following basic structure for your project “foolib”:

./foolib/
./foolib/foolib/ 
    __init__.py
    foo.py
    bar.py

Add COPYING.txt, README.txt, MANIFEST.in, setup.py:

./foolib/
    COPYING.txt
    README.txt
    MANIFEST.in
    setup.py
./foolib/foolib/
    __init__.py
    foo.py
    bar.py

Example setup.py (choose classifier strings from this list):

from setuptools import setup
 
setup(
    name = 'foolib',
    packages = ['foolib'],
    version = '0.0.1',
    description = 'Does something foo',
    author='Your Name',
    author_email='youremail@example.com',
    url='https://github.com/youraccount/foolib',
    classifiers=[
        'Programming Language :: Python',
        'Programming Language :: Python :: 2',
        'License :: OSI Approved :: GNU General Public License (GPL)',
        'Operating System :: OS Independent',
        'Development Status :: 1 - Planning',
        'Environment :: Console',
        'Intended Audience :: Science/Research',
        'Topic :: Scientific/Engineering :: GIS'
    ]
)

Example MANIFEST.in

include COPYING.txt

Step 2: Build distribution files

This is how to publish a binary distribution of your code on PyPi

# check the configuration
python setup.py check
# build Python eggs and publish in PyPi
python setup.py register bdist_egg upload

This has added some new folders in the root of your library:

./build
./dist
...

And now, anyone can install your Python library using PIP or easy_install

pip install foolib

Trying SQLAlchemy for Python

kostas — Fri, 27 Apr 2012 21:49:14 +0000

Install SQLAlchemy

This is easy (using pip):

pip install sqlalchemy

Learn SQLAlchemy

http://docs.sqlalchemy.org/en/latest/core/tutorial.html

This tutorial uses an in memory SQLite database, which is cool in itself.

Various topics:

Indexes

How to install gfortran on Mac OS X

kostas — Fri, 27 Apr 2012 21:25:16 +0000

Why did I install gfortran? Well, not to write Fortran programs. I tried installing SciPy using pip install scipy, and I got a message that a Fortran compiler was needed.

Install

This is how I installed gfortran on my Mac:

Visit hpc.sourceforge.net, and select a binary distribution for your version of Mac OS X, e.g. gfortran-snwleo-intel-bin.tar.gz for Snow Leopard.

Download the file and open the download directory in Terminal.

To install gfortran do the following:

gunzip gfortran-snwleo-intel-bin.tar.gz
tar xvf gfortran-snwleo-intel-bin.tar -C /

Compile a hello world

Let’s do a quick test that gfortran was installed.

gfortran --version

To try compiling a small testprogram, enter the following into a file called radius.for (btw, the whitespace matters):

      program circle
      real r, area
 
c This program reads a real number r and prints
c the area of a circle with radius r.
 
      write (*,*) 'Give radius r:'
      read  (*,*) r
      area = 3.14159*r*r
      write (*,*) 'Area = ', area
 
      stop
      end

Compile and run the program:

gfortran -o radius radius.for
./radius

Enter a number, e.g. 3, to see the area of a circle with this radius.

Uninstall

To uninstall again (warning, I haven’t tried it):

tar -tf gfortran-snwleo-intel-bin.tar | sort -r | (cd /; xargs -p -n 1 rm -d)

or do the following to delete without prompting:

tar -tf gfortran-snwleo-intel-bin.tar | sort -r | (cd /; xargs -t -n 1 rm -d)

Calling init() on all base classes in Python multiple heritance

kostas — Fri, 27 Apr 2012 08:44:23 +0000

Here is an example of calling

__init__()

for all base classes, when doing multiple inheritance in Python (mixins). It uses *args and **kwargs syntax:

mrotest.py:

class A(object):
 
    def __init__(self, *args, **kwargs):
        super(A, self).__init__(*args, **kwargs)
        self.number = kwargs['number']
 
    def helloA(self):
        return self.number
 
class B(object):
 
    def __init__(self, *args, **kwargs):
        super(B, self).__init__()
        self.message = kwargs['message']
 
    def helloB(self):
        return self.message
 
class C(A,B):
 
    def __init__(self, *args, **kwargs):
        # Notice: only one call to super... due to mro
        super(C, self).__init__(*args, **kwargs)
 
    def helloC(self):
        print "Calling A: ", self.helloA()
        print "Calling B: ", self.helloB()  
 
def main():
    c = C(number=42, message="Joe")
    c.helloC()
 
if __name__ == '__main__':
    main()

Running the program gives this output:

$ python mrotest.py 
Calling A:  42
Calling B:  Joe

Getting up to speed with Git

kostas — Thu, 26 Apr 2012 08:38:08 +0000

This is how you can get started successfully using Git.

Basic concepts:
Git Tutorial (by Lars Vogel)
how successful developers are using Git:
A successful branching model for Git (Gitflow)
Online books:
Pro Git and Git Community Book

Additional things to read:

Pyramidal tile cache cheat sheet

kostas — Tue, 24 Apr 2012 17:29:01 +0000

This table lists the number of total tiles for increasing zoom-level in a tile cache. The tile cache is assumed to be pyramidal: .

level 1: 1 tile
level 2: 5 tiles
level 3: 21 tiles
level 4: 85 tiles
level 5: 341 tiles
level 6: 1,365 tiles
level 7: 5,461 tiles
level 8: 21,845 tiles
level 9: 87,381 tiles
level 10: 349,525 tiles
level 11: 1,398,101 tiles
level 12: 5,592,405 tiles
level 13: 22,369,621 tiles
level 14: 89,478,485 tiles
level 15: 357,913,941 tiles
level 16: 1,431,655,765 tiles
level 17: 5,726,623,061 tiles
level 18: 22,906,492,245 tiles
level 19: 91,625,968,981 tiles
level 20: 366,503,875,925 tiles
level 21: 1,466,015,503,701 tiles

Creating plots in LaTeX with Gnuplot and Tikz

kostas — Mon, 23 Apr 2012 12:41:01 +0000

Plotting data in LaTeX is a useful thing to do. Xavier Delaunay shows how to do it on his homepage:

http://d.xav.free.fr/tikz/index.html

The plot looks like this:

The advantage of using Tikz to draw plots is, that things like fonts and arrow-heads will be the same in the document and in the figures.

Below I’ve replicated Xavier Delaunays LaTeX code, in the unlikely case that his site goes offline (Xavier, hope you don’t mind).

\documentclass{article}
\usepackage{tikz}
 
\begin{document}
\pagestyle{empty}
 
\begin{tikzpicture}[x=1cm,y=0.4cm]
 
  \def\xmin{3}
  \def\xmax{9.2}
  \def\ymin{2}
  \def\ymax{15.5}
 
  % grid
  \draw[style=help lines, ystep=2, xstep=1] (\xmin,\ymin) grid
  (\xmax,\ymax);
 
  % axes
  \draw[->] (\xmin,\ymin) -- (\xmax,\ymin) node[right] {$x$};
  \draw[->] (\xmin,\ymin) -- (\xmin,\ymax) node[above] {$y$};
 
  % xticks and yticks
  \foreach \x in {3,4,...,9}
    \node at (\x, \ymin) [below] {\x};
  \foreach \y in {2,4,...,14}
    \node at (\xmin,\y) [left] {\y};
 
  % plot the data from the file data.dat
  % smooth the curve and mark the data point with a dot
  \draw[color=blue] plot[smooth,mark=*,mark size=1pt] file {data.dat}
   node [right] {data};
 
  % generate and plot another a curve y = 0.1 x^2 + 2.5
  % this generates the files figure.parabola.gnuplot and figure.parabola.table 
  \draw[color=red, domain=\xmin:\xmax] plot[id=parabola]
  function{0.1*x**2 + 2.5} node [right] {$y=0.1\,x^2 + 2.5$};
 
\end{tikzpicture}
 
\end{document}

The data.dat file is simply a tab-separated file with coordinates

Simulating the Golden Balls game show

kostas — Sun, 22 Apr 2012 21:22:24 +0000

In this eposide of Golden Balls, an inspired event takes place:

I retold the event at DIKU APL lunch (nice to have a job where game theory is a valid conversation topic), and we had a conversation about it. At first I thought this was prisoners dilemma, but it was quickly revealed that it is a different game. What is cool about it is, that Nick basically forces Abraham to pick split. I don’t think the same approach would work again though. The person in Abrahams position might be tempted to try a counter-steal. The person in Nicks position might be tempted to actually steal the money, which would be a dirty thing to do, but not completely unlikely.

Someone remarked that if each player was a random variable, stealing would be twice as successful as splitting, as on average you get half the money by stealing, while splitting only gets you one quarter of the money on average.

Clearly players are not random variables, so we talked a bit about what a genetic algorithm simulation would reveal. Obviously, if all people were stealers, everyone would loose all the time, so that might inspire people to try splitting. If most people were splitters, stealers would have a field day, which might inspire people to become stealers. Someone speculated that running a simulation would reveal some kind of oscillating pattern. I decided to build a model and implement a simulation in Python.

The way the simulation works is this: A population is initialized, with a certain ratio between stealers and splitters (I use a 50/50 split). A game of golden balls is played, by drawing two participants randomly from the population. If the outcome is a steal, it will have an propability of converting splitters in the population into stealers. If the outcome is a split, it will conversely have a propability of converting stealers into splitters. If the outcome is a mutual steal, and hence loss, it will have a (slightly bigger) propability of converting stealers into splitters.

golden_balls.py:

import random
 
def main():
    # constants
    POP_SIZE = 1000
    INITIAL_STEALERS = 0.5
    population = int(POP_SIZE*INITIAL_STEALERS)*[1]
    population.extend(int(POP_SIZE*(1-INITIAL_STEALERS))*[0])
    GAMES = 100
    CONVERT_TO_STEALER = 0.1
    CONVERT_TO_SPLITTER = 0.05
    CONVERT_TO_SPLITTER_ON_LOOSE = 0.1
 
    ts = [['Generation', 'Percent stealers in population']]
 
    # generations
    for game in range(GAMES):
 
        # calc stealer percent
        stealers = round(reduce(lambda x,y: x+y, population) / float( len(population) ), 5)
 
        # record last generation
        ts.append([game, stealers])
 
        # play game
        p1 = random.random() <= stealers
        p2 = random.random() <= stealers
        new_pop = []
        for person in population:
 
            if (p1 or p2) and not (p1 and p2):
                # steal
                if random.random() <= CONVERT_TO_STEALER:
                    # converted
                    new_pop.append(1)
                else:
                    new_pop.append(person)
 
            elif (not p1) and (not p2):
                # split
                if random.random() <= CONVERT_TO_SPLITTER:
                    new_pop.append(0)
                else:
                    new_pop.append(person)
 
            else:
                # loose
                if random.random() <= CONVERT_TO_SPLITTER_ON_LOOSE:
                    new_pop.append(0)
                else:
                    new_pop.append(person)
 
        population = new_pop
 
 
    print ts
 
if __name__ == '__main__':
    main()

The outcome of the simulation:

Simple programs for simple turing machines

kostas — Sun, 08 Apr 2012 14:36:42 +0000

I’ve always wanted to understand Turing Machines better. I believe that the best understanding comes from the intuition developed by solving puzzles.

So here, I give myself some challenges. Solve simple problems by writing programs for a simple Turing Machine, using this simulator: http://ironphoenix.org/tril/tm/

Challenge 1: Odd or even characters?

Challenge:

Make a program for a Turing Machine, such that the program:

outputs E if there is an even number of sequential x’es on tape

outputs O if there is an odd number of sequential x’es on tape

the program does not have to handle the empty string

running time not important

For example, the program should output ‘E’ for the following input:

xxxx

My solution to challenge 1:

1,x 2,1 >
2,x 3,. <
2,. 2,. >
2,_ 4,_ <
3,. 3,. <
3,0 2,1 >
3,1 2,0 >
4,. 4,_ <
4,0 5,E >
4,1 8,O >
5,_ 6,V, >
6,_ 7,E >
7,_ H,N >
8,_ 9,D >
9,_ H,D >

The program has O(n^2) running time.

Challenge 2: Improve the solution to linear running time

In the above only the character ‘x’ is matched. Matching n characters would mean adding n transitions to state 2. That is straight forward, and I’ll not do it in the following.

Challenge:

Improve the solution to challenge 1:

Algorithm must have O(n) running time

It should handle the empty string (output ‘E’)

My solution to challenge 2:

1,_ 9,_ >
1,x 2,# >
2,x 3,x <
2,_ 6,_ <
3,0 5,0 >
3,1 4,1 >
3,# 4,# >
4,x 2,0 >
5,x 2,1 >
6,0 7,_ <
6,1 8,_ <
6,# 10,_ >
7,0 7,_ <
7,1 7,_ <
7,# 9,_ >
8,0 8,_ <
8,1 8,_ <
8,# 10,_ >
9,_ H,E >
10,_ H,O >

Challenge 3: Sorting

Ok, let’s up the ante a little bit. Let’s do sorting.

Challenge:

Sort a scrambled sequence of numbers from 0-9

Running time should be no worse than bubble sort, O(n^2) worst case

TODO

skipperkongen.dk

Plotting data on maps with matplotlib

Geographical plots

Heatmaps

Making your Python code installable with PIP

Step 1: Structure your code

Step 2: Build distribution files

Trying SQLAlchemy for Python

Install SQLAlchemy

Learn SQLAlchemy

How to install gfortran on Mac OS X

Install

Compile a hello world

Uninstall

Calling __init__() on all base classes in Python multiple heritance

Getting up to speed with Git

Pyramidal tile cache cheat sheet

Creating plots in LaTeX with Gnuplot and Tikz

Simulating the Golden Balls game show

Simple programs for simple turing machines

Challenge 2: Improve the solution to linear running time

Challenge 3: Sorting

Calling init() on all base classes in Python multiple heritance