Benchmark: Reading uncompressed and compressed files from disc

In this post I’ll compare the running time of reading uncompressed and compressed files from disc.

I’ll run a test using two files, data.txt (858M) and data.txt.gz (83M), that have the same content.

About cat and zcat

The well-known command cat, prints the contents of a file. The lesser-known zcat, prints the contents of a GZIP’ed file.

Read more

GDAL and OGR Python tips

Online documentation for GDAL/OGR Python is sparse. Here I show some recommended ways of learning more about GDAL/OGR in Python.

Using Python interpreter

You can learn about GDAL and OGR from inside the Python interpreter.

Start python interpreter:

$ python

Import the modules:

from osgeo import gdal,ogr,osr

Learn about the modules using ‘help’ and ‘dir’ in Python. These built-in functions work for any type of object (module, class, functions etc):

help(ogr) # display help for the ogr module
dir(ogr) # what's contained in ogr module?
help(ogr.Geometry) # display help for ogr.Geometry class
dir(ogr.Geometry) # show contents of an object, like functions on a class

Using is a Q&A site about GIS. Many people here know about GDAL and ORG, also about the Python bindings.

Some good questions I’ve found:

Transformation using OGR in Python

Will add more as I find them…

Importing data from a CSV file into a Postgres table

Simple CSV file import

You have a CSV file called “data.csv”. It has a header line, and is delimited using “;”. You want to import it into Postgres and a table called “your_table”:

Create the database table. Set column-types so the string fields in the CSV file, can be cast to values in columns.

CREATE TABLE your_table
  -- Your columns

Read more

Low tech male/female voice detection system

I was drinking a bootle of water, while listening to a group of people having a debate. As the water level sunk, I noticed something. In the beginning the bottle would resonate/vibrate when the people with higher pitch voices talked. When the bottle was nearly empty, it would resonate/vibrate (and more so) when people with bassier voices talked.

Using bottles of water (with different waterlevels) and a vibration sensor on each, one could make a low tech thing that detects whether a male or female is speaking in the room. Why not use a microphone and a spectrum analyzer? Well, because you are McGyver and you’re caught in some prison, and mission dictates that you find out when the guy with the really bassy voice enters the room. You know, ad hoc spy stuff.

Here is a figure that illustrates the “Low Tech Male/Female Voice Detection System”:

Really easy 3D graphics in LaTeX

Note: The word “easy” is relative to the other solutions I found for doing 3D graphics in LaTeX.

The following template is adapted from the annotated 3d box example.

% Template for drawing with tikz 3D package
% Draw a 3D coordinate system
\begin{tikzpicture}[x  = {(1cm,0cm)},
                    y  = {(0.4cm,0.6cm)},
                    z  = {(0cm,1cm)},
                    scale = 1,
                    color = {lightgray}]
\begin{scope}[canvas is xy plane at z=0]
  % x-axis
  \draw[blue,->] (0,0) -- (5,0) node (x-axis) [right] {$x$};
  % y-axis
  \draw[red,->] (0,0) -- (0,5) node (y-axis) [above] {$y$};
\begin{scope}[canvas is yz plane at x=0]
  % z-axis
  \draw[green,->] (0,0) -- (0,5) node (z-axis) [above] {$z$};

Read more

How to load an ESRI Shapefile into a PostGIS DB

Assuming a shapefile called myshapefile.shp, a table mytable in schema xyz, in a PostGIS enabled database called mydb on localhost. The table is owned by user dbuser who has password “secret”.

Using shp2pgsql

shp2pgsql myshapefile -I xyz.mytable > statements.sql
psql -d mydb -h localhost -U dbuser -f statements.sql

This tip and many more can be read in Making Maps Fast.

Using ogr2ogr

This is even easier with ogr2ogr:

ogr2ogr -f "PostgreSQL" PG:"host=localhost user=dbuser dbname=mydb password=secret" -lco SCHEMA=xyz myshapefile.shp