Uncategorized

How to Draw an Owl

Taken from lecture 1 of Statistical Rethinking course (around the 44 minute mark). The course material is also on Github. How to draw an "owl" version 1: Create generative simulation (GS) Write an estimator Validate estimator using simulated data Analyze real data: … Reuse 1 to compute hypothetical interventions How to draw an "owl" version …

How to Draw an Owl Read More »

How to call an API from PySpark (in workers)

Tested in Databricks import pyspark.sql.functions as F import requests # create dataframe pokenumbers = [(i,) for i in range(100)] cols = ["pokenum"] df_pokenums = spark.createDataFrame(data=pokenumbers, schema=cols) # call API def get_name(rows): # take the first item in list (API doesn't support batch) first = rows[0] url = f'https://pokeapi.co/api/v2/pokemon-form/{first.pokenum}' try: resp = requests.get(url) name = resp.json()['pokemon']['name'] …

How to call an API from PySpark (in workers) Read More »

Create a European city map with population density

Datasets: – Urban morphological zones 2000 (EU): https://www.eea.europa.eu/data-and-maps/data/urban-morphological-zones-2000-2 – Population count (World): http://sedac.ciesin.columbia.edu/data/set/gpw-v4-population-count-rev10/ – Administrative regions (World): http://gadm.org/ The map is European since the “urban” data from the European Environmental Agency (EEA) only covers Europe. Caveats The UMZ data ended up in PostGIS with srid 900914. You can use prj2epsg.org to convert the contents of …

Create a European city map with population density Read More »

How to create a world-wide PostgreSQL database of administrative regions

The GADM database contains geographical data for administrative regions, e.g. countries, regions and municipalities. As always, once you have the data in the right format, it is easy to import it into a database. The data is available from GADM in several formats. All data has the coordinate reference system in longitude/latitude and theWGS84 datum. …

How to create a world-wide PostgreSQL database of administrative regions Read More »