-
How to fill missing dates in Pandas
Create a pandas dataframe with a date column: import pandas as pd import datetime TODAY = datetime.date.today() ONE_WEEK = datetime.timedelta(days=7) ONE_DAY = datetime.timedelta(days=1) df = pd.DataFrame({‘dt’: [TODAY-ONE_WEEK, TODAY-3*ONE_DAY, TODAY], ‘x’: [42, 45,127]}) The dates have gaps: dt x 0 2018-11-19 42 1 2018-11-23 45 2 2018-11-26 127 Now, fill in the missing dates: r = […]
-
Cosine similarity in Python
Cosine similarity is the normalised dot product between two vectors. I guess it is called “cosine” similarity because the dot product is the product of Euclidean magnitudes of the two vectors and the cosine of the angle between them. If you want, read more about cosine similarity and dot products on Wikipedia. Here is how […]
-
How to display a Choropleth map in Jupyter Notebook
Here is the code: %matplotlib inline import geopandas as gpd import matplotlib as mpl # make rcParams available (optional) mpl.rcParams[‘figure.dpi’]= 144 # increase dpi (optional) world = gpd.read_file(gpd.datasets.get_path(“naturalearth_lowres”)) world = world[world.name != ‘Antarctica’] # remove Antarctica (optional) world[‘gdp_per_person’] = world.gdp_md_est / world.pop_est g = world.plot(column=’gdp_per_person’, cmap=’OrRd’, scheme=’quantiles’) g.set_facecolor(‘#A8C5DD’) # make the ocean blue (optional) Here […]
-
(Integer) Linear Programming in Python
Step one: brew install glpk pip install pulp Step two: from pulp import * prob = LpProblem(“test1”, LpMinimize) # Variables x = LpVariable(“x”, 0, 4, cat=”Integer”) y = LpVariable(“y”, -1, 1, cat=”Integer”) z = LpVariable(“z”, 0, cat=”Integer”) # Objective prob += x + 4*y + 9*z # Constraints prob += x+y = 10 prob += […]
-
Python script for geocoding a text file
Assume that you have a file with some locations as text with one location per line. For example, here are some school names in Copenhagen, Denmark, stored in schools.csv: Hyltebjerg Skole Heibergskolen Ellebjerg Skole Katrinedals Skole Peder Lykke Skolen Amager Fælled Skole Tingbjerg Heldagsskole Øster Farimagsgades Skole Sankt Annæ Gymnasiums Grundskole Lykkebo Skole Randersgades Skole […]
-
Easy parallel HTTP requests with Python and asyncio
Python 3.x, and in particular Python 3.5, natively supports asynchronous programming. While asynchronous code can be harder to read than synchronous code, there are many use cases were the added complexity is worthwhile. One such examples is to execute a batch of HTTP requests in parallel, which I will explore in this post. Additionally, the […]
-
How to compute the pagerank of almost anything
Whenever two things have a directional relationship to each other, then you can compute the pagerank of those things. For example, you can observe a directional relationships between web pages that link to each other, scientists that cite each other, and chess players that beat each other. The relationship is directional because it matters in […]
-
Quick introduction to RabbitMQ and Celery
I like to code in Python. I also like the concept of asynchronous workers to build loosely coupled applications. Luckily, RabbitMQ and Celery can help me do exactly that. This post is based on a very nice YouTube video by Max Mautner (the one below). For easy repeatability, I have transcribed the video in this […]
-
How to export CSV file to database with Python
Pandas and SQLAlchemy offer powerful conversions between CSV files and tables in databases. Here is a small example: import pandas as pd from sqlalchemy import create_engine df = pd.read_csv(‘mydata.csv’) engine = create_engine(‘sqlite:///mydata.db’) df.to_sql(‘mytable’, engine) Read more: pandas.DataFrame.to_sql sqlalchemy engines
-
How to use non-default profile in boto3
Given an AWS credentials file that looks like this: [default] aws_access_key_id = DEFAULT aws_secret_access_key = SECRET1 [dev] aws_access_key_id = DEV aws_secret_access_key = SECRET2 [prod] aws_access_key_id = PROD aws_secret_access_key = SECRET3 You can use any profile, say dev, like this in Python: import boto3.session dev = boto3.session.Session(profile_name=’dev’) s3 = dev.resource(‘s3’) for bucket in s3.buckets.all(): print(bucket.name) print(”)