Author: kostas

How to compute and plot Bollinger Bands® in Python

kostas

The aim is to produce a plot like this. The orange line is your data, the green line is the upper "bollinger" band, the blue line is the lower "bollinger" band. The red dots indicate where your data is either above or below the bands.

Copy-paste this code:

import pandas as pd
import numpy as np
from matplotlib import pyplot as plt

N = 100
XMAX = 5
WINMA = 10
ALPHA = 2

def get_bollinger(data, winma=10, alpha=2):
    ser = pd.Series(data)
    ma = ser.rolling(winma).mean()
    std = ser.rolling(winma).std()
    lower = pd.Series(ma - alpha*std).fillna(method='bfill').values
    upper = pd.Series(ma + alpha*std).fillna(method='bfill').values
    return lower, upper

def get_alerts(data, lower, upper):
    low = np.argwhere(data < lower)
    high = np.argwhere(data > upper)
    return low, high

if __name__=='__main__':

    X = np.linspace(0.0, XMAX, num=N)
    data = np.sin(X) + np.random.random(N)
    lower, upper = get_bollinger(data, winma=WINMA, alpha=ALPHA)
    low, high = get_alerts(data, lower, upper)
    for i in low:
        plt.plot(X[i], data[i], 'ro')
    for i in high:
        plt.plot(X[i], data[i], 'ro')
    plt.plot(X, lower)
    plt.plot(X, data)
    plt.plot(X, upper)
    plt.show()

See who is hosting a website

kostas

To see who is hosting a website (e.g. netflix.com), you will need the IP address of the website and the WHOIS record for that IP address. You can obtain both from the command-line.

Obtain the IP address for netflix.com:

# Ping the website
ping netflix.com
# Alternatively, use traceroute and note the last IP address
traceroute netflix.com

For me, the command returned the IP address 52.209.235.141. Now, look up the WHOIS record for the IP address to see who owns it:

# Look up the WHOIS record
whois 52.209.235.141

Here is the record I got from WHOIS for netflix:

OrgName:        Amazon Technologies Inc.
OrgId:          AT-88-Z
Address:        410 Terry Ave N.
City:           Seattle
StateProv:      WA
PostalCode:     98109
Country:        US

So, at the time of writing, it seems that netflix.com was hosted by Amazon.

How to chat between computers using netcat on Mac and Linux

kostas

In our home we have a mix of Mac and Linux computers. My children have a Raspberry Pi in their room and I have a Mac at my desk. Today we chatted with each other through the Terminal running on both computers.

Hi children!
Dudududududududu
Am I talking to a bird?
^v^

(It was a very interesting conversation as you can see)

How to chat using netcat

You can chat between two Unix computers (e.g. Linux or Mac) only using the Terminal and netcat command (i.e. the command nc). The netcat command is slightly different on the two OS'es, but I'll write the variation in comments.

First, you open a terminal window on both computers:

On a Mac, open e.g. the Terminal application

Now, choose one computer to listen (computer #1) and the other to connect to the first (computer #2). We chose that my Mac would listen and the childrens Raspberry Pi would connect.

Set up the listening computer

On the first computer, start netcat listening for incoming messages on port 23456 (or whatever port you prefer). You'll see that the command is slightly different on Mac and Linux.

# On Mac: nc -l <PORT>
nc -l 23456
# On Linux: nc -l -p <PORT>
nc -l -p 23456

Check the IP address of the listening computer:

ifconfig

Look for an entry called "inet", which looks something like this:

...
inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255
...

Shout the IP address you found into the house for all to hear!

Hey, I'm listening on port 23456 and my IP address is 192.168.1.101 !!!

Remember to shout the IP address you found, not the one I wrote above, since yours will probably be different... Now this computer is ready to chat!

Set up the connecting computer

On the second computer, use netcat to connect to the "listening" computer, using the IP address and port that the other person shouted to you.

# Mac/Linux: nc <IP ADDR> <PORT>
nc 192.168.1.101 23456

Again, rememember that the IP address above is just an example. Most likely, the person shouted different numbers to you.

Now the fun part: chat!

Now either computer you can type messages and hit enter. The thing you typed will appear on the other computer. Remember to finish your message by hitting the enter key.

hi, how are you? <enter>

After you hit enter, the message will appear on the other computer, where the person might reply:

hi, how are you?
I am fine thank you. How are you doing? <enter>

Experiment

After you have exchanged a few pleasant greetings, you can start hacking. For example:

  • What happens if you receive a message while you are typing?
  • Does the output always look the same on both computers?
  • What happens if you send special characters (e.g. apple character on a Mac)?
    • You can generally type special characters by using modifier keys, such as alt/option. E.g. to type an Apple icon on a Mac (Danish keyboard) you'll hit option + $/§.

When you are done chatting you can exit netcat with ctrl + c. It doesn't matter who does this first, the other side will exit as well.

How to discover devices on your network (Mac/Linux)

kostas

There are several ways to check what devices are currently logged on to your network (e.g. WiFi at home).

The first option is to use arp, which I have found works the best. There will be some cases where it does not work.

In the following example, I'll assume that your IP address is 192.168.1.xxx.

Using arp and nmap

First, scan your sub-network: 

# see you IP address using ifconfig, here it's assumed to be 192.168.1.0
nmap 192.168.1.0/24

Second, list devices in your ARP cache: 

arp -a

If the output doesn’t look up to date, or if it’s missing an IP you believe should be there, ping the broadcast IP (typically the last result of arp -a ending with “.255”).

# Assuming 239.255.255.250 was the last address in the output from arp -a
ping 239.255.255.250

Now, run arp -a again.

ARP spoofing

Notice that a malicious user can use arp spoofing to mix up the association between MAC addresses and IP addresses. This can be used for man-in-the-middle attacks.

Other ways

You could also use WireShark or even a third or fourth method, but I will not cover those here.

Frk. Slimas Fornemmelse for Snot

kostas

Af Kostas Kefaloukos

Danmark er et lille halvkoldt land, halvlangt mod nord. Det er ikke så koldt at der er særligt meget sne. Til gengæld blæser det meget og regner hele tiden. Disse vejrforhold gør at danskerne ofte bliver forkølede og udvikler store mængder af snot i næse og svælg. På samme måder som grønlænderne har mange ord for sne, har danskerne mange ord for snot. "Grønhakker", "Bussemand", "Løbesnot" osv. Snot forekommer i mange variationer og danskerne kender indgående fornemmelsen af hver eneste form for snot. I denne artikel vil jeg omhyggeligt gennemgå de mest gængse former for snot og relaterede emner.

  • Almindelig snot
  • bussemænd
  • sjældne snotformer.

Almindelig snot

  • Løbesnot. Meget tyndtflydende og oftest klar snot, som typisk forekommer ved allergi og høfeber. Utroligt irriterende og nærmest umulig at slippe af med. Man pudser næse hver 30. sekund og lige lidt hjælper det. Har man ikke et stykke papir, må man snøfte konstant for at holde trit med snotten. Dette er helt utroligt irriterende. Forekommer dette mens man sover, må man på et tidspunkt give op, og acceptere at der kommer snot overalt.
  • Kompakt snot. Typisk ved slem forkølelse. Der dannes et ekstremt overtryk i hovedet når man prøver at blæse den ud og man kan lige så godt give op. Man har det lidt som Arnold Schwarzenegger i Total Recall; scenen hvor hans øjne popper ud og hovedet sprænger i luften. Kompakt snot kommer ikke ud foreløbigt. Man fristes til at prøve igen og igen da fornemmelsen af at kraniet er blevet udstoppet af en satanisk taksidermiker er tæt på uudholdelig. På et tidspunkt kan man fristes til at holde hovedet over en dampende gryde i et desperat forsøg på at opløse snotten, men man kan lige så godt lade være. Det er omsonst.
  • Gul snot. Når forkølelsen er på sit højeste forkommer denne særligt klamme form for snot. Store mængder. Mellem-flydende. Minder lidt om østers. På dette tidspunkt kan området under næsen godt være en smule medtaget, og det svier at tørre snotten væk. Når man pudser næsen i et stykke køkkenrulle eller serviet, kan man betaget stå og se på de enorme mængder der havnede i papiret. Det er lige før man kan spå i det. Det dejlige ved denne form for snot er, at man rent faktisk bliver lidt stolt når man har produceret en ordentlig klump i lommetørklædet.
  • Grønhakker. Som gul snot, bare grønt. Denne form for snot kan næsten gå selv. Meget tilfredsstillende at spytte ud på en fortorvsflise, men samtidigt ekstremt klamt at se på. Ret fascinerende den måde som grønhakkeren lander f.eks. på en fortorvsflise. Fra det sted hvor den lander, bliver den typisk strukket et par centimeter indtil den indre sammenhængskraft og friktion standser processen. Resultatet er ofte en lang grøn streg på flisen. Hvis man er en meget barsk fyr eller pige gøres dette helt åbenlyst og til stor gru for omkringstående personer, som i smug er lidt bange for at komme til at træde i grønhakkeren. Man går selv i en meget stor bue udenom mesterværket.
  • Hvid Gummisnot. Meget viskos snot, der kommer ud i små gelé-agtige klumper. Det burde være forbudt og det er ikke særligt tilfredsstillende at have med at gøre. Snottens svar på Sysifosarbejde.
  • Gul gummisnot. Kan eventuelt hostes op fra lungerne. Dejligt at spytte ud i håndvasken og skylle væk med koldt vand! I forhold til hvor betændt denne snot ser ud, er mængderne som regel ret beherskede. Man tænker måske bekymret på om man udviser tegn på forstadier til slem "gulag" lungebetændelse.
  • Grøn kæmpe-gummisnot. En klar indikator på "gulag" lungebetændelse. Døden er nær, men indtræffer til stor lettelse sjældent. Hostes op fra lungerne i en kompakt klump. Utroligt tilfredsstillende at spytte ud i håndvasken. Man føler at kroppen er blevet renset. Det dummeste man kan gøre er at tygge på det, og genoptage klumpen i systemet. Den vender tilbage!!

Bussemænd

  • Knastør bussemand. Indgår ofte i fusion med næsehår og kan derfor være smertelig at pille ud, da næsehårene typisk rives med ud. Man skulle tro at næsen spontant havde produceret superlim. Denne bussemand er typisk ret flad og grålig i farven. Den er ikke særligt god at trille, da den højst kan foldes lidt sammen og danner ikke en fin kugleform. Følelsen er lidt anti-klimatisk og af denne grund smides bussemanden ofte hvor den pilles. En ret kedelig bussemand.
  • Almindelig bussemand. Dette er en dejlig bussemand. Kan trilles til en perfekt kugle. Farven kan være grønlig til grålig. En almindelig adspredelse er, at trille disse bussemænd mens man sidder i en bilkø eller i sofaen og ser fjernsyn. Selve trilningen må foregå i smug, da det sociale stigma ved åbenlyst at trille bussemænd for de fleste er for stort. Når man er færdig med at trille bussemanden, skal man skille sig af med den. Dette foregår for de flestes vedkommende også i smug. Den kan f.eks. elegant sendes afsted med pegefingeren fra en skjult position. Ved kørsel kan man rulle vinduet ned og måske lade som om armen bare skal dingle lidt. Med en diskret bevægelse smides bussemanden ned på vejen uden at nogen ser det. En lignende manøvre kan foretages i sofaen, hvor hånden der holder bussemanden diskret lægges på ryglænet. Efter en passende pause lader man bussemanden falde. Denne form for bussemand spises ofte af børn, enten rå eller trillet. Det er ikke farligt, men dog ret klamt.
  • Gummi bussemand. Ofte lille og nærmest dej-agtig (våd dej) at røre ved. Lad dig ikke narre af størrelsen. Dette er den mest irriterende form for bussemand. Den er umulig at få pillet ud i ét stykke med fingeren. Man piller og piller og piller og ender med at tvære bussemanden ud på fingeren uden at få det hele med. Resultatet kan i slemme tilfælde være lange bussemandstråde mellem finger og næse. Forsøger man at trille den til en kugle, som en almindelig bussemand, vil man fejle. Man kan med nogen success skrabe denne slemme bussemandstype ud med et lommetørklæde.

Sjældne snotformer

  • Orange snot. snot i sin pureste form. Mellem-flydende. Meget sjælden.

How to scrape images from the web

kostas

I'm interested in object detection and other computer vision tasks. For example, I'm working on a teddy-bear detector with my son.

So, how do you quickly download images for a certain category? You can use this approach that I learned from a course on Udemy.

# pip install icrawler
from icrawler.builtin import GoogleImageCrawler

keywords = ['cat', 'dog']
for keyword in keywords:
    google_crawler = GoogleImageCrawler(
        parser_threads=2,
        downloader_threads=4,
        storage={'root_dir': 'images/{}'.format(keyword)}
    
    )
    google_crawler.crawl(
        keyword=keyword, max_num=10, min_size=(200, 200))

In the above example, the crawler will find images in two categories -- cats and dogs, as if you search for 'cat' and 'dog' on Google images and downloaded what you found.

Let's walk through the parameters used in the code. First, there is the constructor, which is called with three arguments in the example. The most important parameter is storage, which specifies where the images will be stored. Second, we have the call to the crawl function. Here, the max_num parameter is used to specify that at most 10 images per category should be downloaded. The min_size argument specifies that the images must be at least 200 x 200 pixels.

That's it. Happy downloading.

How to fill missing dates in Pandas

kostas

Create a pandas dataframe with a date column:

import pandas as pd
import datetime
 
TODAY = datetime.date.today()
ONE_WEEK = datetime.timedelta(days=7)
ONE_DAY = datetime.timedelta(days=1)
 
df = pd.DataFrame({'dt': [TODAY-ONE_WEEK, TODAY-3*ONE_DAY, TODAY], 'x': [42, 45,127]})

The dates have gaps:

           dt    x
0  2018-11-19   42
1  2018-11-23   45
2  2018-11-26  127

Now, fill in the missing dates:

r = pd.date_range(start=df.dt.min(), end=df.dt.max())
df.set_index('dt').reindex(r).fillna(0.0).rename_axis('dt').reset_index()

Voila! The dataframe no longer has gaps:

          dt      x
0 2018-11-19   42.0
1 2018-11-20    NaN
2 2018-11-21    NaN
3 2018-11-22    NaN
4 2018-11-23   45.0
5 2018-11-24    NaN
6 2018-11-25    NaN
7 2018-11-26  127.0

Cosine similarity in Python

kostas

Cosine similarity is the normalised dot product between two vectors. I guess it is called "cosine" similarity because the dot product is the product of Euclidean magnitudes of the two vectors and the cosine of the angle between them. If you want, read more about cosine similarity and dot products on Wikipedia.

Here is how to compute cosine similarity in Python, either manually (well, using numpy) or using a specialised library:

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
 
# vectors
a = np.array([1,2,3])
b = np.array([1,1,4])
 
# manually compute cosine similarity
dot = np.dot(a, b)
norma = np.linalg.norm(a)
normb = np.linalg.norm(b)
cos = dot / (norma * normb)
 
# use library, operates on sets of vectors
aa = a.reshape(1,3)
ba = b.reshape(1,3)
cos_lib = cosine_similarity(aa, ba)
 
print(
    dot,
    norma,
    normb,
    cos,
    cos_lib[0][0]
)

The values might differ a slight bit on the smaller decimals. On my computer I get:

  • 0.9449111825230682 (manual)
  • 0.9449111825230683 (library)

Terms used in shipping

kostas

Now that I work in shipping, it is necessary to learn a bunch of new terms. Shipping is regulated under Admiralty Law and there are traditional documents and parties involved. Knowing what these are is crucial to understanding shipping.

Legal documents

There are three key documents involved with shipping:

Parties

There are quite a few parties involved in shipping:

  • Carrier
  • Charterer
  • Consignee
  • Consignor
  • Shipbroker
  • Ship-manager
  • Ship-owner
  • Shipper
  • Stevedore

How to sample from softmax with temperature

kostas

Here is how to sample from a softmax probability vector at different temperatures.

import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
import seaborn as sns
 
mpl.rcParams['figure.dpi']= 144
 
trials = 1000
softmax = [0.1, 0.3, 0.6]
 
def sample(softmax, temperature):
    EPSILON = 10e-16 # to avoid taking the log of zero
    #print(preds)
    (np.array(softmax) + EPSILON).astype('float64')
    preds = np.log(softmax) / temperature
    #print(preds)
    exp_preds = np.exp(preds)
    #print(exp_preds)
    preds = exp_preds / np.sum(exp_preds)
    #print(preds)
    probas = np.random.multinomial(1, preds, 1)
    return probas[0]
 
temperatures = [(t or 1) / 100 for t in range(0, 101, 10)]
probas = [
    np.asarray([sample(softmax, t) for _ in range(trials)]).sum(axis=0) / trials
    for t in temperatures
]
 
sns.set_style("darkgrid")
plt.plot(temperatures, probas)
plt.show()

Notice how the probabilities change at different temperatures. The softmax probabilities are [0.1, 0.3, 0.6]. At the lowest temperatures of 0.01, the dominant index (value 0.6) has near 100% probability of being sampled. At higher temperatures, the selection probabilities move towards the softmax values, e.g. 60% probability for the third index.