How to draw lines on map in Databricks

Imports: import plotly.graph_objects as go Plot: fig = go.Figure() fig.add_trace(go.Scattermapbox( mode = "markers+lines", lon = [10, 20, 30], lat = [10, 15,30], marker = {'size': 10})) fig.add_trace(go.Scattermapbox( mode = "markers+lines", lon = [-50, -60,40], lat = [30, 10, -20], marker = {'size': 10})) fig.update_layout( margin ={'l':0,'t':0,'b':0,'r':0}, mapbox = { 'center': {'lon': 10, 'lat': 10}, 'style': […]

How to draw lines on map in Databricks Read More »

How to call an API from PySpark (in workers)

Tested in Databricks import pyspark.sql.functions as F import requests # create dataframe pokenumbers = [(i,) for i in range(100)] cols = ["pokenum"] df_pokenums = spark.createDataFrame(data=pokenumbers, schema=cols) # call API def get_name(rows): # take the first item in list (API doesn't support batch) first = rows[0] url = f'https://pokeapi.co/api/v2/pokemon-form/{first.pokenum}' try: resp = requests.get(url) name = resp.json()['pokemon']['name']

How to call an API from PySpark (in workers) Read More »

Yummy 3D plots

Very nice interactive 3D plots with Plotly. import plotly.graph_objects as go import numpy as np import pandas as pd # Read data from a csv Z = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/api_docs/mt_bruno_elevation.csv').values # Actually not necessary to provide X and Y… X = np.linspace(0, 1000, Z.shape[0]) Y = np.linspace(0, 1000, Z.shape[1]) fig = go.Figure(data=[go.Surface(x=X, y=Y, z=Z)]) fig.update_layout(title='Mt Bruno Elevation',

Yummy 3D plots Read More »

How to create a versioned table from a crud table in Postgres with triggers

Example You have a user table in Postgres: CREATE SCHEMA tutorial; CREATE TABLE tutorial.crud_users ( id INT NOT NULL, username VARCHAR NOT NULL, CONSTRAINT crud_table_pkey PRIMARY KEY (id) ); You insert new users and update usernames for existing users. Update is a destructive operation that overwrites data, so how do you keep track of what

How to create a versioned table from a crud table in Postgres with triggers Read More »