It has been a long time since I’ve updated this blog. My last post was back in February, where I talked about some of the podcasts I listen to. After that, work got busy and I didn’t have time to write another post.
Good thing is, for the past three months, I’ve shipped three (3) open-source projects for work and for personal use. So, instead of dedicating one post for each, I guess it would be better to dump all of them here. Maybe you’ll find something useful!
- Geomancer: Automated Feature Engineering for Geospatial Data
- Tiffany: Convert to TIFF any Google Static Maps Image
- Seagull: Python Library for Conway’s Game of Life
1. Geomancer: Automated Feature Engineering for Geospatial Data
This is one of the first open-source projects I built for Thinking Machines. Given a set of coordinates, Geomancer allows us to extract features/attributes based on a specific set of points-of-interest (PoIs).
We use Geomancer to obtain features such as the “distance to the nearest mall” or “number of embassies within a certain range”. Features are then fed to a machine learning model to enable wealth estimation or poverty prediction, you can see some related work here.
It leverages OpenStreetMaps data and a data warehouse like BigQuery (you can also use Spatialite, and soon, PostGIS), giving you the best of both worlds in terms of compute and data source.
However, my favorite part in Geomancer is how declarative its API is. I was also given the liberty to play with the module names, hence I called Features as Spells:
from geomancer.spells import DistanceToNearest # Load the dataset in a pandas dataframe # df = load_dataset() dist_spell = DistanceToNearest( "embassy", source_table="ph_osm.gis_osm_pois_free_1", feature_name="dist_embassy", dburl="bigquery://project-name", ).cast(df)
In addition, we also added a
SpellBook feature, allowing us to export the
features our analysts found relevant into a JSON file and be shared to others.
from geomancer.spells import DistanceToNearest from geomancer.spellbook import SpellBook spellbook = SpellBook([ DistanceToNearest( "supermarket", source_table="ph_osm.gis_osm_pois_free_1", feature_name="dist_supermarket", dburl="bigquery://project-name", ), DistanceToNearest( "embassy", source_table="ph_osm.gis_osm_pois_free_1", feature_name="dist_embassy", dburl="bigquery://project-name", ), ]) spellbook.to_json("my_spellbook.json")
So if I want to reproduce someone’s features on my own dataset, I don’t need to
re-define the spells programmatically. I just need to get the JSON SpellBook,
load it, and
cast() it on my own data:
from geomancer.spellbook import SpellBook # Load the dataset in a pandas dataframe # df = load_dataset() spellbook = SpellBook.read_json("my_spellbook.json") df_output = spellbook.cast(df)
2. Tiffany: Convert to TIFF any Google Static Maps Image
Got it? tiff…any…Tiffany!
This is my first Golang project and I’m pretty happy with it. Tiffany is just a simple command line tool that eases the way we download and reference Google Static Maps images:
|Original image||Georeferenced||With labels|
When you use tiffany, an API Token is still required, so you need to sign-up and pay for one.
So if you want to download a static map given a coordinate, you just run:
tiffany 14.546943935986324 121.01974525389744
This then downloads a 400x400-sized image that is already georeferenced (you can configure the settings too in the command-line app). Pretty neat! The best thing about it is that you can also pass a shapefile to easily obtain cut-out building footprints for training your computer vision model:
tiffany 14.546943935986324 121.01974525389744 --with-labels=path/to/shapefile.shp
Lastly, you can pass a CSV file with all the coordinates you want to download,
tiffany will download them for you!
tiffany batch coordinates.csv
3. Seagull: Python Library for Conway’s Game of Life
One of my latest personal projects is a simple Python library for Conway’s Game of Life, Seagull. Why Seagull you ask? Well because when you take the first letters of Conway’s Game Of Life, you get C.G.O.L., say it faster and it starts sounding like “seagull”, genius!
I’ve been fascinated with cellular automata and artificial life in general, and I think that creating a fun toy that allows me to experiment and explore patterns might be very helpful.
I made sure that Seagull’s API is intuitive: just set-up your board, add your lifeforms, and put them in a simulator!
import seagull as sg from seagull.lifeforms import Pulsar # Initialize board board = sg.Board(size=(19,60)) # Add three Pulsar lifeforms in various locations board.add(Pulsar(), loc=(1,1)) board.add(Pulsar(), loc=(1,22)) board.add(Pulsar(), loc=(1,42)) # Simulate board sim = sg.Simulator(board) sim.run(sg.rules.conway_classic, iters=1000)
You then just call
sim.animate() to view your animation like so:
I also made a nice collection of lifeforms to choose from, so get crazy!