There are many reasons that you might want to stand up an OpenStreetMap (“OSM”)-based tile cache service – your organization may not have access to external services, your usage may be high enough that you need a more performant system, or you just want to apply custom symbology to create a basemap that better serves your needs.
Creating your own OSM Tile Cache server using the OpenGeo Suite from Boundless is a relatively easy task. Before you do so, there are a few things you need to take into account when implementing the tile cache service based on OSM data. In the following blog we will discuss system sizing, creating and configuring the database, importing the OSM data, and symbolizing the data.
The machine that will process the OSM data will need enough disk space for the OSM data load as well as your resulting tile cache. The OSM worldwide dataset consumes about 120GB of disk space — depending on your tile cache settings, the end result can be 2TB+ of data. For this exercise we used an Ubuntu 14.0.4 VM with 12GB of RAM, 2 cores, and 2TB of disk space.
As our first step, we created /media/boundless/osmdata/ with 246GB of space and have 70GB+ on root for the postgres and geoserver binaries.
boundless@ubuntu:/$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 73G 41G 29G 59% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 5.7G 12K 5.7G 1% /dev
tmpfs 1.2G 3.6M 1.2G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 5.7G 284K 5.7G 1% /run/shm
none 100M 64K 100M 1% /run/user
/dev/sdb 246G 60M 234G 1% /media/boundless/
boundless@ubuntu:/$
Next, install the OpenGeo Suite following the documentation, I recommend Linux (Ubuntu or CentOS) for performance reasons.
The OpenGeo Suite will install GeoServer, GeoWebCache and PostgreSQL, as well as most of the dependencies that we need for OSM2PGSQL (the toolset we will use to load the OSM data into the DB). One item that is not installed is the HSTORE extension for PostgreSQL. Run the following command to install the postgresql-contrib:
sudo apt-get install postgresql-contrib
Next we need create a place for the raw OSM files as well as for the postgres tablespaces (owned by postgres). In the our case we created osmdata and postgres under /media/boundless.
drwxr-xr-x 5 boundless boundless 4096 Oct 15 11:12 osmdata
drwxr-xr-x 3 postgres postgres 4096 Oct 13 19:39 postgresNext step is to start getting some of the data needed for our tile cache server. In this example we will use a subset of the data focused on Colorado. We downloaded the following file into our osmdata directory:
http://download.geofabrik.de/north-america/us/colorado-latest.osm.pbf
[Side comment: If you want to use the entire planet it’s located here – but please note this file will take a while to download. If you’re going to do this you will need to plan for more disk space and longer processing times than this example shows. The Colorado OSM file is a very small subset of the OSM data – if you wish to use the United States data, plan for 120GB+ while the world can be 650GB+. These size of the files is also increasing with every release as more data is contributed to OSM]
Now that we have our OSM data downloading and the libraries needed for the PostGIS and HSTORE extensions, we can setup a DB that will house the OSM data.
Create a user for the DB – in our case we are using the username “boundless” – and then a PostGIS-enabled Database:
su postgres
createuser boundless
cd /media/boundless/postgres mkdir tablespace_1
chown postgres tablespace_1
psql -p 5432
create tablespace tablespace_1 location
'/media/boundless/postgres/tablespace_1';
CREATE DATABASE osm WITH OWNER boundless tablespace tablespace_1;
connect osm;
CREATE EXTENSION postgis;
CREATE EXTENSION postgis_topology;
CREATE EXTENSION hstore;
SELECT postgis_full_version();The output from the posgis_full_version should look similar to this:
POSTGIS="2.1.7 r13414" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.11.2, released 2015/02/10" LIBXML="2.9.1" LIBJSON="UNKNOWN" RASTER
(1 row)Before we start to load the OSM data we need to tell the DB to use the tablespace we created earlier and and set some tuning parameters in PostgreSQL. Edit the postgresql.conf file with the following recommended settings.* Some items above may need to be uncommented in addition to setting the values.
nano /etc/postgresql/9.3/main/postgresql.conf
| option | default | recommended |
| shared_buffers | 24 MB | 4 GB |
| work_mem | 1 MB | 100MB |
| maintenance_work_mem | 16 MB | 4096 MB |
| fsync | on | off |
| autovacuum | on | off (*) |
| checkpoint_segments | 60 | |
| random_page_cost | 4.0 | 1.1 |
| effective_io_concurrency | 1 | 2 |
| temp_tablespace | ’’ | ’tablepsace_1’ |
| listen_address | ‘localhost’ | ‘*’ |
*These may be different depending on your hardware and system configuration.
Edit the pg_hba.conf file to allow login with OS credentials (this can be disabled after the OSM load):
nano /etc/postgresql/9.3/main/pg_hba.conf
change local all all peer
to
local all all trust
change host all all peer
to
host all all trust
Restart postgresql
service postgresql restart
Next we will install OSM2PGSQL:
apt-get install osm2pgsql
Once OSM2PGSQL is instllaed, download the Mapzen style file.
If the steps above have all been successful and the OSM world file and stylesheet are downloaded, un the osm2pgsql command (assuming workding dir is /media/boundless/osmdata):
osm2pgsql -c -k --slim -C 8000 --flat-nodes ./fn -d osm -U boundless -H 127.0.0.1 -S ./mapzen_osm2pgsql.style ./colorado-latest.osm.pbf
Notes:
-C 8000 is set to ~65% of RAM
–slim slows things down but uses less ram by storing intermediate files in the database. This is required if you want to use append to add data later on.
Below is a listing of the objects in the DB after the load.
osm=> d
List of relations
Schema | Name | Type | Owner
——–+——————–+——-+———–
public | geography_columns | view | postgres
public | geometry_columns | view | postgres
public | planet_osm_line | table | boundless
public | planet_osm_nodes | table | boundless
public | planet_osm_point | table | boundless
public | planet_osm_polygon | table | boundless
public | planet_osm_rels | table | boundless
public | planet_osm_roads | table | boundless
public | planet_osm_ways | table | boundless
public | raster_columns | view | postgres
public | raster_overviews | view | postgres
public | spatial_ref_sys | table | postgres
(12 rows)
osm=> select * from geometry_columns;
f_table_catalog | f_table_schema | f_table_name | f_geometry_column | coord_dimension | srid | type
—————–+—————-+——————–+——————-+—————–+——–+————
osm | public | planet_osm_point | way | 2 | 900913 | POINT
osm | public | planet_osm_polygon | way | 2 | 900913 | GEOMETRY
osm | public | planet_osm_line | way | 2 | 900913 | LINESTRING
osm | public | planet_osm_roads | way | 2 | 900913 | LINESTRING
(4 rows)
osm=>
The next step in creating our tile cache server is to load an ocean dataset and administrative boundaries to use in conjunction with the OSM data.
Load the Oceans Polygon (mercator):
wget -bqc http://data.openstreetmapdata.com/water-polygons-split-3857.zip
unzip water-polygons-split-3957.zip
shp2pgsql -g geom -s 900913 -I -D ./water-polygons-split-3857/water_polygons.shp ocean | psql -U boundless osm
Download worldwide boundary files:
unzip ne_10m_admin_0_boundary_lines_land.zip
unzip ne_10m_admin_1_states_provinces_lines.zip
Load the files into the database:
shp2pgsql -g geom -s 4326 -I -D ne_10m_admin_1_states_provinces_lines.shp admin1 | psql -U boundless osm
shp2pgsql -g geom -s 4326 -I -D ne_10m_admin_0_boundary_lines_land.shp admin0 | psql -U boundless osm
The three new layers have an SRID of 4326, we will need to transform them in the DB to 900913. Login to the DB as the table owner and run the following:
alter table admin1 alter column geom type geometry(MULTILINESTRING,900913) using st_transform(geom,900913);
alter table admin0 alter column geom type geometry(MULTILINESTRING,900913) using st_transform(geom,900913);
Download the createDBObjects.sql and run it from the command line
boundless@ubuntu:/media/boundless/osmdata$ psql -d osm -f createDBobjects.sql
psql:createDBobjects.sql:2: NOTICE: table “aero-poly” does not exist, skipping
DROP TABLE
SELECT 278
CREATE INDEX
psql:createDBobjects.sql:17: NOTICE: table “agriculture” does not exist, skipping
DROP TABLE
SELECT 1805
CREATE INDEX
psql:createDBobjects.sql:30: NOTICE: table “amenity-areas” does not exist, skipping
DROP TABLE
SELECT 1300
CREATE INDEX
psql:createDBobjects.sql:44: NOTICE: table “beach” does not exist, skipping
DROP TABLE
SELECT 601
CREATE INDEX
psql:createDBobjects.sql:59: NOTICE: table “building” does not exist, skipping
DROP TABLE
SELECT 200888
CREATE INDEX
psql:createDBobjects.sql:80: NOTICE: table “forest” does not exist, skipping
DROP TABLE
SELECT 8386
CREATE INDEX
psql:createDBobjects.sql:99: NOTICE: table “grass” does not exist, skipping
DROP TABLE
SELECT 17398
CREATE INDEX
psql:createDBobjects.sql:122: NOTICE: table “highway-label” does not exist, skipping
DROP TABLE
SELECT 214823
CREATE INDEX
psql:createDBobjects.sql:143: NOTICE: table “park” does not exist, skipping
DROP TABLE
SELECT 11587
CREATE INDEX
psql:createDBobjects.sql:166: NOTICE: table “parking-area” does not exist, skipping
DROP TABLE
SELECT 16952
CREATE INDEX
psql:createDBobjects.sql:181: NOTICE: table “placenames-medium” does not exist, skipping
DROP TABLE
SELECT 105
CREATE INDEX
psql:createDBobjects.sql:196: NOTICE: table “route-bridge-0” does not exist, skipping
DROP TABLE
SELECT 4930
CREATE INDEX
psql:createDBobjects.sql:221: NOTICE: table “route-bridge-1” does not exist, skipping
DROP TABLE
SELECT 6742
CREATE INDEX
psql:createDBobjects.sql:246: NOTICE: table “route-bridge-2” does not exist, skipping
DROP TABLE
SELECT 130
CREATE INDEX
psql:createDBobjects.sql:271: NOTICE: table “route-bridge-3” does not exist, skipping
DROP TABLE
SELECT 40
CREATE INDEX
psql:createDBobjects.sql:296: NOTICE: table “route-bridge-4” does not exist, skipping
DROP TABLE
SELECT 4
CREATE INDEX
psql:createDBobjects.sql:321: NOTICE: table “route-bridge-5” does not exist, skipping
DROP TABLE
SELECT 3
CREATE INDEX
psql:createDBobjects.sql:346: NOTICE: table “route-fill” does not exist, skipping
DROP TABLE
SELECT 424383
CREATE INDEX
psql:createDBobjects.sql:397: NOTICE: table “route-line” does not exist, skipping
DROP TABLE
SELECT 325307
CREATE INDEX
psql:createDBobjects.sql:432: NOTICE: table “route-tunnels” does not exist, skipping
DROP TABLE
SELECT 106
CREATE INDEX
psql:createDBobjects.sql:453: NOTICE: table “route-turning-circles” does not exist, skipping
DROP TABLE
SELECT 28105
CREATE INDEX
psql:createDBobjects.sql:462: NOTICE: table “water-outline” does not exist, skipping
DROP TABLE
SELECT 44146
CREATE INDEX
psql:createDBobjects.sql:483: NOTICE: table “water” does not exist, skipping
DROP TABLE
SELECT 44146
CREATE INDEX
psql:createDBobjects.sql:504: NOTICE: table “wetland” does not exist, skipping
DROP TABLE
SELECT 1283
CREATE INDEX
boundless@ubuntu:/media/boundless/os
Using the GeoServer admin UI, create a workspace in GeoServer (localhost:8080/geoserver admin/geoserver) called osm with a http://osm.org namespace.
Create a PostGIS datastore called openstreetmap that connects to the osm database:
Next we will use the REST API to connect to GeoServer and create layers, symbology, and associate between the symbology and layers.
- Download the SLD scripts to your OSM data directory and unzip it.
- Run the SLD_create script in the SLD folder. This will create the styles and layers in GeoServer, associate the style to the corresponding layer, and then create a group layer that is the base map.
./SLD_create.sh
Once the SLD create script has run successfully you will see the layers and styles in your GeoServer instance. To see the basemap, go to layer preview and find the OSM group layer. It should look similar this:
If you zoom into Denver, Colorado you can see the results.
At a tight zoom level we can see more detail and get a feel for how the data looks.
So what have we accomplished? We have created and are now serving an OSM Tile Cache – We configured our Postgres/GIS instance to facilitate the import of OSM data, loaded both OSM and world wide boundary data, created feature extracts, published the data to geoserver and applied a basic styling for our basemap.
This example can be expanded upon for your organization’s uses. Maybe you want to incorporate your own data into the basemap, alter the symbology to meet your needs, load the entire world’s data, or all of the above to provide the perfect base map for your use.