All GeoServer

How to Add Layers to GeoServer Using the REST API


GeoServerAt OpenGeo we’re often asked how to ingest large amounts of data into a system for storage (on disk, or in PostGIS) and publishing (through GeoServer and/or GeoWebCache).

GeoServer’s graphical administrative tools offer a convenient UI for managing data as workspaces, stores, layers and styles but will only handle them one at a time. If you have a significant amount of data to upload into GeoServer anew, or migrate from an existing implementation from another vendor, one-at-a-time just won’t cut it.

Enter GeoServer’s REST API (REST stands for REpresentational State Transfer). There’s a ton of general information online about REST but here it is in 50 words or less: REST lets you GET a representation of the current state of something through a URI, which you are then free to modify as/if needed, and finally PUT that representation back to the URI, where (if the request is authenticated and/or valid) the object’s state is modified.
In the GeoServer world, you might GET (a true HTTP GET) the current representation of a datastore, update something in that datastore configuration (correct a username, password or path), and then PUT the corrected representation back onto the server to update the configuration. If the service permits, you can also POST a new configuration to create a new object, or fire off a DELETE to an endpoint to remove an object from it. The specific methods you can apply to a REST endpoint and the objects you can apply those REQUESTS are often advertised by the endpoint itself. A RESTful interface should also offer written (prose) documentation of its capabilities.

GeoServer is no exception to these norms and written documentation is available in the GeoServer user manual. If you have GeoServer installed, the REST endpoint is available at:

http://<your.geoserver.url>:<port>/geoserver/rest

If you hit the base REST URL, you’ll notice that the format is HTML by default. The links, to workspaces, stores, layers, etc. should be consistent with your understanding of how (your) data is laid out in GeoServer. This is a nice way to discover GeoServer’s resource hierarchy, but it’s a bit thin on the details for any particular resource. Don’t worry, read on.

Specific object representations can also be requested through GeoServer’s REST API as XML or JSON that the requesting process can easily parse and/process by most simply appending the format type to the request (E.g. .xml or .json). We previously mentioned the term API (application programming interface) and the name is pretty self-descriptive. An API just means that there is an access point through which client applications can send and receive data. In this case the client can be something deeply nerdy like Java (etc) code, an application (coded in whatever) that wraps the API in an alternate UI (the OpenGeo Suite’s GeoExplorer is an application that shoots commands to GeoServer’s REST API), or something in- between like a shell script that uses cURL to automate requests to a REST interface based on virtually any rules or logic required.

Now for some examples …

One of our  clients came to us with a large number of shapefiles (over 900) that they wanted to publish with GeoServer. While there are a number of options for loading data through the graphical interface, all of these options would have been time-intensive. We suggested the REST interface to them, and provided a script employing the following utilities:

  • shp2pgsql (shapefile loader/dumper),
  • psql (command-line PostGIS client), and
  • cURL (a command-line URL handler).

Most of these commands were inspired by GeoServer’s user manual. The proposed solution uses cURL on BASH in Linux, but you could do very similar things in Windows with Powershell, in the Mac OSX or UNIX shells, or in PHP, ASP.NET, C#, Visual Basic, Python, (etc!) with virtually the same tool-set or concepts.

We started with shapefiles loaded en masse into PostGIS. Thought there are a variety of ways to do this, we prefer this method from the shell:

> for f in *.shp; do
> shp2pgsql -s 4326 -g geom $f ${f%.shp} | psql earth;
> done

In line 1 we use a handy for loop to iterate over all .shp files in the current directory as the variable f.

In line 2 we iterate within our collection of .shp files and call the shp2pgsql command on each current file ($f); the output of which is sent to the earth database through the command-line PostgreSQL tool psql.

shp2pgsql is a command-line tool that ships with PostGIS (and the OpenGeo Suite) and converts a shapefile into the SQL instructions needed to load that geographic data into PostGIS. All of the options and parameters for this tool are displayed when you type shp2pgsql without arguments at the command line. We use -s 4326 (add the SRID (EPSG:)4326 to the feature meta-tables) and -g geom to make sure the geometries are added to a column called, well ‘geom’.

Normally, shp2pgsql sends its SQL directly to standard out. You could ‘redirect’ (>) this output to a file and then run it into PostGIS separately; however, in this case we ‘pipe’ (|) the output directly to the psql command so that it is executed against the database ‘earth’. When these commands have finished up, we have loaded into PostGIS, but they’re not registered in GeoServer for exposure to clients through OGC compliant web-services (WMS, WFS, WCS and WFS).

You should know that there are several steps to registering data with GeoServer, and they all can be completed through the REST API.

Creating a workspace (~namespace~) in GeoServer for these data is pretty straightforward:

> curl -u admin:geoserver -v -XPOST -H 'Content-type:text/xml' 
> -d '<workspace><name>earthws</name></workspace>' 
> http://localhost:8080/geoserver/rest/workspaces

Line 1 shoots off a curl request with authentication (-u), in verbose mode (-v) so we can bear witness to this electronic miracle, with a (-X) POST request type. It also gives the server a (-H) header-type heads-up that it should expect a payload in the format text/xml.

Line 2 passes data (-d) (in text/xml) that instruct GeoServer REST how the workspace should be created. You’ll notice that the workspace definition is pretty minimal; it only contains the
<name/>; within a similarly terse <workspace/>; wrapper. This is normal with the GeoServer REST API. Often, you only need to specify the minimal definitions, GeoServer defaults the rest sensibly.

Line 3 gives the command a place to happen – This is the REST endpoint for a local GeoServer. Since we only needed one workspace to hold all of our layers, this command was a one-off. In other situations, there’s nothing stopping us from using some script-fu to iterate over a collection of multiple workspace names.

Our next step: creating a datastore that provided the connection information to our PostGIS database was also a standard operation:

> curl -u admin:geoserver -v -XPOST -H 'Content-type:text/xml' -T earth_datastore.xml 
> http://localhost:8080/geoserver/rest/workspaces/earthws/datastores

You’ll notice a different approach in Line 1 of this command,. Now we use an external XML file referenced with the -T option, rather than inline XML with the -d option (as we did previously). Admittedly, it took some effort to get all the necessary parameters right (the example from the documentation link above is a bit sparse).

The XML for the datastore definition looks like this:

<dataStore>
<name>earthds</name>
<connectionParameters>
     <host>localhost</host>
     <port>5432</port>
     <database>earth</database>
     <schema>public</schema>
     <user>postgres</user>
     <passwd>postgres</passwd>
     <dbtype>postgis</dbtype>
</connectionParameters>
</dataStore>

It’s useful to know that this XML sample derived from a GET request on an existing PostGIS datastore through REST. We were able to save this file locally, nab the (minimal) parts we needed, edit as needed to define a new store, and then upload it to create that store. That’s the spirit of REST: Transferring various REpresentations of an object’s State. REST easy.

Finally, we incanted some more iterative BASH magic to ingest all of the tables from the PostGIS store/connection into GeoServer as distinct layers:

> for f in *shp; do
> curl -u admin:geoserver -v -XPOST -H 'Content-Type:text/xml'
> -d "${f %.shp}"
> http://localhost:8080/geoserver/rest/workspaces/earthws/datastores/earthds/featuretypes;
> done

There’s that BASH iteration coming back up on us. In this example, we rely on the fact that the list of files in the source directory is synced with the list of tables in PostGIS (minus .shp using ${f%.shp}). Naturally, there would be other ways to drive this iteration.

What was that line from “Field of Dreams”? If you build it (as an array, a collection or a manifest file etc.), they will come (and loop over ever item in the collection until the iteration is complete).” Not also that the “quotes” around the -d argument were needed to force that expression to be evaluated properly.

This all gets us close to, but not quite all of the way to GeoServer published data bliss. The data originally from shapefiles, and now in PostGIS, are available through GeoServer to any browser client that can connect to the server, but they don’t have any specific styling information added to them . They will all have been assigned the default style according to their geometry type (point, line or polygon). The default style is probably sufficient if you’re uploading rasters.

Adding SLDs as styles, and assigning them to a layer by REST is also possible, but automating the whole cartographic design process might be a bit tough. In this snippet a new style will be created and assigned to the layer created in the previous example. The following creates a new style named roads_style:

curl -u admin:geoserver -XPOST -H 'Content-type: text/xml' 
-d '<style><name>roads_style</name><filename>roads.sld</filename></style>'
http://localhost:8080/geoserver/rest/styles

You’d execute this request to upload the file roads.sld:

curl -u admin:geoserver -XPUT -H 'Content-type: application/vnd.ogc.sld+xml'  -d @roads.sld
http://localhost:8080/geoserver/rest/styles/roads_style

And do something like this to apply the newly created style to a layer:

curl -u admin:geoserver -XPUT -H 'Content-type: text/xml' 
-d '<layer><defaultStyle><name>roads_style</name></defaultStyle></layer>' 
http://localhost:8080/geoserver/rest/layers/acme:roads

As well, the newly registered layers are only available individually. Often, GeoServer layers are arranged into layer-groups to simplify display and consolidate requests for access.

Adding multiple files to a layer-group is possible by POST’ing a layergroup XML file to the relevant REST endpoint, and you could similarly build this layergroup manifest by iterating off of the files in a directory; however, order is important in a layergroup (first listed = first rendered = bottom of the drawing stack), so you’re going to want to inform your list of members so that they are layered properly.

<layerGroup>
  <name>earth</name>
  <layers>
     <layer>layer_1</layer>
     <layer>layer_N</layer>
  </layers>
  <styles>
     <style>style_1</style>
     <style>style_N</style>
  </styles>
</layerGroup>

This short, contrived snippet shows how you’d create a layer group through the GeoServer REST API. In well-formed XML, give it a name, give it a collection of layers, and (importantly) give each layer a corresponding style.

Now’s the perfect time to upload that mess of layers you’ve been sitting on. Try using the REST API and see how it goes. If you hit a snag  don’t hesitate to contact us with any questions.

 


cropped-ICON-01.png

Boundless Geo

Boundless has over 14 years of experience in the open GIS technology industry; serving as the leading open mapping software and solutions provider to government and commercial sector clients. As an active member of the open source community, including OSGeo and LocationTech, Boundless contributes to over 140 projects repositories. Boundless resides in the USA and has over 80 employees spread across the globe.