Adding Layers to GeoServer Using the REST API

GeoServerAt OpenGeo we’re often asked how to ingest large amounts of data into a system for storage (on disk, or in PostGIS) and publishing (through GeoServer and/or GeoWebCache).

GeoServer’s graphical administrative tools offer a convenient UI for managing data as workspaces, stores, layers and styles but will only handle them one at a time. If you have a significant amount of data to upload into GeoServer anew, or migrate from an existing implementation from another vendor, one-at-a-time just won’t cut it.

Enter GeoServer’s REST API (REST stands for REpresentational State Transfer). There’s a ton of general information online about REST but here it is in 50 words or less: REST lets you GET a representation of the current state of something through a URI, which you are then free to modify as/if needed, and finally PUT that representation back to the URI, where (if the request is authenticated and/or valid) the object’s state is modified.
In the GeoServer world, you might GET (a true HTTP GET) the current representation of a datastore, update something in that datastore configuration (correct a username, password or path), and then PUT the corrected representation back onto the server to update the configuration. If the service permits, you can also POST a new configuration to create a new object, or fire off a DELETE to an endpoint to remove an object from it. The specific methods you can apply to a REST endpoint and the objects you can apply those REQUESTS are often advertised by the endpoint itself. A RESTful interface should also offer written (prose) documentation of its capabilities.

GeoServer is no exception to these norms and written documentation is available in the GeoServer user manual. If you have GeoServer installed, the REST endpoint is available at:

http://<your.geoserver.url>:<port>/geoserver/rest

If you hit the base REST URL, you’ll notice that the format is HTML by default. The links, to workspaces, stores, layers, etc. should be consistent with your understanding of how (your) data is laid out in GeoServer. This is a nice way to discover GeoServer’s resource hierarchy, but it’s a bit thin on the details for any particular resource. Don’t worry, read on.

Specific object representations can also be requested through GeoServer’s REST API as XML or JSON that the requesting process can easily parse and/process by most simply appending the format type to the request (E.g. .xml or .json). We previously mentioned the term API (application programming interface) and the name is pretty self-descriptive. An API just means that there is an access point through which client applications can send and receive data. In this case the client can be something deeply nerdy like Java (etc) code, an application (coded in whatever) that wraps the API in an alternate UI (the OpenGeo Suite’s GeoExplorer is an application that shoots commands to GeoServer’s REST API), or something in- between like a shell script that uses cURL to automate requests to a REST interface based on virtually any rules or logic required.

Now for some examples …

One of our  clients came to us with a large number of shapefiles (over 900) that they wanted to publish with GeoServer. While there are a number of options for loading data through the graphical interface, all of these options would have been time-intensive. We suggested the REST interface to them, and provided a script employing the following utilities:

  • shp2pgsql (shapefile loader/dumper),
  • psql (command-line PostGIS client), and
  • cURL (a command-line URL handler).

Most of these commands were inspired by GeoServer’s user manual. The proposed solution uses cURL on BASH in Linux, but you could do very similar things in Windows with Powershell, in the Mac OSX or UNIX shells, or in PHP, ASP.NET, C#, Visual Basic, Python, (etc!) with virtually the same tool-set or concepts.

We started with shapefiles loaded en masse into PostGIS. Thought there are a variety of ways to do this, we prefer this method from the shell:

> for f in *.shp; do
> shp2pgsql -s 4326 -g geom $f ${f%.shp} | psql earth;
> done

In line 1 we use a handy for loop to iterate over all .shp files in the current directory as the variable f.

In line 2 we iterate within our collection of .shp files and call the shp2pgsql command on each current file ($f); the output of which is sent to the earth database through the command-line PostgreSQL tool psql.

shp2pgsql is a command-line tool that ships with PostGIS (and the OpenGeo Suite) and converts a shapefile into the SQL instructions needed to load that geographic data into PostGIS. All of the options and parameters for this tool are displayed when you type shp2pgsql without arguments at the command line. We use -s 4326 (add the SRID (EPSG:)4326 to the feature meta-tables) and -g geom to make sure the geometries are added to a column called, well ‘geom’.

Normally, shp2pgsql sends its SQL directly to standard out. You could ‘redirect’ (>) this output to a file and then run it into PostGIS separately; however, in this case we ‘pipe’ (|) the output directly to the psql command so that it is executed against the database ‘earth’. When these commands have finished up, we have loaded into PostGIS, but they’re not registered in GeoServer for exposure to clients through OGC compliant web-services (WMS, WFS, WCS and WFS).

You should know that there are several steps to registering data with GeoServer, and they all can be completed through the REST API.

Creating a workspace (~namespace~) in GeoServer for these data is pretty straightforward:

> curl -u admin:geoserver -v -XPOST -H 'Content-type:text/xml' 
> -d '<workspace><name>earthws</name></workspace>' 
> http://localhost:8080/geoserver/rest/workspaces

Line 1 shoots off a curl request with authentication (-u), in verbose mode (-v) so we can bear witness to this electronic miracle, with a (-X) POST request type. It also gives the server a (-H) header-type heads-up that it should expect a payload in the format text/xml.

Line 2 passes data (-d) (in text/xml) that instruct GeoServer REST how the workspace should be created. You’ll notice that the workspace definition is pretty minimal; it only contains the
<name/>; within a similarly terse <workspace/>; wrapper. This is normal with the GeoServer REST API. Often, you only need to specify the minimal definitions, GeoServer defaults the rest sensibly.

Line 3 gives the command a place to happen – This is the REST endpoint for a local GeoServer. Since we only needed one workspace to hold all of our layers, this command was a one-off. In other situations, there’s nothing stopping us from using some script-fu to iterate over a collection of multiple workspace names.

Our next step: creating a datastore that provided the connection information to our PostGIS database was also a standard operation:

> curl -u admin:geoserver -v -XPOST -H 'Content-type:text/xml' -T earth_datastore.xml 
> http://localhost:8080/geoserver/rest/workspaces/earthws/datastores

You’ll notice a different approach in Line 1 of this command,. Now we use an external XML file referenced with the -T option, rather than inline XML with the -d option (as we did previously). Admittedly, it took some effort to get all the necessary parameters right (the example from the documentation link above is a bit sparse).

The XML for the datastore definition looks like this:

<dataStore>
<name>earthds</name>
<connectionParameters>
     <host>localhost</host>
     <port>5432</port>
     <database>earth</database>
     <schema>public</schema>
     <user>postgres</user>
     <passwd>postgres</passwd>
     <dbtype>postgis</dbtype>
</connectionParameters>
</dataStore>

It’s useful to know that this XML sample derived from a GET request on an existing PostGIS datastore through REST. We were able to save this file locally, nab the (minimal) parts we needed, edit as needed to define a new store, and then upload it to create that store. That’s the spirit of REST: Transferring various REpresentations of an object’s State. REST easy.

Finally, we incanted some more iterative BASH magic to ingest all of the tables from the PostGIS store/connection into GeoServer as distinct layers:

> for f in *shp; do
> curl -u admin:geoserver -v -XPOST -H 'Content-Type:text/xml'
> -d "${f %.shp}"
> http://localhost:8080/geoserver/rest/workspaces/earthws/datastores/earthds/featuretypes;
> done

There’s that BASH iteration coming back up on us. In this example, we rely on the fact that the list of files in the source directory is synced with the list of tables in PostGIS (minus .shp using ${f%.shp}). Naturally, there would be other ways to drive this iteration.

What was that line from “Field of Dreams”? If you build it (as an array, a collection or a manifest file etc.), they will come (and loop over ever item in the collection until the iteration is complete).” Not also that the “quotes” around the -d argument were needed to force that expression to be evaluated properly.

This all gets us close to, but not quite all of the way to GeoServer published data bliss. The data originally from shapefiles, and now in PostGIS, are available through GeoServer to any browser client that can connect to the server, but they don’t have any specific styling information added to them . They will all have been assigned the default style according to their geometry type (point, line or polygon). The default style is probably sufficient if you’re uploading rasters.

Adding SLDs as styles, and assigning them to a layer by REST is also possible, but automating the whole cartographic design process might be a bit tough. In this snippet a new style will be created and assigned to the layer created in the previous example. The following creates a new style named roads_style:

curl -u admin:geoserver -XPOST -H 'Content-type: text/xml' 
-d '<style><name>roads_style</name><filename>roads.sld</filename></style>'

http://localhost:8080/geoserver/rest/styles

You’d execute this request to upload the file roads.sld:

curl -u admin:geoserver -XPUT -H 'Content-type: application/vnd.ogc.sld+xml'  -d @roads.sld

http://localhost:8080/geoserver/rest/styles/roads_style

And do something like this to apply the newly created style to a layer:

curl -u admin:geoserver -XPUT -H 'Content-type: text/xml' 
-d '<layer><defaultStyle><name>roads_style</name></defaultStyle></layer>' 

http://localhost:8080/geoserver/rest/layers/acme:roads

As well, the newly registered layers are only available individually. Often, GeoServer layers are arranged into layer-groups to simplify display and consolidate requests for access.

Adding multiple files to a layer-group is possible by POST’ing a layergroup XML file to the relevant REST endpoint, and you could similarly build this layergroup manifest by iterating off of the files in a directory; however, order is important in a layergroup (first listed = first rendered = bottom of the drawing stack), so you’re going to want to inform your list of members so that they are layered properly.

<layerGroup>
  <name>earth</name>
  <layers>
     <layer>layer_1</layer>
     <layer>layer_N</layer>
  </layers>
  <styles>
     <style>style_1</style>
     <style>style_N</style>
  </styles>
</layerGroup>

This short, contrived snippet shows how you’d create a layer group through the GeoServer REST API. In well-formed XML, give it a name, give it a collection of layers, and (importantly) give each layer a corresponding style.

Now’s the perfect time to upload that mess of layers you’ve been sitting on. Try using the REST API and see how it goes. If you hit a snag  don’t hesitate to contact us with any questions.

 

EmailTwitterFacebookGoogle+tumblrLinkedIn

9 thoughts on “Adding Layers to GeoServer Using the REST API

  1. Pingback: Month in Review: October 2012 – GeoServer Blog

  2. Hi,

    thanks a lot for this very useful tutorial on GeoServer’s REST API!
    I can succesfully reconstruct everything by myself except for the creation of a new layer from the PostGIS table.

    It seems you are passing only the table name as XML content to the curl request, but this doesn’t work for me. I’m wondering if it does for you? I’d guess that the call expects an entire XML featuretype file. Do I really have to build a new featuretype XML manually (similarly to the datastore or grouplayer feature)?

    Maybe you can help on how this worked for you.

    Cheers,
    Matthias

    • Hey Matthias (and Jeff) …

      Sorry it took so long for me to reply … I was on holidays for a while there.

      I mucked up the example for creating a featuretype/layer by name in a datastore. Try this:

      for f in *shp; do
      curl -v -u admin:geoserver -X POST -H 'content-type:text/xml'
      -d "${f/.shp/}"
      http://localhost:8080/geoserver/rest/workspaces/foobar/datastores/foobar/featuretypes;
      done

      You’ll notice that my string substitution is a bit different (the original code, used a shell extension), AND …

      The two enclosing tags are indeed missing – DOH! I’m not sure where they got stripped off, but they’re gone.

      Hope that helps – Good luck, and ping me if you need any more info.

      Sam

      You only need the name, but it does need to be wrapped in something … I’m note sure where those

  3. I’m using the PHP method of using curl to post to the rest api and had the same question as Mathias.
    Something like this seems to be working for me after a lot of brute force testing of various xml:
    $xmlStr = ‘
    ‘ . $layer_name . ‘
    true
    EPSG:4326

    ‘;

    $layer_name is the table name in your postgis DB.
    posted to: …/geoserver/rest/workspaces/earthws/datastores/earthds/featuretypes

  4. grrr, xml in websites… the square bracket below should be angle ones of course, changed to display better, $layer_name and srid are php variables that I’m putting in the xml:
    $xml = ‘[featureType]
    [name]‘ . $layer_name . ‘[/name]
    [enabled]true
    [srs]EPSG:’ . $srid . ‘[/srs]
    [attributes][/attributes]
    [/featureType]‘

  5. Hi,

    i am very new to this technology, i wonder if we can use geoserver to develop geoscience database portal without using arcgis for server which our department can’t think of buying,..its too expensive !!. any suggestions or help to my pertaining problem.

    rdgs

    lalit

Comments are closed.