#multigeometry | Explore Tumblr posts and blogs

kitwallace · 8 years ago

Text

Accessing GIS data

In addition to the open data provided by Bristol City Council’s Socrata-based https://opendata.bristol.gov.uk portal, BCC also provide documentation on many datasets via https://data.gov.uk which we can find by searching for ‘Bristol’ The list of over 600 sources (actually a large number of Bathymetric data from the UK Hydrographic Office) contains many which are available in CSV or XML format and these are straightforward to access since the documentation contains the URL to fetch the actual data file. Of course there are also some in PDF -nasty!

However geospatial data such as boundaries of wards or the location of libraries is accessed from the BCC GIS system directly. These services are described by WFS (Web Feature Service) and WMS (Web Map Service) meta-data files returned as XML. I have to confess I was confused about how to use these and gave up, until the other day when I needed to access the boundary data for parks and openspaces to be integrated with BristolTrees. With the help of my friend MT @agreensteam and one of the BCC GIS team, I have finally made some progress.

Park Boundaries

On data.gov.uk, I found an entry called ‘Bristol City Council INSPIRE WFS https://gov.uk/dataset/bristol-city-council-inspire-wfs [ I confess I’m still unclear what INSPIRE is -seems to be a EU standard?]

which in turn contains a link to the WFS service on the BCC arcgis server (ArcGIS is the leading commercial GIS platform developed by ESRI).

http://maps.bristol.gov.uk/arcgis/services/ext/INSPIRE/MapServer/WFSServer?service=WFS&request=GetCapabilities

In the WFS document, I can see a feature-type called ‘ Parks and greens spaces ’ in the features list but I had no idea how to construct a URL to access this data. My plea for help was answered and MT offered to extract the data for me, and, using a program called QGIS, she sent me the parks data in two formats: shapefile and geojson. Shapefiles are a directory of multiple files and looked frightening. geojson is just a JSON file and I have some code for parsing JSON with XQuery. However on a file of this size, full parsing failed and although I could drop back to a less formatted version and extract polygon data, processing and conversion of every point to latitude and longitude using my XQuery module was slow.

QGIS to the rescue

Finally I realised that QGIS was free and downloaded the latest version 2.18.4 which includes a WFS interface. The browser version displays GIS information but to export you need the desktop version.

Under the WFS menu, it was simple to add a new connection by providing the WFS URL. The metadata is loaded and an entry created for each layer ( wfs:FeatureType). Selecting a layer loads the data and displays the areas (wonder where I get a map as a backdrop?).

Layers can be exported in a wide range of formats. Being familiar with KML, I started with this format. I changed the CRS (Coordinate Reference System) to Conversion to WGS84 and extraction was very fast. I stumbled around trying to get the attributes exported in the name and description fields, following several stale trails from stackoverflow-like advice which pertained to earlier versions of QGIS. Finally I actually looked at the screen and saw the dropdown list ‘Set Fields to Export ..”. Attribute values are held in a generic XML structure:

<SchemaData schemaUrl="#parks1"> <SimpleData name="Site_name">Queen Square</SimpleData> <SimpleData name="Site_code">QUEESQ</SimpleData> </SchemaData>

This is difficult to index so after generating the kml and loading into eXist, I reformatted the xml to make selection of a given area faster and more convenient.

These areas are now included in the pages on BristolTrees where there was match. XQuery converts the polygons to JavaScript in the format needed by google.map.Polygon for access by the JavaScript mapping code. This is the area of the Avon New Cut (bit short of located trees at present!)

In retrospect, perhaps I shouldn’t have jumped at the first format I recognised. QGIS also exports GPX and GML. GPX is no use for this data since it doesn’t support multiple polygons, needed for many of the areas, but GML is a candidate.

QGIS is an amazing piece of software. Amongst other capabilities, QGIS can read and display local KML, GML and geojson files, with multiple layers for comparison. I’m very glad I’ve been introduced to it - thanks MT.

Plain REST URL

Now that I understood more about the interface, I looked again at the meta data and the brief guidance in Wikipedia. A bit of experimentation led to the URL

http://maps.bristol.gov.uk/arcgis/services/ext/INSPIRE/MapServer/WFSServer?service=WFS&request=GetFeature&TypeName=INSPIREWFS:Parks_and_greens_spaces

(where the value for the TypeName parameter is the wfs:Name of the wfsFeatureType in the WFS document)

to download the GML file directly. Conversion to lat/long is not supported directly and would have to be done in XQuery, but this does allow GIS data to be loaded directly into the XML database. Moreover the format of attribute data is more convenient:

<ogr:OBJECTID>4075</ogr:OBJECTID> <ogr:Site_name>River Avon Trail West</ogr:Site_name> <ogr:Site_code>AVONWAWE</ogr:Site_code> <ogr:UPRN>000000303853</ogr:UPRN>

The guide to the REST URL protocol is here

http://enterprise.arcgis.com/en/server/latest/publish-services/windows/communicating-with-a-wfs-service-in-a-web-browser.htm

Hand-draw boundaries

For areas for which BCC does not provide boundary data, these may have to be created. One way is to use Google Earth to hand-draw a path, taking care to close the path, and save as KML. KML exported from Google Earth represents the path by a LineString element rather than a Polygon as exported by QGIS, but the core coordinates element is compatible (that weird longitude, latitude ordering) with an additional altitude value so it needs to be converted to the MultiGeometry structure used in the QCIS export.

Here is the boundary for our residents-only Morley Square.

opendata.arcgis.com

Joe Brickley at BCC later supplied me with a direct link into the opendata access point on the arcgis platform for the parks data:

http://opendata.arcgis.com/datasets/a37403b31c0e4a5baae47385156efa44_10?geometry=-2.632%2C51.452%2C-2.496%2C51.471

from where the data can be downloaded either as Shapefile (shudder!) or KML. This is even simpler than getting the GML since the coordinate conversion has already been done. I found that this is an open portal only requiring a free registration. Once logged in and searching amongst the thousands of open datasets there for Bristol, I found the parks data but also datasets on TPOs and a dataset I hadn’t seen before, of Blue Plaques. This is point data so can be expressed in CSV and here too the coordinates are lat/long.

There are actually several datasets describing parks and openspaces. Impressively there are the boundaries of every BCC asset, including council houses, and every allotment plot. My next task is to extract schools to add to BristolTrees but the addition of over 350 boundaries is a nice enrichment.

conclusion

So now I have an embarrassment of ways of getting GIS data - QGIS using WFS metadata to get any of multiple formats, direct URLs to the BCC arcgis server for GML and access to those sets published to opendata.arcgis.com for KML. Progress!

Postscript May 2019

FILTER

With large datasets, it is either inconvenient or impossible to load the whole feature set (due to limits on the number of returned Features. There seem to be several URL parameters for filtering available but the only way I have found with the version of arcgis set up for BCC is to use the filter parameter with an XML filter definition.

For example to retrieve an asset from the list of BCC assets, this filter selects the asset with a Property Reference of 820:

<Filter%20xmlns="http://www.opengis.net/ogc"><PropertyIsEqualTo><PropertyName>INSPIREWFS:Property_reference</PropertyName><Literal>820</Literal></PropertyIsEqualTo></Filter>

Viewing Asset layer

add the Inspire Asset layer

add the OpenStreet Map layer

select the asset layer

use the layer style tab to change the opacity of the asset layer so the OpenStreetMap shows thru

use the Layer/Labeling tab to select which attribute to use as a label - I will use PropertyReference

#trees #GIS

0 notes