Monday, 9 December 2013

Storm surge

Last week the geography, planetary alignment and weather combined to cause misery to hundreds of families around Britain.

Tides around Britain naturally have a big range, the second highest tidal range in the world is in the Bristol Channel in Britain's south west. Tides vary depending on the alignment of the sun and moon, when the Sun and Moon are at the same side of the Earth (new moon) or opposite sides of the Earth (full moon) the tides are higher than when they are not in alignment. The highest tides are called spring tides, whatever time of year they occur.

The North Sea is roughly V-shaped, getting much narrower at the southern end. Last week a storm swept across the country, driving very strong winds from the north east down the North Sea coast forcing water in the sea towards the narrow part. The storm was, as usual, a deep low pressure system. With the low pressure over the sea, the water level will rise in a so-called storm surge.

The spring tide, the storm surge and the extra water blown down the North Sea created a lot of water pushing up against sea defences along the North Sea coast, particularly the southern end. In addition the waves thrown up by the stormy winds made topping the sea walls inevitable. Flooding followed.

Earlier this year I surveyed a new, small road close to the Humber Bridge, Wintersgill Place. Sadly the road was flooded. The houses now look finished, but there are three for-sale signs and one sold sign for the six houses. I wonder just how planning permission was granted for these houses when the area is clearly a flood risk. Now the small field next to these new houses is also set to be developed.

I tried to look at the local council's forward planning map to see if they agree that the area is a flood risk. The map was curiously off-line over the period of the storm and just after. Now it has lost the most detailed zoom level and the newly built houses are not on the map at all. I hope the planners have better information available, but since they have allowed houses to be built that have flooded before they were even sold, maybe they don't.

Wednesday, 27 November 2013

Using Leaflet with a database

The previous two posts created a map with markers. The marker information was stored in a fixed geojson file. For the few markers that don't change much this is fine, but it would be much more flexible if the markers were in a database. If there are a large number of markers, say thousands, browsers might slow down showing them, even though many might not actually be visible. One way to help with this is to work out which markers would be visible in the current view and only show them. To do this we need to use some features of Leaflet and introduce Ajax. We will also need to store the marker information in a database, write some code to extract it and format it into the geojson format that we know works so well.

Ajax is a means of exchanging data between the client browser and the server without forcing a page reload. I tend to use jQuery to simplify the process of using Ajax and jQuery ensures that the process works on a wide range of browsers. We will request some data from the server with Ajax which can return data in a json format, which works with geojson too.

In the examples so far the files from the server have been simple files, not needing scripting or a database. In my examples I'm using PHP for script and MySQL for the database as this is a very common combination available from many hosts. In the GitHub repository there is a SQL file, plaques.sql, you can use to create a table called plaques in a MySQL database and import the same data that we have seen already.

To extract the data from the database we'll use a PHP script. It needs to receive a request for the bounding box and it will extract that, format the geojson result and return it to the client. The client then can display the markers. If the user scrolls the map or changes the zoom then a new Ajax request will get the markers that are in the new view and display them. This isn't really needed for the seventy or so markers in this example but it is very useful for a large number of markers.

Let's start with the PHP script to extract the data:


// uncomment below to turn error reporting on
ini_set('display_errors', 1);
error_reporting(E_ALL);

/*
 * ajxplaque.php
 * returns plaque points as geojson
 */

// get the server credentials from a shared import file
$idb= $_SERVER['DOCUMENT_ROOT']."/include/db.php";
include $idb;

if (isset($_GET['bbox'])) {
    $bbox=$_GET['bbox'];
} else {
    // invalid request
    $ajxres=array();
    $ajxres['resp']=4;
    $ajxres['dberror']=0;
    $ajxres['msg']='missing bounding box';
    sendajax($ajxres);
}
// split the bbox into it's parts
list($left,$bottom,$right,$top)=explode(",",$bbox);

// open the database
try {
    $db = new PDO('mysql:host=localhost;dbname='.$dbname.';charset=utf8', $dbuser, $dbpass);
} catch(PDOException $e) {
    // send the PDOException message
    $ajxres=array();
    $ajxres['resp']=40;
    $ajxres['dberror']=$e->getCode();
    $ajxres['msg']=$e->getMessage();
    sendajax($ajxres);
}

//$stmt = $db->prepare("SELECT * FROM hbtarget WHERE lon>=:left AND lon<=:right AND lat>=:bottom AND lat<=:top ORDER BY targetind");
//$stmt->bindParam(':left', $left, PDO::PARAM_STR);
//$stmt->bindParam(':right', $right, PDO::PARAM_STR);
//$stmt->bindParam(':bottom', $bottom, PDO::PARAM_STR);
//$stmt->bindParam(':top', $top, PDO::PARAM_STR);
//$stmt->execute();


try {
    $sql="SELECT plaqueid,lat,lon,plaquedesc,colour,imageid FROM plaques WHERE lon>=:left AND lon<=:right AND lat>=:bottom AND lat<=:top";
    $stmt = $db->prepare($sql);
    $stmt->bindParam(':left', $left, PDO::PARAM_STR);
    $stmt->bindParam(':right', $right, PDO::PARAM_STR);
    $stmt->bindParam(':bottom', $bottom, PDO::PARAM_STR);
    $stmt->bindParam(':top', $top, PDO::PARAM_STR);
    $stmt->execute();
} catch(PDOException $e) {
    print "db error ".$e->getCode()." ".$e->getMessage();
}
   
$ajxres=array(); // place to store the geojson result
$features=array(); // array to build up the feature collection
$ajxres['type']='FeatureCollection';

// go through the list adding each one to the array to be returned   
while ($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
    $lat=$row['lat'];
    $lon=$row['lon'];
    $prop=array();
    $prop['plaqueid']=$row['plaqueid'];
    $prop['plaquedesc']=$row['plaquedesc'];
    $prop['colour']=$row['colour'];
    $prop['imageid']=$row['imageid'];
    $f=array();
    $geom=array();
    $coords=array();
   
    $geom['type']='Point';
    $coords[0]=floatval($lon);
    $coords[1]=floatval($lat);
   
    $geom['coordinates']=$coords;
    $f['type']='Feature';
    $f['geometry']=$geom;
    $f['properties']=$prop;

    $features[]=$f;
   
}
   
// add the features array to the end of the ajxres array
$ajxres['features']=$features;
// tidy up the DB
$db = null;
sendajax($ajxres); // no return from there

function sendajax($ajx) {
    // encode the ajx array as json and return it.
    $encoded = json_encode($ajx);
    exit($encoded);
}
?>



This is called ajxplaques.php in the folder ajax on the server, available in the GitHub repository.  The script needs a query string with bbox= in it. This defines the west,south,east and north longitude and latitude that bounds the current view of the map. It then queries the database for these items and returns the geojson of these limited markers. If the bounding box (BBOX) is big enough then all the markers will be returned and if the BBOX contains no markers then none are returned and that is fine too. I'm using MySQL and ignoring GIS functions as selecting points is quick and easy. If I was extracting polygons and using a powerful GIS database such as PostrgreSQL with the PostGIS extension then I would consider using a GIS function to find the polygons that intersect the BBOX.

To call the script from the JavaScript (example3.js) I use the ajax functions that are part of jQuery:

function askForPlaques() {
    var data='bbox=' + map.getBounds().toBBoxString();
    $.ajax({
        url: 'ajax/ajxplaques.php',
        dataType: 'json',
        data: data,
        success: showPlaques
    });
}


This creates the query string by using map.bounds() and formats into the format we need with toBBoxString(). The $.ajax() function uses the query string, requests json (of which geojson is just a special case) and will call the function showPlaques() when the data is returned.

function showPlaques(ajxresponse) {
    lyrPlq.clearLayers();
    lyrPlq.addData(ajxresponse);
}


The showPlaques() function is called when the data is returned from the script. The geojson data is in the ajxresponse. We delete all of the existing markers with clearLayers() and add the new data to the geojson layer. To trigger this process we need to call askForPlaques() every time the view of the map changes. We can ask the map object to trigger an event whenever this occurs. So after the map is displayed we add

map.on('moveend', whenMapMoves);

This calls the function whenMapMoves() when the event is triggered. That function simply calls  askForPlaques() to get the correct data for the view.

Two more things have changed. Firstly, when the geojson layer is created no data is added - it is called with null - so the plaques.js is not used at all. When the map is first displayed we need to call askForPlaques() once to get the first set of markers before the map is moved.

Now we have a much more dynamic map, using data from a database and potentially using a part of thousands of markers without overloading the browser.

Thursday, 21 November 2013

Using Leaflet part 2

In the last post I described how to create a map with markers on it. I'm going to build on this to make some improvements. Firstly, the default markers are useful but it would be good to have some alternatives. I copied the example1.htm, .js and .css files to example2.x and made some changes there. I created three images which are blue, green and white disks and saved them in the images folder. To use these images we need to create a Leaflet Icon for each marker type. The variables used to store the Icon objects are declared in the global area and then the Icon is created before anything else is created so it is available whenever we need it.

    blueicon=L.icon({
        iconUrl: 'images/blueplaque.png',
        iconSize:[24, 24], // size of the icon
        iconAnchor:[12, 23] // point of the icon which will correspond to marker's location
    });
    greenicon=L.icon({
        iconUrl: 'images/greenplaque.png',
        iconSize:[24, 24], // size of the icon
        iconAnchor:[12, 23] // point of the icon which will correspond to marker's location
    });
    whiteicon=L.icon({
        iconUrl: 'images/whiteplaque.png',
        iconSize: [24, 24], // size of the icon
        iconAnchor:[12, 23] // point of the icon which will correspond to marker's location
    });


The icons use the images from the images folder. The icon anchor is important to make a marker work well especially when zooming. If your marker seems to slide around as you zoom in or out you probably don't have the anchor set correctly. It defaults to [0,0] which is top left and rarely what you want.

In the last example we used events to add popups to each markers, here we want to substitute the markers, so we need to change the behaviour, but not much. When we create the geojson layer we need to remove the onEachFeature and replace it with pointToLayer:

  lyrPlq = L.geoJson(plaques, {
        pointToLayer: setIcon
        }
    );


The pointToLayer event allows you to provide the marker you want to display for each feature. The function you define, setIcon in this case, gets passed the feature and the position that the marker need to be as a leaflet LngLat object. In the function you create a marker or some other Leaflet object and return that. Leaflet adds what you return to the layer.

function setIcon(feature,ll) {
    var plq;
    if (feature.properties.colour=='green') {
        plq=L.marker(ll, {icon: greenicon});
    }
    else if (feature.properties.colour=='white') {
        plq=L.marker(ll, {icon: whiteicon});
    }
    else {
        plq=L.marker(ll, {icon: blueicon});
    }
    plq.bindPopup(feature.properties.plaquedesc);
    return p;
}


In this case we use one of the properties, colour, to choose the marker to display. We now need to add the popup to our newly created marker and return the marker for Leaflet to use.

You can see an example here. The source code is on GitHub.

More examples to come ...

Using Leaflet v0.7

The Leaflet JavaScript library has changed the way OpenStreetMap is being used, making it easy to use and offering all kinds of additional features and functions as plug-ins. I blogged about Leaflet soon after it was first released and that post has been read by a lot of people and has generated more comments than any other. Leaflet version 0.7 has just been released and when I was asked about using it I realised that my original post was badly out of date. I decided to use some local data to describe using Leaflet, including some plug-ins. I decided to use jQuery for a few features. It is widely used and is cross-platform, just like Leaflet. The jQuery files are in a folder called jquery, and the Leaflet files are all in a folder called leaflet.

All the example files are in a GitHub repository: http://github.com/chillly/plaques

Leaflet displays a slippy map in an HTML div. It uses JavaScript to control the way the map behaves. The style, as you would expect, is controlled by CSS. Our first example displays a base map with an overlay of markers on it to show where blue plaques are around the UK city of Hull.  Take a look here. The HTML is really straightforward, take a look in the GitHub repository above.

In the head section there is a style sheet for leaflet (leaflet.css) and a script (leaflet.js). These are used in every example. I have also included leaflet-hash.js which is an example of a Leaflet plug-in. I like to store CSS and JavaScript in separate files, not in the HTML file, so I have also included example1.css and example1.js. The JavaScript names plaques.js holds the locations of the plaques to display, formatted as a geojson file. The CSS simply makes the div, with the id “mapdiv”, fill the page. The real work is in the javascript:


/* 
* global variables 
*/ 
var map; // global map object 
var lyrOsm; // the Mapnik base layer of the map 
var lyrPlq; // the geoJson layer to display plaques with 

// when the whole document has loaded call the init function 
$(document).ready(init);

function init() { 
  // map stuff 
  // base layer 
  var osmUrl='http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png';
  var osmAttrib='Map data © OpenStreetMap contributors';
  lyrOsm = new L.TileLayer(osmUrl, { minZoom: 9, maxZoom: 19, attribution: osmAttrib });

  // a geojson 
  layer lyrPlq = L.geoJson(plaques,{ 
    onEachFeature: makePopup
    } 
  ); 

  // set the starting location for the centre of the map 
  var start = new L.LatLng(53.7610,-0.3529); 
  
  // create the map 
  map = new L.Map('mapdiv', { // use the div called mapdiv 
    center: start, // centre the map as above 
    zoom: 12, // start up zoom level 
    layers: [lyrOsm,lyrPlq] // layers to add
   }); 

  // create a layer control 
  // add the base layers 
  var baseLayers = { "OpenStreetMap": lyrOsm }; 

  // add the overlays 
  var overlays = { "Plaques": lyrPlq }; 

  // add the layers to a layer control 
  L.control.layers(baseLayers, overlays).addTo(map); 

  // create the hash url on the browser address line 
  var hash = new L.Hash(map);


function makePopup(feature, layer) { 
  // create a popup for each point 
  if (feature.properties && feature.properties.plaquedesc) {
    layer.bindPopup(feature.properties.plaquedesc);
  } 
}

The file starts with global variables. The map variable is the core of Leaflet, any name would do, but it is called map by convention. The map has two layers, one to display the base map and one to show the markers. These are defined by two variables lyrOsm and lyrPlq.

The first use for jQuery is:

    $(document).ready(init);

This means that when the document is completely loaded and ready call the function init. Doing this is very useful as on a slow link, such as some mobile connections. It makes sure that all of the elements of the page are available before trying to use them.

The init function needs to create the layers, create the map itself and add the layers to the map. We need two different layers one is the base layer which is a set of square tiles  that Leaflet requests from the provider and arranges in the right place. We need to tell the layer where to get the tiles from, in this case the main OSM tile server.

    var osmUrl='http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png'; 
We also need to add attribution – all OSM-based maps need attribution as part of the licence to use the data.

    var osmAttrib='Map data © OpenStreetMap contributors';

The tile provider may also require or request attribution. In one of the later examples will will use a different tile provider with a different attribution. We can now make a layer:

lyrOsm = new L.TileLayer(osmUrl, {
    minZoom: 9,
    maxZoom: 19,
    attribution: osmAttrib
});


This creates the layer of tiles, from the tile provider, which will zoom out only to zoom level 9 and in to level 19 and add the attribution to the bottom right of the map, by default.

We also need a layer to display the plaque markers. In the HTML a file called plaques.js was loaded. That contains a geojson format file which we will use in a geojson layer. Geojson is a very useful format that I use frequently and is supported well by Leaflet. Geojson allows a variety of objects to be passed to Leaflet for displaying, including points, lines, multiple lines, polygons and multipolygons and these can be mixed together as needed. With a simple set of points the loaded file can be simply used as a layer. The file plaques.js creates a variable called plaques that can be used directly:

    lyrPlq = L.geoJson(plaques, {
        onEachFeature: makePopup
        }
    );


The onEachFeature is an example of responding to an event. In this case as each feature of the geojson file is added the function makePopup is called. This allows us to use one of the properties in each feature in the geojson file to be used to make a popup if the marker is clicked.

We now create a LatLng object to use to centre the map and then the map object is created:

map = new L.Map('mapdiv', {        // use the div called mapdiv
    center: start,                   // centre the map as above
    zoom: 12,            // start up zoom level
    layers: [lyrOsm,lyrPlq]        // layers to add
});


This creates the map, centres it, zooms to level 12 and adds the two layers we created above.

That would be enough to create a slippy map, but I added a couple of extra features which are often useful. The first is a layers control which allows the layers on the map to be selected and hidden. There are two types of layer a base layer and an overlay and we have one of each. The two layers are created, with names that will appear in the layer control, and the control is then created with the layers added and then added to the map.

The last feature is a leaflet plug-in. I added the leaflet-hash.js file to the leaflet folder and loaded it in the HTML. The hash plug-in changes the URL displayed in the browser address line as the map is scrolled and zoomed so the address can always be used as a bookmark. It replaces the permalink used on earlier versions. A simple line adds the plug-in to your map.

Following posts will show how to change the icons that appear, customise the popup, use database data to display the markers, deal with a large number of markers including clustering them and how to respond to click or tap in other ways than just displaying a popup.

Friday, 18 October 2013

Land Registry INSPIRE polygons

The UK Land Registry has released some open data as part of the INSPIRE directive. I say open data, but not very open, indeed hardly even ajar.

The Land Registry has released various data sets as recorded in this blog post, but there is a serious issue with some of their data, the polygons that show the cadastre, or property bounds, data. As you might expect, the Land Registry use the UK's national mapping agency, Ordnance Survey, to base any mapping on. So when they reproduce any GIS-based data, it is based on OS maps and, as usual, OS claims that their viral licensing clause applies. So even though the Land Registry maintain the cadastre data and want to publish it under the liberal Open Government licence,  OS prevent that by adding a personal-only, no publishing clause. This is a further example of how OS viral licensing is harming innovation and thus damaging business and potential open data users.

I have been using the Land Registry polygons to research the accuracy of creating Openstreetmap cadastre data simply by using surveys, permitted aerial imagery and local knowledge. I have decided to publish an example of what I have found under fair dealing of OS copyright and database rights. If OS are not happy about this I'm sure they will contact me and we could discuss it.

Some time ago I added all of the buildings in the postcode area HU14, Melton, North Ferriby and Swanland and added their address. In North Ferriby I also added all of the hedges, fences etc around properties. It took a lot of doing, using aerial imagery and some local knowledge. Having done it I decided that I wouldn't do any more places like that - I would only add buildings and address at most.

Now I have the means to check how good a job I managed to do, by comparing my boundary lines with the cadastre polygons from the Land Registry. I extracted a small area from their data, reprojected it to Mercator and used TileMill to render a simple view of part of North Ferriby with the cadastre polygons laid over it. Here are the three images:
OpenStreetMap data showing gardens and buildings

Land Registry cadastre polygons

OpenStreetmap data overlaid with Land Registry cadastre polygons

I am very pleased with the way the two datasets coincide. The biggest area of error in creating the Openstreetmap boundaries is the alignment of the aerial imagery - that seems to be very good in this instance. I did use multiple GPS traces to create an average trace and aligned the imagery to that. That seems to work well.

The Openstreetmap images © Chris Hill, CC-BY SA 2.0 which is based on Openstreetmap data © Openstreetmap contributors ODbL 1.0

The Land Registry data require the following statements:
This information is subject to Crown copyright and is reproduced with the permission of Land Registry

© Crown copyright and database rights [2013] Ordnance Survey 100026316

Thursday, 3 October 2013

Tram depot and Boothferry Park

Recently, sladen added some tram lines close to Wheeler Street off Anlaby Road in Hull. I was surprised to see railway=tram appearing in Hull as I hadn't heard the city was joining the enlightened ones adding smart new public transport options for the commuters. It turns out that that was the site of the old tram network depot. Trams were progressively phased out by mid 1940's in favour of trolleybuses, using the same overhead infrastructure but without the rails. They ran until 1964. After a short conversation on IRC with sladen I set out to see if I could find any signs of track at the site and any other clues about the depot.

I arrived at the Wheeler Street depot tram entrance on Anlaby Road. The tracks were clearly visible in the old cobblestone roadway. It is now a private yard of some kind. I walked a little way down the roadway before being stopped by a large chap who wanted to know what I wanted. I explained about it having been a tram depot and asked if I could see the rest of the tram lines, he said no, and told me to f*** off where I'd come from. I left.

I suppose the camera round my neck and the GPS in my hand might have been intimidating, but I was polite. It is quite unusual in my experience to get a response like that when out mapping - it just made me wonder just what was going on on the site.

I tried to see into the yard from Wheeler Street but the site has buildings and high walls around it, so I couldn't see any more details.
Overall I could only confirm what sladen had seen elsewhere already, there are a few tram lines left, but not the extent of them. Judging by recent aerial images the building had gone. Older aerial images showed the shell of a building with no roof on it. I'm not able to add to that at all.

I checked the restored oneway section of Wheeler street near Wheeler primary school, confirmed that the opposite cycleway has been restored after being removed during some building work. I then moved along Anlaby road to where Boothferry Park, the football ground of Hull City, used to be. It is in the process of being turning into a housing estate. The first houses are occupied now but there is still a lot more to build. The main drag is called Legends Way with the first side road to the east called Boothferry Park Halt. Boothferry Park was unusual because it had its own railway station, only used for match days and only used until the mid 1980's. I'll need to check the site some more in a few months to catch up with the new developments.

Thursday, 12 September 2013

Open data? Hmmmm

I have been impressed with SK53's use of open data in Nottingham, so I thought I'd see what was available for my home county of the East Riding of Yorkshire and for nearby Hull. The answer has been disappointing and frustrating.

I decided to look at the web site of the two councils in the hope that either of them had started to publish any open data. Not a trace of anything vaguely resembling publishing anything as open data on either site. For example, ERoY site has a map showing current road works. It needs MS Silverlight to see it, and when you do look, the data is copyright. Hull City website has a link to food hygiene data on the landing page, but that only sends you to the Food Standards Agency site, not a local data set and, as before, everything on their website seems to be copyright.

So, using SK53's example, I asked for food hygiene data from the two councils, in the hope that I would find a list of addresses just like he did. I sent an email request on Saturday 17th August to each council. The request said that I wanted to analyse food hygiene data and I would like to know if they publish such data as open data, preferably under the Open Government Licence. As I usually do in such matters I asked them to treat this as request under the Freedom of Information act.

Given that the request was sent on a Saturday, it is reasonable to expect that no one read it until the following Monday, 19th August, so I would expect the twenty working days clock that applies to FoI requests to start then. Allowing of the public holiday in August I would expect an answer by 17th September and both councils beat that deadline comfortably.

Hull City council responded quickly. On 21st August I got a reply from Garry of the Food and Health & Safety (sic) team to say they had been busy and would I please send my request to the Information Governance Team. He did also direct me to the FSA website too. This is directly contrary to the FoI act. It is also contrary to the Hull City Council policy on the FoI act. I replied pointing this out. The Information Governance Team responded by saying that the officer was wrong to direct me to them, apologised and said that the email should just have been forwarded to them internally. They confirmed that the deadline for their response would be 16th September. On the 29th August, well within the deadline, the Information Governance Team answered with the same idea as Garry: look at the FSA site.

ERoY council first replied on the 22nd of August. The email had an attached MS Word document. The first page had headings but was otherwise blank, the second page acknowledged my request, repeating my email, and the third page confirmed that they would reply by the 17th September, as I expected. On the 9th September I got a response. That too was an MS Word document attached to an email. That too recommended that I look at the FSA website to find such data. They gave a link to a council website page that includes a list of registered food premises. This is a simple list, with no description and no licence or copyright information, so it is not possible to assume that it is open data.

When I was disappointed with the lack of open data on either councils' websites I also sent an email to the Department for Communities and Local Government. They have suggested in the past that councils should expect to publish any data they have under an open licence. I asked if the DfCLA or my local Member of Parliament might be able to do anything to apply pressure to the councils to publish any data as open data as a driving force for openness and innovation. Today I got a response from Rt Hon David Davis MP, my MP, with an attached letter from Malcolm Sims, Director of Corporate Resources, East Riding of Yorkshire council. Mr Sims confirms that they have received an FoI request and that they will reply by 20th September, which they already have. Neither Mr Davis nor Mr Sims seem to grasp the underlying point: they don't publish open data, and don't seem to want to.

The art of politics is to give a response to a question that wasn't asked, but that you can answer.

Sunday, 18 August 2013

Rights of Way

I saw mention of some open data about rights of way in various English and Welsh counties on http://www.rowmaps.com/. I downloaded the shapefile and took a look at it. The rights of way are stored as MultiLineString because a few are broken into two sections, though most are single lines. I wanted to see how they relate to any rights of way in OSM. I am not sure about the licence - there is no statement from East Riding of Yorkshire council about the licence so I cannot load this in any way into OSM. I do have other reservations about the data too, but more of that later.

My web site provider, like many cheap one, provide PHP and MySQL to create a site. I have not used the spatial functions in MySQL before so I decided to see how well they work and in the process make an overlay for an OSM map with these rights of way on it. I created a simple python script to load the data into a very simple MySQL table, loading the MultiLineString into a geometry field. The functions that work with the geometry fields in MySQL are much less capable than PostGIS in PostgreSQL. MySQL allows the geometry be to selected based on a bounding box. This is ideal for me as I want to select the items that show on a section of map that is currently being displayed. Its bounding box is easy to find. The MySQL function is MBRIntersects. The bounding box of the current map being displayed is converted into a polygon and this is compared to the bounding box that encloses each of the MultiLineString in each row and if the two bounding boxes intersect then the row is selected. This means that some part of the right-of-way that the data represent should be displayed on the displayed window of the map. So how to display it?

I like the Leaflet library to display maps and Leaflet has good support for displaying geoJson as an overlay. When the map is first displayed or scrolled or panned I use the new bounding box to create an ajax request which returns the data selected as a geojson. This is a PHP script which uses the bounding box to convert into a simple SELECT statement and parses the returned data into geoJson. The existing overlay is cleared and the freshly returned data are added to the geoJson layer.

A geoJson layer is a collection of features, in my case each feature is a MultiLineSegment from the original shapefile. It also has properties which we got from the shapefile too. As each feature is processed the style can be processed dynamically. In my case the footpaths are made red, the bridleways made green and the byways are blue; one of the properties is the right of way type. I also use the onFeature event to call a function that adds an onClick event to each feature. When a feature is clicked the properties can be displayed in a side bar. I added a hash to the address line too to match the standard OSM map. You can see the result here.

So now I can compare the OSM map with the data from the council. The first thing to say is that there are a lot of rights of way on the overlay (1762) and a significant number are clearly missing from OSM. On closer examination though the data does not look very accurate nor very up to date. In West Ella a footpath seems to go through a couple a of houses. Elsewhere a local footpath was moved many years ago, but the old route is still shown. OSM shows the surveyed and signed route. There are other anomalies too.

Having the overlay does provide a good place to start surveying what is really on the ground and signed, but loading the data verbatim doesn't look a very good idea even if the licence was published and acceptable.

I think I may yet extend the overlay to show which rights-of-way have been surveyed and matched with OSM ways. It looks like there's some walking and cycling ahead, which in the East Riding countryside is never a hardship.

Sunday, 11 August 2013

Comparing layers

About three years ago I created a simple map using OpenLayers to display a base layer and overlay some layers on it. It features layers useful to some GB mappers. From the logs I can see that it is used quite a lot.  All of the slippy maps I've created recently have used Leaflet rather than OpenLayers so I thought I should update my comparison map in Leaflet. You can see the result here.

It has base layers of MapQuest open, the Standard OSM layer or MapQuest roads. For overlays you can choose from the ITO World tiles of OS OpenData Locator anomalies, the OS OpenData StreetView or my tiles showing the Office of National Statistics postcode centroids. Whichever of the overlays are shown you can move the slider (top-left) to change the opacity to make the base or overlay readable.

The browser address line shows the constantly-updated location and zoom, in the new style of the OSM landing page. This makes bookmarking the position easy. There is also a lat / lon display bottom right which shows you the location the mouse is currently pointing at.

I intend to leave the existing oscompare unchanged and maintain this new leaflet-based one in the future. If anyone would like any other layers or if there is any problem please comment.

Thanks to MapQuest and ITO World for their map tiles and their processes for updating them, thanks to Ordnance Survey for their OpenData and thanks to the OSM admins who manage the OS StreetView tiles.

Tuesday, 9 July 2013

Locating bridges

I haven't been able to get out mapping much recently, but a lovely day tempted me out. I did some OS Locator name chasing and as is often the case, the OS Locator names led me to a development with far more changes than just the ones OS Locator picks up. I will need to return again as the development grows in Kingswood. There's lots of area obviously earmarked for development but presumably waiting for the cash generated from selling the current houses to fund the next stage.

I then went to look at a newly opened crossing of the river Hull, Scale Lane bridge. It is a pedestrian and cycle route across the river very close to the museum quarter. It lead directly to a large fairly new hotel and towards The Deep, Hull's aquarium, or Submarium as they call it. If you are in the city take a look at The Deep, it is interesting. You walk slowly down to see lower and lower levels in the aquarium and then a glass lift carries you from the bottom back to the top to looking out at the large aquarium as you go.

Scale Lane Bridge, the black part swings
The new bridge is interesting too. If it opens for river traffic, gates shut at the east end and this end swings around to open for boats, but anyone on the bridge can stay on and people can still get on and off the bridge at the west end. In the middle of the bridge is a building which will be a café eventually and a small exhibition space too I think. These is a sloped cycleway and a flatter footway, except the footway has steps along it. All along the footway there are seats with views looking across the river. It didn't swing open while I was there, but it still looks quite different for a bridge and I like it.
The Arctic Corsair trawler, now a museum, beside the river Hull walkway


The main road crossing of the river Hull and beyond the tidal barrier

I then took a look at a new development close to home. The Bridgehead development is close to the north end of the Humber bridge. It looks like it will be the usual for commercial buildings: steel structured, clad with glass and shuttering. The first two buildings are under way with room for a few more. The roads are accessible so I grabbed a trace. It is interesting that cycle lanes are being included in the road layouts, since the entrance is off the A15 dual carriageway where there are no cycle lanes. Maybe a new cycleway will join up from the development to the existing cycle lanes nearer the bridge and beyond.

Wednesday, 3 July 2013

OS Locator chasing

I finally got around to sorting out some new OS Locator anomalies close to home. I went to look at a new small road under the shadow of the Humber bridge. When I got there the road was dug up for pipe laying. The tarmac has been down less than 3 months and already the utilities are digging it up again. As I was standing looking down the road, wondering if I should brave the excavators, a woman in a hard hat and fluorescent jacket approached me smiling. She asked if I was interested in the new houses, which are not finished. When I explained I was making a map the smile faded but she agreed I could walk to the end of the road with my GPS 'at my own risk'. Since everything I do tends to be at my own risk I collected my trace. On the way out I asked the once more smiling woman what the road name is going to be and she confirmed the same name that OS Locator has.

Next I went to Welton. Here a slightly bigger development has been squeezed into what was a farmyard many years ago. Here too the road was uneven, but only because the final surface has not been laid. None of the houses looked finished, indeed some were barely started. Here too I was approached by a florescent jacket, this time draped around a young man with a clipboard. He too wanted to sell me a house, but when he heard that I was making a map he asked me to leave as it was still a building site and not safe for the public. As I already had my trace I left, this time snapping a picture of the new road's name board - about the only part of the development that was complete.

I am used to roads showing up in OS Locator as they are close to complete, so seeing ones so obviously early in their development is a little different. Maybe OS are trying to get ahead of the game.

Monday, 6 May 2013

So do the OSMF working groups work?

I do wonder if the process of working groups really works. I have asked the Licence Working Group to make a simple action to prevent a falsehood being repeated. They did nothing and did not even respond to my emails.

I have now asked the Data Working Group to revert undiscussed mass edits. Again, no action and not even an acknowledgement email. If we are serious about stopping undiscussed mass-edits, the only real way forward is to revert them as quickly as possible to force people to follow the guidelines. If an ordinary mapper, like me does the revert that risks an edit war, especially if the ordinary mapper has already engaged with the mass-editor to no effect as in my case. The longer it takes to revert a mass-edit the harder the revert becomes because of other small-scale edits.

If the working groups are not going to take action or at least acknowledge a request or contact, then they should resign and close down the group so everyone knows that the only process open to an ordinary mapper is down to him.

Friday, 5 April 2013

Enclosures

I had a meeting today about a project in our village. Ian had a map that he has assembled of Swanland & North Ferriby that shows the villages pre-Enclosure Acts. It would be good to create an overlay with the modern OSM map for people to see how things have changed in nearly 200 years. In case you haven't guessed, the answer is lots.

I need to warp Ian's map to fit the EPSG:3857 projection, make its background transparent and then build a little app to show one layer over the other. I'll see how I get on.

Sunday, 3 February 2013

A Park, Hawthorn and Rugby

I've been trying to fill the gaps in land use in Hull. I think that, generally, everywhere in a city can and should have a landuse tag and that land use areas probably should join at their edges. I have been joining, say, a residential land use to a commercial land use, sharing the nodes, so there is no gap between them. I have not been joining land use areas to roads. That is just wrong. A few land use areas stop at a boundary with a fence or a hedge, so I sometimes reuse the land use nodes to add a barrier tag, but never a road. Roads are, I think, too complex to be shared with a land use edge. Roads do fall into the land use. I residential road is part of the residential area along with the houses and occasional shop and pub too. Parks, schools, recreation grounds, cemeteries and other large areas I've generally left out of other land use - layering land use on top of, but another land use seems wrong to me. I have tried not use too many multipolygons with land use, so a pond in a park will not be a hole in a multipolygon, though it would work well if it was.

This had involved a lot of staring at aerial imagery followed by surveys to try to determine where one land use type gives way to another. There have also been a lot of areas that seem to have no land use. I have resorted to landuse=grass for space between a river bank and, say, an industrial area nearby. I'm not happy that 'grass' is a land use, but then many OSM tags have grown to be useful without their actual keys or values being completely logical. OSM is about a map database not some kind of taxonomy after all. I have also used natural=scrub in some places too. Again not sure but I think it is better than nothing. It is really urban waste ground, but that doesn't seem to be in the wiki. I have used landuse=railway for the inaccessible land fenced off beside railways which seems right to me. I think there should be a landuse=highway tag for major roads and motorways too. They often have an area of inaccessible land reserved beside them, sometimes beyond the verge perhaps up an embankment. In Britain this is usually grassed sometimes with bushes or small trees on to too. They have become useful wildlife havens.

I noticed on the aerial images a largish piece of grass on the outskirts of the city that I didn't recognise. When we went to look we found a large grassy park so I added that. It was very, very wet - flooded in parts, so I didn't venture onto it, though I will later in the year just for a look around at what lies beyond some trees and bushes and whether they too are part of the park or not.

Part of my work has been focussed on areas of Hull that are being redeveloped - and I haven't completed them yet. Hawthorn Avenue had a lot of terraced streets of it on both sides. Some had fallen into disrepair and even dereliction and the council declared the will to redevelop some of the area. I don't want to give the impression that it was or is a slum. Just some areas where becoming a mess and the houses were getting harder to maintain and costing more to heat, for example. Yesterday we took a look at the work that is progressing. Areas of Woodcock street have been transformed, with a mixture of size and styles of houses, a small park, sensible car parking, cycle ways and foot ways and the whole area looks impressive. I don't know anyone who lives there to see if there is a community spirit building, but somehow there is a feeling of a place that could be very pleasant to live in.
Greek Street boarded up

Further north along the eastern side Hawthorn Avenue the work is much less well advanced. Some streets have been partly demolished and others are boarded up ready for demolition. Sadly there are a few people who have refused to leave their homes yet and are living in tiny islands surrounded by derelict houses.  

To the western side of Hawthorn Avenue little remains of the old buildings with new buildings steadily going up, again in a mixture of styles. I have marked the areas boarded up as brownfield until construction starts. This does currently include the few houses still occupied, but I suspect not for long.

We also took a look at the old Hull FC rugby league ground, the Boulevard. The main entrance lies off Airlie Street, hence the team was nicknamed the Airlie Birds. They moved to share the KC stadium with Hull City football club in 2003. The old ground was used for greyhound racing for a while but now construction is under way for a new school, The Boulevard Academy. I wonder if they will have any rugby league teams?

Friday, 1 February 2013

Local recycling

I wanted to take some stuff to the local recycling site and was turned away because they were rebuilding it. I hadn't seen any warning about this so I searched for it by name - only I got the name wrong. I had added the site to OSM in 2008 and added the name from memory as Humberdale Waste Site when it is really Humberfield Household Waste Recycling Site.

What surprised me was that the top of my Google search came back with the OSM Node Browse results for the site - presumably because nothing else matched my erroneous name that I searched for. It probably won't work now anyway because I've corrected the name.

I should just add that I used the word Google above without stating that Google is a registered trademark. I didn't have to say that because I was not attempting to provide any services that Google offer, nor was I trying to pass off my goods or services as though they were Google's. I mention it because of a recent OSMF missive about a newly trademarked word that they are frantically removing all references to from our Wiki and Help system. We gather data and publish it under an open licence. I'm not sure I see how that might breach a patent troll's dubious US trademark that was refused in the EU, but then I'm not a lawyer, all the happier for that.

Saturday, 12 January 2013

Soft style

I'm working on a project for the Hull area. I want to display overlays on a map. I want the map to be there, but not in your face. I don't want any shops, pubs or other such objects to detract from the overlay, but I do want people to be able to relate the overlay to places in the city and to the road layout. I needed my own render for the city.

At one time I might have turned to Maperitive to create a custom render. It works well but it uses .net so is best used in Windows. Windows leaves me cold. I have used the excellent Mapnik in the past to customise map rendering. Although the end results from Mapnik are superb, wrangling the xml to produce the required result was not easy to understand, so my customisations were tweaks rather than rewrites. Then MapBox released TileMill. I quickly saw how powerful it was. It uses a new style of coding called Carto to define the map style and uses easily defined layers to determine how the map is built up. Under the skin it uses Mapnik to render the example each time you save a change. One of the options is to export the Mapnik XML of your current design so you can throw that at a Mapnik instance and either generate a static image or a set of tiles for a slippy map - which is what I want.

Like any new environment it does take a little understanding. OSM data is best used by importing it into a PostgreSQL database using a recent version of osm2pgsql. This is a lossy process, intended to populate a database schema laid out for use by a renderer. Coastline is created by using a huge shapefile. I have a cut-down version just for the UK which is smaller and quicker to use. If you want to generate images at low zoom levels there are other short-cut shapefiles such as a simplified coastline and named places for example. These speed up processing at low zoom levels when massive amounts of OSM data would otherwise be needed.

Using TileMill is really about getting to grips with Carto and its relationship with the layers. Layers are each a definition of a data source such as a shapefile or a database table or a selection of part of a database table for example. The order they are listed defines the order that image of the map is built up. Carto, which is a lot like CSS, then applies the detailed use of the data in a layer to define the style of each object, such as its colour, opacity and so on. Some objects on the map may use separate layers to define them and be rendered by separate Carto statements. A road may be made up of a casing (its edge), the centre space which may well be coloured differently from the casing, and text for the name of the road to be displayed. These will be built up in different layers to make the joints between roads seamless and to ensure the casing is covered by the filler and then the text lays on top again. Then there are tunnels and bridges and all the other options the real world throws at cartographers. Layers and specific Carto statements together sort all of this out.

I have been helped in my growing understanding of the best ways to do some of this by reading examples, both that come as part of the download of TileMill and by reading the details of Andy Allan's excellent recreation of the Standard OSM map in TileMill.

To create my map, I downloaded the OSM data I needed from Geofabrik's download server, which is a simple and reliable way to get OSM data that is only a day old. I loaded it into a PostgreSQL database with the PostGIS extensions using osm2pgsql just using the default schema. It probably has all sorts of stuff I won't render, but it is quick and easy to do. Then I pointed TileMill at that and the UK coastline shapefile and set to work. Just to check all was well, when I had some of the style working I exported the Mapnik XML file to render some tiles and it quite quickly produced zoom 11 to zoom 18 for the area. I uploaded the tiles just to see how my web server responds with the tiles and all is well. The style needs a lot more work, but the process works very well, so I'm happy to put the hours in to become more of a cartographer as well as a mapper. I'm just not sure how good I'll be at it.

You can see the crude first pass here. The city will also look better not only when I've improved my style but also when the landuse gets tidied up with fewer gaps and overall more careful coverage - something Bing imagery will help with a lot.

Monday, 7 January 2013

Road search fixed

A friend from my school days pointed out that my previous post about finding a road had a browser problem. The drop-down list of road names get hidden by the map. All things MS and Internet Explorer especially are irritating to me. Quite how a company can grow to be so huge based on such poor quality products still confounds me, but still. The problem is that most people by far who use a computer use an MS operating system on it and a large number of them still use a version of Internet Explorer. If I want wide range of people to be able to use a web page it has to work on IE 7+, there is no option.

It seems that amongst the plethora of inconsistencies that IE demonstrates, there is a horrible process of not supporting the z-index style. z-index should determine which item on a web page should appear above another, with the highest z-index being on top and visible. In IE, anything that uses relative or absolute positioning can cause a separate z-index set to be created, starting at zero which causes the z-index of other objects to be seemingly disregarded. The map tiles use absolute positioning so the drop-down can have a huge z-index and yet it is hidden behind the map. There seems to be various ways to overcome this, but forcing the z-index of the map div to -1 seems to be the simplest solution. Hopefully now anyone trying this simple page will see the list of suggested names above the map.

Some good has come from this. Apart for becoming aware of this problem and trying out the various solutions, I have set up a much better test environment for Windows and IE. A Twitter exchange with Iván Sánchez Ortega also turned up with the concept that Win8 should really be renamed to Fail8 which made me chuckle.