Rights of Way and fruit

I been wanting to get back to doing some cycling, I've tried before and struggled with the hills. I live in a village where every route back home is up hill. I went to the very interesting and very exciting Fully Charged Live show, all about renewable energy and low emission vehicles, such as electric cars. One of the areas there was about electric bicycles and electric skateboards and mountain boards. My brother loved the electric boards - he's a kite surfer and snowboarder, so he took to it straight away. I really, really liked the e-bikes. When I got home I checked out a local bike shop, borrowed an e-bike, sailed up the hill home with ease and bought the mountain bike version. It is fantastic.

I'm happy with how courteous and respectful most drivers are, but there's always a few idiots and it only takes one to cause a serious problem, so I like to get off the roads. I've been wandering the paths and tracks near home on my new bike and loving the fact that hills are now not obstacles but instead are fun.

Today I followed a route I've walked and cycled for years and I thought I knew very well, but as usual, when you take a close look at familiar things new stuff emerges. Mapping makes you look closely.

I have the file of Public Rights of Way that the East Riding of Yorkshire council have finally released under OGLv3. I am grateful to Robert Whittaker for his persistent and skillful chasing of the council. He got the data released under the open licence where I had failed.

In general the data shows where rights of way are present and shows the footpaths, bridleways and byways as expected. Today I followed a route which used various rights of way and all were signed with the yellow and green arrows and some fingerboards all put up by the council. You would expect these signs on the ground to match the data in the file from the council, but no.

There is a part of Elloughton Dale where the path drops down a steep slope to the road through the valley. You can then follow the road for a couple of hundred metres before turning off to climb the other side of the valley. The descent is through woods and is steep enough that steps with wooden risers have been cut to prevent erosion. This however is not the route I took, because there is a more direct route that comes out of the woods opposite the point where the ascent starts, thus it avoids the steps and there is no need to use the road except to cross it. When I first used this more direct route, probably forty years ago, it was unofficial and the land owner put up barbed wire to stop it being used, but after a while a gate appeared instead (he was probably fed up with replacing the wire) and this was the preferred route to take. More recently, at least twenty five years ago, public right of way signs appeared both at the roadside and at the point in the woods where the route down the steps and this newer route diverge.

This new route is not part of the data the East Riding of Yorkshire council publish and it is not on their 'Walking the Riding' map they publish on their website. So which to believe, council signs on the ground or the data they reluctantly published? Maybe their reluctance to publish was partly because they know the data is out of date. I have used what I know on the ground to map the route, though I can't add a prow_ref tag because that's not displayed on any of the signs.

Hawthorn looking good

It has rained recently, but the long, dry summer means that there's hardly any sign of damp on the

No fruit on the brambles 

Elder looking shrivelled

ground. In the woods the ground is dry and even dusty in places and very few muddy spots even in places that are usually muddy. All this dry weather has taken its toll on wild fruit, but some more so than others. Bullaces seem a bit small but are abundant, hawthorn looks to be a fairly average number and a good size. Elder has few berries and they are small and many shrivelled one. The big loser this year are brambles (blackberries). There is hardly a single fruit to be seen with all the remnants of flowers hardly showing as any berries at all. Any wildlife that usually feasts on the masses of brambles around the paths and tracks here will have a to find something else if they want to survive.

Have you moved ...

... or was it just your postcode?

The new open version of GB postcodes has some strange anomalies in it. There are nearly 1.7 million postcodes, but there are over 45,000 that have moved location since the November 2016 version. Most that have moved have moved less than 20 metres but 1,520 have moved 500m or more. There are 15 that have moved more than 10km, with the furthest moving over 62km.

The source of these open postcodes have changed, maybe the process that generates them has a problem in it. I'm going to look into this some more and try to get some information from Ordnance Survey too.

Maybe this is normal but it feels odd to me.

New Postcodes

I have thought about a tool to see what postcodes have been added recently for a while, so now I've finally got around to looking at it. The UK Office of National Statistics provide all kind of open data under the Open Government Licence and one of these is a list of postcode centroids for Great Britain. The dataset includes the date a postcode was added. 

When an area is quite well mapped it needs to be kept up-to-date. A mapper may have visited somewhere to map it when it's not normally somewhere she might visit, so any new developments might not get mapped as it is built. These new developments all get new postcodes, so showing a map with all the recently added postcodes might help mappers find places that need another survey.

I've put together a map that shows all of the postcodes added since the beginning of 2016. You can see it here http://pcdates.raggedred.net/#10/52.7159/-1.3149.  There would be quite a lot of markers in some areas so I use the excellent MarkerCluster plugin to make the markers more manageable. Each marker is a single centroid. If you click on it it shows a popup with the postcode and start date in it. 

If it is useful please let me know what you think. Would you like anything changed? If people like it I will maintain it as each new release of postcode data  id available, which is currently four times each year.

Postcode changes

I have provided a tile layer of postcode centroids based on the Codepoint Open postcodes since they were first published in 2010. I also provide a similar layer for postcodes based on the Office of National Statistics dataset called ONSPD. I have just updated the two layers with the recently updated data.

The ONSPD dataset contains almost a million more records than the Codepoint Open one. Most of these extra codes are retired postcodes. There are also some entries for BT codes. These are Northern Irish postcodes which are not released under OGL, so I don't publish them - I wouldn't want them to be used as a source for Northern Irish postcodes in OSM as that would violate their copyright. There are some IM codes, but without any coordinates. These are Isle of Man codes. I don't believe they are released under an open licence so again I don't publish them, but having no coordinates makes them useless for this purpose anyway.

The ONSPD dataset has a column for the start date and one for the end date. The earliest start date (just year and month) seems to be 1996-06 which most active postcodes have, the most recent start date is 2017-01.

Edit: The spread of dates is much wider than I first thought, with the earliest is 1973-08.

I think showing any newly added postcodes would be useful to help people see where there may be new developments that need surveying. Some new postcodes will just be yet another code on a building that already has many postcodes. These are places that get lots of post (after all that's what postcodes are for), often royal mail buildings where there are PO Box addresses.

The recently published files have changed their formats (again). Now the Codepoint Open dataset has fixed length postcode fields, so the natural spacing is lost unless they are reformatted, which is very easy. In the past the list of postcodes from Codepoint Open exactly matched the processed list of postcodes from ONSPD, now they don't. I'm examining what it is that is different.

Many of the centroids have moved a few metres. I can understand that a centroid might move if the street or section of street had some new houses (delivery points) added or removed, but that is not the cause of most of the changes. It seems the reason for the change is that the source of the centroids has changed. The data used to come from Address Point (which is now discontinued), now it comes from the Postal Address Location Feed of Geoplace. This is interesting because it removes the connection with Royal Mail and gets the postcode from local authority data. Local authorities are where addresses are created, though Royal Mail must have a part to play in adding the postcode. Getting the data from Geoplace may be one small step towards releasing UK address data as open data. I hope so.

To see how to use the tile layers follow the links for Codepoint Open and ONSPD.

Edit:
I have discovered there are 111 postcodes in the Codepoint Open dataset that are not in the ONSPD dataset after I have processed it. These are 'large user' postcodes and they don't have useful coordinates. In the Codepoint Open dataset there is a field called positional quality indicator. 10 indicates the best positional quality, 90 indicates the worst. All 111 are 90. In future I'll ignore these entries.

Go Pokémon

I'm always keen to see new contributors to OSM. They bring details that interest them to OSM, be it their local area, their sport, their hobby, their work or whatever. Having an open tagging scheme that allows all kinds of objects to be mapped is part of the strength of OSM.

I follow new mappers in my area and sometimes add comments to their changesets or send them a welcome message. Sometimes a new mapper adds one thing and never adds more, sometimes they add one type of thing over wide areas. One thing the East Riding of Yorkshire needs is more fine detail of rural footpaths and bridleways. A few new mappers appeared starting to add footways and I was encouraged that this may spread. Sadly I was wrong and what I hoped was something useful is turning into something less so.

It seems that someone has discovered that some details in OSM may be being used in Pokémon Go, the virtual reality game by Niantic to find imaginary creatures. These imaginary creatures appear more near water and in parks near footpaths. The new mappers were adding footpaths in parks to encourage these beasties to appear. Their new footpaths were in good places and were welcome, but then word spread.

People then started adding more parks and ponds. Some were real parks and ponds that were just missing from the map, but some were plonked anywhere, over existing buildings and roads, over schools or whatever. A few people realised that changing existing areas' tagging was easier, so suddenly a building becomes a park or a pond.

It is not clear to me that Niantic are using OSM data, but if they are, adding phoney stuff won't work very well because it will be found and cleaned up quickly. If Niantic are using OSM data they will probably only incorporate changes to OSM data occasionally, if at all.

It's great to see new mappers. Maybe a few will become more interested in mapping than Pokémon. All the real parks, paths and waterways that are being added are all helpful. Tidying up the junk is nuisance but maybe I'll spot other stuff to improve too.

Back to the new users feed. Oh, there's another one ...

Leaflet plugin

I have been trying to replace a long-standing map based on the OpenLayers 2 library to LeafletJS. The main sticking point was that OpenLayers has a useful permalink function that changes the URL to include a query string that identifies the map location and which layers should be shown. 

Leaflet has a neat plugin to show the location as a hash but that doesn't include the layers to show. After a bit of investigation I decided to try to write a Leaflet plugin to make all this easier.

If a map has optional base layers and some overlays it often will use the Layers control to make the choices easy to select. The layers are not easily accessible from the map object as only the currently displayed layers, not the hidden ones, are available. The Layers control has access to all the layers, but that control is not accessible from the central map object. The only way to have access to all of the layers is from the Layers control, so a new control, inherited from the Layers control seemed to be the right answer. HashLayers control was developed.

HashLayers does all that the Layers control does, of course, but adds the functionality of the URL hash. The hash looks like this:

#zoom/longitude/latitude/layers

This gets appended to the URL of the page that the map is on. Using the hash means the position and layers can be changed dynamically as the map is zoomed or scrolled around and as different layers are selected. Changing the hash value of a URL doesn't force a reload of the page and doesn't show up in browser history.

The zoom is the current zoom level and longitude and latitude are the centre of the map. The layers is a string representing the base layers and any overlays. If a base layer is selected 'B' is shown, 'O' shows an available base layer not selected. The overlays are shown as a string of 'T' or 'F' (true or false) showing if they are shown or not. A typical hash might be:

#12/-0.393105/53.775095/BOFFTF

 This has the first base layer visible, the second one not showing and four overlays available, but with only the third one visible. These selections will be reflected in the visible selection control too.

If a map is created with the HashLayers control on it and a hash is passed in the URL, the map will centre on the location and zoom specified and show the requested layers. If no hash is passed then the default location and zoom will be used from the setView() or map options as normal. Layers will be shown as they were added to the map. A hash to reflect all this will be appended to the URL which gets updated with every change. There is no need for the OpenLayers permalink control.

I now need to test it more carefully and decide if it is worth publishing as a Leaflet plugin.

Replacing OpenLayers

When OS released their Open Data I created a simple map that showed a few of their offerings as overlays on an OSM base layer oscompare.raggedred.net. At the time my maps were driven by OpenLayers. OpenLayers v2 works well but is a lot harder to use than LeafletJS. As soon as Leaflet arrived I started to use that and it has only got better as time has passed. There is OpenLayers v3 available I think, but I doubt I'll use it.

OpenLayers has a permalink option that changes the URL in the browser to include a query string describing the location of the centre of the map and the zoom level. This lets you bookmark or send a link that will open the map in the right place. Leaflet has a plugin to create a hash on the URL that does a similar thing.  It works well but there is a bit of a problem.

The OpenLayers permalink adds a 'layers=' bit to the URL. An example would be BOTFF. This means the first base layer is shown and not the second one, then the first overlay is visible but not the second or third. This great, because not only does the URL make map show the right place, but the overlay you wanted to show will also appear too. Leaflet's hash plugin doesn't do this and that is a problem for me.

I decided to replace the map showing the OS overlays using Leaflet. It was quick to do and works well, but I can't use it as the replacement for the OpenLayers map because Leaflet doesn't have a built-in way to show the layers that were selected. People link to my map, according to the logs, with the query string to select the place and layers. If I just replaced it with the Leaflet-based map their links would all fail. They couldn't create a new bookmark that includes the layers to show either. I decided to fix this.

I looked into the documentation for the latest version of Leaflet (1.0.2 at the time of writing). The documentation is very useful. If there are overlays on a map a layers control can be added that allows the overlays to be selected. There can also be more than one base layer that the control allows you to select.

I thought about a generic way to create a similar effect to the OpenLayers permalink. For a one-off map I can code up the layers to do as I need, but a generic version would be useful. I can get at the layers that a map is currently showing, but Leaflet's way of hiding a layer is to remove it from the map. The definition of that layer is then not accessible from the map object. I couldn't see how to find the list of layers including the ones not displayed. The layers control has a list of the layers it uses, which would be ideal, but I couldn't find any way to find a a reference to that from the map object. There doesn't seem to be any way to find any controls from the map object. A map method that returns a list of controls might be useful.

So far I have created a bespoke version of the Leaflet map I wanted. It will accept an OpenLayers-style URL, display the map and create a Leaflet-style URL. I'm thinking about how to duplicate the OpenLayers feature more generically. I have ideas ...