Tuesday 30 October 2012

Office of National Statistics

The UK Office of National Statistics announced they have released some OpenData under the Open Government Licence. This is all based around GB postcodes, excluding Northern Ireland.

Edit: Postcodes for Northern Ireland (BT codes) are included. They are based on the Irish National Grid and need to be transformed to the projection OSM uses with the EPSG geodetic parameter EPSG:29902; OSGB needs EPSG: 27700.

I downloaded the data as a CSV and checked what the format looked like. A quick Python script using the proj4 extension for Python and the OSGB northings and eastings turned into longitude and latitude. Checking for expired postcodes and missing northings and eastings and I had a set of data in the same format as my existing CodePoint Open data. I reused the tile rendering process that I'd used with the CodePoint Data. So now I have GB postcode tiles based on Open Government Licenced data to use as an overlay on JOSM and Potlatch2. They work as an overlay on Leaflet or OpenLayer too. There's some information here, including an overlay. So now I have GB postcode tiles based on Open Government Licence data to use.

The work I put in to get permission to use the CodePoint Open data from Royal Mail seems to have been ignored by the Licence Working Group. I passed the written permission on to them so they could remove the statement excluding the CodePoint Open data on the OS OpenData page. When I removed the statement it was quickly reverted by the guy who caused the phoney fuss about the OS OpenData licence in the first place. Rather than start an edit war I asked the LWG to make the change, but that seems to have been too difficult for them to do, as does answering my emails.

Still, now UK postcodes are available and I hope people will find them useful and uncontroversial.

Sunday 14 October 2012

GeoJSON conversions

Looking some more at converting route relations into GeoJSON it quickly became clear that some options are less useful than others. One thing that causes problems are braids. A braid is where a route splits into two alternate routes. I thought about this in advance and didn't think they would be much of a problem, but I was wrong. If a route is a two-way route where you could travel along it in either direction there are braids more often than I expected. Dual carriageways immediately cause problems as the route would take separate carriageways depending on the direction. Roundabouts with flares similarly have alternate routes depending on direction and a few complex junctions also have alternative routes too. One-way systems in a town also forces different routes to be used. Then there are the genuine alternative routes giving an option such as a picturesque versus quicker route.

My original plan was to try to create a single GeoJSON LineString to represent the whole route. This is clearly not possible as each braid cannot fit into a single LineString and braids are the norm not the exception. Another way is to treat each way in the OSM relation as a separate GeoJSON LineString.

GeoJSON can have a single object in it or multiple objects. Each object can also be a multiple object in itself. A LineString has a single line, a MultiLineString has multiple lines, but, confusingly, a MultiLineString counts as a single object. Each single object is a feature and GeoJSON can hold a collection of features, known as a FeatureCollection. These can be mixed types of points, lines and polygons, though for now I'm only interested in lines. If an object is described as a feature, the feature can also have properties, indeed it must have properties.

I'm going to experiment with the options and see what works.

Saturday 13 October 2012

Basic mapping

Talking today to our local parish council chairman, Geoff, while we both tidied our allotments, I discovered that he has been using OpenStreetMap map of the village to look up where planning applications in the village apply to. Adding all of the buildings and addresses in the village certainly helps him, which is great.


There has been some talk about changing the style of the OSM home page and the map that gets presented there. I'm not sure the home page should be filled with a map - I think it sends the wrong message. We should provide renders certainly, but maybe a few shown with thumbnails rather than the drop-down menu on the home page. That menu, and the other changes to the style, are better than the plain OpenLayers style that was there before. Thanks, I think, to Tom MacWright for those changes. The problem is that maps mean different things to different people. Some people see the house address numbers and names as clutter, whereas Geoff clearly finds them useful. Some people want to see more points of interest, others see too many shops, postboxes, businesses and so on as clutter. Too many points of interest mean there is not enough room to write all of their names so detail is not shown. Some people want to see routes, such as bus routes, or cycle routes others want to see clearly named roads, not covered by routes they don't use. Some people like boundaries to be drawn, others are confused by these lines appearing that they don't understand. Some people would like contours or hill shading, others think it gets in the way.

I'm up for a change - it seems overdue to me. It is clear that a single map can't be useful to everyone, and that many maps will still fall short for many people. There will continue to be specialist maps and specialist overlays, but it seems that a map to provide most power for OSM should be simple, with little detail beyond features like waterways, roads, railways and place outlines. This can then be used with a series of overlays that can be selected to supply the user with what is required. A simple base map would also be very useful for other people's specialist overlays too. I'd like to see thumbnail maps of various types on the OSM landing page so a detailed map is never more than a click away which can then be bookmarked for later use.

If the base map is changed, people like Geoff need to find their overlay ('Buildings with Addresses' for example) as easily as possible and it needs to be easy to bookmark so he can find it again next time.

I don't know if this is possible - the simple base map sounds possible, until you think of the email threads that will ensue about  what should or should not be included. The overlays will each need to either be created on the fly or re-rendered as the objects they show get changed, in much the same way as map tiles get rendered now when something changes. Does OSM have the resources to deliver this?

I'm really interested to see what happens, I just hope the debate can be positive, productive and open.

Tuesday 9 October 2012

GeoJSON from Relations

I've been working on a couple of small projects and both need overlays on maps. Using the lovely LeafletJS library overlays as points, lines and polygons are easy if your overlay comes as GeoJSON. I often store the data in a database and generate the GeoJSON on the fly to download to the overlay using Ajax. The issue sometimes can be where to get the data from to create the overlay and how then to store it in the database in the best way to create the dynamic download.

I have written some bespoke scripts for each situation so far so I thought I'd see how easy it would be be to write something a bit more general. My first thought was how to make a GeoJSON file that describes a route from a relation. Often route relations are in OSM data but they don't render on the Standard render. These routes might be commuter or tourist cycle routes, long-distance walks, tourist walks in a city, bus routes and so on. A series of overlays would then allow these to be selectively displayed.

To create a simple GeoJSON overlay for a route I could create a LineString feature. That needs a series of points with latitude and longitude for each point in the correct order to draw the whole route. These come from the nodes in the ways in the relation. The nodes in each way can be listed in either direction along the way, depending on how it was drawn. The nodes that make the join between the ways will appear in the two joining ways.

Another way would be to create a MultiLineString feature. This could be made of each way that makes up the relation, so each point in each LineString is simply a node along the corresponding way. The duplicate nodes that join two ways just make the two LineStrings join up. The direction each LineString is drawn does not matter to the way it looks. Using these for simple overlays would also allow only some of the LineStrings to be used for a limited area. This is simpler, but it might not be what is sometimes needed. One big disadvantage is that if the overlay is drawn with a transparent line, where two lines overlap at the joining node the doubled-up more opaque colour will be obvious.

The ideal route relation would only have ways in it. If there are nodes in the relation they can easily be ignored, but there is the case of super relations, where a long or complex route relation might be made of other, smaller relations. I'm going to ignore that for now. The ideal route relation would have a series of ways listed in the relation so that each follows the other correctly. This is where the whole process breaks down most often. Ways in relations are often out of sequence. Often one ways leads directly into the end of another, but sometimes that is not so. A tee junction in OSM is entirely normal, but when a route takes the tee, the way may not be broken at the junction so a spur continues past the junction. These really ought to be fixed, but in reality not all have been. In city centres with a variety of bus routes and cycles routes the streets can get broken into many small sections in a few places and just occasionally someone who doesn't spot or understand the relations may try to 'mend' the ways either breaking the relations or leaving these spurs as they go. There may also be gaps in the route. Then there are roundabouts. They are closed so there is no way to join one end of that to the end of another way. Some routes have one-way sections too.

I need to work out the best way to deal with these anomalies. I'm hopeful the result will be useful. Then I need to work out the best way to store GeoJSON in databases such as MySQL and SQLite to then be useful.

Saturday 6 October 2012

More local cycle routes

After looking at the routing of NCN 1 yesterday, I thought I'd check out the braids of NCN 65 today. When something is on your doorstep and you know it well it is easy to ignore the signage, but today I put myself in the shoes of a visitor trying to follow the National Cycle Network route 65 in my local area.

Firstly I hope such a visitor would have a lovely map based on OSM or a GPS route based on OSM because then they would make sense of the route without looking at the signs. Along the route it is generally signed well, but at a couple of important decision points the signs could be better. The section of the NCN 65 I'm interested in here is also shared with the Trans Pennine Trail. You can see it here on Andy Allan's lovely cycle map. The route, travelling west to east enters North Ferriby and arrives at the crossroads, as shown in more detail from Andy's map here:

As you approach from the west you can see this sign:


The sign seems to say that the NCN 65 turns left (along Swanland Hill) and local cycle route turns right (along Church Road). This is actually where the two braids of NCN 65 split up, but the sign doesn't make it clear that route 65 could be either way.

If you do turn left you are taking a longer braid that goes through the villages, the shorter, flatter route is to the right and follows the Humber bank to Hessle. Both routes are fairly well signed along their length and I think I could find my way just following the signs, but you might be confused if you did turn right to find you are still following NCN 65.

Suppose you are travelling east to west and you have followed either braid to this point you want to be sure you turn west along High Street to continue correctly and in both cases there is a sign to help. 

But look again, where did that '/1' come from on the second sign? This shows that if you are travelling north up Church Road and you turn left you will be on NCN 65 (correctly) and also on NCN 1. NCN 1 is not on that road and if you head left you will be heading away from the nearest point of NCN, which is in Hessle where you just came from if you are following NCN 65, and you will not meet NCN 1 until you have circumnavigated the world.

So what about the other point at which the route diverges into braids? That is in Hessle as Andy's map shows:




The route from the east divides at the junction of First Lane and Hull Road. If I am travelling west the sign one sees is this (sorry for the photo):
This shows the route north, through the villages is NCN 65 and the route west appears to be a local cycle route to (along?) the Humber bank. Again if you travel straight on here you be surprised to find you are still on NCN 65, which is well signed. If you arrive at the junction from the west a sign clearly points you to carry on eastwards as expected (no photo available) if you arrive along the braid through the villages south down First Lane you are greeted by a this sign:
This is informative, but slightly confusing. If you are following the whole route and chose the villages braid you need to turn left to continue, if you turn right you will return to North Ferriby where you made the choice by the Humber bank route. Since both are labelled 65 this is not entirely clear.





There are other anomalies along the route. Where NCN 1 joins with the Humber bank braid there is a sign showing the route 65 in brackets, it is not in brackets anywhere else that I have seen. About four kilometres away in North Ferriby the signs that show 65/1 are repeated, even though this is certainly not NCN 1 and the newer sign in the opposite direction has this right.

As I mentioned in my last post, Sustrans want to change the braids to make them uniquely numbered or named. That is a good idea, but will mean changes to some signage and especially at the places braid join up.

As usual a trip out even to a very local area revealed something I had missed before and I have added the name to a very small park at the point in North Ferriby that the braid join.

Maps on this post are © openstreetmap contributors for the data; the map tiles are by Andy Allan.

Friday 5 October 2012

Cycle data and cycle routes

Not long after I started adding to OpenStreetMap I added some of the local Cycle network. Sustrans is a charity that has grown to encourage safe cycling in the UK and part of that is to specify routes that people might like to follow on a bike. Some of it is off-road, some on cycleways beside roads and some are on roads. Sometimes these roads have cycle lanes, sometimes not. Sustrans works with local authorities to put up the signage for the routes along roads and has volunteers to help with the routes too.

Cyclestreets is a route planner for cyclists. It uses OpenStreetMap data and so getting OSM as accurate as possible is important to them, particularly regarding cycling infrastructure. They recently received some open data from the Department for Transport which we can compare to OSM and improve it where possible.

Andy Allan has modified the Potlatch 2 editor to create a version that assists with the process of adding useful stuff from the DfT data to OSM. You can see more here. I have used it to confirm that some of the routes in my area as well tagged, often adding a surface tag and lit tag to improve things. (Do we really need to tag surface=asphalt on most UK roads?) I then quickly noticed that something was odd - part of the NCN 1 relation was missing. I discovered that it had been removed as part of a rationalization of the NCN numbering at Sustrans.

Now, I like to survey what I add to OSM. On the ground there are signs for NCN 1 (and NCN 65 and the Trans-Pennine Trail) along the route that had been removed. After a discussion and a look at the Sustrans rule book it is clear that there is a problem.

Sometimes the Sustrans routes have alternatives. In this case there are two ways to to get from the Humber bridge at Hessle to Beverley, both following NCN 1. One route takes you into Hull and Cottingham and the other takes a more direct route through Kirk Ella to Cottingham and then both on to Beverley. This idea that two routes could both have the same number on the signs has always struck me as daft and confusing, possibly leading to a long detour by mistake (more later). It seems that Sustrans now agree and they are trying to sort it out either by making it clear that one is a loop, or alternate route or by renumbering one of the routes if they diverge for a long way. Sounds good, but on the ground none of the signs have changed yet here, so changing OSM to reflect the Sustrans policy does not help people trying to find the route on the ground. I'm not sure how we would stand with respect to Sustrans's copyright either. We clearly can't copy their maps, only glean what we know from surveys and local knowledge. Some Sustrans rangers are OSM contributors too, but not around here. They do have inside knowledge for their area. I have to stick to what I see on the ground.

When I first checked out the NCN 1 route through Kirk Ella I found it from a handful of signs, some of which have now been removed. I was dubious about it at the time - it involved using a very muddy bridleway and going through a school campus. It seems the signs have been removed because the route has been changed. We checked out the new route today. The signage is poor to say the least. The signs are stuck to the poles of road signs or lamp posts and some are peeling off. None are the flat metal or plastic signs that are used elsewhere and stand out so well. At a couple of junctions the signs are missing, so a left-or-right choice at a tee junction becomes a guessing game in the traffic. Overall the new route is fine - it just needs some more signs.

At the point the two routes join in Cottingham there is a much bigger problem and one that I suspect is why Sustrans want to eliminate multiple routes with the same number.  The tile from Andy Allan's excellent cycle map shows the situation.


The two routes (in pink) converge along Northgate; the direct route east along Northgate and the route through Hull west along Northgate. Both should turn north along Park Lane. The problem is there are no signs to say so. So in either case the next sign a cyclist will see will be merrily guiding them back toward the point in Hessle they left many kilometres ago, and no way for him to know that from the signs. Signage is very important and seems to not be very good here. It is clear that the route numbers do need to be changed, but that must be reflected on the ground too. As a compromise I have broken the NCN 1 relation in OSM and shown the shorter route as an separate relation - I don't know if that is right yet, I hope Sustrans help me to get it right.

Then there is NCN 65 ...