Finding a road

I've been working on a project in the City of Hull. Part of that needs to find a specific road in the city. I wanted to use a type-ahead style search, showing a list of roads that meet a partial match of whatever is typed - a fairly normal modern requirement. In OSM we have the Nominatim search, but it is intended for searching the World - my requirements are more modest.

I extracted a list of road names from an OSM extract for Hull to take a look at. The first hassle is that there seemed to be loads and loads of duplicates. These mostly turn out to be roads that have small spurs with the same name and small 'T' or 'Y' endings of dead ends which again get tagged with the road name. The next next hassle is that I need to be able to zoom a map to show the selected road, so I thought I needed to know the centre point of each road. I thought about creating a bounding rectangle for the road and then calculating the centre point. Suddenly that rang a bell.

That format is what the open data OS Locator is presented as. So I decided to see what was needed to use that. First I extracted the roads for Hull from the GB dataset, converting the OSGB grid references to longitude and latitude in the OSM projection. There were a few duplicate road names. OS break a road at an administrative boundary, so as a road crosses a ward boundary it is split. There were about 160 of these (a tiny fraction of the OSM duplicate names) which I recombined into a single rectangle and recalculated the centre point.

One thing I know about in OS Locator data is that there are mistakes. Having surveyed all of Hull and recorded the anomalies with the not:name tags I could use these to correct the OS Locator data to reflect the names on the ground. I then changed the OS names to title case (lowercase with capitalised names) and removed apostrophes which would make searching harder. I then added all these roads with their centre points, about 2700, into a database table. Python scripts made short work of all of this.

I wrote a quick couple of lookups to make the jQuery ajax type ahead work, one returns the list of matches of road names based on a search term and one returns the details for a specific named road. I used the longitude and latitude of the centre point to zoom to that location on the map. Quickly I realised that didn't work well for all roads, because the zoom level needed to vary to show the whole road, some are much longer than others.  Fortunately the OS Locator data holds the rectangle that encloses the road, so I zoomed the map to the maximum zoom that shows that rectangle. The Leaflet library makes all of this pretty simple.

You can see an example of the end result here.  I chose the Mapquest Open tiles because they will suit another part of the project, more of that another time ...

St Nicholas gate

Beverley, in East Yorkshire, is a market town, which like many others, feels like it is all jammed into a small space. It has expanded into two adjoining villages, Molescroft and Woodmansey but the space within Beverley is still at a premium. The town has had industry in the past, especially a large tannery and I expect the good people of the town are now probably happier not to have the place as the smell was awful. The site became a museum for a while but for some years it has lain as rubble strewn waste ground. Now a new housing development has started and the first houses are complete.

I have been biding my time, waiting for access before I surveyed the roads. When the latest OS Locator comparisons highlighted three of the roads as missing from OSM I decided it must be time to take a look. One road, Simmonds Close, is quite conventional, but the other two are a little different. The front door of these houses face onto a short pedestrian road with a service road behind each one with garages and access to the rear of the houses. I wandered up each road, GPS in hand and chatted to the builders who were finishing some of the houses off, then went on my way.

When I got home I discovered that OS Locator has labelled the service roads as the missing roads (Hamilton Walk and Dickinson Walk). I suspect the clue is the name guys, the walking bits are the named roads, not the service roads.

There was a good view of Beverley Minster from the end of one road.

OS Opendata in OSM

Prompted by a comment from Socks, I have looked at some of the GB regions where Ordnance Survey road names don't match the names in OSM. The OS Opendata names on roads are provided by OS StreetView and a textual version is provided by OS Locator. ITO World and Musical Chairs both use OS Locator to create a graphical tool to help people identify where the OS & OSM names for roads differ.

Sometimes the name OS has for a road does not match the name displayed on a name board for the road. OSM uses the name from the name board. To help with the processing of anomalies the wrong, OS name can be added as a not:name tag. The occurrence of the not:name tag is a useful indicator to the origin of the names in OSM. If the names closely match the OS list then the OS names may have been used and the actual names may not have been surveyed. In this way the errors in the OS datasets find their way in to OSM.

One possibility is that mappers in an area may not use the not:name tag. They may not like it or maybe they don't know about it. If they survey their road names and don't use the erroneous OS names then these anomalies should show up in any comparison.

I looked at some of the data about OS Locator and threw some data into a list for the most complete 250 authorities. ( I lost the will to live after 250.) You can see the list at osloclist.raggedred.net. If you click the headings it will sort that column, allowing you to see where how each authority compare.

Short cuts

The latest version of OS Locator open data has been released. Musical Chairs and ITOWorld both published updated information. There are five new names in Hull and seventeen in East Yorkshire all to be visited, checked and then updated.

Of course there is a short cut. I could just use the names OS supply and update the roads in OSM without checking. That would be easy, I could do it from home without spending any time out in the cold and the city and county would quickly look complete again. I could get most of this from the OS StreetView, copying the road names from the tiles overlaid in the editors. Would that be a good idea?

I have checked some of the areas that have slowly had the number of anomalies whittled away and it seems that about 3½ to 4% of the roads named get a not:name tag to show that what is shown on the ground is not the name OS think it is. So if I just copied the OS names  without checking them I would be introducing about a 4% error into the names in OSM.

Looking at a few places where the names have suddenly had a huge reduction in the list of anomalies it seems that the checking may not have been done. Sheffield has a not:name ration of 0.5%, Penwith has a ratio of 0.9%. Taunton Dean has no not:names at all. It is possible that this ratio is right, but it seems unlikely. I hope that people do manage to sort this out over time, but a road with a name is not likely to have its name resurveyed, so I suspect most of these will persist.

I like the OS Locator data and I'm grateful to the people who process it and make it available to mappers. It is a great way to find out the roads that have been added or changed recently (thanks OS) but please use OS (or any other data) sceptically, it is not perfect, it needs checking. Short cuts are only useful if they preserve the quality.

Office of National Statistics

The UK Office of National Statistics announced they have released some OpenData under the Open Government Licence. This is all based around GB postcodes, excluding Northern Ireland.

Edit: Postcodes for Northern Ireland (BT codes) are included. They are based on the Irish National Grid and need to be transformed to the projection OSM uses with the EPSG geodetic parameter EPSG:29902; OSGB needs EPSG: 27700.

I downloaded the data as a CSV and checked what the format looked like. A quick Python script using the proj4 extension for Python and the OSGB northings and eastings turned into longitude and latitude. Checking for expired postcodes and missing northings and eastings and I had a set of data in the same format as my existing CodePoint Open data. I reused the tile rendering process that I'd used with the CodePoint Data. So now I have GB postcode tiles based on Open Government Licenced data to use as an overlay on JOSM and Potlatch2. They work as an overlay on Leaflet or OpenLayer too. There's some information here, including an overlay. So now I have GB postcode tiles based on Open Government Licence data to use.

The work I put in to get permission to use the CodePoint Open data from Royal Mail seems to have been ignored by the Licence Working Group. I passed the written permission on to them so they could remove the statement excluding the CodePoint Open data on the OS OpenData page. When I removed the statement it was quickly reverted by the guy who caused the phoney fuss about the OS OpenData licence in the first place. Rather than start an edit war I asked the LWG to make the change, but that seems to have been too difficult for them to do, as does answering my emails.

Still, now UK postcodes are available and I hope people will find them useful and uncontroversial.

GeoJSON conversions

Looking some more at converting route relations into GeoJSON it quickly became clear that some options are less useful than others. One thing that causes problems are braids. A braid is where a route splits into two alternate routes. I thought about this in advance and didn't think they would be much of a problem, but I was wrong. If a route is a two-way route where you could travel along it in either direction there are braids more often than I expected. Dual carriageways immediately cause problems as the route would take separate carriageways depending on the direction. Roundabouts with flares similarly have alternate routes depending on direction and a few complex junctions also have alternative routes too. One-way systems in a town also forces different routes to be used. Then there are the genuine alternative routes giving an option such as a picturesque versus quicker route.

My original plan was to try to create a single GeoJSON LineString to represent the whole route. This is clearly not possible as each braid cannot fit into a single LineString and braids are the norm not the exception. Another way is to treat each way in the OSM relation as a separate GeoJSON LineString.

GeoJSON can have a single object in it or multiple objects. Each object can also be a multiple object in itself. A LineString has a single line, a MultiLineString has multiple lines, but, confusingly, a MultiLineString counts as a single object. Each single object is a feature and GeoJSON can hold a collection of features, known as a FeatureCollection. These can be mixed types of points, lines and polygons, though for now I'm only interested in lines. If an object is described as a feature, the feature can also have properties, indeed it must have properties.

I'm going to experiment with the options and see what works.

Basic mapping

Talking today to our local parish council chairman, Geoff, while we both tidied our allotments, I discovered that he has been using OpenStreetMap map of the village to look up where planning applications in the village apply to. Adding all of the buildings and addresses in the village certainly helps him, which is great.

There has been some talk about changing the style of the OSM home page and the map that gets presented there. I'm not sure the home page should be filled with a map - I think it sends the wrong message. We should provide renders certainly, but maybe a few shown with thumbnails rather than the drop-down menu on the home page. That menu, and the other changes to the style, are better than the plain OpenLayers style that was there before. Thanks, I think, to Tom MacWright for those changes. The problem is that maps mean different things to different people. Some people see the house address numbers and names as clutter, whereas Geoff clearly finds them useful. Some people want to see more points of interest, others see too many shops, postboxes, businesses and so on as clutter. Too many points of interest mean there is not enough room to write all of their names so detail is not shown. Some people want to see routes, such as bus routes, or cycle routes others want to see clearly named roads, not covered by routes they don't use. Some people like boundaries to be drawn, others are confused by these lines appearing that they don't understand. Some people would like contours or hill shading, others think it gets in the way.

I'm up for a change - it seems overdue to me. It is clear that a single map can't be useful to everyone, and that many maps will still fall short for many people. There will continue to be specialist maps and specialist overlays, but it seems that a map to provide most power for OSM should be simple, with little detail beyond features like waterways, roads, railways and place outlines. This can then be used with a series of overlays that can be selected to supply the user with what is required. A simple base map would also be very useful for other people's specialist overlays too. I'd like to see thumbnail maps of various types on the OSM landing page so a detailed map is never more than a click away which can then be bookmarked for later use.

If the base map is changed, people like Geoff need to find their overlay ('Buildings with Addresses' for example) as easily as possible and it needs to be easy to bookmark so he can find it again next time.

I don't know if this is possible - the simple base map sounds possible, until you think of the email threads that will ensue about  what should or should not be included. The overlays will each need to either be created on the fly or re-rendered as the objects they show get changed, in much the same way as map tiles get rendered now when something changes. Does OSM have the resources to deliver this?

I'm really interested to see what happens, I just hope the debate can be positive, productive and open.