I wanted to take some stuff to the local recycling site and was turned away because they were rebuilding it. I hadn't seen any warning about this so I searched for it by name - only I got the name wrong. I had added the site to OSM in 2008 and added the name from memory as Humberdale Waste Site when it is really Humberfield Household Waste Recycling Site.
What surprised me was that the top of my Google search came back with the OSM Node Browse results for the site - presumably because nothing else matched my erroneous name that I searched for. It probably won't work now anyway because I've corrected the name.
I should just add that I used the word Google above without stating that Google is a registered trademark. I didn't have to say that because I was not attempting to provide any services that Google offer, nor was I trying to pass off my goods or services as though they were Google's. I mention it because of a recent OSMF missive about a newly trademarked word that they are frantically removing all references to from our Wiki and Help system. We gather data and publish it under an open licence. I'm not sure I see how that might breach a patent troll's dubious US trademark that was refused in the EU, but then I'm not a lawyer, all the happier for that.
Friday, 1 February 2013
Saturday, 12 January 2013
Soft style
I'm working on a project for the Hull area. I want to display overlays on a map. I want the map to be there, but not in your face. I don't want any shops, pubs or other such objects to detract from the overlay, but I do want people to be able to relate the overlay to places in the city and to the road layout. I needed my own render for the city.
At one time I might have turned to Maperitive to create a custom render. It works well but it uses .net so is best used in Windows. Windows leaves me cold. I have used the excellent Mapnik in the past to customise map rendering. Although the end results from Mapnik are superb, wrangling the xml to produce the required result was not easy to understand, so my customisations were tweaks rather than rewrites. Then MapBox released TileMill. I quickly saw how powerful it was. It uses a new style of coding called Carto to define the map style and uses easily defined layers to determine how the map is built up. Under the skin it uses Mapnik to render the example each time you save a change. One of the options is to export the Mapnik XML of your current design so you can throw that at a Mapnik instance and either generate a static image or a set of tiles for a slippy map - which is what I want.
Like any new environment it does take a little understanding. OSM data is best used by importing it into a PostgreSQL database using a recent version of osm2pgsql. This is a lossy process, intended to populate a database schema laid out for use by a renderer. Coastline is created by using a huge shapefile. I have a cut-down version just for the UK which is smaller and quicker to use. If you want to generate images at low zoom levels there are other short-cut shapefiles such as a simplified coastline and named places for example. These speed up processing at low zoom levels when massive amounts of OSM data would otherwise be needed.
Using TileMill is really about getting to grips with Carto and its relationship with the layers. Layers are each a definition of a data source such as a shapefile or a database table or a selection of part of a database table for example. The order they are listed defines the order that image of the map is built up. Carto, which is a lot like CSS, then applies the detailed use of the data in a layer to define the style of each object, such as its colour, opacity and so on. Some objects on the map may use separate layers to define them and be rendered by separate Carto statements. A road may be made up of a casing (its edge), the centre space which may well be coloured differently from the casing, and text for the name of the road to be displayed. These will be built up in different layers to make the joints between roads seamless and to ensure the casing is covered by the filler and then the text lays on top again. Then there are tunnels and bridges and all the other options the real world throws at cartographers. Layers and specific Carto statements together sort all of this out.
I have been helped in my growing understanding of the best ways to do some of this by reading examples, both that come as part of the download of TileMill and by reading the details of Andy Allan's excellent recreation of the Standard OSM map in TileMill.
To create my map, I downloaded the OSM data I needed from Geofabrik's download server, which is a simple and reliable way to get OSM data that is only a day old. I loaded it into a PostgreSQL database with the PostGIS extensions using osm2pgsql just using the default schema. It probably has all sorts of stuff I won't render, but it is quick and easy to do. Then I pointed TileMill at that and the UK coastline shapefile and set to work. Just to check all was well, when I had some of the style working I exported the Mapnik XML file to render some tiles and it quite quickly produced zoom 11 to zoom 18 for the area. I uploaded the tiles just to see how my web server responds with the tiles and all is well. The style needs a lot more work, but the process works very well, so I'm happy to put the hours in to become more of a cartographer as well as a mapper. I'm just not sure how good I'll be at it.
You can see the crude first pass here. The city will also look better not only when I've improved my style but also when the landuse gets tidied up with fewer gaps and overall more careful coverage - something Bing imagery will help with a lot.
At one time I might have turned to Maperitive to create a custom render. It works well but it uses .net so is best used in Windows. Windows leaves me cold. I have used the excellent Mapnik in the past to customise map rendering. Although the end results from Mapnik are superb, wrangling the xml to produce the required result was not easy to understand, so my customisations were tweaks rather than rewrites. Then MapBox released TileMill. I quickly saw how powerful it was. It uses a new style of coding called Carto to define the map style and uses easily defined layers to determine how the map is built up. Under the skin it uses Mapnik to render the example each time you save a change. One of the options is to export the Mapnik XML of your current design so you can throw that at a Mapnik instance and either generate a static image or a set of tiles for a slippy map - which is what I want.
Like any new environment it does take a little understanding. OSM data is best used by importing it into a PostgreSQL database using a recent version of osm2pgsql. This is a lossy process, intended to populate a database schema laid out for use by a renderer. Coastline is created by using a huge shapefile. I have a cut-down version just for the UK which is smaller and quicker to use. If you want to generate images at low zoom levels there are other short-cut shapefiles such as a simplified coastline and named places for example. These speed up processing at low zoom levels when massive amounts of OSM data would otherwise be needed.
Using TileMill is really about getting to grips with Carto and its relationship with the layers. Layers are each a definition of a data source such as a shapefile or a database table or a selection of part of a database table for example. The order they are listed defines the order that image of the map is built up. Carto, which is a lot like CSS, then applies the detailed use of the data in a layer to define the style of each object, such as its colour, opacity and so on. Some objects on the map may use separate layers to define them and be rendered by separate Carto statements. A road may be made up of a casing (its edge), the centre space which may well be coloured differently from the casing, and text for the name of the road to be displayed. These will be built up in different layers to make the joints between roads seamless and to ensure the casing is covered by the filler and then the text lays on top again. Then there are tunnels and bridges and all the other options the real world throws at cartographers. Layers and specific Carto statements together sort all of this out.
I have been helped in my growing understanding of the best ways to do some of this by reading examples, both that come as part of the download of TileMill and by reading the details of Andy Allan's excellent recreation of the Standard OSM map in TileMill.
To create my map, I downloaded the OSM data I needed from Geofabrik's download server, which is a simple and reliable way to get OSM data that is only a day old. I loaded it into a PostgreSQL database with the PostGIS extensions using osm2pgsql just using the default schema. It probably has all sorts of stuff I won't render, but it is quick and easy to do. Then I pointed TileMill at that and the UK coastline shapefile and set to work. Just to check all was well, when I had some of the style working I exported the Mapnik XML file to render some tiles and it quite quickly produced zoom 11 to zoom 18 for the area. I uploaded the tiles just to see how my web server responds with the tiles and all is well. The style needs a lot more work, but the process works very well, so I'm happy to put the hours in to become more of a cartographer as well as a mapper. I'm just not sure how good I'll be at it.
You can see the crude first pass here. The city will also look better not only when I've improved my style but also when the landuse gets tidied up with fewer gaps and overall more careful coverage - something Bing imagery will help with a lot.
Monday, 7 January 2013
Road search fixed
A friend from my school days pointed out that my previous post about finding a road had a browser problem. The drop-down list of road names get hidden by the map. All things MS and Internet Explorer especially are irritating to me. Quite how a company can grow to be so huge based on such poor quality products still confounds me, but still. The problem is that most people by far who use a computer use an MS operating system on it and a large number of them still use a version of Internet Explorer. If I want wide range of people to be able to use a web page it has to work on IE 7+, there is no option.
It seems that amongst the plethora of inconsistencies that IE demonstrates, there is a horrible process of not supporting the z-index style. z-index should determine which item on a web page should appear above another, with the highest z-index being on top and visible. In IE, anything that uses relative or absolute positioning can cause a separate z-index set to be created, starting at zero which causes the z-index of other objects to be seemingly disregarded. The map tiles use absolute positioning so the drop-down can have a huge z-index and yet it is hidden behind the map. There seems to be various ways to overcome this, but forcing the z-index of the map div to -1 seems to be the simplest solution. Hopefully now anyone trying this simple page will see the list of suggested names above the map.
Some good has come from this. Apart for becoming aware of this problem and trying out the various solutions, I have set up a much better test environment for Windows and IE. A Twitter exchange with Iván Sánchez Ortega also turned up with the concept that Win8 should really be renamed to Fail8 which made me chuckle.
It seems that amongst the plethora of inconsistencies that IE demonstrates, there is a horrible process of not supporting the z-index style. z-index should determine which item on a web page should appear above another, with the highest z-index being on top and visible. In IE, anything that uses relative or absolute positioning can cause a separate z-index set to be created, starting at zero which causes the z-index of other objects to be seemingly disregarded. The map tiles use absolute positioning so the drop-down can have a huge z-index and yet it is hidden behind the map. There seems to be various ways to overcome this, but forcing the z-index of the map div to -1 seems to be the simplest solution. Hopefully now anyone trying this simple page will see the list of suggested names above the map.
Some good has come from this. Apart for becoming aware of this problem and trying out the various solutions, I have set up a much better test environment for Windows and IE. A Twitter exchange with Iván Sánchez Ortega also turned up with the concept that Win8 should really be renamed to Fail8 which made me chuckle.
Friday, 28 December 2012
Finding a road
I've been working on a project in the City of Hull. Part of that needs to find a specific road in the city. I wanted to use a type-ahead style search, showing a list of roads that meet a partial match of whatever is typed - a fairly normal modern requirement. In OSM we have the Nominatim search, but it is intended for searching the World - my requirements are more modest.
I extracted a list of road names from an OSM extract for Hull to take a look at. The first hassle is that there seemed to be loads and loads of duplicates. These mostly turn out to be roads that have small spurs with the same name and small 'T' or 'Y' endings of dead ends which again get tagged with the road name. The next next hassle is that I need to be able to zoom a map to show the selected road, so I thought I needed to know the centre point of each road. I thought about creating a bounding rectangle for the road and then calculating the centre point. Suddenly that rang a bell.
That format is what the open data OS Locator is presented as. So I decided to see what was needed to use that. First I extracted the roads for Hull from the GB dataset, converting the OSGB grid references to longitude and latitude in the OSM projection. There were a few duplicate road names. OS break a road at an administrative boundary, so as a road crosses a ward boundary it is split. There were about 160 of these (a tiny fraction of the OSM duplicate names) which I recombined into a single rectangle and recalculated the centre point.
One thing I know about in OS Locator data is that there are mistakes. Having surveyed all of Hull and recorded the anomalies with the not:name tags I could use these to correct the OS Locator data to reflect the names on the ground. I then changed the OS names to title case (lowercase with capitalised names) and removed apostrophes which would make searching harder. I then added all these roads with their centre points, about 2700, into a database table. Python scripts made short work of all of this.
I wrote a quick couple of lookups to make the jQuery ajax type ahead work, one returns the list of matches of road names based on a search term and one returns the details for a specific named road. I used the longitude and latitude of the centre point to zoom to that location on the map. Quickly I realised that didn't work well for all roads, because the zoom level needed to vary to show the whole road, some are much longer than others. Fortunately the OS Locator data holds the rectangle that encloses the road, so I zoomed the map to the maximum zoom that shows that rectangle. The Leaflet library makes all of this pretty simple.
You can see an example of the end result here. I chose the Mapquest Open tiles because they will suit another part of the project, more of that another time ...
I extracted a list of road names from an OSM extract for Hull to take a look at. The first hassle is that there seemed to be loads and loads of duplicates. These mostly turn out to be roads that have small spurs with the same name and small 'T' or 'Y' endings of dead ends which again get tagged with the road name. The next next hassle is that I need to be able to zoom a map to show the selected road, so I thought I needed to know the centre point of each road. I thought about creating a bounding rectangle for the road and then calculating the centre point. Suddenly that rang a bell.
That format is what the open data OS Locator is presented as. So I decided to see what was needed to use that. First I extracted the roads for Hull from the GB dataset, converting the OSGB grid references to longitude and latitude in the OSM projection. There were a few duplicate road names. OS break a road at an administrative boundary, so as a road crosses a ward boundary it is split. There were about 160 of these (a tiny fraction of the OSM duplicate names) which I recombined into a single rectangle and recalculated the centre point.
One thing I know about in OS Locator data is that there are mistakes. Having surveyed all of Hull and recorded the anomalies with the not:name tags I could use these to correct the OS Locator data to reflect the names on the ground. I then changed the OS names to title case (lowercase with capitalised names) and removed apostrophes which would make searching harder. I then added all these roads with their centre points, about 2700, into a database table. Python scripts made short work of all of this.
I wrote a quick couple of lookups to make the jQuery ajax type ahead work, one returns the list of matches of road names based on a search term and one returns the details for a specific named road. I used the longitude and latitude of the centre point to zoom to that location on the map. Quickly I realised that didn't work well for all roads, because the zoom level needed to vary to show the whole road, some are much longer than others. Fortunately the OS Locator data holds the rectangle that encloses the road, so I zoomed the map to the maximum zoom that shows that rectangle. The Leaflet library makes all of this pretty simple.
You can see an example of the end result here. I chose the Mapquest Open tiles because they will suit another part of the project, more of that another time ...
Thursday, 15 November 2012
St Nicholas gate
Beverley, in East Yorkshire, is a market town, which like many others, feels like it is all jammed into a small space. It has expanded into two adjoining villages, Molescroft and Woodmansey but the space within Beverley is still at a premium. The town has had industry in the past, especially a large tannery and I expect the good people of the town are now probably happier not to have the place as the smell was awful. The site became a museum for a while but for some years it has lain as rubble strewn waste ground. Now a new housing development has started and the first houses are complete.
I have been biding my time, waiting for access before I surveyed the roads. When the latest OS Locator comparisons highlighted three of the roads as missing from OSM I decided it must be time to take a look. One road, Simmonds Close, is quite conventional, but the other two are a little different. The front door of these houses face onto a short pedestrian road with a service road behind each one with garages and access to the rear of the houses. I wandered up each road, GPS in hand and chatted to the builders who were finishing some of the houses off, then went on my way.
When I got home I discovered that OS Locator has labelled the service roads as the missing roads (Hamilton Walk and Dickinson Walk). I suspect the clue is the name guys, the walking bits are the named roads, not the service roads.
There was a good view of Beverley Minster from the end of one road.
I have been biding my time, waiting for access before I surveyed the roads. When the latest OS Locator comparisons highlighted three of the roads as missing from OSM I decided it must be time to take a look. One road, Simmonds Close, is quite conventional, but the other two are a little different. The front door of these houses face onto a short pedestrian road with a service road behind each one with garages and access to the rear of the houses. I wandered up each road, GPS in hand and chatted to the builders who were finishing some of the houses off, then went on my way.
When I got home I discovered that OS Locator has labelled the service roads as the missing roads (Hamilton Walk and Dickinson Walk). I suspect the clue is the name guys, the walking bits are the named roads, not the service roads.
Tuesday, 13 November 2012
OS Opendata in OSM
Prompted by a comment from Socks, I have looked at some of the GB regions where Ordnance Survey road names don't match the names in OSM. The OS Opendata names on roads are provided by OS StreetView and a textual version is provided by OS Locator. ITO World and Musical Chairs both use OS Locator to create a graphical tool to help people identify where the OS & OSM names for roads differ.
Sometimes the name OS has for a road does not match the name displayed on a name board for the road. OSM uses the name from the name board. To help with the processing of anomalies the wrong, OS name can be added as a not:name tag. The occurrence of the not:name tag is a useful indicator to the origin of the names in OSM. If the names closely match the OS list then the OS names may have been used and the actual names may not have been surveyed. In this way the errors in the OS datasets find their way in to OSM.
One possibility is that mappers in an area may not use the not:name tag. They may not like it or maybe they don't know about it. If they survey their road names and don't use the erroneous OS names then these anomalies should show up in any comparison.
I looked at some of the data about OS Locator and threw some data into a list for the most complete 250 authorities. ( I lost the will to live after 250.) You can see the list at osloclist.raggedred.net. If you click the headings it will sort that column, allowing you to see where how each authority compare.
Sometimes the name OS has for a road does not match the name displayed on a name board for the road. OSM uses the name from the name board. To help with the processing of anomalies the wrong, OS name can be added as a not:name tag. The occurrence of the not:name tag is a useful indicator to the origin of the names in OSM. If the names closely match the OS list then the OS names may have been used and the actual names may not have been surveyed. In this way the errors in the OS datasets find their way in to OSM.
One possibility is that mappers in an area may not use the not:name tag. They may not like it or maybe they don't know about it. If they survey their road names and don't use the erroneous OS names then these anomalies should show up in any comparison.
I looked at some of the data about OS Locator and threw some data into a list for the most complete 250 authorities. ( I lost the will to live after 250.) You can see the list at osloclist.raggedred.net. If you click the headings it will sort that column, allowing you to see where how each authority compare.
Friday, 9 November 2012
Short cuts
The latest version of OS Locator open data has been released. Musical Chairs and ITOWorld both published updated information. There are five new names in Hull and seventeen in East Yorkshire all to be visited, checked and then updated.
Of course there is a short cut. I could just use the names OS supply and update the roads in OSM without checking. That would be easy, I could do it from home without spending any time out in the cold and the city and county would quickly look complete again. I could get most of this from the OS StreetView, copying the road names from the tiles overlaid in the editors. Would that be a good idea?
I have checked some of the areas that have slowly had the number of anomalies whittled away and it seems that about 3½ to 4% of the roads named get a not:name tag to show that what is shown on the ground is not the name OS think it is. So if I just copied the OS names without checking them I would be introducing about a 4% error into the names in OSM.
Looking at a few places where the names have suddenly had a huge reduction in the list of anomalies it seems that the checking may not have been done. Sheffield has a not:name ration of 0.5%, Penwith has a ratio of 0.9%. Taunton Dean has no not:names at all. It is possible that this ratio is right, but it seems unlikely. I hope that people do manage to sort this out over time, but a road with a name is not likely to have its name resurveyed, so I suspect most of these will persist.
I like the OS Locator data and I'm grateful to the people who process it and make it available to mappers. It is a great way to find out the roads that have been added or changed recently (thanks OS) but please use OS (or any other data) sceptically, it is not perfect, it needs checking. Short cuts are only useful if they preserve the quality.
Of course there is a short cut. I could just use the names OS supply and update the roads in OSM without checking. That would be easy, I could do it from home without spending any time out in the cold and the city and county would quickly look complete again. I could get most of this from the OS StreetView, copying the road names from the tiles overlaid in the editors. Would that be a good idea?
I have checked some of the areas that have slowly had the number of anomalies whittled away and it seems that about 3½ to 4% of the roads named get a not:name tag to show that what is shown on the ground is not the name OS think it is. So if I just copied the OS names without checking them I would be introducing about a 4% error into the names in OSM.
Looking at a few places where the names have suddenly had a huge reduction in the list of anomalies it seems that the checking may not have been done. Sheffield has a not:name ration of 0.5%, Penwith has a ratio of 0.9%. Taunton Dean has no not:names at all. It is possible that this ratio is right, but it seems unlikely. I hope that people do manage to sort this out over time, but a road with a name is not likely to have its name resurveyed, so I suspect most of these will persist.
I like the OS Locator data and I'm grateful to the people who process it and make it available to mappers. It is a great way to find out the roads that have been added or changed recently (thanks OS) but please use OS (or any other data) sceptically, it is not perfect, it needs checking. Short cuts are only useful if they preserve the quality.
Subscribe to:
Posts (Atom)