House or road

More name checks today, the target was the Barmby Moor area near Pocklington. As we have seen before, the OS Locator data is very useful. It showed up a few small new developments in Barmby Moor. It also showed up a number of oddities which we have sorted out.

When I originally surveyed Barmby Moor we did it quite quickly and some street names were missing. Today I wanted to correct that. One area that was a bit awkward was the main road through the village. It only bears a name plate at the north east end, where it says Front Street. According to OS Locator it is called Main Street. We couldn't see any other name boards, so if the name changes it was difficult to see where. I looked at the house numbers for clues, but for once that didn't work - the houses have names and no numbers. I decided to ask the locals.

A very helpful elderly chap was tidying the garden of his bungalow and he explained, at some length, how the name of the street had changed in his lifetime. He explained where Front Street changes to Main Street and how, when the bungalow he lives in was built 30 years ago, the end of the street became West End. Sure enough the few bungalows at the end of the street have a numbering scheme that fits.

The main A1079 that runs by the village is called York Road on OS Locator. There are no signs to that effect that I can see, but it does go to York. Oddly there is a small piece of the road that seem to have a name Gale Hill within the section that is also called York Road. Looking closely I spotted a house called Gales Hill in just the right spot. I have seen this elsewhere with OS data. Near Molescroft there is a road OS think is Constitution Hill, but that is the name of a nearby farm. The road is called Malton Road, I know someone who lives there.

If you are using OS Locator for road names, beware of surveyors who can't tell a house name from a road name.

Sympathy

I like the tools that encourage people to get out and look at their surroundings. Some might call it surveying, but often it's just an extra to some other journey or visit. Sadly some people don't get out and look what's there, they just use the tools to blithely add data to the OSM database regardless of how up-to-date it is or how valid it is.

Today we wandered around part of the East Riding looking for names where OSM and OS Locator don't agree. The rain had stopped but the promised brightness didn't appear. I found a new housing development with a few houses on it developed since our last visit (one of the best uses of the OS Locator dataset). We found a couple of roads OS Locator has names for where we couldn't see a name and a few roads where the OS Locator data was wrong. Coppleflat Lane seems to be everywhere and actually it is nowhere. I think the OS surveyor (or whoever they got their data from) must have had a bad day.

I toyed with venturing down a named, very muddy track that we didn't have a trace for and decided that a trace from Bing, well aligned with our GPS track would do. The Bing photos are a bit old, but I estimate the track is hundreds of years old so they will do nicely.

One name I didn't know was Bluestone Bottoms. I can feel some sympathy with the road - this hard chair that I'm on now leaves me feeling a bit like that sometimes.

A whole village?

We took a tour around Melton, a small village not far from home. My aim was to add all of the addresses for the place, from collection to editing, in an afternoon and I failed. We wandered around the place in a howling gale, struggling to see some of the addresses. The addresses of some houses which don't display a number can be determined just by looking at the number either side, but on some of the older parts of village streets that doesn't work. Some houses only have names, some are recent additions to the street and the numbers don't fit and sometimes there are gaps in the numbers for no apparent reason.

We completed the whole village quite quickly. Except we didn't - I had forgotten that part of the village lies to the south of the A63. So, addresses for a village in an afternoon? Not quite.

Binary files

I have downloaded lots of OSM data from the API, XAPI, the Export tab and from other sources such as Geofabrik and Cloudmade. This has usually been XML files which are often compressed and are easy to understand and deal with, if a bit bulky. The ubiquitous XML format means there are XML parsers available for the languages I use and choices in the way of approaching parsing depending on what you are trying to do, though XML parsing remains a fairly slow process especially given that it is rarely an end in itself so the real use of the data cannot start until the XML has been deciphered.

The volume of data for a given area has steadily increased as detail has increased and the one thing XML is not good at is being compact. The process of moving OSM data around involves moving data across networks and this is sometimes a slow process - my own broadband is not very quick and I have no choice of supplier so there is no competitive pressure to improve the performance or the dreadful customer service, but that's another story. One way, potentially, to improve this is to use a binary file format which could be much smaller even than a compressed XML file and might be quicker to encode and decode. Such a format has been created by Scott Crosby based on the Google Protocol Buffers (protobuf). The files are known as .pbf files.

I have not used the protobuf file formats before so I read the Google pages which seemed to make some sense, though the examples are simplistic.  I turned to the OSM wiki page describing how Scott has created the protobuf layouts and it left me baffled, so I looked at the source code of various utilities that now incorporate support for .pbf files.

I wanted to write something using a .pbf file and, having done this kind of thing before, I know that it is easy to copy other people's code and use it without really understanding it; understanding the use of the .pbf files was an important objective for me. Google provide direct support for using protobuf in C, Java and Python, but there were no Python OSM examples to be found, so I decided to start by writing a Python pbf parser so I couldn't just copy someone else's code verbatim.

I downloaded a .pbf file from Geofabrik to work on. Examining the layout of the file was not easy. If you try to use a hex viewer things are not clear because chunks of the data can be compressed and even uncompressed parts are difficult because the protobuf format squashes data, especially numbers, into the minimum number of bytes to save space. Most of the text used in tags are in a string table, so each string only occurs once in each block of data. In the end I simply wrote code to work through the file extracting and examining each part step by step. The OSM wiki page did help in some places, but I got most help by looking at  other code, the protobuf definitions and the Python files the protobuf compiler creates.

I now have a Python script to parse .pbf files, so I can use the data in the same way as I would having parsed it from XML. I have used the Python OSM classes that I created some time ago to store the data so I can write XML as a quick test. If you are interested in seeing the result of my work you can download it from here.

Footnote:
I have tried to parse the downloaded file for England from Geofabrik. It couldn't load all of the nodes in my 3Gb memory and was killed. I removed the code to store the nodes, ways and relations just to let the code run through the whole file. After more than two hours it had run through, but I couldn't have done very much with it as none of the expanded data was saved. It works well for a smaller area, such as a county or a city, but it doesn't handle big files very well at all.

Surveys

This mapping makes me notice changes that I think would have washed over me before, but I'm sure I would have noticed what has happened to Heron Foods warehouse. They have moved to a shiny new warehouse outside of town and the old one has been demolished, but I'm not sure what the plans are for the site yet.

It will be a while before any of the other maps display the site correctly, for me it is easy, I can just draw the building from the Bing aerials (the one with the mouse cursor on it) and tag it as a brownfield site until I know more. There is a fence around the site, where before there were the walls of the building, so I can add that too.

Rarely does a trip out not reveal something that has been missed or has changed, but not always such a dramatic change as this. Roll on the spring and the chance to do more surveys.

Using GB postcodes

As mentioned in the last post I have been looking at the Code-Point Open data supplied as part of the Opendata released by Ordnance Survey. The data describes the centroids for each postcode which is given as a list of post codes for England, Scotland and Wales. There are about 1.7 million post codes in the dataset, broken up into CSV files, one for each top level code such as HU. Each file has a lot of fields with no data or dummy data in them, but the postcode and the easting and northing are what I need. 

I wanted to make an overlay displaying the postcode centroids to see if they would be useful in working out the postcode for OSM addresses. Postcodes are part of the jealously-guarded Royal Mail database, sold as the PAF, which makes a couple of million pounds profit each year for Royal Mail. I would have liked to the detailed address data or at least the polygon that a postcode covers but the centroids are the best we have for now.

The process of making an overlay started by converting the OS eastings and northings to longitude and latitude. The original file is pruned to remove the excess and then I used gdaltransform to make the conversion, using the the OSGB datum EPSG:27700, before loading this into a database table.

Next I set about drawing a transparent set of tiles to use. I wanted to draw a dot for the centroid and the postcode as text beneath it. If I retrieve a list of centroids for the tile then any point close to the edge of a tile would have text that extends beyond the edge. The next-door tile, however, wouldn't know about that centroid so the text would be cut off. The simple way to deal with this is to draw each tile a bit bigger retrieving data for a larger area, draw the centroids and text then trim each tile to it's proper size. That way the text for points close to an edge will appear on tiles either side of the edge.

I rendered a set of zoom level 16 tiles for my local area and took a look to see how useful the centroid information is. It varies. For some roads the postcode is obvious from the centroid, some it is not clear where a postcode changes. I think it is of some use, so I pressed on.

I did not want to fill my on-line disk space with tiles that might not get used, so I decided to try creating the tiles on the fly to see if the response would be good enough. It is pretty good and local caching of the tiles helps too. I did have a small problem with the HTTP headers not wanting to work, but now that is sorted out. I can load the tiles as an overlay http://www.raggedred.net/codepoint/ and they work as a layer in JOSM and Potlatch 2 by using http://www.raggedred.net/tiles/codepoint as the place to get the tiles from.

I have only loaded the HU postcode area so far so, I will need to load areas for other areas if people want them. I do hope no one uses this to enter centroids into OSM, which seems a pointless exercise to me, but it might help people work out post codes for addressing.

Addresses again

Some time ago I started adding addresses for my local area. It was tedious so I didn't continue. Then, along came the Ordnance Survey Opendata followed by Bing imagery which together mean I could trace buildings. The OS data is deliberately crude so OS still have something better to sell and the locations are a bit flaky. I have found it is not all that up-to-date in parts too, but still much better than nothing.

The Bing imagery is fairly good, but about nine years old. It is more detailed by one zoom level over Hull though but for most buildings it is good enough here. For the recent additions not on Bing, OS sometimes helps, for others a camera, GPS and guestimates are the best I can do for now. Today I went out locally to gather the addresses of a few streets and ended up talking to a pleasant chap who uses maps and is interested in maps, but had not heard of OSM. I gave him a leaflet and promised to send him more information about getting OSM data onto his GPS.

Adding addresses to buildings seems far less arbitrary and is easier to survey if the buildings are traced and printed to take out to write on. I have so far not attained the level of detail that blackadder has achieved. Part of my, more modest attempts are here.

The one set of data that are a struggle are postcodes. When the OS Opendata was released it included postcode centroids. These are the centre of the area a postcode covers. I ignored it, but now I'm working on visualising these centroids in an editor to see if the postcode for an area can be reliably discerned. That is going well, but rendering the tiles may produce far too much data for my web site to handle, so I'm weighing up the options: limited coverage, dynamic tile generation or find somewhere else to host the tiles. First though I'm going to see if they are useful.