Today I received two DVDs with Lidar data on them. They cover Hull and some of the surrounding area data showing centimetre height records at 50cm spacing. It shows DTM, so no buildings or trees, though DSM is available too.
I've just finished making a replacement window for the garage - the old one was completely rotten - so I fitted it this afternoon while the weather was fine. Now that's done I can turn to this detailed Lidar data.
Thursday, 25 June 2015
Saturday, 6 June 2015
When does a postcode start?
I have mapped a fairly large area, Hull and most of East Yorkshire. When there was nothing on the map I just added what I found but now there is a fairly complete road network. Keeping that up to date can be hard work - how do you know what has been added since the last time I looked? OS Locator is really valuable for this as new roads appear on there as OS maps them. It would be better to know developments are starting and get in early and I think there may be a way: postcodes.
New postcodes are allocated all the time. Around a thousand are created every month in GB. The Office of National Statistics publish a postcode list based on Codepoint Open data but with some extra stuff in there. One thing is the month the postcode was set up. Looking at the recent ones may let us find places that need surveying.
I've created a map to visualise these: http://pcage.raggedred.net
I hope this proves useful.
New postcodes are allocated all the time. Around a thousand are created every month in GB. The Office of National Statistics publish a postcode list based on Codepoint Open data but with some extra stuff in there. One thing is the month the postcode was set up. Looking at the recent ones may let us find places that need surveying.
I've created a map to visualise these: http://pcage.raggedred.net
I hope this proves useful.
Wednesday, 3 June 2015
New postcode layer
I've just finished extracting the data from a full download of OS Open Names and setting up a layer to see the new postcodes. The postcode data is only a centroid for each postcode area.
I maintain a layer of postcodes from the Office of National Statistics, one from Codepoint Open and now Open Names. The Codepoint Open and ONS postcodes are in the same locations, though Codepoint Open data is a bit more up-to-date right now. The OS Open Names postcode locations are now sub-metre, so I expected that the centroid might be in a slightly different location, but some are substantially different as you can see here.
The Codepoint Open are in red, the OS Open Names are in magenta.
I don't store any of these tiles, I render them on the fly. Real tiles are often stored in a hierarchy of folders in the form of
I also extracted the road name and place name data, reprojected all of the location data from OS grid references to lon and lat and stored them for later use. If anyone wants to see this I'll happily pass it on, just ask.
If you want to see the new postcodes you can see the new layer on oscompare. Make sure you open the layer selection (blue+white +) and select what you want to see. You need to zoom in to see the postcodes.
I maintain a layer of postcodes from the Office of National Statistics, one from Codepoint Open and now Open Names. The Codepoint Open and ONS postcodes are in the same locations, though Codepoint Open data is a bit more up-to-date right now. The OS Open Names postcode locations are now sub-metre, so I expected that the centroid might be in a slightly different location, but some are substantially different as you can see here.
I don't store any of these tiles, I render them on the fly. Real tiles are often stored in a hierarchy of folders in the form of
tilename-base/zoom/xposition/yposition.pngSince I don't store any of these, when someone requests a tile they would normally get a "Not found (404)" error. I trap these errors and use the folder hierarchy to extract the postcode centroids from a database for the zoom, x and y area of the tile. This is used to render a tile with a transparent background and centroid markers on it. I do this because I didn't want to use the considerable disk space the real tiles would take up, especially as I create to to zoom level 21 to make editing easier with the layer switched on.
I also extracted the road name and place name data, reprojected all of the location data from OS grid references to lon and lat and stored them for later use. If anyone wants to see this I'll happily pass it on, just ask.
If you want to see the new postcodes you can see the new layer on oscompare. Make sure you open the layer selection (blue+white +) and select what you want to see. You need to zoom in to see the postcodes.
Tuesday, 2 June 2015
Twenty by twenty
It seems that there are not missing sections in the OS Open Names downloads from OS OpenData. The sections are 20x20 km squares.
Monday, 1 June 2015
OS Open Names
As I noted in the previous post Ordnance Survey have said that OS Locator open data is being withdrawn and being replaced by OS Open Names. The file is a strange mixture of three types of data jumbled up. So I needed to separate them into something useful. The three types are postcode centroids, place names and road names. All of the locations are in the projection that Ordnance Survey use, OSGB36 or EPSG:27700. The great thing about the values used in this projection is that they are measured in metres from a fixed point. The previous OS open data, such as OS Locator or CodePoint Open (postcode centroids) use this projection too of course which fixes a location to a square metre. OS Open Names however has a decimal component so the location is now specified to the nearest millimetre - more accurate than OSM works to. The east-west specification in OS parlance is know as eastings and the north-south specification is known as northings. Eastings and northings can be converted to longitude and latitude for use in OSM using, for example, GDAL libraries.
The postcode centroids are very simple records. It needs the postcode and the easting and northing for the location of the centroid. Comparing a few OS Open Names centroids with Codepoint Open records the centroids are in slightly different locations.
OS Locator lists named roads with its name, a centroid and a bounding box for the road. There is also a hierarchy of place, borough, county or unitary authority. All of this is in OS Open Names and the increased millimetre accuracy is there too.
I've not used the gazetteer of place names, but the name and location data are available as you would expect.
All of the data types have some URIs in the files too for many of the fields. Many that I have tried to open are dead links, but some show the hierarchy of data. I'm not sure why this is useful.
I think I'm going to write a routine to unzip and process all of the OS Open Names data. I'll load the data into a database, reproject it for OSM and make the processed data available if anyone wants it.
First, however, OS need to supply the missing sections of their open data.
The postcode centroids are very simple records. It needs the postcode and the easting and northing for the location of the centroid. Comparing a few OS Open Names centroids with Codepoint Open records the centroids are in slightly different locations.
OS Locator lists named roads with its name, a centroid and a bounding box for the road. There is also a hierarchy of place, borough, county or unitary authority. All of this is in OS Open Names and the increased millimetre accuracy is there too.
I've not used the gazetteer of place names, but the name and location data are available as you would expect.
All of the data types have some URIs in the files too for many of the fields. Many that I have tried to open are dead links, but some show the hierarchy of data. I'm not sure why this is useful.
I think I'm going to write a routine to unzip and process all of the OS Open Names data. I'll load the data into a database, reproject it for OSM and make the processed data available if anyone wants it.
First, however, OS need to supply the missing sections of their open data.
OS Locator is to be withdrawn
I just received an email from Ordnance Survey, telling me that OS Locator, part of the OS Open Data, is to be withdrawn in a year's time. They say that OS Open Names has been published and that replaces OS Locator. They hint that it may replace CodePoint Open too, though there's no notice of withdrawal for that yet. I thought I should look at OS Open Names to see what it offers.
The first thing I saw was that the data is broken into the OS large-scale grid squares, which break the country into a 13x7 grid with 55 squares with actual data in them. Each one of these needs to be downloaded separately to cover GB. That is a pain to start with. I requested SE and TA which are both needed to cover the village I live in. After downloading them the .zip file contains each area broken down into 100 separate sections as I would expect, but I quickly realised that there were only 25 in SE and 10 in TA. Much of TA covers the North Sea, so there should be fewer sections, but clearly many were still missing. I downloaded another area and again the same 75 sections were missing. I emailed customer service at OS to point this out, with no answer yet.
I pressed on to look at the data in the sections that were there. The data is in a CSV format (there was another choice). There is a summary of column headings, but the data is a strange muddle of three types of data. Much of the data is a url to an OS website with the data summarised on a page for each column of each line of the text. It appears that there is a record type for place names, a record type for postcode centroids and a record type for named roads all freely mixed up throughout the file. There doesn't seem to be a field to simply identify what the record type is for each line. The first field is an id field. For place names it seems to start with osgb and has a number after that, for postcodes a postcode type id, with spaces removed, is in the id field and for road names an id that looks like a GUID is in the field. It doesn't seem to be a pattern for why the data is in the order that it is.
I'm going to throw some code together to disentangle these record types and see what useful data is then available. Hopefully we will not have lost anything useful in this process, though it does look as though processing this open data is going to be a bit harder than it used to be.
Anyone would think OS don't want to release open data.
Edit: There are two fields which distinguishes these different record types.
The first thing I saw was that the data is broken into the OS large-scale grid squares, which break the country into a 13x7 grid with 55 squares with actual data in them. Each one of these needs to be downloaded separately to cover GB. That is a pain to start with. I requested SE and TA which are both needed to cover the village I live in. After downloading them the .zip file contains each area broken down into 100 separate sections as I would expect, but I quickly realised that there were only 25 in SE and 10 in TA. Much of TA covers the North Sea, so there should be fewer sections, but clearly many were still missing. I downloaded another area and again the same 75 sections were missing. I emailed customer service at OS to point this out, with no answer yet.
I pressed on to look at the data in the sections that were there. The data is in a CSV format (there was another choice). There is a summary of column headings, but the data is a strange muddle of three types of data. Much of the data is a url to an OS website with the data summarised on a page for each column of each line of the text. It appears that there is a record type for place names, a record type for postcode centroids and a record type for named roads all freely mixed up throughout the file. There doesn't seem to be a field to simply identify what the record type is for each line. The first field is an id field. For place names it seems to start with osgb and has a number after that, for postcodes a postcode type id, with spaces removed, is in the id field and for road names an id that looks like a GUID is in the field. It doesn't seem to be a pattern for why the data is in the order that it is.
I'm going to throw some code together to disentangle these record types and see what useful data is then available. Hopefully we will not have lost anything useful in this process, though it does look as though processing this open data is going to be a bit harder than it used to be.
Anyone would think OS don't want to release open data.
Edit: There are two fields which distinguishes these different record types.
Road Closure
The road I live on is closed. A house needs some work done on it. The house wall is tight against the road and that part of the road is also narrow. Scaffolding to work on the house now stands to the middle of the road, so the road is closed to vehicles for two weeks
Should I change the road in Openstreetmap to reflect this temporary closure? I have decided not to change it. Anyone trying to use the road in a vehicle will be directed by another route, by the signs. Most are locals so they will know the alternatives anyway. If I make a change to OSM and someone downloads a snapshot of the data with that change in place they may not download a new version for a while and have the break in the road long after it has actually reopened. Anyway the road exists - it's just not accessible to vehicles for a while.
This closure has caused some very poor driving standards to surface. The alternative routes are small residential roads with parked cars and tight and narrow corners. People using these routes seem to need to take out some annoyance with their normal route being closed by roaring through these streets at ludicrous speeds with no intention to give way to anyone else.
The closure was a surprise to me. I saw a sign go up announcing the closure a couple of days before it happened. When I tried to find out why the road was being closed I drew a blank. The council didn't respond to my request for information and searching their website produced nothing at all. A neighbour pointed out that there was a notice in the local newspaper which was reproduced online. Few few people still read the Hull Daily Mail - it has been steadily descending way below mediocrity for many years. The online notice didn't appear in my searches because it is not sensibly indexed. The HDM website is a nightmare to use, with any page constantly bouncing around as adverts and videos randomly pop up and freeze the pages.
Councils are obliged to publish public notices about some things, such as road closures. It's clear to me that publishing in the local newspaper is not a viable way to do this as they no longer reach much of the population. A sensible addition would be publish the notice on the council's website. I what the East Riding of Yorkshire council will respond to my suggestion of this.
Should I change the road in Openstreetmap to reflect this temporary closure? I have decided not to change it. Anyone trying to use the road in a vehicle will be directed by another route, by the signs. Most are locals so they will know the alternatives anyway. If I make a change to OSM and someone downloads a snapshot of the data with that change in place they may not download a new version for a while and have the break in the road long after it has actually reopened. Anyway the road exists - it's just not accessible to vehicles for a while.
This closure has caused some very poor driving standards to surface. The alternative routes are small residential roads with parked cars and tight and narrow corners. People using these routes seem to need to take out some annoyance with their normal route being closed by roaring through these streets at ludicrous speeds with no intention to give way to anyone else.
The closure was a surprise to me. I saw a sign go up announcing the closure a couple of days before it happened. When I tried to find out why the road was being closed I drew a blank. The council didn't respond to my request for information and searching their website produced nothing at all. A neighbour pointed out that there was a notice in the local newspaper which was reproduced online. Few few people still read the Hull Daily Mail - it has been steadily descending way below mediocrity for many years. The online notice didn't appear in my searches because it is not sensibly indexed. The HDM website is a nightmare to use, with any page constantly bouncing around as adverts and videos randomly pop up and freeze the pages.
Councils are obliged to publish public notices about some things, such as road closures. It's clear to me that publishing in the local newspaper is not a viable way to do this as they no longer reach much of the population. A sensible addition would be publish the notice on the council's website. I what the East Riding of Yorkshire council will respond to my suggestion of this.
Subscribe to:
Posts (Atom)