Friday, 27 November 2015

When is a town not a town?

I live a few miles away from a rapidly growing settlement called Brough. I see it mentioned from time to time as a town in the local press, but I've always thought of it as a large village. It has been a settlement for a long time, it was a town, Peturia, when Rome ruled this area.

The idea of not knowing whether a place in England is a town or a village seems easy to resolve: does it have a town council or a parish council? Well in the case of Brough it is not quite so easy. The local council is a town council, but it covers both Elloughton and Brough and is called Elloughton-cum-Brough Town Council. It is not that uncommon to find a civil parish or town council with multiple settlements within its bounds, but the council is a town council, surely one of the settlements needs to be a town.

It is clear to me there are two settlements here. Road signs show both places sometimes in different directions, OS Locator shows the street names as streetname : Brough : Elloughton-cum-Brough and streetname : Elloughton : Elloughton-cum-Brough.

OS Open Names shows both places as populatedPlace: village. So is it possible to have a town council presiding over an area with two villages and no towns in it?

I've asked the Local Authority, who, after all, provide the data for OS Locator and OS Open Names, what they think.

The answer is a political hot potato at the moment, as Elloughton-cum-Brough town council have recently decided to spend £4000 on a mayoral-style chain for the head of the council. It seems he can legitimately call himself mayor after the town council made the appropriate change in 2011. That's a lot of money to spend on what some call trinkets at a time when there are council cuts elsewhere.

Monday, 28 September 2015

Extracting building heights from LIDAR

The UK Environment Agency have released some LIDAR data in Digital Elevation Model (DEM) format. It includes Digital Terrain Model (DTM) data and Digital Surface Model data. The DSM data includes buildings and trees while the DTM is processed to remove these so the underlying terrain is visible. Tim Waters asked if you subtract the DTM from the DSM would you be left with just building and tree heights? I'd started to look at this and it turns out building heights are extractable in this way.

I have written a script to do the subtraction and create both the difference file and an SQL file to load the data into a postgresql table. I created a hill shading image from one of the difference files to see what features have been stripped from the DTM data. Here's a jpg version of it:

You can see that the buildings and some other features are all well defined. Phil Endecott suggested using the DSM data to create building outlines which could be traced in OSM. His images look good. I would suggest starting his process with this difference data as all the terrain detail has been removed so it may be even better.

Once the SQL file of height data has been loaded into a Postgresql database, which has the PostGIS extension installed, we can then do some queries on it. I selected some OSM building polygons in the range of the loaded data and found which of the height points fell within each polygon. The highest of these is the highest point of the building above the surrounding terrain. I've written a script as a place-holder to extract all the heights for a rectangular area. Someone could extend this to make a file to upload to OSM to add the heights for every building in the defined area. I feel this is clearly an import and so the usual OSM import process needs to be gone through before the import takes place.

I think there is real merit in using this data to extract building heights, which are needed for the 3D images of city buildings.

The two scripts are available here: The first ( needs matching DSM and DTM files and outputs a difference file and the SQL to load into a database table. This will work for any of the resolutions published by the EA. The second, much less polished, script ( defines a rectangle, extracts the OSM buildings for that rectangle and then finds the height data for each building. I wrote it as a script so it can be extended to create a data file for processing or uploading or so overlay tiles could be made from it. I loaded some OSM data with osm2pgsql (which would normally be used for rendering) and added the table for the heights data to the database. The SQL for the table is:

  hid serial NOT NULL,
  height double precision,
  locn geometry(Point,27700),
  CONSTRAINT eaheight_prim PRIMARY KEY (hid)

CREATE INDEX eadata_index
  ON eadata
  USING gist
The output SQL data can be loaded into this with the command:
psql -f
where and are whatever you used. The table name eadata and field names are hard-coded in 

It is important to say that I would not use a rendering database to create the upload data from as some fields will be missing. osm2pgsql is a lossy process. You can use the OSM ID to extract a current version from the API or from Overpass to add the height data to. I used the rendering data for convenience as I already had it available and to satisfy myself that the process works.  

I hope this is useful to someone. Please feel free to ask for more information if I've not made anything clear.

Sunday, 20 September 2015

More LIDAR goodness

I looked into LIDAR data from the UK Environment Agency some weeks ago. I needed it help a local group who are investigating flood mitigation options. The data was listed as being about £26,500 but we got a 100% discount if we used it for restricted, research use, so we could afford it! A few weeks after I'd used the height data for the group I got an email from the Environment Agency. They said the data was being made available as Open Data under the Open Government Licence. So now I could use it for any other purposes at no cost. You can get the data from

I decided to make a detailed relief map of part of the area close to home. The data doesn't cover the whole country, only parts that are deemed at risk of flooding. All of Hull and the river Hull catchment area are included in this. I've only looked at my local area so other areas may vary.

The data is downloaded as Ordnance Survey 10km grid tiles. There are 2m, 1m, 50cm and 25cm options and digital terrain model and digital surface model options too, so let's look at these options, but first a bit about LIDAR.

LIDAR is a technology that uses laser light to measure a distance repeatedly over an area to create a 3D model of an area. If the LIDAR transceiver is mounted at a fixed point it can pan around to record a very detailed image in 3D of everything that can be seen from that point. It works very well in this way inside a building or a cave to make a very accurate model. The US Space Shuttle flew a mission to use a variant of LIDAR to record the height of the surface of the Earth from space. This is available as SRTM.

More recently LIDAR equipment has been flown in aircraft. The difficulty of making useful measurements from an aircraft should not be underestimated. The only data LIDAR returns is distance to the target, so knowing PRECISELY where the aircraft is in 3D is the real problem. GPS is hopeless at altitude measurement and scarcely good enough for lateral location, barometric height measurements vary over time and location and inertial dead-reckoning accuracy falls off with time. A combination of all of these plus post-processing can result in useful data.

The Environment Agency LIDAR distance options specify the distance between the sample points, the 2m option having less detail than the 25cm option. The area that these options cover varies with the highest detail covering the smallest area. I chose the 50cm option as it covered the area I wanted at the highest level of detail. The detail does make for larger datasets and more processing needed to do anything with it.

Clearly the LIDAR measures the distance to the first object it encounters from the aircraft, so it measures tree tops, building roofs and even vehicles. This is known as the digital surface model. This is often a composite from multiple images, as this data is, to compensate for location inaccuracies and to help remove things like vehicles. To get a useful model of the real landscape, without trees and buildings, the data is post processed to create the digital terrain model. This is the data I have used.

The OGL data was different from the data the Environment Agency originally supplied. The original data was in smaller grid squares and the height was rounded to the nearest centimetre. The OGL data is in bigger squares which makes it a bit easier to process but seems to use 18 decimal places of a metre, which is smaller than the diameter of an atom.

I wanted to create a relief map and make contours from the data and, not for the first time, GDAL had the tools. The data uses the UK Ordnance Survey projection, known as OSGB36 or ESPG:27700, so to use any OSM data with it I would need to reproject to WGS84 or EPSG:4326.

To make a relief map I used gdaldem with the hillshade option on each of the datafiles. These need to be joined together to make a larger image, so the option -compute-edges is also needed. The complete command is:
gdaldem hillshade -compute_edges infile relieftiff/outfile.tif
The output is geoTIFF files which can be merged into a single (large) geoTIFF with the command -o big.tif *.tif
This creates a geoTIFF file which has the image of the relief in a TIFF image and also has the locations of the edges in the original OS projection.

The next step is to use gdalwarp to reproject the large tiff file to one in the WGS84 projection. The command describes the source and target projection and filenames. There are significant missing pieces in the large TIFF as the available data was not rectangular. The -srcnodata 0 and -dstalpha makes missing data transparent rather than black.
gdalwarp -s_srs epsg:27700 -t_srs epsg:4326 -srcnodata 0 -dstalpha big.tif bigr.tif
The new TIFF file is what we want to see, but now it needs turning into tiles to be displayed on a slippy map. I decided that zoom level 13 to 18 would give a useful display. To make these tiles I used, specifying the reprojected TIFF image, the zoom levels and the folder to put the tiles into. -z13-19 bigrt.tif tiles
This makes a set of tiles in the TMS format in the specified folder, in this case tiles.

Another way to visualise the LIDAR data is contours. I decided to create a set of overlay tiles that are transparent except for the contours. These can have a different density of contours at each zoom level. I chose the smallest contour step to be shown at the highest zoom level to be 0.2 metres. The GDAL tool for the job is gdal_contour which makes a shapefile with a linestring for each contour. The command is
gdal_contour -i 0.2 -a height infile outshapefile.shp
The resulting shapefile needs to be reprojected to WGA84. The tool to reproject shapefiles is ogr2ogr
ogr2ogr -t_srs epsg:4326 -s_srs epsg:27700 outshape.shp new.shp
I decided to use Mapnik to make the contour overlay tiles. Mapnik can use shapefiles but specifying the long list of shapefiles created above would be a problem so I loaded the shapefiles into a postgresql table in a database with PostGIS enabled. Postgresql comes with shp2pgsql to do this:
shp2pgsql -a -g way new.shp eacontours > new.sql
This makes SQL to load the shapefile into eacontours table, putting the geometry in the field called way. To load this into a database called ukcontours which already has the postgis extension installed the command is
psql -d ukcontours -f new.sql
I then designed the overlay with Tilemill to create the transparent tiles with more contours at higher zoom levels.

You can see the results at I added a water overlay and a roads overlay (thanks to MapQuest for the roads) to help position the relief imagery.

Tuesday, 11 August 2015


Hull is a low-lying city, as are some of the towns and villages that surround it. I have blogged before about the serious flooding in the area in 2007, just before I started mapping in OSM. What a lot of people won't know, even locals, is that flooding is a regular occurrence in some parts of the area. This flooding is on a much smaller scale than happened in 2007, but it still causes misery for anyone whose house is flooded. The Cottingham Flood Action Group have tried to understand what causes this regular flooding and then campaign to fix the causes. I have helped a bit by providing maps and overlaying data from various sources. Some of these sources are not open so I don't want to publish them here.

One source of contention is riparian ownership. This can apply when a water course runs through your land and you may then be required to maintain it. For example a ditch carrying surface water away may need to be cleaned out so water can actually flow in the ditch and not back up causing a flood elsewhere.

The East Riding of Yorkshire council is strapped for cash. They are pushing people to maintain ditches and in some cases ditches in culverts to prevent flooding, claiming these people are riparian owners of the ditches. To be a riparian owner the watercourse must be wholly within the property or form the boundary of the property. If the watercourse forms a boundary, then the property owner owns the watercourse to the mid line.

I have matched the Land Registry Inspire dataset to my surveys of a few of these and it clearly shows that the ditch is many metres outside of the privately owned land and well within the area of the public highway. This means that no matter how strapped the council is for cash, they must maintain the ditch or culvert. Of course, my survey probably isn't enough to convince the council, but it certainly should prompt people to get more evidence.

Don't believe your council. If they claim you are responsible for something unusual, make them prove it and make sure you get enough facts to stand up to them. The council may be right, but they be just trying to get you to pay for something they are responsible for.

Thursday, 25 June 2015


Today I received two DVDs with Lidar data on them. They cover Hull and some of the surrounding area data showing centimetre height records at 50cm spacing. It shows DTM, so no buildings or trees, though DSM is available too.

I've just finished making a replacement window for the garage - the old one was completely rotten - so I fitted it this afternoon while the weather was fine. Now that's done I can turn to this detailed Lidar data.

Saturday, 6 June 2015

When does a postcode start?

I have mapped a fairly large area, Hull and most of East Yorkshire. When there was nothing on the map I just added what I found but now there is a fairly complete road network. Keeping that up to date can be hard work - how do you know what has been added since the last time I looked? OS Locator is really valuable for this as new roads appear on there as OS maps them. It would be better to know developments are starting and get in early and I think there may be a way: postcodes.

New postcodes are allocated all the time. Around a thousand are created every month in GB. The Office of National Statistics publish a postcode list based on Codepoint Open data but with some extra stuff in there. One thing is the month the postcode was set up. Looking at the recent ones may let us find places that need surveying.

I've created a map to visualise these:

I hope this proves useful.

Wednesday, 3 June 2015

New postcode layer

I've just finished extracting the data from a full download of OS Open Names and setting up a layer to see the new postcodes. The postcode data is only a centroid for each postcode area.

I maintain a layer of postcodes from the Office of National Statistics, one from Codepoint Open and now Open Names. The Codepoint Open and ONS postcodes are in the same locations, though Codepoint Open data is a bit more up-to-date right now. The OS Open Names postcode locations are now sub-metre, so I expected that the centroid might be in a slightly different location, but some are substantially different as you can see here.
 The Codepoint Open are in red, the OS Open Names are in magenta.

I don't store any of these tiles, I render them on the fly. Real tiles are often stored in a hierarchy of folders in the form of
Since I don't store any of these, when someone requests a tile they would normally get a "Not found (404)" error. I trap these errors and use the folder hierarchy to extract the postcode centroids from a database for the zoom, x and y area of the tile.  This is used to render a tile with a transparent background and centroid markers on it. I do this because I didn't want to use the considerable disk space the real tiles would take up, especially as I create to to zoom level 21 to make editing easier with the layer switched on.

I also extracted the road name and place name data, reprojected all of the location data from OS grid references to lon and lat and stored them for later use. If anyone wants to see this I'll happily pass it on, just ask.

If you want to see the new postcodes you can see the new layer on oscompare. Make sure you open the layer selection (blue+white +) and select what you want to see. You need to zoom in to see the postcodes.