Thursday, 21 May 2015

Not Alone

I went out looking for new developments in East Yorkshire today. Looks like I was not alone.

Monday, 11 May 2015

Heights & OS

Working with Digital Elevation Models (DEM) is an interesting extension to creating maps, which are usually a flat representation of part of the world. I really want to find a way to show elevation in a way that is a bit different from a flat map. Working with the OS DEM data has whet my appetite to try something new, but first I need a map to work with.

The most detailed DEM data I have is based on Ordnance Survey OpenData, so creating a map in the OS projection will be useful. I use TileMill to create maps from OSM data.

Firstly I needed OSM data in the Ordnance Survey projection. That means loading some OSM data into a fresh PostgreSQL database. I created a PostgreSQL database and, as usual, add the extension for PostGIS. this creates a table called spacial_ref_sys that includes the OS projection, amongst many others. I often add the hstore extension too, but this is a simple map so I didn't need it.

createdb -E UTF8 EYOS
echo "CREATE EXTENSION postgis;" | psql -d EYOS
I loaded an extract of OSM data using the usual osm2pgsql utility except the projection was needed too to convert the data to OS projection as it is loaded.

osm2pgsql --slim -d EYOS -C 1024 ey.osm.pbf --proj 27700
I decided to add a coastline, so that needed to be in OS projection too. OSM coastlines are handled differently from all other data. They are extracted from the main DB, checked for consistency and created into a shapefile for the world. This is known as processed_p.shp. I have my own copy with a cut-down version with only the British Isles in it to make rendering a bit quicker. I reprojected that to a copy in OS projection using OGR2OGR, part of the Geospatial Data Abstraction Library

ogr2ogr -t_srs 'EPSG:27700' -s_srs 'EPSG:3857' coast_bi_os.shp coast_bi.shp

Armed with all of this I could now start TileMill and add the layers I need for the map. Each of the layers, including the coast shapefile, needed a custom projection. This is:
+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.999601 +x_0=400000 +y_0=-100000 +ellps=airy  +units=m +towgs84=446.448,-125.157,542.060,0.1502,0.2470,0.8421,-20.4894 +units=m +nodefs
 I got this from the PostgreSQL table postgis created above. Once I had designed the map as I wanted it I exported the Mapnik XML and ran it through Mapnik. I discovered that the Mapnik XML was not quite right. It needed to have the third line changed so the srs part matches the custom projection above. There doesn't seem to be a way to set this in TileMill, so a manual edit was needed.

To run the Mapnik XML through Mapnik I used the following python code:


# generate a map image in OS projection epsg:27700

import mapnik
import sys, os
def drawMap(filename, west,south,east,north):
    sz = 5000
    ossrs = "+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +towgs84=446.448,-125.157,542.06,0.1502,0.247,0.8421,-20.4894 +units=m +no_defs"
    m = mapnik.Map(sz,sz,ossrs)
    bbox = mapnik.Envelope(west,south,east,north)
    im = mapnik.Image(sz,sz)
    mapnik.render(m, im)
    view = im.view(0,0,sz,sz) # x,y,width,height,'png')

if __name__ == '__main__':

 That long-winded projection was needed again. Notice the coordinates in the drawMap function are OS coordinates, not longitude and latitude. Everything must match the chosen projection.

This gives me an image of the map in the OS projection, but the style could be any style you choose, though I'd be wary of copying the OS style too closely. This will now match the DEM data if they are combined. My style is still a bit stark and only renders a few objects, but it is something to work with.

Next I need to use it imaginatively.

Friday, 8 May 2015


I've been working on something locally for a while that benefits from maps. It needs height information displayed so I thought I'd take a closer look at what was available, especially Digital Elevation Model (DEM) data

OSM doesn't hold much height information, so when people want to display heights they turn to outside information. One such source is Shuttle Radar Topography Mission data or SRTM. One Space Shuttle mission flew around the world and mapped the heights of the ground below using radar. This data has been published as open data. It is 1 second of arc data points for the USA and 3 seconds of arc data points for the rest of the world. This gives a height data point about every 90m for the UK. There are issues with this data with some places having voids where the radar return didn't register. It is usual for people who use this with OSM to render this data as contour lines or as hill shading or both as a way of visualising the height. I thought I'd do some simple processing to be sure I understood the data format.

The SRTM is published as a 1° square. I read the height values and displayed them as a shade of green since human eyes can distinguish more share variations in green than any other colour. Any voids I show as black There were nine pixels in this square) and any value with a small negative value (small so not a void) I show as blue. There's a lot of interest in this which I'd not noticed looking at contours. The dark area top left is the Vale of York, the green area top centre is the bottom end of the Yorkshire Wolds You can just make out the Humber estuary just above the centre and to the right. The bright green area bottom right is part of the Lincolnshire Wolds. The valleys with tributaries feeding into the Vale of York are interesting. None of those exist as rivers or streams today, so I expect they are remnants of the retreating ice caps about ten thousand years ago when the ground was still permafrost so any melting water cut river channels. Today the water table is much lower with the chalk of the Wolds allowing water to drain into it.

Next I looked at Ordnance Survey (OS) OpenData. They release height data as contours and spot heights in shape file format and DEM data too. The DEM is 50m spacing and should be free of voids. They use their own projection (EPSG:27700) for all of their data and this works better for the UK for some jobs. OS release some of their data in parcels based on their own grid. I am interested in a section of including Cottingham, a large village west of Hull. The OS square TA03 has Cottingham in the middle of it, so that is helpful.

OS TA03 square
I created a similar, image from the OS DEM data. I deliberately emphasised the height differences more than the SRTM image. The area is much smaller than the SRTM area but more detailed. bright green on the left is the edge of the Yorkshire Wolds. The blue line is the river Hull which cuts through the middle of the city of Hull. For comparison the bottom of the the blue smudge on the SRTM image is approximately where the OS image is. Again valleys are shown, though this time running west to east. Again they are dry (though very occasionally not which is part of what I'm investigating). I've decided that there is enough detail in the OS area and that it is big enough, perhaps with one more alongside it, to show what I want. so I'll work with that.

More of what to do with it later.

The python code to produce the SRTM image is here:
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import struct
from PIL import Image

def getpix(val):
    if val == -32768:
        return (0,0,0)
    if val < 0:
        return (0,0,255)
    return (0,int(val),0)

if __name__ == "__main__":
    tile = "N53W001.hgt"
    #make the new empty (white) image
    im ="RGB", (1201, 1201), "white")
    with open(tile, "rb") as f:
        #print get_sample(tile, n, e)
        # scan through each of the heights in the file and colour a pixel
        for n in range(1201):
            for e in range(1201):
                buf =
                hite=struct.unpack('>h', buf)
                #print '{0} {1} {2}'.format(n,e,hite[0])
                if pt == -32768:
                    print 'VOID {0} {1} {2}'.format(n,e,pt)
                if pt                    lowest=pt
                if pt>highest:
    print 'lowest:{0}, highest:{1}'.format(lowest,highest)'h.png')
The code to produce the OS image is here:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import struct
from PIL import Image

def getpix(num):
    if val < 0:
        return (0,0,255)
    return (0,int(val)*3,0)

if __name__ == "__main__":
    osf = "TA03.asc"
    #make the new empty (white) image
    im ="RGB", (200, 200), "white")
    with open(osf, "rb") as f:
        for i in range(5,205):
            s=lines[i].split(' ')
            for idx, val in enumerate(s):

Monday, 16 February 2015

Drone deliveries

The US FAA has ruled that drones need to remain within sight of the operator. This is a major obstacle to companies planning deliveries in the US by drone. I had wondered what kind of mapping such an enterprise would need. Where would such a delivery be made? If the address has a garden then that might be useful, but what if there are plants or garden furniture in the way? If the address has a driveway then that could be a good landing site, but if the delivery sits on the driveway what's to stop it being stolen or driven over by car arriving? If the address is an apartment on a street front will the parcel just be dumped on the street? How could a drone safely land on a sidewalk?

When a delivery is made by hand it is handed over at a doorway, posted into a mailbox or left by a thinking person in a suitable place. A person can gain access to places like a shared lobby too. Can a drone do any of this?

Maybe would-be delivery addresses need to designate a landing site for drones, maybe that would need signage and access restrictions and possibly even extra insurance.

The more I think about it the mapping needed to control this would be very detailed and very specific. I suspect that the FAA ruling is the least of the problems this idea has.

Saturday, 29 November 2014

Save a village

I've been checking out a few new roads in the area. That took me to North Ferriby, a place I know well as I lived there for many years. A housing development, known as Melton Fields, was proposed about three years ago for a piece of land that is not actually in Ferriby, but in the next-door parish of Welton. The land is next to a wood in an open field and the developers dared to suggest that a footpath and cycleway would connect the development to Ferriby through the wood, as well as by a road joined into a recently modified junction - modified with this purpose in mind in my opinion. The development caused outrage that the extra people would swamp the village services and destroy the character of the village.

Three years on and the planning application has been turned down by the local authority and there's an enquiry under way. Enquiries like this seem pointless to me, everyone knows the dice is heavily loaded in favour of the developers, so why waste millions on an enquiry that will ultimately allow the development to go ahead anyway? We need new houses.

The one thing I have noticed over the past few years is the number of very ugly signs scattered all over the village, protesting about the development. This ugliness is something the village should be saved from - it is a real mess. I wonder if the owners of these signs got planning permission to erect the signs as they don't seem temporary to me. It is interesting that some recent developments in the village have these signs outside their properties (the one pictured is not a recent development). So squeezing poky little in-fills in that don't match the surroundings and all strain the services is OK but building an integrated new development outside of the village, which includes extra services is not. Save the village, but from nimbys not sensible development.

Tuesday, 9 September 2014

Lock in

I've been adding lots of buildings in Hull. Tracing from imagery is a bit tedious, but I have got into a rhythm. Adding the house numbers I find easier with a printed map with the buildings already drawn then I can note significant points like house numbers near junctions, any extras (like 4a), any gaps in numbering and whether there is a 13 or not. I was working on Beverley Road. Wanted some shopping so I decided to go shopping there, so I could see the layout of the shops as well as any numbering. I also remembered a couple of developments that were worth looking at to see if they were accessible.

I went to The Jacobs Homes off Askew Avenue. The place was complete, smart
and easy to go round, number one complete. Then I went to The Sidings. This is still only partly complete, with two road names (neither called The Sidings of course, this was a railway goods yard many, many years ago). One small road joined these two named ones - I got its name from a house number with the street name under it. Number two complete. I then set off towards the junction between Beverley Road and Cottingham Road to look at the shops and buy a few things.

Before I got to the shops I saw a yellow board with a development name on it I didn't recognise, Scholars Gate, so I followed it. The development is far from complete but a substantial number of houses have been built and many look occupied. Once again no name board, but once again a house number had the road name on it so I could get the details. Number three complete.

When I got to Cottingham Road shops I took lots of photos of the shop fronts from across the street, got asked what I was doing and handed out a leaflet about OSM. When I'd done my shopping I set off for home.

I noticed another new development off Cottingham Road. When I had turned round and got back to it I realised it was gated and the gate was shut. The sign said Chancellor's Court (private road). As I sat in front of the gate I saw a sensor on a wall inside the gate and guessed that if I could get in, the sensor would open the gates to let me out. Just then a van pulled up near the gates at the otherside and they opened, so I drove in, hoping I was right. I drove down the road, turned at the end and set off out. The gates didn't open. I sat near them waiting for someone else to open the gates and a few seconds later a car pulled past me and the gates opened. I followed the car out and my lock in was over. Number four complete.

As I drove home I saw another yellow board for a development, this time in Cottingham at Cleminson Gardens. I had added bit of the development when it was first accessible. Now the whole road loop is accessible and most of the houses look complete and occupied. That completed number five.

Monday, 1 September 2014


In 2007 large parts of Hull and surrounding areas were flooded. The reasons why are still being debated. Drainage capacity, pumping capacity, pumping failures and poor coordination of various agencies have been blamed in reports.

Most flooding in the UK in the past has been either due to rivers bursting their banks or coastal flooding. Coastal flooding along the east coast is often somewhat predictable as it is often the combination of a spring tide, a deep depression and winds from the north or north west blowing down the North Sea. The low pressure allows the sea level to rise, the wind blows the water into the bottle-neck of the southern North Sea and combined with a spring tide the winds blow waves over sea defences which quickly floods land behind the defences.

Rivers bursting their banks is a bit more obvious. Heavy rainfall runs off land and fills water courses, which may burst their banks downstream and flood surrounding land. What we are beginning to see in Britain are more cases of another kind of flooding: direct flooding from heavy rainfall. Here saturated ground and impermeable ground cannot hold any more rainfall, so water sweeps across the surface of land to a low point where it forms a temporary flood, as it did in Hull in 2007. Britain's existing drainage infrastructure tries to deal with this using ditches and other water courses but they are now unable to cope more and more often.

I think more concentrated, heavier rainfall is causing part of this, some of which is down to climate change no doubt. Some of the flooding is man-made in other ways too. Ditches and other water courses are badly maintained and even being filled in. Building causes more run off from roofs, roads and other impermeable surfaces.

One decision made, I think, in the 1950s is also having a devastating effect and that is where any land run off water goes. In most cases the water boards of the 1950 decided to direct run off into sewers. At the time most areas with access the coast simply dumped untreated sewage into the sea and land run off was seen as a way to dilute it. Now all sewage is supposed to be treated before discharge. I said 'supposed to be treated' because the Water Act allows water companies some discretion to pump untreated sewage into the sea when their infrastructure is overwhelmed in an excessive rainfall event. At Bridlington a new pipe has been built for this purpose.

Adding relatively clean land run off to sewage has two big effects. Land run off water ends up getting treated as sewage in expensive treatment works. This water could reasonably be discharged directly into natural water courses or the sea without much problem, instead sewage works with greatly inflated capacity have to be built and run to deal with it. The second big effect is that when unusually heavy rainfall events overwhelm the drainage system, the bottlenecks are the sewage pipes which overflow, putting not just land run off into streets, gardens and homes but sewage too. Fixing this is not easy and will be very, very expensive and disruptive but as climate change progresses some changes will be needed.

I have been helping Cottingham Flood Action Group to understand the drainage and sewer network for that low-lying village by using their knowledge and surveys to add the ditches, drains, culverts, sewers and manholes to OSM and produce a map to help them understand the issues. You can see the map here. So far only a small part of the sewer network of the village has been added but various people in the group have enough knowledge to add the whole network I think. Then the onward link to Hull's network needs adding, which is a much bigger task. All of the network ends up in a single treatment works at the east of Hull at Salt End.

I hope CFAG find my map useful.