tag:blogger.com,1999:blog-1793259266112993762024-03-06T06:02:24.198+00:00Open MapsMaking not just the best free map, but the best mapChris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.comBlogger330125tag:blogger.com,1999:blog-179325926611299376.post-72943253398115753302018-09-14T12:28:00.001+01:002018-09-14T20:39:43.414+01:00Rights of Way and fruitI been wanting to get back to doing some cycling, I've tried before and struggled with the hills. I live in a village where every route back home is up hill. I went to the very interesting and very exciting Fully Charged Live show, all about renewable energy and low emission vehicles, such as electric cars. One of the areas there was about electric bicycles and electric skateboards and mountain boards. My brother loved the electric boards - he's a kite surfer and snowboarder, so he took to it straight away. I really, really liked the e-bikes. When I got home I checked out a local bike shop, borrowed an e-bike, sailed up the hill home with ease and bought the mountain bike version. It is fantastic.<br />
<br />
I'm happy with how courteous and respectful most drivers are, but there's always a few idiots and it only takes one to cause a serious problem, so I like to get off the roads. I've been wandering the paths and tracks near home on my new bike and loving the fact that hills are now not obstacles but instead are fun.<br />
<br />
Today I followed a route I've walked and cycled for years and I thought I knew very well, but as usual, when you take a close look at familiar things new stuff emerges. Mapping makes you look closely.<br />
<br />
I have the file of Public Rights of Way that the East Riding of Yorkshire council have finally released under OGLv3. I am grateful to Robert Whittaker for his persistent and skillful chasing of the council. He got the data released under the open licence where I had failed.<br />
<br />
In general the data shows where rights of way are present and shows the footpaths, bridleways and byways as expected. Today I followed a route which used various rights of way and all were signed with the yellow and green arrows and some fingerboards all put up by the council. You would expect these signs on the ground to match the data in the file from the council, but no.<br />
<br />
There is a part of Elloughton Dale where the path drops down a steep slope to the road through the valley. You can then follow the road for a couple of hundred metres before turning off to climb the other side of the valley. The descent is through woods and is steep enough that steps with wooden risers have been cut to prevent erosion. This however is not the route I took, because there is a more direct route that comes out of the woods opposite the point where the ascent starts, thus it avoids the steps and there is no need to use the road except to cross it. When I first used this more direct route, probably forty years ago, it was unofficial and the land owner put up barbed wire to stop it being used, but after a while a gate appeared instead (he was probably fed up with replacing the wire) and this was the preferred route to take. More recently, at least twenty five years ago, public right of way signs appeared both at the roadside and at the point in the woods where the route down the steps and this newer route diverge.<br />
<br />
This new route is not part of the data the East Riding of Yorkshire council publish and it is not on their 'Walking the Riding' map they publish on their website. So which to believe, council signs on the ground or the data they reluctantly published? Maybe their reluctance to publish was partly because they know the data is out of date. I have used what I know on the ground to map the route, though I can't add a prow_ref tag because that's not displayed on any of the signs.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://2.bp.blogspot.com/-K6dYvHpTfbM/W5uatKctewI/AAAAAAAADa8/DxrSPNW80pU69tXrsAR7wzfIEqe2Gjf8wCLcBGAs/s1600/IMG_20180914_102451.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1200" height="320" src="https://2.bp.blogspot.com/-K6dYvHpTfbM/W5uatKctewI/AAAAAAAADa8/DxrSPNW80pU69tXrsAR7wzfIEqe2Gjf8wCLcBGAs/s320/IMG_20180914_102451.jpg" width="240" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Hawthorn looking good</td></tr>
</tbody></table>
It has rained recently, but the long, dry summer means that there's hardly any sign of damp on the<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-pudfx02232A/W5uat3vl_KI/AAAAAAAADbA/x2v5_pWANfU4urmqA_OtTGAOVZPU7KRmgCLcBGAs/s1600/IMG_20180914_102818.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1200" height="320" src="https://3.bp.blogspot.com/-pudfx02232A/W5uat3vl_KI/AAAAAAAADbA/x2v5_pWANfU4urmqA_OtTGAOVZPU7KRmgCLcBGAs/s320/IMG_20180914_102818.jpg" width="240" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">No fruit on the brambles </td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-NngsGNxXScc/W5uaswfrTvI/AAAAAAAADa0/Gy7f7u9UceYqowQENmjETeWYFn3OzfeAQCLcBGAs/s1600/IMG_20180914_102612.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1200" height="320" src="https://3.bp.blogspot.com/-NngsGNxXScc/W5uaswfrTvI/AAAAAAAADa0/Gy7f7u9UceYqowQENmjETeWYFn3OzfeAQCLcBGAs/s320/IMG_20180914_102612.jpg" width="240" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Elder looking shrivelled</td></tr>
</tbody></table>
ground. In the woods the ground is dry and even dusty in places and very few muddy spots even in places that are usually muddy. All this dry weather has taken its toll on wild fruit, but some more so than others. Bullaces seem a bit small but are abundant, hawthorn looks to be a fairly average number and a good size. Elder has few berries and they are small and many shrivelled one. The big loser this year are brambles (blackberries). There is hardly a single fruit to be seen with all the remnants of flowers hardly showing as any berries at all. Any wildlife that usually feasts on the masses of brambles around the paths and tracks here will have a to find something else if they want to survive.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com0tag:blogger.com,1999:blog-179325926611299376.post-79820256675686087642017-03-07T22:38:00.002+00:002017-03-07T22:38:25.724+00:00Have you moved ...... or was it just your postcode?<br />
<br />
The new open version of GB postcodes has some strange anomalies in it. There are nearly 1.7 million postcodes, but there are over 45,000 that have moved location since the November 2016 version. Most that have moved have moved less than 20 metres but 1,520 have moved 500m or more. There are 15 that have moved more than 10km, with the furthest moving over 62km.<br />
<br />
The source of these open postcodes have changed, maybe the process that generates them has a problem in it. I'm going to look into this some more and try to get some information from Ordnance Survey too.<br />
<br />
Maybe this is normal but it feels odd to me.Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com2tag:blogger.com,1999:blog-179325926611299376.post-14678006506644393872017-03-05T15:14:00.000+00:002017-03-05T15:14:09.779+00:00New PostcodesI have thought about a tool to see what postcodes have been added recently for a while, so now I've finally got around to looking at it. The UK Office of National Statistics provide all kind of open data under the Open Government Licence and one of these is a list of postcode centroids for Great Britain. The dataset includes the date a postcode was added. <div>
<br /></div>
<div>
When an area is quite well mapped it needs to be kept up-to-date. A mapper may have visited somewhere to map it when it's not normally somewhere she might visit, so any new developments might not get mapped as it is built. These new developments all get new postcodes, so showing a map with all the recently added postcodes might help mappers find places that need another survey.</div>
<div>
<br /></div>
<div>
I've put together a map that shows all of the postcodes added since the beginning of 2016. You can see it here <a href="http://pcdates.raggedred.net/#10/52.7159/-1.3149">http://pcdates.raggedred.net/#10/52.7159/-1.3149</a>. There would be quite a lot of markers in some areas so I use the excellent MarkerCluster plugin to make the markers more manageable. Each marker is a single centroid. If you click on it it shows a popup with the postcode and start date in it. </div>
<div>
<br /></div>
<div>
If it is useful please let me know what you think. Would you like anything changed? If people like it I will maintain it as each new release of postcode data id available, which is currently four times each year.</div>
Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com4tag:blogger.com,1999:blog-179325926611299376.post-49389063764654468992017-03-04T12:25:00.001+00:002017-03-04T16:18:53.422+00:00Postcode changesI have provided a tile layer of postcode centroids based on the Codepoint Open postcodes since they were first published in 2010. I also provide a similar layer for postcodes based on the Office of National Statistics dataset called ONSPD. I have just updated the two layers with the recently updated data.<br />
<br />
The ONSPD dataset contains almost a million more records than the Codepoint Open one. Most of these extra codes are retired postcodes. There are also some entries for BT codes. These are Northern Irish postcodes which are not released under OGL, so I don't publish them - I wouldn't want them to be used as a source for Northern Irish postcodes in OSM as that would violate their copyright. There are some IM codes, but without any coordinates. These are Isle of Man codes. I don't believe they are released under an open licence so again I don't publish them, but having no coordinates makes them useless for this purpose anyway. <br />
<br />
The ONSPD dataset has a column for the start date and one for the end date. The earliest start date (just year and month) seems to be 1996-06 which most active postcodes have, the most recent start date is 2017-01.<br />
<br />
Edit: The spread of dates is much wider than I first thought, with the earliest is 1973-08.<br />
<br />
I think showing any newly added postcodes would be useful to help people see where there may be new developments that need surveying. Some new postcodes will just be yet another code on a building that already has many postcodes. These are places that get lots of post (after all that's what postcodes are for), often royal mail buildings where there are PO Box addresses.<br />
<br />
The recently published files have changed their formats (again). Now the Codepoint Open dataset has fixed length postcode fields, so the natural spacing is lost unless they are reformatted, which is very easy. In the past the list of postcodes from Codepoint Open exactly matched the processed list of postcodes from ONSPD, now they don't. I'm examining what it is that is different.<br />
<br />
Many of the centroids have moved a few metres. I can understand that a centroid might move if the street or section of street had some new houses (delivery points) added or removed, but that is not the cause of most of the changes. It seems the reason for the change is that the source of the centroids has changed. The data used to come from Address Point (which is now discontinued), now it comes from the Postal Address Location Feed of Geoplace. This is interesting because it removes the connection with Royal Mail and gets the postcode from local authority data. Local authorities are where addresses are created, though Royal Mail must have a part to play in adding the postcode. Getting the data from Geoplace may be one small step towards releasing UK address data as open data. I hope so.<br />
<br />
To see how to use the tile layers follow the links for <a href="http://codepoint.raggedred.net/" target="_blank">Codepoint Open</a> and <a href="http://onspd.raggedred.net/" target="_blank">ONSPD</a>.<br />
<br />
Edit:<br />
I have discovered there are 111 postcodes in the Codepoint Open dataset that are not in the ONSPD dataset after I have processed it. These are 'large user' postcodes and they don't have useful coordinates. In the Codepoint Open dataset there is a field called positional quality indicator. 10 indicates the best positional quality, 90 indicates the worst. All 111 are 90. In future I'll ignore these entries.Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com0tag:blogger.com,1999:blog-179325926611299376.post-89448260393613775992017-01-26T15:43:00.002+00:002017-01-26T16:09:25.735+00:00Go PokémonI'm always keen to see new contributors to OSM. They bring details that interest them to OSM, be it their local area, their sport, their hobby, their work or whatever. Having an open tagging scheme that allows all kinds of objects to be mapped is part of the strength of OSM.<br />
<br />
I follow new mappers in my area and sometimes add comments to their changesets or send them a welcome message. Sometimes a new mapper adds one thing and never adds more, sometimes they add one type of thing over wide areas. One thing the East Riding of Yorkshire needs is more fine detail of rural footpaths and bridleways. A few new mappers appeared starting to add footways and I was encouraged that this may spread. Sadly I was wrong and what I hoped was something useful is turning into something less so.<br />
<br />
It seems that someone has discovered that some details in OSM may be being used in Pokémon Go, the virtual reality game by Niantic to find imaginary creatures. These imaginary creatures appear more near water and in parks near footpaths. The new mappers were adding footpaths in parks to encourage these beasties to appear. Their new footpaths were in good places and were welcome, but then word spread.<br />
<br />
People then started adding more parks and ponds. Some were real parks and ponds that were just missing from the map, but some were plonked anywhere, over existing buildings and roads, over schools or whatever. A few people realised that changing existing areas' tagging was easier, so suddenly a building becomes a park or a pond.<br />
<br />
It is not clear to me that Niantic are using OSM data, but if they are, adding phoney stuff won't work very well because it will be found and cleaned up quickly. If Niantic are using OSM data they will probably only incorporate changes to OSM data occasionally, if at all.<br />
<br />
It's great to see new mappers. Maybe a few will become more interested in mapping than Pokémon. All the real parks, paths and waterways that are being added are all helpful. Tidying up the junk is nuisance but maybe I'll spot other stuff to improve too.<br />
<br />
Back to the new users feed. Oh, there's another one ...Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com0tag:blogger.com,1999:blog-179325926611299376.post-73413192061218538482016-12-12T14:05:00.000+00:002016-12-12T15:18:18.030+00:00Leaflet plugin<div class="tr_bq">
I have been trying to replace a long-standing map based on the OpenLayers 2 library to LeafletJS. The main sticking point was that OpenLayers has a useful permalink function that changes the URL to include a query string that identifies the map location and which layers should be shown. </div>
<br />
Leaflet has a neat plugin to <a href="https://github.com/mlevans/leaflet-hash">show the location as a hash</a> but that doesn't include the layers to show. After a bit of investigation I decided to try to write a Leaflet plugin to make all this easier.<br />
<br />
If a map has optional base layers and some overlays it often will use the Layers control to make the choices easy to select. The layers are not easily accessible from the map object as only the currently displayed layers, not the hidden ones, are available. The Layers control has access to all the layers, but that control is not accessible from the central map object. The only way to have access to all of the layers is from the Layers control, so a new control, inherited from the Layers control seemed to be the right answer. HashLayers control was developed.<br />
<br />
HashLayers does all that the Layers control does, of course, but adds the functionality of the URL hash. The hash looks like this:<br />
<blockquote class="tr_bq">
<span style="background-color: #eeeeee;">#zoom/longitude/latitude/layers</span></blockquote>
This gets appended to the URL of the page that the map is on. Using the hash means the position and layers can be changed dynamically as the map is zoomed or scrolled around and as different layers are selected. Changing the hash value of a URL doesn't force a reload of the page and doesn't show up in browser history. <br />
<br />
The zoom is the current zoom level and longitude and latitude are the centre of the map. The layers is a string representing the base layers and any overlays. If a base layer is selected 'B' is shown, 'O' shows an available base layer not selected. The overlays are shown as a string of 'T' or 'F' (true or false) showing if they are shown or not. A typical hash might be:<br />
<blockquote>
<span style="background-color: #eeeeee;">#12/-0.393105/53.775095/BOFFTF</span></blockquote>
This has the first base layer visible, the second one not showing and four overlays available, but with only the third one visible. These selections will be reflected in the visible selection control too.<br />
<br />
If a map is created with the HashLayers control on it and a hash is passed in the URL, the map will centre on the location and zoom specified and show the requested layers. If no hash is passed then the default location and zoom will be used from the setView() or map options as normal. Layers will be shown as they were added to the map. A hash to reflect all this will be appended to the URL which gets updated with every change. There is no need for the OpenLayers permalink control.<br />
<br />
I now need to test it more carefully and decide if it is worth publishing as a Leaflet plugin.Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com3tag:blogger.com,1999:blog-179325926611299376.post-12599805330657376032016-12-03T23:56:00.003+00:002016-12-04T00:03:30.051+00:00Replacing OpenLayersWhen OS released their Open Data I created a simple map that showed a few of their offerings as overlays on an OSM base layer <a href="http://oscompare.raggedred.net/">oscompare.raggedred.net</a>. At the time my maps were driven by OpenLayers. OpenLayers v2 works well but is a lot harder to use than LeafletJS. As soon as Leaflet arrived I started to use that and it has only got better as time has passed. There is OpenLayers v3 available I think, but I doubt I'll use it.<br />
<br />
OpenLayers has a permalink option that changes the URL in the browser to include a query string describing the location of the centre of the map and the zoom level. This lets you bookmark or send a link that will open the map in the right place. Leaflet has a plugin to create a hash on the URL that does a similar thing. It works well but there is a bit of a problem.<br />
<br />
The OpenLayers permalink adds a 'layers=' bit to the URL. An example would be BOTFF. This means the first base layer is shown and not the second one, then the first overlay is visible but not the second or third. This great, because not only does the URL make map show the right place, but the overlay you wanted to show will also appear too. Leaflet's hash plugin doesn't do this and that is a problem for me.<br />
<br />
I decided to replace the map showing the OS overlays using Leaflet. It was quick to do and works well, but I can't use it as the replacement for the OpenLayers map because Leaflet doesn't have a built-in way to show the layers that were selected. People link to my map, according to the logs, with the query string to select the place and layers. If I just replaced it with the Leaflet-based map their links would all fail. They couldn't create a new bookmark that includes the layers to show either. I decided to fix this.<br />
<br />
I looked into the documentation for the latest version of Leaflet (1.0.2 at the time of writing). The documentation is very useful. If there are overlays on a map a layers control can be added that allows the overlays to be selected. There can also be more than one base layer that the control allows you to select.<br />
<br />
I thought about a generic way to create a similar effect to the OpenLayers permalink. For a one-off map I can code up the layers to do as I need, but a generic version would be useful. I can get at the layers that a map is currently showing, but Leaflet's way of hiding a layer is to remove it from the map. The definition of that layer is then not accessible from the map object. I couldn't see how to find the list of layers including the ones not displayed. The layers control has a list of the layers it uses, which would be ideal, but I couldn't find any way to find a a reference to that from the map object. There doesn't seem to be any way to find any controls from the map object. A map method that returns a list of controls might be useful.<br />
<br />
So far I have created a bespoke version of the Leaflet map I wanted. It will accept an OpenLayers-style URL, display the map and create a Leaflet-style URL. I'm thinking about how to duplicate the OpenLayers feature more generically. I have ideas ...Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com0tag:blogger.com,1999:blog-179325926611299376.post-3025400538567588932016-11-14T23:10:00.000+00:002016-11-15T09:04:50.748+00:00A Map in an AppMaps come in all shapes and sizes. I've made maps for a few different uses and since Openstreetmap came along there's always been some part of OSM in my maps. I offered to make a map for <a href="http://www.clockenflap.com/">Clockenflap Music and Arts Festival</a> in Hong Kong, so I expected to use OSM as the base. What I've ended up with uses the tools to create and display the map: JOSM, osm2pgsql, Tilemill, Mapnik and Leaflet, but none of OSM's greatest asset, its data.<br />
<br />
Clockenflap, like most festivals, is largely a temporary venue. This year it's a new venue to Clockenflap on the <a href="http://www.openstreetmap.org/#map=17/22.28401/114.16299">Central Harbour-front Event Space</a>. I looked at the area on OSM and how to map the temporary stuff that the event will put up. There's seven stages, food stalls, bars, attractions, art installations and all the usual support stuff like toilets, first aid and entrance and exit gates. That's a lot of stuff to add to OSM temporarily. Worst of all, when I started to look at it the site design was still not fixed. I quickly decided that it needed to be stored locally and not uploaded to OSM at all.<br />
<br />
The map is embedded in an app, which I'm not writing. The idea is to run the map in a Webview with a JavaScript Leaflet-based display and it works well. The map has a base layer with all the services on it and layers of overlays with icons that can be clicked. In the apps (one iOS and one Android) the JavaScript in the Webview calls a native iOS or Android function when an icon is touched. So instead of a web popup the app can respond with a dialog or a more detailed response. For example, if a stage icon is touched the app can show what acts are on the stage now and on next. The reverse is true too. For example, if the user scrolls through the list of food suppliers in the app and chooses one, the map is called and shows where the food stall is on the site. I can't show the apps yet, but a working example of the map is <a href="http://clockenflap.raggedred.net/">here</a>.<br />
<br />
The festival team provided the icons and the look of the map. I think it looks very good. There are sponsors who need to see their logos and colour schemes and the map needs to reflect this, as well as being useful for getting around on a site that no one knows their way around because it is only a day or two old. <br />
<br />
A few things are quite different from most of the maps I've made before. Driving the map from a phone screen is a challenge. Trying to get specific icons on the map so they are big enough to read on smaller screen, but don't overlap too much is tricky. At the lowest zoom level (17) very little is shown by default. If all the overlays are show it is a jumble, there's just no choice.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://clockenflap.raggedred.net/" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="226" src="https://2.bp.blogspot.com/-Wmv7Vb1IYsU/WCo_zEFb46I/AAAAAAAAChQ/430qu4-sRo0YoDkVO4fp1dCautm03PyjQCLcB/s400/jumble.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Lots to fit on at low zoom if everything is turned on</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://3.bp.blogspot.com/-1_kQ3izeq-E/WCpBylk9ffI/AAAAAAAAChc/9q0bG-ZPWcMQ7UHi9QgLeFAt_nGkqSx3wCLcB/s1600/foodstalls.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://3.bp.blogspot.com/-1_kQ3izeq-E/WCpBylk9ffI/AAAAAAAAChc/9q0bG-ZPWcMQ7UHi9QgLeFAt_nGkqSx3wCLcB/s320/foodstalls.png" width="199" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Food stalls need room for their much longer real names </td></tr>
</tbody></table>
By the time we get to the highest zoom level (22) there's lots of space in most places. I needed that zoom level to comfortably resolve the food stalls, which are 3m x 3m, with an icon and room for their name. I needed to display a lot of food supplier names close together and the names needed to respond to a touch (or click). I used the Leaflet plugin Leaflet.Label for this. If I had used Leaflet 1.0, rather than 0.7, I think I could have used built-in tool tips instead.<br />
<br />
There is an optical illusion when you zoom in on an icon, especially on a phone where you use pinch zoom. If you show the higher zoom level with the same size icon on it the icon seems to have shrunk. People expect pinch zoom to make things bigger, so I increase the size of the icons as each level zooms in. That is easy for the stuff on the base map, Tilemill makes that simple, but scaling the icons on overlays took a bit of code I'd not needed to use before. This involves resizing the icon and using setIcon() to use the newly sized icon in a marker. setIcon() is not documented in Leaflet 0.7, but works just the same.<br />
<br />
The standard Leaflet layer control is much too small for a mobile phone. I tweaked the size and position with CSS. I also discovered that you don't need a base layer on the control and you can force it to always be open. Both are perfectly plainly described in the really good documentation of Leaflet, I'd just not read that bit before. <br />
<br />
I control the markers on the map by describing them in geoJson files, one for each overlay. To position the many markers on the map I created a simple editor which uses the Leaflet.Draw plugin, which is really simple to use in code and produces a very useful editor.<br />
<br />
I have enjoyed creating this map and working with creative people who have an absolute deadline to work to. If you are in Hong Kong on 25 - 27th November enjoy Clockenflap and enjoy using my map to find all the stuff Clockenflap has to offer. Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com1tag:blogger.com,1999:blog-179325926611299376.post-55351318371231073082016-02-05T16:30:00.003+00:002016-02-05T16:30:38.786+00:00Drains and meadowsI mapper jumped into mapping the rural areas of East Yorkshire more than a year ago but, sadly, seems to have stopped. This mapper has added farmland and woods in much more detail than anything before and has drawn the outlines of lots of buildings in villages too. There are a couple of things that I've decided to change in any areas that I take a look at now. That is meadows and drains.<br />
<br />
Meadows are a special thing in Britain, with most having disappeared. They are not just areas of grassland but a very specific kind of management, involving no fertiliser, a careful cutting of the grass for hay more than once a year and some animal grazing, but not all year round. The timing of the hay cuts allows wild flowers to flourish as they have time to set seed before they are cut down. This makes a beautiful landscape that has all but vanished from the English countryside. The mapper has made the mistake that almost any green field in the aerial imagery is a meadow, so I'm checking as best I can on the ground and changing it to farmland unless it really is a meadow. The farmland is drawn with each field outlined which means I can try to add hedges when I see them. This too improves the quality of the rural mapping.<br />
<br />
The other problem is that every field-side ditch has been mapped as a drain. This is further worsened by another mapper doing the same thing in Holderness, the flat land to the east of Hull. There are some waterways I would call a drain there, but most are ditches in my view. A ride out doesn't cover much ground to correct this and imagery is ambiguous at best. So there's lots to check and much will change.<br />
<br />
The East Riding countryside is a great place to spend time; having an excuse to carefully explore and examine more of it in detail is no bad thing, so I'm not complaining.Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com0tag:blogger.com,1999:blog-179325926611299376.post-52916909384058071652016-02-02T12:08:00.000+00:002016-02-02T13:23:31.040+00:00SchoolsPeople meet up in a few areas across Britain drawn together to share views and ideas about OSM. One of these is known as Mappa Mercia, meeting in the old Kingdom of Mercia, roughly the Midlands. They started a quarterly series of projects which encourages people to focus on improving specific areas, such as nature reserves. This quarter it is schools.<br />
<br />
People across Britain have taken to the suggestion of checking schools and improving the quality of recording schools. This is a simple idea that will greatly improve the quality of data in OSM, something that the quarterly projects have always aimed at. It's a great idea and something I support.<br />
<br />
There is a list of UK schools available as open data which Robert has used to show a map of how well the schools in the list are represented in OSM. It uses postcode areas as that is how the open data is structured. This is a very good indicator of areas that still need work.<br />
<br />
This is all useful, but then I think the project took a wrong turn. People were encouraged to add a hashtag to changeset comments. This has then been used to produce a list of people who have contributed the most changes to the project. I don't like this. OSM is not a competition and turning it into one encourages edits just to climb the list (I'm not accusing anyone of that, just that the pressure is created). We have seen that with the list of anomalies published between OS Locator road names and OSM road names. To push an area up the list people just use the OS Locator names without checking. I know that OS Locator contains errors and there is a mechanism to record these, but many areas have very few recorded errors, so erroneous OS data is now in OSM.<br />
<br />
I like the idea of focussing on improving specific things in OSM. It should encourage surveying. I would prefer it is not be seen as a competition.Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com2tag:blogger.com,1999:blog-179325926611299376.post-34676634735853634692015-11-27T16:43:00.002+00:002015-12-01T17:38:55.837+00:00When is a town not a town?I live a few miles away from a rapidly growing settlement called <a href="http://www.openstreetmap.org/#map=14/53.7333/-0.5720">Brough</a>. I see it mentioned from time to time as a town in the local press, but I've always thought of it as a large village. It has been a settlement for a long time, it was a town, Peturia, when Rome ruled this area.<br />
<br />
The idea of not knowing whether a place in England is a town or a village seems easy to resolve: does it have a town council or a parish council? Well in the case of Brough it is not quite so easy. The local council is a town council, but it covers both Elloughton and Brough and is called Elloughton-cum-Brough Town Council. It is not that uncommon to find a civil parish or town council with multiple settlements within its bounds, but the council is a town council, surely one of the settlements needs to be a town.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-xlJaJbhV8zc/VliIHMH1IpI/AAAAAAAAB8o/A8kf_Bo-AEc/s1600/ell-cum-brough.JPG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="318" src="http://1.bp.blogspot.com/-xlJaJbhV8zc/VliIHMH1IpI/AAAAAAAAB8o/A8kf_Bo-AEc/s320/ell-cum-brough.JPG" width="320" /></a></div>
It is clear to me there are two settlements here. Road signs show both places sometimes in different directions, OS Locator shows the street names as streetname : Brough : Elloughton-cum-Brough and streetname : Elloughton : Elloughton-cum-Brough.<br />
<br />
OS Open Names shows both places as populatedPlace: village. So is it possible to have a town council presiding over an area with two villages and no towns in it?<br />
<br />
I've asked the Local Authority, who, after all, provide the data for OS Locator and OS Open Names, what they think.<br />
<br />
The answer is a political hot potato at the moment, as Elloughton-cum-Brough town council have recently decided to spend £4000 on a mayoral-style chain for the head of the council. It seems he can legitimately call himself mayor after the town council made the appropriate change in 2011. That's a lot of money to spend on what some call trinkets at a time when there are council cuts elsewhere.<br />
<br />
Edit:<br />
The clerk to the Elloughton-cum-Brough has contacted me and kindly given me information about the process that resulted in the parish council becoming a town council. She also confirmed that both Brough and Elloughton are villages. That means that the town council area has two villages and no towns in it.<br />
<br />Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com3tag:blogger.com,1999:blog-179325926611299376.post-11010286690535096992015-09-28T14:36:00.001+01:002015-09-28T14:36:59.856+01:00Extracting building heights from LIDARThe UK Environment Agency have released some LIDAR data in Digital Elevation Model (DEM) format. It includes Digital Terrain Model (DTM) data and Digital Surface Model data. The DSM data includes buildings and trees while the DTM is processed to remove these so the underlying terrain is visible. Tim Waters asked if you subtract the DTM from the DSM would you be left with just building and tree heights? I'd started to look at this and it turns out building heights are extractable in this way.<br />
<br />
I have written a script to do the subtraction and create both the difference file and an SQL file to load the data into a postgresql table. I created a hill shading image from one of the difference files to see what features have been stripped from the DTM data. Here's a jpg version of it:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-A8irnPsuREM/Vgk2NZO443I/AAAAAAAAB5k/fu5mrBI8HKc/s1600/sub.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="http://2.bp.blogspot.com/-A8irnPsuREM/Vgk2NZO443I/AAAAAAAAB5k/fu5mrBI8HKc/s400/sub.jpg" width="400" /></a></div>
<br />
You can see that the buildings and some other features are all well defined. Phil Endecott suggested using the DSM data to create building outlines which could be traced in OSM. His images look good. I would suggest starting his process with this difference data as all the terrain detail has been removed so it may be even better.<br />
<br />
Once the SQL file of height data has been loaded into a Postgresql database, which has the PostGIS extension installed, we can then do some queries on it. I selected some OSM building polygons in the range of the loaded data and found which of the height points fell within each polygon. The highest of these is the highest point of the building above the surrounding terrain. I've written a script as a place-holder to extract all the heights for a rectangular area. Someone could extend this to make a file to upload to OSM to add the heights for every building in the defined area. I feel this is clearly an import and so the usual <a href="http://wiki.openstreetmap.org/wiki/Automated_Edits_code_of_conduct">OSM import process</a> needs to be gone through before the import takes place.<br />
<br />
<br />
I think there is real merit in using this data to extract building heights, which are needed for the 3D images of city buildings.<br />
<br />
The two scripts are available here: <a href="http://www.raggedred.net/shared/heights.zip">http://www.raggedred.net/shared/heights.zip</a> The first (s-t.py) needs matching DSM and DTM files and outputs a difference file and the SQL to load into a database table. This will work for any of the resolutions published by the EA. The second, much less polished, script (match.py) defines a rectangle, extracts the OSM buildings for that rectangle and then finds the height data for each building. I wrote it as a script so it can be extended to create a data file for processing or uploading or so overlay tiles could be made from it. I loaded some OSM data with osm2pgsql (which would normally be used for rendering) and added the table for the heights data to the database. The SQL for the table is:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">CREATE TABLE eadata<br />(<br /> hid serial NOT NULL,<br /> height double precision,<br /> locn geometry(Point,27700),<br /> CONSTRAINT eaheight_prim PRIMARY KEY (hid)<br />)<br />WITH (<br /> OIDS=FALSE<br />);<br /><br />CREATE INDEX eadata_index<br /> ON eadata<br /> USING gist<br /> (locn);</span></blockquote>
The output SQL data can be loaded into this with the command:<br />
<blockquote class="tr_bq">
psql <dbname> -f <filename></filename></dbname></blockquote>
where <dbname> and <filename> are whatever you used. The table name eadata and field names are hard-coded in s-t.py. </filename></dbname><br />
<br />
It is important to say that I would not use a rendering database to create the upload data from as some fields will be missing. osm2pgsql is a lossy process. You can use the OSM ID to extract a current version from the API or from Overpass to add the height data to. I used the rendering data for convenience as I already had it available and to satisfy myself that the process works. <br />
<br />
I hope this is useful to someone. Please feel free to ask for more information if I've not made anything clear.Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com6tag:blogger.com,1999:blog-179325926611299376.post-43642591020001784252015-09-20T18:31:00.001+01:002017-03-14T21:22:12.480+00:00More LIDAR goodnessI looked into LIDAR data from the <a href="https://www.gov.uk/government/organisations/environment-agency">UK Environment Agency</a> some weeks ago. I needed it help a local group who are investigating flood mitigation options. The data was listed as being about £26,500 but we got a 100% discount if we used it for restricted, research use, so we could afford it! A few weeks after I'd used the height data for the group I got an email from the Environment Agency. They said the data was being made available as Open Data under the <a href="http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/">Open Government Licence</a>. So now I could use it for any other purposes at no cost. You can get the data from <a href="http://environment.data.gov.uk/ds/survey">http://environment.data.gov.uk/ds/survey</a><br />
<br />
I decided to make a detailed relief map of part of the area close to home. The data doesn't cover the whole country, only parts that are deemed at risk of flooding. All of Hull and the river Hull catchment area are included in this. I've only looked at my local area so other areas may vary.<br />
<br />
The data is downloaded as Ordnance Survey 10km grid tiles. There are 2m, 1m, 50cm and 25cm options and digital terrain model and digital surface model options too, so let's look at these options, but first a bit about LIDAR.<br />
<br />
<a href="https://en.wikipedia.org/wiki/Lidar">LIDAR</a> is a technology that uses laser light to measure a distance repeatedly over an area to create a 3D model of an area. If the LIDAR transceiver is mounted at a fixed point it can pan around to record a very detailed image in 3D of everything that can be seen from that point. It works very well in this way inside a building or a cave to make a very accurate model. The US Space Shuttle flew a mission to use a variant of LIDAR to record the height of the surface of the Earth from space. This is available as <a href="http://www2.jpl.nasa.gov/srtm/">SRTM</a>.<br />
<br />
More recently LIDAR equipment has been flown in aircraft. The difficulty of making useful measurements from an aircraft should not be underestimated. The only data LIDAR returns is distance to the target, so knowing PRECISELY where the aircraft is in 3D is the real problem. GPS is hopeless at altitude measurement and scarcely good enough for lateral location, barometric height measurements vary over time and location and inertial dead-reckoning accuracy falls off with time. A combination of all of these plus post-processing can result in useful data.<br />
<br />
The Environment Agency LIDAR distance options specify the distance between the sample points, the 2m option having less detail than the 25cm option. The area that these options cover varies with the highest detail covering the smallest area. I chose the 50cm option as it covered the area I wanted at the highest level of detail. The detail does make for larger datasets and more processing needed to do anything with it.<br />
<br />
Clearly the LIDAR measures the distance to the first object it encounters from the aircraft, so it measures tree tops, building roofs and even vehicles. This is known as the digital surface model. This is often a composite from multiple images, as this data is, to compensate for location inaccuracies and to help remove things like vehicles. To get a useful model of the real landscape, without trees and buildings, the data is post processed to create the digital terrain model. This is the data I have used.<br />
<br />
The OGL data was different from the data the Environment Agency originally supplied. The original data was in smaller grid squares and the height was rounded to the nearest centimetre. The OGL data is in bigger squares which makes it a bit easier to process but seems to use 18 decimal places of a metre, which is smaller than the diameter of an atom.<br />
<br />
I wanted to create a relief map and make contours from the data and, not for the first time, <a href="http://www.gdal.org/">GDAL</a> had the tools. The data uses the UK Ordnance Survey projection, known as OSGB36 or ESPG:27700, so to use any OSM data with it I would need to reproject to WGS84 or EPSG:4326.<br />
<br />
To make a relief map I used gdaldem with the hillshade option on each of the datafiles. These need to be joined together to make a larger image, so the option -compute-edges is also needed. The complete command is:<br />
<blockquote class="tr_bq">
<span style="font-family: inherit;">gdaldem hillshade -compute_edges infile relieftiff/outfile.tif</span></blockquote>
The output is geoTIFF files which can be merged into a single (large) geoTIFF with the command<br />
<blockquote class="tr_bq">
gdal_merge.py -o big.tif *.tif</blockquote>
This creates a geoTIFF file which has the image of the relief in a TIFF image and also has the locations of the edges in the original OS projection.<br />
<br />
The next step is to use gdalwarp to reproject the large tiff file to one in the WGS84 projection. The command describes the source and target projection and filenames. There are significant missing pieces in the large TIFF as the available data was not rectangular. The -srcnodata 0 and -dstalpha makes missing data transparent rather than black.<br />
<blockquote class="tr_bq">
gdalwarp -s_srs epsg:27700 -t_srs epsg:4326 -srcnodata 0 -dstalpha big.tif bigr.tif</blockquote>
The new TIFF file is what we want to see, but now it needs turning into tiles to be displayed on a slippy map. I decided that zoom level 13 to 18 would give a useful display. To make these tiles I used gdal2tiles.py, specifying the reprojected TIFF image, the zoom levels and the folder to put the tiles into.<br />
<blockquote class="tr_bq">
gdal2tiles.py -z13-19 bigrt.tif tiles</blockquote>
This makes a set of tiles in the TMS format in the specified folder, in this case tiles.<br />
<br />
Another way to visualise the LIDAR data is contours. I decided to create a set of overlay tiles that are transparent except for the contours. These can have a different density of contours at each zoom level. I chose the smallest contour step to be shown at the highest zoom level to be 0.2 metres. The GDAL tool for the job is gdal_contour which makes a shapefile with a linestring for each contour. The command is<br />
<blockquote class="tr_bq">
gdal_contour -i 0.2 -a height infile outshapefile.shp</blockquote>
The resulting shapefile needs to be reprojected to WGS84. The tool to reproject shapefiles is ogr2ogr<br />
<blockquote class="tr_bq">
ogr2ogr -t_srs epsg:4326 -s_srs epsg:27700 outshape.shp new.shp</blockquote>
I decided to use Mapnik to make the contour overlay tiles. Mapnik can use shapefiles but specifying the long list of shapefiles created above would be a problem so I loaded the shapefiles into a postgresql table in a database with PostGIS enabled. Postgresql comes with shp2pgsql to do this:<br />
<blockquote class="tr_bq">
shp2pgsql -a -g way new.shp eacontours > new.sql</blockquote>
This makes SQL to load the shapefile into eacontours table, putting the geometry in the field called way. To load this into a database called ukcontours which already has the postgis extension installed the command is<br />
<blockquote class="tr_bq">
psql -d ukcontours -f new.sql</blockquote>
I then designed the overlay with Tilemill to create the transparent tiles with more contours at higher zoom levels.<br />
<br />
You can see the results at <a href="http://relief.raggedred.net/">http://relief.raggedred.net</a>. I added a water overlay and a roads overlay (thanks to MapQuest for the roads) to help position the relief imagery.Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com2tag:blogger.com,1999:blog-179325926611299376.post-25859716432632787462015-08-11T18:14:00.001+01:002015-11-17T22:21:07.099+00:00StrappedHull is a low-lying city, as are some of the towns and villages that surround it. I have blogged before about the serious flooding in the area in 2007, just before I started mapping in OSM. What a lot of people won't know, even locals, is that flooding is a regular occurrence in some parts of the area. This flooding is on a much smaller scale than happened in 2007, but it still causes misery for anyone whose house is flooded. The Cottingham Flood Action Group have tried to understand what causes this regular flooding and then campaign to fix the causes. I have helped a bit by providing maps and overlaying data from various sources. Some of these sources are not open so I don't want to publish them here.<br />
<br />
One source of contention is riparian ownership. This can apply when a water course runs through your land and you may then be required to maintain it. For example a ditch carrying surface water away may need to be cleaned out so water can actually flow in the ditch and not back up causing a flood elsewhere.<br />
<br />
The East Riding of Yorkshire council is strapped for cash. They are pushing people to maintain ditches and in some cases ditches in culverts to prevent flooding, claiming these people are riparian owners of the ditches. To be a riparian owner the watercourse must be wholly within the property or form the boundary of the property. If the watercourse forms a boundary, then the property owner owns the watercourse to the mid line.<br />
<br />
I have matched the Land Registry Inspire dataset to my surveys of a few of these and it clearly shows that the ditch is many metres outside of the privately owned land and well within the area of the public highway. This means that no matter how strapped the council is for cash, they must maintain the ditch or culvert. Of course, my survey probably isn't enough to convince the council, but it certainly should prompt people to get more evidence.<br />
<br />
Don't believe your council. If they claim you are responsible for something unusual, make them prove it and make sure you get enough facts to stand up to them. The council may be right, but they be just trying to get you to pay for something they are responsible for. Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com0tag:blogger.com,1999:blog-179325926611299376.post-56147578588705411632015-06-25T17:52:00.003+01:002015-06-25T17:52:38.851+01:00LidarToday I received two DVDs with Lidar data on them. They cover Hull and some of the surrounding area data showing centimetre height records at 50cm spacing. It shows DTM, so no buildings or trees, though DSM is available too.<br />
<br />
I've just finished making a replacement window for the garage - the old one was completely rotten - so I fitted it this afternoon while the weather was fine. Now that's done I can turn to this detailed Lidar data. Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com2tag:blogger.com,1999:blog-179325926611299376.post-23884061616526960252015-06-06T20:48:00.001+01:002015-06-06T20:48:10.918+01:00When does a postcode start?I have mapped a fairly large area, Hull and most of East Yorkshire. When there was nothing on the map I just added what I found but now there is a fairly complete road network. Keeping that up to date can be hard work - how do you know what has been added since the last time I looked? OS Locator is really valuable for this as new roads appear on there as OS maps them. It would be better to know developments are starting and get in early and I think there may be a way: postcodes.<br />
<br />
New postcodes are allocated all the time. Around a thousand are created every month in GB. The Office of National Statistics publish a postcode list based on Codepoint Open data but with some extra stuff in there. One thing is the month the postcode was set up. Looking at the recent ones may let us find places that need surveying.<br />
<br />
I've created a map to visualise these: <a href="http://pcage.raggedred.net/">http://pcage.raggedred.net</a> <br />
<br />
I hope this proves useful.Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com2tag:blogger.com,1999:blog-179325926611299376.post-43935617074059365242015-06-03T20:39:00.002+01:002015-06-03T20:44:06.361+01:00New postcode layerI've just finished extracting the data from a full download of OS Open Names and setting up a layer to see the new postcodes. The postcode data is only a centroid for each postcode area. <br />
<br />
I maintain a layer of postcodes from the Office of National Statistics, one from Codepoint Open and now Open Names. The Codepoint Open and ONS postcodes are in the same locations, though Codepoint Open data is a bit more up-to-date right now. The OS Open Names postcode locations are now sub-metre, so I expected that the centroid might be in a slightly different location, but some are substantially different as you can see here.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-Qm2kVEPyiCI/VW9Uw803IPI/AAAAAAAAB30/HB90N1eRZUU/s1600/postcodes.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-Qm2kVEPyiCI/VW9Uw803IPI/AAAAAAAAB30/HB90N1eRZUU/s1600/postcodes.png" /></a></div>
The Codepoint Open are in red, the OS Open Names are in magenta.<br />
<br />
I don't store any of these tiles, I render them on the fly. Real tiles are often stored in a hierarchy of folders in the form of<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">tilename-base/zoom/xposition/yposition.png</span></blockquote>
Since I don't store any of these, when someone requests a tile they would normally get a "Not found (404)" error. I trap these errors and use the folder hierarchy to extract the postcode centroids from a database for the zoom, x and y area of the tile. This is used to render a tile with a transparent background and centroid markers on it. I do this because I didn't want to use the considerable disk space the real tiles would take up, especially as I create to to zoom level 21 to make editing easier with the layer switched on.<br />
<br />
I also extracted the road name and place name data, reprojected all of the location data from OS grid references to lon and lat and stored them for later use. If anyone wants to see this I'll happily pass it on, just ask.<br />
<br />
If you want to see the new postcodes you can see the new layer on <a href="http://oscompare.raggedred.net/?zoom=13&lat=53.74575&lon=-0.48529&layers=0B0FFTFT">oscompare</a>. Make sure you open the layer selection (blue+white +) and select what you want to see. You need to zoom in to see the postcodes.Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com6tag:blogger.com,1999:blog-179325926611299376.post-39997191274484574032015-06-02T17:49:00.001+01:002015-06-02T17:49:38.194+01:00Twenty by twentyIt seems that there are not missing sections in the OS Open Names downloads from OS OpenData. The sections are 20x20 km squares. Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com0tag:blogger.com,1999:blog-179325926611299376.post-29901575518904541672015-06-01T22:09:00.001+01:002015-06-01T22:09:15.203+01:00OS Open NamesAs I noted in the previous post Ordnance Survey have said that OS Locator open data is being withdrawn and being replaced by OS Open Names. The file is a strange mixture of three types of data jumbled up. So I needed to separate them into something useful. The three types are postcode centroids, place names and road names. All of the locations are in the projection that Ordnance Survey use, OSGB36 or EPSG:27700. The great thing about the values used in this projection is that they are measured in metres from a fixed point. The previous OS open data, such as OS Locator or CodePoint Open (postcode centroids) use this projection too of course which fixes a location to a square metre. OS Open Names however has a decimal component so the location is now specified to the nearest millimetre - more accurate than OSM works to. The east-west specification in OS parlance is know as eastings and the north-south specification is known as northings. Eastings and northings can be converted to longitude and latitude for use in OSM using, for example, GDAL libraries.<br />
<br />
The postcode centroids are very simple records. It needs the postcode and the easting and northing for the location of the centroid. Comparing a few OS Open Names centroids with Codepoint Open records the centroids are in slightly different locations. <br />
<br />
OS Locator lists named roads with its name, a centroid and a bounding box for the road. There is also a hierarchy of place, borough, county or unitary authority. All of this is in OS Open Names and the increased millimetre accuracy is there too.<br />
<br />
I've not used the gazetteer of place names, but the name and location data are available as you would expect.<br />
<br />
All of the data types have some URIs in the files too for many of the fields. Many that I have tried to open are dead links, but some show the hierarchy of data. I'm not sure why this is useful. <br />
<br />
I think I'm going to write a routine to unzip and process all of the OS Open Names data. I'll load the data into a database, reproject it for OSM and make the processed data available if anyone wants it.<br />
<br />
First, however, OS need to supply the missing sections of their open data. Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com2tag:blogger.com,1999:blog-179325926611299376.post-9245922863718426812015-06-01T17:12:00.002+01:002015-06-02T16:54:21.208+01:00OS Locator is to be withdrawnI just received an email from Ordnance Survey, telling me that OS Locator, part of the OS Open Data, is to be withdrawn in a year's time. They say that OS Open Names has been published and that replaces OS Locator. They hint that it may replace CodePoint Open too, though there's no notice of withdrawal for that yet. I thought I should look at OS Open Names to see what it offers. <br />
<br />
The first thing I saw was that the data is broken into the OS large-scale grid squares, which break the country into a 13x7 grid with 55 squares with actual data in them. Each one of these needs to be downloaded separately to cover GB. That is a pain to start with. I requested SE and TA which are both needed to cover the village I live in. After downloading them the .zip file contains each area broken down into 100 separate sections as I would expect, but I quickly realised that there were only 25 in SE and 10 in TA. Much of TA covers the North Sea, so there should be fewer sections, but clearly many were still missing. I downloaded another area and again the same 75 sections were missing. I emailed customer service at OS to point this out, with no answer yet.<br />
<br />
I pressed on to look at the data in the sections that were there. The data is in a CSV format (there was another choice). There is a summary of column headings, but the data is a strange muddle of three types of data. Much of the data is a url to an OS website with the data summarised on a page for each column of each line of the text. It appears that there is a record type for place names, a record type for postcode centroids and a record type for named roads all freely mixed up throughout the file. There doesn't seem to be a field to simply identify what the record type is for each line. The first field is an id field. For place names it seems to start with osgb and has a number after that, for postcodes a postcode type id, with spaces removed, is in the id field and for road names an id that looks like a GUID is in the field. It doesn't seem to be a pattern for why the data is in the order that it is.<br />
<br />
I'm going to throw some code together to disentangle these record types and see what useful data is then available. Hopefully we will not have lost anything useful in this process, though it does look as though processing this open data is going to be a bit harder than it used to be.<br />
<br />
Anyone would think OS don't want to release open data.<br />
<br />
Edit: There are two fields which distinguishes these different record types. Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com0tag:blogger.com,1999:blog-179325926611299376.post-44903747174487223342015-06-01T12:38:00.002+01:002015-06-01T12:38:56.910+01:00Road ClosureThe road I live on is closed. A house needs some work done on it. The house wall is tight against the road and that part of the road is also narrow. Scaffolding to work on the house now stands to the middle of the road, so the road is closed to vehicles for two weeks<br />
<br />
Should I change the road in Openstreetmap to reflect this temporary closure? I have decided not to change it. Anyone trying to use the road in a vehicle will be directed by another route, by the signs. Most are locals so they will know the alternatives anyway. If I make a change to OSM and someone downloads a snapshot of the data with that change in place they may not download a new version for a while and have the break in the road long after it has actually reopened. Anyway the road exists - it's just not accessible to vehicles for a while.<br />
<br />
This closure has caused some very poor driving standards to surface. The alternative routes are small residential roads with parked cars and tight and narrow corners. People using these routes seem to need to take out some annoyance with their normal route being closed by roaring through these streets at ludicrous speeds with no intention to give way to anyone else.<br />
<br />
The closure was a surprise to me. I saw a sign go up announcing the closure a couple of days before it happened. When I tried to find out why the road was being closed I drew a blank. The council didn't respond to my request for information and searching their website produced nothing at all. A neighbour pointed out that there was a notice in the local newspaper which was reproduced online. Few few people still read the Hull Daily Mail - it has been steadily descending way below mediocrity for many years. The online notice didn't appear in my searches because it is not sensibly indexed. The HDM website is a nightmare to use, with any page constantly bouncing around as adverts and videos randomly pop up and freeze the pages.<br />
<br />
Councils are obliged to publish public notices about some things, such as road closures. It's clear to me that publishing in the local newspaper is not a viable way to do this as they no longer reach much of the population. A sensible addition would be publish the notice on the council's website. I what the East Riding of Yorkshire council will respond to my suggestion of this.Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com4tag:blogger.com,1999:blog-179325926611299376.post-909749606328676402015-05-21T21:27:00.005+01:002015-05-21T21:27:55.127+01:00Not AloneI went out looking for new developments in East Yorkshire today. Looks like I was not alone.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-HIIwwgScjPs/VV4_us1-R7I/AAAAAAAAB3E/WuFis5CPHv0/s1600/googcar.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="http://4.bp.blogspot.com/-HIIwwgScjPs/VV4_us1-R7I/AAAAAAAAB3E/WuFis5CPHv0/s400/googcar.JPG" width="318" /></a></div>
<br />Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com5tag:blogger.com,1999:blog-179325926611299376.post-88549952713701201752015-05-11T15:37:00.002+01:002015-05-11T15:39:11.422+01:00Heights & OSWorking with Digital Elevation Models (DEM) is an interesting extension to creating maps, which are usually a flat representation of part of the world. I really want to find a way to show elevation in a way that is a bit different from a flat map. <a href="http://chris-osm.blogspot.co.uk/2015/05/heights.html">Working with the OS DEM data</a> has whet my appetite to try something new, but first I need a map to work with.<br />
<br />
The most detailed DEM data I have is based on Ordnance Survey OpenData, so creating a map in the OS projection will be useful. I use <a href="https://www.mapbox.com/tilemill/">TileMill</a> to create maps from OSM data. <br />
<br />
Firstly I needed OSM data in the Ordnance Survey projection. That means loading some OSM data into a fresh PostgreSQL database. I created a PostgreSQL database and, as usual, add the extension for PostGIS. this creates a table called spacial_ref_sys that includes the OS projection, amongst many others. I often add the hstore extension too, but this is a simple map so I didn't need it. <br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">createdb -E UTF8 EYOS<br />echo "CREATE EXTENSION postgis;" | psql -d EYOS</span></blockquote>
I loaded an extract of OSM data using the usual osm2pgsql utility except the projection was needed too to convert the data to OS projection as it is loaded.<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">osm2pgsql --slim -d EYOS -C 1024 ey.osm.pbf --proj 27700 </span></blockquote>
I decided to add a coastline, so that needed to be in OS projection too. OSM coastlines are handled differently from all other data. They are extracted from the main DB, checked for consistency and created into a shapefile for the world. This is known as processed_p.shp. I have my own copy with a cut-down version with only the British Isles in it to make rendering a bit quicker. I reprojected that to a copy in OS projection using OGR2OGR, part of the <a href="http://www.gdal.org/">Geospatial Data Abstraction Library</a>. <br />
<br />
<br />
<br />
<blockquote class="tr_bq">
<span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">ogr2ogr -t_srs 'EPSG:27700' -s_srs 'EPSG:3857' coast_bi_os.shp coast_bi.shp</span></span> </blockquote>
<br />
Armed with all of this I could now start TileMill and add the layers I need for the map. Each of the layers, including the coast shapefile, needed a custom projection. This is:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.999601 +x_0=400000 +y_0=-100000 +ellps=airy +units=m +towgs84=446.448,-125.157,542.060,0.1502,0.2470,0.8421,-20.4894 +units=m +nodefs </span></blockquote>
I got this from the PostgreSQL table postgis created above. Once I had designed the map as I wanted it I exported the Mapnik XML and ran it through Mapnik. I discovered that the Mapnik XML was not quite right. It needed to have the third line changed so the srs part matches the custom projection above. There doesn't seem to be a way to set this in TileMill, so a manual edit was needed.<br />
<br />
To run the Mapnik XML through Mapnik I used the following python code:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">#!/usr/bin/python<br /><br /># generate a map image in OS projection epsg:27700<br /><br />import mapnik<br />import sys, os<br />def drawMap(filename, west,south,east,north):<br /> print(filename)<br /> sz = 5000<br /> ossrs = "+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +towgs84=446.448,-125.157,542.06,0.1502,0.247,0.8421,-20.4894 +units=m +no_defs"<br /> m = mapnik.Map(sz,sz,ossrs)<br /> mapnik.load_map(m,"osproj.xml")<br /> bbox = mapnik.Envelope(west,south,east,north)<br /> m.zoom_to_box(bbox)<br /> im = mapnik.Image(sz,sz)<br /> mapnik.render(m, im)<br /> view = im.view(0,0,sz,sz) # x,y,width,height<br /> view.save(filename,'png')<br /><br />if __name__ == '__main__':<br /> drawMap('cott.png',500000,430000,510000,440000)</span> </blockquote>
<br />
<a href="http://3.bp.blogspot.com/-q1KxTY7-vtQ/VVC-YJbqKcI/AAAAAAAAB2w/jItfj1F679A/s1600/cott.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="320" src="http://3.bp.blogspot.com/-q1KxTY7-vtQ/VVC-YJbqKcI/AAAAAAAAB2w/jItfj1F679A/s320/cott.png" width="320" /></a> That long-winded projection was needed again. Notice the coordinates in the drawMap function are OS coordinates, not longitude and latitude. Everything must match the chosen projection.<br />
<br />
This gives me an image of the map in the OS projection, but the style could be any style you choose, though I'd be wary of copying the OS style too closely. This will now match the DEM data if they are combined. My style is still a bit stark and only renders a few objects, but it is something to work with.<br />
<br />
Next I need to use it imaginatively. Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com0tag:blogger.com,1999:blog-179325926611299376.post-78797266956502909052015-05-08T16:52:00.003+01:002015-05-08T17:01:31.116+01:00HeightsI've been working on something locally for a while that benefits from maps. It needs height information displayed so I thought I'd take a closer look at what was available, especially Digital Elevation Model (DEM) data<br />
<br />
OSM doesn't hold much height information, so when people want to display heights they turn to outside information. One such source is <a href="http://www2.jpl.nasa.gov/srtm/">Shuttle Radar Topography Mission data</a> or SRTM. One Space Shuttle mission flew around the world and mapped the heights of the ground below using radar. This data has been published as open data. It is 1 second of arc data points for the USA and 3 seconds of arc data points for the rest of the world. This gives a height data point about every 90m for the UK. There are issues with this data with some places having voids where the radar return didn't register. It is usual for people who use this with OSM to render this data as contour lines or as hill shading or both as a way of visualising the height. I thought I'd do some simple processing to be sure I understood the data format.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbgYGbqTHDesB8KCtWHzs5aaXP6XhDRD0f6cMqwtUkSG3TjAlgfd4jdcpWp_NCRuBTET89rarTL4jLDOINSDYj_VgBgPqyWgQEuQG-BynV0umSzRA9f3mXkHqWd2FghvOQMKGQ8C-GgNE/s1600/h.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbgYGbqTHDesB8KCtWHzs5aaXP6XhDRD0f6cMqwtUkSG3TjAlgfd4jdcpWp_NCRuBTET89rarTL4jLDOINSDYj_VgBgPqyWgQEuQG-BynV0umSzRA9f3mXkHqWd2FghvOQMKGQ8C-GgNE/s320/h.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">SRTM</td><td class="tr-caption" style="text-align: center;"></td></tr>
</tbody></table>
The SRTM is published as a 1° square. I read the height values and displayed them as a shade of green since human eyes can distinguish more share variations in green than any other colour. Any voids I show as black There were nine pixels in this square) and any value with a small negative value (small so not a void) I show as blue. There's a lot of interest in this which I'd not noticed looking at contours. The dark area top left is the Vale of York, the green area top centre is the bottom end of the Yorkshire Wolds You can just make out the Humber estuary just above the centre and to the right. The bright green area bottom right is part of the Lincolnshire Wolds. The valleys with tributaries feeding into the Vale of York are interesting. None of those exist as rivers or streams today, so I expect they are remnants of the retreating ice caps about ten thousand years ago when the ground was still permafrost so any melting water cut river channels. Today the water table is much lower with the chalk of the Wolds allowing water to drain into it.<br />
<br />
Next I looked at Ordnance Survey (OS) OpenData. They release height data as contours and spot heights in shape file format and DEM data too. The DEM is 50m spacing and should be free of voids. They use their own projection (EPSG:27700) for all of their data and this works better for the UK for some jobs. OS release some of their data in parcels based on their own grid. I am interested in a section of including Cottingham, <a href="http://www.openstreetmap.org/#map=13/53.7776/-0.4249&layers=C">a large village west of Hull</a>. The OS square TA03 has Cottingham in the middle of it, so that is helpful.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-M4suWzndaTM/VUzYDCEWEjI/AAAAAAAAB2c/lZQSUU3_5eU/s1600/o.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="http://1.bp.blogspot.com/-M4suWzndaTM/VUzYDCEWEjI/AAAAAAAAB2c/lZQSUU3_5eU/s1600/o.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">OS TA03 square</td></tr>
</tbody></table>
I created a similar, image from the OS DEM data. I deliberately emphasised the height differences more than the SRTM image. The area is much smaller than the SRTM area but more detailed. bright green on the left is the edge of the Yorkshire Wolds. The blue line is the river Hull which cuts through the middle of the city of Hull. For comparison the bottom of the the blue smudge on the SRTM image is approximately where the OS image is. Again valleys are shown, though this time running west to east. Again they are dry (though very occasionally not which is part of what I'm investigating). I've decided that there is enough detail in the OS area and that it is big enough, perhaps with one more alongside it, to show what I want. so I'll work with that.<br />
<br />
More of what to do with it later.<br />
<br />
The python code to produce the SRTM image is here:<br />
<blockquote class="tr_bq">
<span style="font-size: xx-small;"><span style="font-family: "Courier New",Courier,monospace;">#!/usr/bin/env python<br /># -*- coding: utf-8 -*-<br /><br />import struct<br />from PIL import Image<br /><br />def getpix(val):<br /> if val == -32768:<br /> return (0,0,0)<br /> if val < 0:<br /> return (0,0,255)<br /> return (0,int(val),0)<br /><br />if __name__ == "__main__":<br /> top=54<br /> left=5<br /> highest=-5000<br /> lowest=5000<br /> tile = "N53W001.hgt"<br /> <br /> #make the new empty (white) image<br /> im = Image.new("RGB", (1201, 1201), "white")<br /> <br /> with open(tile, "rb") as f:<br /> <br /> #print get_sample(tile, n, e)<br /> <br /> # scan through each of the heights in the file and colour a pixel<br /> for n in range(1201):<br /> for e in range(1201):<br /> buf = f.read(2)<br /> hite=struct.unpack('>h', buf)<br /> #print '{0} {1} {2}'.format(n,e,hite[0])<br /> pt=hite[0]<br /> if pt == -32768:<br /> print 'VOID {0} {1} {2}'.format(n,e,pt)<br /> if pt<lowest and="" br="" pt=""> lowest=pt<br /> if pt>highest:<br /> highest=pt<br /> im.putpixel((e,n),getpix(pt))<br /> <br /> print 'lowest:{0}, highest:{1}'.format(lowest,highest)<br /> im.save('h.png')</lowest></span></span></blockquote>
The code to produce the OS image is here:<br />
<blockquote class="tr_bq">
<br />
<span style="font-size: xx-small;"><span style="font-family: "Courier New",Courier,monospace;">#!/usr/bin/env python<br /># -*- coding: utf-8 -*-<br /><br />import struct<br />from PIL import Image<br /><br />def getpix(num):<br /> val=round(num)<br /> if val < 0:<br /> return (0,0,255)<br /> return (0,int(val)*3,0)<br /><br />if __name__ == "__main__":<br /> top=54<br /> left=5<br /> highest=-5000<br /> lowest=5000<br /> osf = "TA03.asc"<br /> <br /> #make the new empty (white) image<br /> im = Image.new("RGB", (200, 200), "white")<br /> <br /> with open(osf, "rb") as f:<br /> lines=f.readlines()<br /> for i in range(5,205):<br /> s=lines[i].split(' ')<br /> for idx, val in enumerate(s):<br /> im.putpixel((idx,i-5),getpix(float(val)))<br /> <br /> im.save('o.png')</span></span> </blockquote>
<br />Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com0tag:blogger.com,1999:blog-179325926611299376.post-44656218876090208112015-02-16T12:53:00.000+00:002015-02-16T12:53:00.914+00:00Drone deliveriesThe US FAA has ruled that drones need to remain within sight of the operator. This is a major obstacle to companies planning deliveries in the US by drone. I had wondered what kind of mapping such an enterprise would need. Where would such a delivery be made? If the address has a garden then that might be useful, but what if there are plants or garden furniture in the way? If the address has a driveway then that could be a good landing site, but if the delivery sits on the driveway what's to stop it being stolen or driven over by car arriving? If the address is an apartment on a street front will the parcel just be dumped on the street? How could a drone safely land on a sidewalk?<br />
<br />
When a delivery is made by hand it is handed over at a doorway, posted into a mailbox or left by a thinking person in a suitable place. A person can gain access to places like a shared lobby too. Can a drone do any of this?<br />
<br />
Maybe would-be delivery addresses need to designate a landing site for drones, maybe that would need signage and access restrictions and possibly even extra insurance. <br />
<br />
The more I think about it the mapping needed to control this would be very detailed and very specific. I suspect that the FAA ruling is the least of the problems this idea has.Chris Hillhttp://www.blogger.com/profile/02951528269028953589noreply@blogger.com2