Open data? Hmmmm

I have been impressed with SK53's use of open data in Nottingham, so I thought I'd see what was available for my home county of the East Riding of Yorkshire and for nearby Hull. The answer has been disappointing and frustrating.

I decided to look at the web site of the two councils in the hope that either of them had started to publish any open data. Not a trace of anything vaguely resembling publishing anything as open data on either site. For example, ERoY site has a map showing current road works. It needs MS Silverlight to see it, and when you do look, the data is copyright. Hull City website has a link to food hygiene data on the landing page, but that only sends you to the Food Standards Agency site, not a local data set and, as before, everything on their website seems to be copyright.

So, using SK53's example, I asked for food hygiene data from the two councils, in the hope that I would find a list of addresses just like he did. I sent an email request on Saturday 17th August to each council. The request said that I wanted to analyse food hygiene data and I would like to know if they publish such data as open data, preferably under the Open Government Licence. As I usually do in such matters I asked them to treat this as request under the Freedom of Information act.

Given that the request was sent on a Saturday, it is reasonable to expect that no one read it until the following Monday, 19th August, so I would expect the twenty working days clock that applies to FoI requests to start then. Allowing of the public holiday in August I would expect an answer by 17th September and both councils beat that deadline comfortably.

Hull City council responded quickly. On 21st August I got a reply from Garry of the Food and Health & Safety (sic) team to say they had been busy and would I please send my request to the Information Governance Team. He did also direct me to the FSA website too. This is directly contrary to the FoI act. It is also contrary to the Hull City Council policy on the FoI act. I replied pointing this out. The Information Governance Team responded by saying that the officer was wrong to direct me to them, apologised and said that the email should just have been forwarded to them internally. They confirmed that the deadline for their response would be 16th September. On the 29th August, well within the deadline, the Information Governance Team answered with the same idea as Garry: look at the FSA site.

ERoY council first replied on the 22nd of August. The email had an attached MS Word document. The first page had headings but was otherwise blank, the second page acknowledged my request, repeating my email, and the third page confirmed that they would reply by the 17th September, as I expected. On the 9th September I got a response. That too was an MS Word document attached to an email. That too recommended that I look at the FSA website to find such data. They gave a link to a council website page that includes a list of registered food premises. This is a simple list, with no description and no licence or copyright information, so it is not possible to assume that it is open data.

When I was disappointed with the lack of open data on either councils' websites I also sent an email to the Department for Communities and Local Government. They have suggested in the past that councils should expect to publish any data they have under an open licence. I asked if the DfCLA or my local Member of Parliament might be able to do anything to apply pressure to the councils to publish any data as open data as a driving force for openness and innovation. Today I got a response from Rt Hon David Davis MP, my MP, with an attached letter from Malcolm Sims, Director of Corporate Resources, East Riding of Yorkshire council. Mr Sims confirms that they have received an FoI request and that they will reply by 20th September, which they already have. Neither Mr Davis nor Mr Sims seem to grasp the underlying point: they don't publish open data, and don't seem to want to.

The art of politics is to give a response to a question that wasn't asked, but that you can answer.

Rights of Way

I saw mention of some open data about rights of way in various English and Welsh counties on http://www.rowmaps.com/. I downloaded the shapefile and took a look at it. The rights of way are stored as MultiLineString because a few are broken into two sections, though most are single lines. I wanted to see how they relate to any rights of way in OSM. I am not sure about the licence - there is no statement from East Riding of Yorkshire council about the licence so I cannot load this in any way into OSM. I do have other reservations about the data too, but more of that later.

My web site provider, like many cheap one, provide PHP and MySQL to create a site. I have not used the spatial functions in MySQL before so I decided to see how well they work and in the process make an overlay for an OSM map with these rights of way on it. I created a simple python script to load the data into a very simple MySQL table, loading the MultiLineString into a geometry field. The functions that work with the geometry fields in MySQL are much less capable than PostGIS in PostgreSQL. MySQL allows the geometry be to selected based on a bounding box. This is ideal for me as I want to select the items that show on a section of map that is currently being displayed. Its bounding box is easy to find. The MySQL function is MBRIntersects. The bounding box of the current map being displayed is converted into a polygon and this is compared to the bounding box that encloses each of the MultiLineString in each row and if the two bounding boxes intersect then the row is selected. This means that some part of the right-of-way that the data represent should be displayed on the displayed window of the map. So how to display it?

I like the Leaflet library to display maps and Leaflet has good support for displaying geoJson as an overlay. When the map is first displayed or scrolled or panned I use the new bounding box to create an ajax request which returns the data selected as a geojson. This is a PHP script which uses the bounding box to convert into a simple SELECT statement and parses the returned data into geoJson. The existing overlay is cleared and the freshly returned data are added to the geoJson layer.

A geoJson layer is a collection of features, in my case each feature is a MultiLineSegment from the original shapefile. It also has properties which we got from the shapefile too. As each feature is processed the style can be processed dynamically. In my case the footpaths are made red, the bridleways made green and the byways are blue; one of the properties is the right of way type. I also use the onFeature event to call a function that adds an onClick event to each feature. When a feature is clicked the properties can be displayed in a side bar. I added a hash to the address line too to match the standard OSM map. You can see the result here.

So now I can compare the OSM map with the data from the council. The first thing to say is that there are a lot of rights of way on the overlay (1762) and a significant number are clearly missing from OSM. On closer examination though the data does not look very accurate nor very up to date. In West Ella a footpath seems to go through a couple a of houses. Elsewhere a local footpath was moved many years ago, but the old route is still shown. OSM shows the surveyed and signed route. There are other anomalies too.

Having the overlay does provide a good place to start surveying what is really on the ground and signed, but loading the data verbatim doesn't look a very good idea even if the licence was published and acceptable.

I think I may yet extend the overlay to show which rights-of-way have been surveyed and matched with OSM ways. It looks like there's some walking and cycling ahead, which in the East Riding countryside is never a hardship.

Comparing layers

About three years ago I created a simple map using OpenLayers to display a base layer and overlay some layers on it. It features layers useful to some GB mappers. From the logs I can see that it is used quite a lot.  All of the slippy maps I've created recently have used Leaflet rather than OpenLayers so I thought I should update my comparison map in Leaflet. You can see the result here.

It has base layers of MapQuest open, the Standard OSM layer or MapQuest roads. For overlays you can choose from the ITO World tiles of OS OpenData Locator anomalies, the OS OpenData StreetView or my tiles showing the Office of National Statistics postcode centroids. Whichever of the overlays are shown you can move the slider (top-left) to change the opacity to make the base or overlay readable.

The browser address line shows the constantly-updated location and zoom, in the new style of the OSM landing page. This makes bookmarking the position easy. There is also a lat / lon display bottom right which shows you the location the mouse is currently pointing at.

I intend to leave the existing oscompare unchanged and maintain this new leaflet-based one in the future. If anyone would like any other layers or if there is any problem please comment.

Thanks to MapQuest and ITO World for their map tiles and their processes for updating them, thanks to Ordnance Survey for their OpenData and thanks to the OSM admins who manage the OS StreetView tiles.

Locating bridges

I haven't been able to get out mapping much recently, but a lovely day tempted me out. I did some OS Locator name chasing and as is often the case, the OS Locator names led me to a development with far more changes than just the ones OS Locator picks up. I will need to return again as the development grows in Kingswood. There's lots of area obviously earmarked for development but presumably waiting for the cash generated from selling the current houses to fund the next stage.

I then went to look at a newly opened crossing of the river Hull, Scale Lane bridge. It is a pedestrian and cycle route across the river very close to the museum quarter. It lead directly to a large fairly new hotel and towards The Deep, Hull's aquarium, or Submarium as they call it. If you are in the city take a look at The Deep, it is interesting. You walk slowly down to see lower and lower levels in the aquarium and then a glass lift carries you from the bottom back to the top to looking out at the large aquarium as you go.

Scale Lane Bridge, the black part swings

The new bridge is interesting too. If it opens for river traffic, gates shut at the east end and this end swings around to open for boats, but anyone on the bridge can stay on and people can still get on and off the bridge at the west end. In the middle of the bridge is a building which will be a café eventually and a small exhibition space too I think. These is a sloped cycleway and a flatter footway, except the footway has steps along it. All along the footway there are seats with views looking across the river. It didn't swing open while I was there, but it still looks quite different for a bridge and I like it.

The Arctic Corsair trawler, now a museum, beside the river Hull walkway

The main road crossing of the river Hull and beyond the tidal barrier

I then took a look at a new development close to home. The Bridgehead development is close to the north end of the Humber bridge. It looks like it will be the usual for commercial buildings: steel structured, clad with glass and shuttering. The first two buildings are under way with room for a few more. The roads are accessible so I grabbed a trace. It is interesting that cycle lanes are being included in the road layouts, since the entrance is off the A15 dual carriageway where there are no cycle lanes. Maybe a new cycleway will join up from the development to the existing cycle lanes nearer the bridge and beyond.

OS Locator chasing

I finally got around to sorting out some new OS Locator anomalies close to home. I went to look at a new small road under the shadow of the Humber bridge. When I got there the road was dug up for pipe laying. The tarmac has been down less than 3 months and already the utilities are digging it up again. As I was standing looking down the road, wondering if I should brave the excavators, a woman in a hard hat and fluorescent jacket approached me smiling. She asked if I was interested in the new houses, which are not finished. When I explained I was making a map the smile faded but she agreed I could walk to the end of the road with my GPS 'at my own risk'. Since everything I do tends to be at my own risk I collected my trace. On the way out I asked the once more smiling woman what the road name is going to be and she confirmed the same name that OS Locator has.

Next I went to Welton. Here a slightly bigger development has been squeezed into what was a farmyard many years ago. Here too the road was uneven, but only because the final surface has not been laid. None of the houses looked finished, indeed some were barely started. Here too I was approached by a florescent jacket, this time draped around a young man with a clipboard. He too wanted to sell me a house, but when he heard that I was making a map he asked me to leave as it was still a building site and not safe for the public. As I already had my trace I left, this time snapping a picture of the new road's name board - about the only part of the development that was complete.

I am used to roads showing up in OS Locator as they are close to complete, so seeing ones so obviously early in their development is a little different. Maybe OS are trying to get ahead of the game.

So do the OSMF working groups work?

I do wonder if the process of working groups really works. I have asked the Licence Working Group to make a simple action to prevent a falsehood being repeated. They did nothing and did not even respond to my emails.

I have now asked the Data Working Group to revert undiscussed mass edits. Again, no action and not even an acknowledgement email. If we are serious about stopping undiscussed mass-edits, the only real way forward is to revert them as quickly as possible to force people to follow the guidelines. If an ordinary mapper, like me does the revert that risks an edit war, especially if the ordinary mapper has already engaged with the mass-editor to no effect as in my case. The longer it takes to revert a mass-edit the harder the revert becomes because of other small-scale edits.

If the working groups are not going to take action or at least acknowledge a request or contact, then they should resign and close down the group so everyone knows that the only process open to an ordinary mapper is down to him.

Enclosures

I had a meeting today about a project in our village. Ian had a map that he has assembled of Swanland & North Ferriby that shows the villages pre-Enclosure Acts. It would be good to create an overlay with the modern OSM map for people to see how things have changed in nearly 200 years. In case you haven't guessed, the answer is lots.

I need to warp Ian's map to fit the EPSG:3857 projection, make its background transparent and then build a little app to show one layer over the other. I'll see how I get on.