Maps of memorials

I spent a bit of time looking at the UK National Inventory of War Memorials web site looking for memorials to add to OSM. I realised they need a lot of work to improve their data so I volunteered to help.

I received a set of instructions so I waded in. The job is to update a holding database that will be used to update the main database in time. The holding database has a list of memorials and the idea is to enter the information about people who are mentioned on the memorial, adding as much detail as is available. The holding database has links to data, either .TIF files that hold scanned images or PDFs that hold photgraphs. The scanned images are often hand-written notes from people who have visited the site. This is screaming out for crowd-sourcing. The disconnect between the person who visited and the person who edits is what has caused the errors and omissions that volunteers are trying to solve, but using the same disconnected system.

The first .TIF file I opened had 1667 pages of scanned images, relating to even more memorials and because they are just bitmaps there's no searching or index available. The first PDF I opened had a horrible photo that was barely readable. The memorials I am part-way through has 1200+ names on it, all need amendments.

It is, however, very satisfying being able to correctly name and identify these men (almost exclusively men in WWI) making it possible for people to trace relatives' memorials. I will stick at it.

The coordinator does seem interested in adding a map to their web site. I knocked up an example OSM map based on extracted UK, OSM data using historic=memorial and that quickly showed that differentiating war memorials from all kinds of other memorials is not straightforward. If you are adding a war memorial please add a tag memorial=war_memorial to differentiate it from others such as blue plaques. 

I want to encourage them to use an OSM-based map, using their data and then they may release their data for use in OSM generally. I have yet to break it to them that every memorial will need a location and the 6 figure OS grid reference they hold for some of them is nothing like accurate enough since it only ties the memorial down to 10,000 square metres, and that's if it's correct. A crowd-sourcing approach to position the memorials more accurately might be in order.

Shrines, what shrines?

Following a project of the week earlier this month to add Memorials we added a few. In order to some idea where there might be war memorials we could survey I googled 'war memorials' and came up with the UK National Inventory of War Memorials which is supported by the Imperial War Museum.

I expected to find a few memorials in Hull, but there were pages and pages of them. Looking closer, many entries were individual memorials attached to grave stones, so I'm ignoring these. Most of what remained were street shrines, which I'd not heard of. The database has a field to enter any reference relating to each entry and most of the Hull ones seem to have come from a single book: HULL STREET SHRINES AND ROLL'S OF HONOUR (sic) by M&M Mann.

I tried to find out more about the book and discovered it was in the local library reference section, which is part of the Hull History Centre. The Hull History Centre is a friendly place, allowing laptops in and providing easy access to their books and map and a complete contrast to the East Yorkshire Treasure House which is plagued by stupid rules (you need a PAT test certificate before you can take a laptop in for example). We went to take a look in the History Centre.

It turned out to be a bit of an odd book. There was no reference information about the book at all, no publisher, no ISBN, no copyright or publishing date and it had been bound by the Hull Library Service. There were indeed a long list of the street shrines in the book, but almost all of it was a series of copies of newspaper cuttings with some that are hard to read being transcribed. These clippings were mostly about the ceremonies to mark the erection of street shrines to commemorate local men who had been killed in the 1914-18 war. There were also articles about how these shrines didn't mention all of the people killed, mostly because they were erected before the end of the war, so more people needed to be added by its end.

One important thing that the web site authors seem to have missed is the introduction. Here, in a bit of a ramble, Malcolm Mann explains that all but six of the shrines no longer exist, for a variety of reasons. He describes the six shrines that do exist, so today we set off to look for them. After searching for four of them in the rain and with the daylight fading fast we gave up. There was no sign of the shrines we looked for. I was expecting to be able to add these to OSM and produce a simple overlay to possibly help the web site. I can show conventional war memorials so maybe that will be useful.

I contacted the web site authors to see if they need any help in our area, but I've not had any response yet. I wonder if they want to include these shrines since they don't exist on the ground and many have not done so for nearly eighty years.

Graveyards and sliding churches

A wonderfully crisp and sunny afternoon drew me out into a few local villages looking for more war memorials. The three we found were all in church yards, one being the isolated one at Rowley. Rowley is a very small hamlet, which has a rather nice country house hotel, Rowley Manor surrounded by a farm and a few houses and the church of St Peters. The actual civil parish of Rowley is quite large and takes in the village of Little Weighton. We walked around the perimeter wall to get a trace for it and looked out over the strange lych gate. It is has a path in the church yard to it, but outside of the boundary wall there is just a paddock.

Skidby

Rowley

Walkington

The war memorials are easy to spot because they have the wreaths of poppies on them. I've contacted the UK National Inventory of War Memorials to see if they want to include any mapping on their web site, but I've not had a reply yet.

When I came to enter my traces into the OSM DB, I used JOSM and had a slippy map overlay for the Ordnance Survey OpenData. The area I was looking at shows how badly misaligned some of the OS OpenData overlay is. I wanted to add the churches but the (crude) building outlines in the OS overlay were way off according to my traces. This is a common and consistent problem, both with my current GPS and its predecessor which fell to bits. The overlay always shows objects further south than my traces. I trust the traces more than I trust the overlay's position. In the end I traced a church outline to get the orientation right and slid it to a more reasonable place based on my traces.

Memorials

The project of the week has not really rung my bell before, but this week it is to add war memorials, so I added a few local ones (sorry for the quality of the photos, but the weather was bad and the light poor).

Welton

Brough

Brantingham

South Cave

North Ferriby

Plans

I heard on the local radio that a large development is being proposed in Brough, a few miles from my home. There was a display in the Brough library about the proposed new homes and commercial area, so I thought I'd go and take a look.

Brough has been very substantially extended over the past ten years. When I was at school there was a very large area of greenhouses given over largely to growing tomatoes and cucumbers and in the summer there were always employment opportunities picking the fruit for anyone who didn't mind being roasted alive in the greenhouses and scratched to bits on the cucumber plants. All of the glasshouses, warehousing and packing sheds have long ago been demolished and a huge new housing estate and a school built there. There have been other new facilities built too, such as a supermarket, health centre and, bucking the national trend, a pub. One of the new facilities is a library and community centre where the plans were on show.

The plans were laid out on display boards, with black and white maps showing Brough and the new roads and building areas sketched over as coloured blocks. I can't reproduce them here for fear of breaking the developer's copyright. The maps were very old OS ones. They showed the market gardens as though they still existed, with the existing, new estate blocked in colour which made it look part of the development. All of the existing, new facilities and some existing roads were drawn in as though they were to be added by the developer. The development that really is to be added is largely on open fields but it is partly on the airfield currently owned and operated by BAe. This is fiercely guarded and is not available for private aircraft to use, so only a few aircraft use it each week. One interesting part of the proposal is to build a new supermarket. Tesco already have expressed an interest in building a supermarket near the A63, not far from this proposed development and the developers are latching on to the hostility to Tesco's plans by saying that their supermarket would be instead of Tesco's planned one.

I have no particular objections to the development, but it is interesting to see how maps are being used to portray a developer's specific agenda and attempt to put them in a good light. As to replacing Tesco's proposal, well Tesco have the cash and planning experience to usually get what they want.

Bridges & stuff

When I look at at KeepRight it flags up all sorts of things including where a road crosses a waterway when there's no bridge. There is a small waterway running through Hull called the Beverley and Barmston Drain. I wasn't sure that all of roads that pass over it did so on bridges rather than it simply running through a culvert of some kind, so we went to take a look. The simple answer is there were bridges everywhere, though the pictured one is actually one we already knew about.

As usual when we go out looking for one thing we discover something else. In this case we saw a new industrial estate which is still being completed and a housing estate that is much bigger than it was the last time we checked. The industrial estate didn't have any road names nor a name for the estate, but I expect this to change as the place is finished. The housing estate (pictured) is also still being developed but at least has a road name and also plenty of mud.

The autumn colours in the trees and bushes have been great this year and in spite of a windy day there are still lots of leaves to be seen.

How fast is Autumn

The roundabout on the A1079 is complete, but the link road from it to the caravan works is not accessible yet. The temporary speed limits have been removed. The new traffic signals on the junctions between the A164 and the A1079 are working and all of this is now in the OSM database.

The leaves are changing colour here, a bit late this year, but they look particularly good. I have heard that spring spreads from the south of England northwards at about walking pace, but does autumn spread south at a similar pace?

Link roads, bugs and limits

We needed some compost for potting up winter plants and I know that a new link road is being built near to the garden centre, so I thought I'd check it out. I know the road is being built because the A1079 has a frustrating new 30 mph speed limits and speed cameras on it. A roundabout is being added to help the link road join into the main road. The link load is, I think, only going to Swift caravans to help them move their raw materials onto their site and especially move the large mobile homes and caravans they produce off site, directly onto a substantial road and not through Cottingham village. The new roads cuts across North Moor Lane, and would have provided a short cut into Cottingham from the A1079, something the planners want to avoid, so North Moor Lane has been cut in half to prevent this rat-run being created.

Part of the reason I looked at this area is that a newbie had added a huge stretch of 30 mph on the A1079 in response to a Skobbler bug report. The 30 mph limit had already been added, but rather than accept that the editor just added extra 30 mph areas. Bug reporting is a great idea, but it would be nice if people would actually visit the reported area before 'fixing' it. I have sorted it out now, but the new roundabout will still need adding when I can get a GPS trace around it.

A little bit of mapping

It's been a quiet mapping time, but today after a discussion about some areas of Hull I found myself checking out some of the ten foots off Spring Bank West. A ten foot is the local name for an alley. They run behind rows of houses which are often terraces so these ten foots give access to the back of the houses. The land that fronts onto the ten foot are almost all garages, maybe at one time it would have been a garden. A few of these alleys have been closed to public access with gates. In fact I have no idea of the access status of an ungated ten foot. They tend to be untidy and unkempt and now the wheelie bins just make things worse. 

Quiet time filled with toads

I've not been out gathering new OSM data very much in the past few weeks. I have been running through the OS Locator list of street names in Hull. Following it up on the ground means I have found streets I had missed first time around and ones that have been built since I last visited. All of this improves the quality of the map. I've been doing the same sort of thing in the villages and towns around Hull, but the East Riding of Yorkshire still has a large amount to do to get to the same level of completeness.

I have been experimenting with different ways to place markers on a slippy map. This was prompted by trying to display the locations of some statues of toads placed in and around Hull as part of the celebrations remembering the poet Philip Larkin who died 25 years ago. You can see one example of my efforts here.

I decided to document the process. I'm learning a lot about OpenLayers and that, as you might expect, there are various ways to do it. I am considering how to build a prompted process that helps people create a map on a web site with markers on it and how to help them make the markers respond to actions. It is interesting and in due course, if I get anywhere, I'll post more here. Someone might like to see the documentation at least but some kind of prompt-and-use system could be useful.

Lancing the boil

SteveC recently wrote a blog entry about the nuisance factor that a small, noisy number of people can cause and how enough is enough. I agree with him that the OSM email lists are almost unusable, being swamped by a few people who seem to want to bring OSM down rather than see it prosper.

One guy, probably using a stupidly obvious pseudonym, comments on almost every email anyone sends, sometimes with more than one reply to the same email and often replies to his own email. You begin to wonder if he has some kind of disorder that needs some help.

Someone else repeats the same mantra over and over. Repeating the same objection and asking the same question over and over doesn't make you insightful, or right and doesn't make your objection or question any more reasonable, it just makes you someone who repeats yourself.

Another guy it seems is a serial project disrupter, having been ejected from Wikipedia. His stock-in-trade seems to be circular arguments backed up with hollow threats to set something up as an alternative to OSM. Sadly he doesn't follow through and remains in OSM to continue to cause trouble.

Some people don't like the proposed change to the new ODbL licence and the accompanying contributor terms.  One way of reinforcing their argument against this change is to spread FUD (fear, uncertainty and doubt) about how much of the imported data will have to be removed from OSM because ODbL is not compatible with the imported data. This is, of course, rubbish; why would the sensible, knowledgeable and very hard-working team who have worked on the new licence recommend something that would mean we lose so much? These trouble-makers are free to reject the new licence when they get asked, but they don't want to be Billy Nomates left in an empty space with their data and no prospect of growth, so they want to scare some others into joining them. A pretty despicable tactic if you ask me.

There are other small-time disrupters, people who argue with each other via the email lists, as though a public argument somehow strengthens their point of view. It doesn't of course, it just shows how childish they can be.

The mailing lists are there to ask for help, disseminate information and to discuss ideas and problems. Almost all of this has been swamped by these trolls. Sadly ordinary OSMers get dragged it to this too. Sometimes the mainstream OSMers have tried to engage with the trolls, but that is just what they want. It's like pouring petrol on a fire. The trolls don't back down in the face of reasoned argument or even firm evidence they just go into email overdrive; they would deny the laws of physics just to fill their sad little worlds with their form of entertainment.

Well I say their entertainment has to come to an end. The lists need moderating. When people are deemed to be outside the norms of behaviour they should be privately warned and if they continue they should be suspended from the list for a few days. When they return, if they continue to behave unacceptably they should be asked (or forced) to leave for at least a month. I know this is fraught with difficulties, but OSM is a do-ocracy, so lets just do it and more than that, tell the world we've done it. Lets make it clear that we are about map data, not a home for trolls.

The needs of the many outweigh the needs of the few, these few must act reasonably or expect a reasonable response from the many. If they feel badly treated, let them form their own mailing list or even their own project, and fill that with as much blather as they like.

Crossover

I write two blogs, one about openstreetmap and one about our allotment. Today I decided to post the same entry in both blogs because I surveyed the plots on our small allotment site. You can see the results here: Allotment map.

More detail is still possible, more sheds, the position of the water butts, and perhaps the plot numbers, but the detail of which crop is where I'll leave alone.

Strange name change

We've been working through the names in Hull and East Yorks where OSM names differ from the OS Locator names. Today we looked at Mollison Road.

The existing data just showed Mollison Road but the OS Locator claims that the west-most end is Mollison Road West. We took a look and indeed there is a sign which we had missed saying Mollison Road West. At first sight it would make sense that the road might change it's name at the boundary, which has Hull to the East and East Riding of Yorkshire to the west. When I looked closely this didn't add up, the boundary (from OS BoundaryLine data, so probably fairly accurate) would be near the junction with Ulverston Road. Looking at the house numbers this doesn't work. I did a small bit of address gathering and used the buildings from OS StreetView to position them.

If the street name actually changes then the house numbers would have to reflect this, but the house numbers just carry on. There is a small kink in the road that my new GPS shows up nicely, and it seems that if the name really does change then it does so at the kink. This is bonkers. There are no buildings that front onto the tiny part of the road called Mollison Road West, so what is the point of the change of name?

One thing that I have thought about before is that the boundary of Hull has moved a little. I haven't been able to determine if this is true yet, but it would account for this kind of oddity. There is a centre in both Hull and Beverley (one for the UA of Hull and one for the UA of East Yorkshire) with old maps and documents so maybe one rainy day I'll do some more research.

OS road names

Shaun sent me an email about a Cyclestreets problem report. There is a cycleway that crosses a railway line on a foot bridge with a cycle ramp up the steps. The steps were not in the database, as Shaun suggested. I went out to check it and knock a few OS Locator name issues off.  The new GPS gave a much better track than the old one along the cycleway which is under a lot of trees beside a railway embankment, so not only did I add the steps but improved the location of the path too. I added the steps, a bike ramp and a bollard too.

At one end of the cycleway is Perth Street West. OS Locator seems to think that a part of it is called Perth Street. This is clearly nonsense, there is a street called Perth Street to the east of Chanterlands Avenue. It contines to the west of the avenue where it is called Perth Street West. Further west it passes under a railway bridge and, according to OS, becomes Perth Street once again. There are no name plates for this section and no houses either. There are a couple of businesses on the road, including S&R Motors and on the back of their van was the address: Perth Street West, so that's what I'll take.

The OS names of roads become definative, which I don't like. In the village where I live there is a road known as Beech Hill. At one end of the road there is an old sign for Beech Hill, but at the other end there is a new sign for Beech Hill Road. All of the long-standing locals call it Beech Hill, but OS call it Beech Hill Road.  Persuading the council that the OS name is wrong is likely to be hard, so I am seeking evidence to establish the long-standing name.

More checks and a correction

A couple years ago we were checking out streets in the Anlaby Common area and I wrote a blog about it. When the OS Locator data was made easy to use by ITO's tiles I saw that Ordnance Survey were repeating the same stuff that was on Google's Map two years ago, maybe they share the same surveyors.  Since it was two years ago I thought I'd check again, as well as some other anomalies. I'm glad I did check, someone has corrected a sign board, though OS still have it wrong.

Two roads run parallel to one another, one is called Plantation Drive and one Plantation Drive West. When we last were there the most easterly road carried the name Plantation Drive West, which I commented about at the time, now the name boards have been swapped. OS still think the most easterly road is Plantation Drive East, which is not what the name board says.

Nearby in Anlaby Common there is a road with a loop called Spring Gardens. OS name all the parts separately as Spring Gardens East, West and South. The signs all say Spring Gardens and the house numbers work for a single street, even though they dot about a bit.

We did find a few small roads that were missing from OSM and a couple of names that were wrong, but there were a few other names that OS had wrong.  I'm finding the process very useful to check what we entered years ago and helping find new roads and extentions that we otherwise wouldn't know about.

Comparisons

I have used the tiles provided by ITO World for a few short sorties to improve the names in OSM. I always use what I find on the ground when I can, but many country roads don't have name boards on display. The Ordnance Survey data that has been released often has names for these roads. I add these from the OS StreetView dataset with a tag source:name=OS_OpenData_StreetView.

I found that I could only look up OS StreetView names and ITO World's tile in an editor. I wanted to see them on a web page. I like the transparent page that sautter.com provide, so I thought I'd do the same kind of page too.

The end result (http://oscompare.raggedred.net) has either a Mapnik layer or Osmarender layer at opacity 1, with either the ITO World tiles or OS StreetView or both over the base layer. The slider changes the opacity of the overlays from 0 to 1 (hidden to fully opaque). I have only tested it on Firefox; comments are, as always, welcome.

New kit and old problems

I ordered a new Garmin eTrex Vista HCx GPS and it arrived today. My old Garmin is dropping to bits, I've had it for about seven years so it has worked hard. The rubber is coming off which stops the buttons working, the screen sometimes doesn't work well and now it doesn't always download its data without switching off.

The new model is shorter and fatter, with a wider screen. The colour screen seems easy to read. It also beeps which should be better for navigating. The way of moving around the various screens is a little different and I have yet to explore it completely. I don't yet have a spare microSD card - I'll get one tomorrow, so I can't load a map yet.

I took it out for a test run and the quickly became apparent that it has a better receiver chipset. The satellite lock was very quick and the accuracy was showing +/- 8ft to 10ft for the whole test. When the old unit finally got a lock it varied from 19ft to 45ft. Not a scientific test, but I'm happy. Once I got home I could still get a lock inside the house with +/-20ft. I downloaded the track in a moment instead of the few minutes with the old unit.

As to the test route, we went out to check some more of the name anomalies thrown up by the OS Locator data, this time in Beverley. Most of the differences were small roads and especially tracks that were missing. There were a few typing errors and a few that were errors in the OS Locator data. I think the data extract has been changed to include apostrophe differences. There are many examples of inconsistencies with apostrophes on name boards and I found some today, including a small road that had two name boards facing each other, one with an apostrophe and one without. I just accepted the OS Locator version (without).

I used the new GPS track to move a few roads to the seemingly more accurate position, and in the process lined a couple up with the OS Boundary Line data, which is probably right.

New GPS: happy.
Checking apostrophes: no fun.

Wrong road names

The Ordnance Survey data that has been released is quite a mixed bag. One of the things I have overlooked is OS Locator. This gives a list of roads with a containing rectangle and some location data, such as settlement and county. The coordinates are in OS northings and eastings of course. Itoworld have taken this data and compared it to OSM data in Britain. They created an overlay showing the locations where OSM and OS road names differ, using the OS rectangle to highlight the area. They have also produced a summary of the number of differences in districts, counties and unitary authorities. As usual with Itoworld's work it is useful and accurate but carries the small penalty of taking a couple of days for changes to be reflected.

I took a look at my local area and immediately found errors. This is not a surprise, my work is not being checked on the ground by anyone else and I am as prone to errors as the next man.

When I looked closely it was clear that I couldn't tell if the OS data was wrong or if there were mistakes in the OSM data without checking it on the ground. I selected a dozen or so fairly close to home to check out.

I found a small residential road I had missed, a couple of tracks that were not on our map, one with a clear name plate I had missed. I found a small industrial site with an apparently public road through it that was missing from OSM. There were a few mis-spellings all on my part when I haven't transcribed from the photograph accurately.

Lastly I found two roads where the OS data had a fault. One was easy, OS call a road in Cottingham Middle Dike Lane, the name board shows (and we know from our local knowledge) it is Middledyke Lane. One photo of the name board and a check that there are no other name boards and we were sure. 

The second was a bit harder. There is a road called Endike Lane that runs from Hull to Cottingham, at least that's how it looks, but the name board in Cottingham says Endyke Lane and the name boards in Hull say Endike Lane. When I mapped the area I assumed that the name changed at the boundary of the city, but now I know that is not quite true. We found the break point by looking at the house numbers, they number from each end towards the join, so 601 Endike Lane is next door to 42 Endyke Lane and the break must lie between them. This is not as OS describes it, but OSM was wrong too, the boundary is about 200m away. I suspect that at sometime in the past the city boundary did lie where the name changes but has moved since.

To prevent the difference being highlighted again, even though we now know that the OSM data is correct, Peter Miller of Itoworld suggests adding a not:name=Middle Dike Lane. I was unhappy at first about adding data to OSM to show this, but now I accept that it is just metadata like a few other widely used tags, and, unlike any others, it is a negative tag which could be useful elsewhere.

All things considered, this new overlay will improve the quality of OSM, especially in areas where lone mappers are working and I welcome it.

ODbL Deniers

My weekly copy of New Scientist arrived a couple of days ago. Amongst the usual news and articles there is a series of longer articles about denial. You know the sort of thing: climate change is a myth, there was no Holocaust, HIV doesn't cause AIDS, MMR jabs cause autism, evolution is an atheists' lie, smoking doesn't cause cancer and so the list goes on. 

All of these are well known in at least some circles. One at least, Holocaust denial, is illegal in some parts of the world. Some, for example HIV denial, has killed hundreds of thousands of people by denying them treatment.

The very interesting articles describe the similarities between these quite different groups. It outlines the way these denial groups form, spread mis-information, resist and twist the truth and some of the effects they have, including drawing ordinary, sensible people into believing their stories.

As I read these articles, I couldn't help but see the similarities to the process OSM is going through in moving to a different licence. At first there was a lot of discussion about the technicalities, was it the right licence, what might the process be and so on. Once the licence and process had been thrashed out - at great length I might add, the deniers popped up. They had not taken part in the process, they knew nothing about the law behind the licence, they didn't understand the benefits of ODbL or the weakness of the existing CC-BY-SA licence - they just know the change is wrong. So they look for the 'real' reason for change. It must be driven by commercial interests: it must benefit CloudMade or it must be so Google can take over OSM. The wording of the licence is difficult to understand so it must be hiding something and so on and so on.

I know that the OSM licence is not as important as climate change or HIV/AIDS, but it does matter to many people. If you are tempted to grandly delete the bit of the map you have added or send that email accusing someone of hiding commercial bias, ask your yourself this: Am I being sensibly sceptical or stupidly led down a blind alley? Have I studied this or am I reacting without thinking? Do I have real evidence there is a problem or am I making a fool of myself?

I am a natural sceptic. Sceptics question everything, but they let the answers they find lead them on, usually to more questions. Deniers are lead by their agenda and rubbish the parts that don't fit their agenda, however wrong it is, ignoring any contrary evidence. I am a sceptic about ODbL and I think it is the right way to go. CC-BY-SA doesn't fit OSM, it's time to move on. Let's hope the deniers can let go and move on too.

Poles apart

A couple of short trips out today have helped me to understand the way power lines are represented by the Ordnance Survey VectorMap shape files. The shape file gives no clues about the power lines, they are all labelled the same, but there are two types of power line represented.

The first type is lower voltage that is supported by two wooden poles. It does seem that the nodes do represent the actual poles, at least most of the time.

The first photo shows where two lines join with a bit of a crossover and this  to be represented by the ways shown in JOSM.  The crossover seems to be to relieve the strain in the cables. I have followed a couple of lines where they were accessible and checked the poles and they seem to be where the nodes are. This is very far from exhaustive, so I'm only going to add lines that I have checked.  I also followed a short bit of power line on a single pole and that turned out not to be part of the OS data set.

The second kind of power lines have steel pylons. I have checked a couple of lines and the gaps in the ways do seem to be the position of the pylons. I think there are different sizes of pylons. If the gap in the way is proportional to the size of the pylon it might be possible to deduce the type of power line, but again I haven't been able to confirm this yet.

The biggest problem I can see is that it will be a fiddly job to join each short section together with the node in the middle for the tower. I'll see if I can sort out a program to help out ...

More OS data

I have written up notes about using OS shape files in the wiki here. I hope it's useful for anyone else who wants to use them.

I thought I'd take a look at a few of the other features that might be useful from the VectorMap data set. One thing that I glanced at before was the Settlement_Line. It seems to be electricity transmission lines, at least all the ones I've seen so far only contain this. Loading these power lines could be useful, but in OSM we mark the pylons as well as the lines. 50k OS maps mark the lines but they don't show the locations of the pylons, so you can only be sure there is one at corners and junctions, so I thought these might be the same.

I did the usual re-projection of a shape file for one of the 10x10 km areas close to home. I then modified a python script to extract all of the polylines in a shape file and ran it. When I loaded the osm file in JOSM I realised there are at least two types of power line. One way is in longish sections which are not straight. There is also another type; this is made of single sections between two nodes with a gap of more than ten metres between each section.

I think these are different types of power line. The continuous lines are lower voltage lines, with the gaps marking where the pylons are in a high voltage line. I need to go and look at the lines I have identified locally and see what I find. If I'm right it will take more work to join the sections of line together, since OSM only uses a single node to mark a pylon. It doesn't make the location of power poles on the low voltage clear yet either.

Power lines, especially high voltage power lines, can run large distances from power stations to towns and cities and the 10km squares that the data is supplied in breaks up the lines into annoying sections too.

Sign of the times

My local council (East Riding of Yorkshire) have a rolling programme of improvements that they apply to all of the 280 or so villages in the county. It's called Street Scene and it is a way of fixing the small jobs, like small road repairs, fixing or replacing signs, bins, fences and anything else that the council is responsible for. Like all council schemes it starts out as a good idea then descends into a money-wasting exercise which is basically budget led. In the end jobs get done because there is money that needs to be spent by a certain time or lost and not because the job really needed doing.

All of the street name signs in the village were 'reviewed' and ones that were shabby were replaced. The sign pictured is a new replacement. The sign faces a T junction on one of the main routes into the village. It had an arrow pointing left for West End and an arrow pointing right for Main Street, which is wrong. The break between these two roads is not at this junction and some of West End lies to the right.

I emailed the council explaining that the new sign was misleading, they refused to fix it saying that it was the same as the old sign (not true) and that it was not wrong. I gave up, knowing the real reason was that the money had run out.

Some months later I bumped into Geoff, our local parish council chairman, near the sign. I mentioned that it was wrong; after some thought he agreed and I suggested that all that needed to happen was that the incorrect arrow be painted out. Today I noticed that that is exactly what has happened. I don't know if Geoff did it, an expensive council workman did it or someone else did it who had the same idea as me and had some white enamel paint to hand, but whoever painted out the arrow, thank you.

Using the VectorMap

I decided to use some of the OS VectorMap data. The stuff that makes the most sense is the natural features: water and woodland. There are some fish farms near Driffield alongside the river Hull that have loads of little lakes and streams, so that looked like a simple target that would add some value.

Having re-projected the VectorMap shape file of the area, I selected a group of lakes and ponds in the shape file using QGIS and saved them as a small shape file. Next a little python scrip was thrown together to extract all of the polygons and change them into an osm file to import into JOSM.

There were a few things to change, the river Hull needed to be sorted into riverbanks, some of the small streams needed to be joined up properly, but really it was very easy, and much better than tracing the ponds.

OS VectorMap District

Using the experience gained with the OS BoundaryLine data I thought I'd take a look at the VectorMap District data that the Ordnance Survey have just released. It is organised into chunks based, not surprisingly, on the OS grid. The grid has 100km squares labelled with letters (you can give a grid ref without the letters, but the VectorMap uses the letters). Within each square there is a simple grid system with two parts, known as northings and eastings. The VectorMap uses a single digit of each to define a 10km square. When you have worked out which square you want and opened the folder for that square there is a variety of shape files in it. My village happens to not only fall across two 10km squares but two 100km squares, so I need SE92 and TA02 to cover the whole village.

The shape files cover a range of features: administrative boundaries, community services, heights, natural features by area and line, railway lines and points, roads as lines, settlements by area and line, text and tidal boundaries. I haven't investigated any of these exhaustively and some only cursorily and my findings so far are mixed.

The administrative boundaries seem to be the some of the same ones in the BoundaryLine dataset, but with the disadvantage that they are spread over many 10km tiles, so using the BoundaryLine ones is easier and has more boundary types.

Community services seem to be schools, I haven't found anything else. I might have expected all public buildings, for example libraries, which get highlighted on the StreetView rasters, but not as far as I can see. They are only shown as a point and with no name or other description.

Heights seem to be spot heights. The feature description is 'Heighted Point', which is neither descriptive nor English.

The natural features is interesting. It seems to contain woodland and water features, both seem useful. It also contains 'custom landform' which I still haven't identified yet. On tidal waters it seems to show the low-tide level, i.e. areas that always have water cover.

Railway lines shows the railway lines! It seems to show at least some sidings and multiple track sections. Some are labelled as multi track railway, some as single track railway.

Railway points shows stations with their names.

Road_line has the roads in the 10km square. They are listed with their type (A road, B road, Minor road, Local street - there may others elsewhere). The A and B roads have numbers, some roads have names, but most do not, including residential and minor roads, which would be the most useful. When I looked closely at the Road_line detail it was clear that some newer roads were missing, including a local estate built about ten years old. Also most private roads are missing too. In contrast, a small estate completed only about 18 months ago is in the Roads_line data. I would not trust this to be complete, and the lack of names limits its value.

Settlements by area seems to attempt to show where buildings are. It is showing this by blocking in areas, but these seem very arbituary. There are gaps between the blocks that don't really make much sense. It is more up-to-date than the Road_line data, since the estates that are missing from the Road_lines are present in the Settlement_area, including leaving the space for the missing roads. It does distinguish some building types, such as glasshouses.

Settlement_line seem to show electricity transmission lines, though not any more detail like their voltage, nor the position of the posts or towers.

Text is a list of points with the text that should be written there. The font size and display angle is listed too. The list of named items includes, places, waterway names, farms, bridges, in fact all sorts of things which may be useful to see on a map. Without the context of the object it seems useless to me.

Tidal_boundaries show the high and low water lines in some detail. I need to compare this to the high water mark in the BoundaryLine data, which I think is somewhat out-of-date. I'll post those results later.

Over all there is some interesting data in here, but, like the rest of the OS data, it is a mixed bag. I think OS have released a lot of data in a fragmented way that makes it hard to use without a lot of work to glue it all together. OSM seems like the perfect platform to put the useful data into, but only with quite some effort and, as always, best tempered with local knowledge.

Getting boundaries right

I've got to grips with the boundary data that the Ordnance Survey released earlier this month. The data was in shape files and, with the help of OSM::SK53, I have extracted OSM style data from them. Shape files are a twenty-year-old format that can include a projection file (*.prj). This was where the problem lay that stopped me using the data before. With Jerry's help the projection file was altered and then the process of changing the projection OS use to the one OSM uses was easy.

If you are thinking of using OS shape files be warned: don't trust their *.prj files - they are incomplete.

Once the shape files were in the OSM projection I could then extract polygons or polylines from them. I created a python script that will extract a named parish boundary polygons or numbered polylines that make up the coastline. I spent a few mind-numbing hours loading each of the parishes that fall on the outside edge of the county of the East Riding of Yorkshire. I created a relation for each parish, and deleted each of the sections of ways that were duplicated between parishes. The outer edges of each parish also marked the edge of the county too, so the relations for the county and the region has improved too.

I have also updated the coastline ways from the Boundary Line data set from OS. I managed to leave them broken over night, as OSM::PA94 realised and helped to fix. The coastline is not often updated - it's not part of the normal rendering process for Mapnik, so I'll have to wait to see what Mapnik makes of it.

I have noticed a few things as part of this. The boundaries sometimes follow a stream or river, but sometimes the boundary leaves the river briefly, probably because the river has moved, but the boundary hasn't. The boundaries do not follow the centre line of roads. They clearly lie at one side or the other and at certain points you can see where the boundary jinks across to the other side.

Lastly I have to say the match between the boundaries and the surveyed areas is very close and I'm very comfortable in using the OS data for boundaries, especially because there is no better way to survey this data on the ground.

I will write up the detail of the steps involved if there is any interest.

OS Data part 3

The OS data is correct. I have checked a few points in polygons in the shp file in the OS projection and they are the same as a paper map, so somehow the transformation from OS to OSM is translating the result.

As an example a point on the transformed shp file is 53.76098, -0.48939 should be 53.76129, -0.49106.  As a bodge I'll add the translation to the OSM output and then test a few more places to see if the translation is consistent across the country.

OS Data part 2

I've checked the transformation from the OS projection to the OSM projection and it seems to be correct. The resulting boundary is the right shape and size, but it is shifted to the east. I'm going to check out exactly how much - it would be possible to correct it during the creation of the OSM file.

Why would this data be translated? The transformation could be wrong, the OS data could be wrong, the existing OSM data that I'm comparing it to could be wrong or something else. I have tried the transformation with two separate programs and they both produce the same results. The existing OS data has been gathered by various means and is likely to be mostly right. There is a chance that the OS has an error, I wonder if anyone else has looked at it yet.

If the OS data is OK, then what else have I missed?

Ordnance survey boundaries

The UK Ordnance Survey has generously released some of its data for unlimited use. One of the data sets is about boundaries and included in it are parish boundaries. Boundaries are not painted on the ground so adding them to OSM is hard. Some people have added some parish boundaries, or bits of them, from the out-of-copyright NPE maps from OS, but a lot has changed in over sixty years.

The OS boundary data comes in the form of ESRI: shape files, about 179MB to make up the parish boundaries. I loaded it in QGIS to take a look at it - there where about 14,000 polygons. Dealing with this is not going to be a small task.

I dug about T'Internet to work out how shape files work and after a bit of work I've extracted a list of the polygons in the shape file, then a method to create an OSM file for each polygon suitable for loading into JOSM.

I loaded the polygon for my village and at first sight it looks good, but when I compared it to a real OS map, the parish boundary doesn't quite match - I've not transformed the shape file from its original OS projection to the OSM projection correctly.

This is  not the only problem of course. If I load a single parish boundary that's fairly easy. If I want to load an adjacent boundary some of the nodes need to be shared nodes along parts of the boundary. In addition the parish boundaries will share nodes with the county boundaries too. There are district boundaries available too, so these will also share nodes. On top of this there is the problem of dealing with the existing data that people have already loaded which might need to be merged, deleted or tagged as an historic boundary. This alone makes it important that each boundary is processed by someone who can make decisions about what to merge etc.

I'm going to sort out the projection issue, then import a single boundary via JOSM and see what that looks like. If that goes well I'll import an adjacent boundary and see what it takes to merge the shared nodes.

If I can make this work well, I'll write up a process and consider how to offer parish boundary files to other people for them to import in their local area, but supplying 14,000 files might be too big a task. Then there are the other boundaries types too ...

Ringtone ?

We made a short trip out to Howden, so on the way we took a second look at Eastrington, Or should that be East Ringtone?

These second visits always yield extra information, so they are worthwhile.  In Howden I noticed a large school that we had somehow missed. We took loads of photos (it's the Easter break so no indignant people complaining about taking photos near a school) and used what evidence we could find to estimate the boundary of the school.

Smooth

We went to Long Riston today on a wet and breezy day. On the way, one thing I noticed was a smooth new road. Stavely road is used as a part of a short cut between Hedon Road and Holderness Road. It used to be a spine crushing route full of pot holes and badly filled pot holes, with some ridges in the concrete slabs adding to the problem. The new surface is smooth and flat. It's not the usual lick of tar and stones, but a proper new layer of tarmac.

The Ordnance Survey recently asked for comments on the idea of releasing some of its data for public use. The process ended last week. Today, No. 10 announced that a substantial amount of OS data will be released for use without restriction. Details will follow in a week or so. So what was the consultation period for? Government processes take months not days to evaluate things, so it is not possible for a standard assessment process (if one really exists) to have been applied to the submissions, many of which came towards the end of the consultation period. I can only conclude that what ever data is released must have been decided some time ago. If that is the case, then why the consultation period?

Of course the imminent election may have led to short cuts in the process - I just hope that the route ahead will be a smooth as the new Stavely Road.

If the OS data is useful (that might be a big 'if') and freely available I wonder how it will be used in OSM. It largely depends on how it is released. If it is raster data then the process of tracing from a map already exists and is already well understood. It is also suited to our crowd sourced model - lots of people each add a bit and we get a great database.

If it is vector data (which I think is more likely) then we will need to process the vector data into a form that it can be used in OSM, probably not a difficult job given the expertise we have. Then we just need to decide how to use it ...

Numbers and canal

Another foray into Driffield has given us another chunk of the town with a decent basic cover. One of the areas I wanted to look at was the canal head. OSM's resident canal expert sent me some information on the refurbishment of the Driffield Navigation. Today we only drove past, but another day I want to map it in detail.

The rest of the residential area we checked out was fairly straightforward. The changeovers between some road names were a little unclear until we looked at the house numbers, especially The Mount, Southfield Road and Southfield Close. Checking house numbers is something I have used many times to be sure if a street name has changed and today it worked well.

Tile viewer

Now that I have the beginnings of a renderer, it needs a GUI. Well, that's what I told myself. Since the the path to a renderer has really been a journey towards understanding Java, I need to explore more of Java and its classes. One of the options Java gives is a graphical user interface - Swing. I have had a fiddle with Swing using NetBeans and I'm pleasantly surprised how easy it has been.  I am very familiar with GUI programming in other languages and it seems that Java follows a similar route.  I now have a rudimentary viewer for a tile, though rendering that tile nicely is still a long way off.

Pubs and coast

A wet sunday afternoon saw us in Driffield. The satellite reception was awful most of the time, but we got lots of photos which help to make up for it. The rivers and becks around are very full and the ground is sodden, so a few days of dry weather would be useful. All over the country pubs are closing, but not, it seems, in Driffield. There were loads dotted around the market place and only one that looked closed down. There's still quite a lot of the town to complete - I hope the satellite cover is better next time we go there, it would be nice to be able to predict good coverage for an area ...

I've been working on rendering coastlines - I think I've finally sorted it out. Drawing coastlines made up of ways that are not closed polygons, when there are many possibilities of how they can be arranged, and including islands and lakes has been tough. It has forced me to revisit some long forgotten maths, including calculating cross products of line segments to decide if a closed way turns clockwise or anti-clockwise.

These two pictures show tiles with fairly complex coastlines drawn accurately - the Pacmen are lakes and islands. Making test data with JOSM has been very useful - testing with live data never gives you all the possible circumstances, but it will give me volume data tests later. Text placement next I think.

Coastlines again

I really am an untidy person. Without the pressure from my better half I would live under a huge pile of stuff that should be in drawers or boxes or whatever. Code is, however, a different world and it has to be tidy.

I've just finished tidying up the coastline routines on my renderer. It is much better now, distilled into sensible classes and built around a simple state engine. There are a few tasks still to do but they are documented and have slots to put them in.  There is a lot of testing to do yet too.  I have been creating test .osm files that cover as many eventualities as needed, which for coastlines is more than I expected.

Now the tidy itch has been scratched I really can move on.

Crick in the neck

A cold gray day didn't tempt me out, except in the car.  We went to take a look at the villages to the east of Hull, and check a few bus stops against the NaPTAN import at the same time.

The quality of signing was very poor, only one stop had the NaPTAN code on it and a few stops with shelters had no signs at all - I guess these are working stops.  One of the stops had a sign we nearly missed, perched on top of a pole, level with the first floor windows, probably about five metres up. 

We have checked nearly 50% of the bus stops in East Yorkshire, but those that remain are getting further from home.

Coastlines

I've been looking at how to render my own maps. It started as an exercise in learning Java but has become a tussle with coastlines.  In OSM the coastlines are stored as a collection of ways with natural=coastline tags.  They can have other tags too of course.  These ways should join up.  The big issue is that they don't individually form a polygon which can be filled to make up the sea. 

My renderer-to-be is working on a .osm file.  Depending on where that file covers it might contain coastline ways and they might be complex.  It has proved harder than I thought create a filled-in coastline.  Drawing the ways is very straightforward. I found a flood-fill routine (Java doesn't seem to have one built in) and tried to use it to fill from the way to the edges. This is harder than I expected because the twists and turns of coastlines.

I then decided to work differently and create polygons (which Java is good at filling) that are made up from the way and the edges of the canvas to draw on.  Because the land is always on the left of a coastline way, I can work out which part to fill.  Finally this seems to work.

There is more work to do to improve the handling of multiple, unconnected sections of coastline on a map, such as drawing a section of a peninsular or a section of an estuary. There is still an issue of a closed way: is it an island or a lake (some lakes are drawn with coastline rather than natural=water). Clockwise closed ways are lakes and anti-clockwise closed ways are islands. I want to review the way I manage the classes around this too. 

I'm going to move on to other stuff for now and return to coastlines later.  Next is a style sheet.  I've looked at MapCSS and I've looked at Maperitive's style sheet. I think I prefer the latter.  It seems to use a system of identifying what an object is and then how to draw it.  This allows similar things to be treated similarly: amenity=graveyard and landuse=cemetery could both be identified as a graveyard and drawn in the same way.  This is exactly how I expected to work.  Maperitive also has built in support for zoom levels, which don't have to be integer values. So that looks the next task.

I don't know how far I'll get before the weather improves.  Then there will be more draws to be outside: walking, cycling, mapping, working the allotment etc. We have managed to work on Driffield.  There's also the town of Hornsea and about 40+ more villages to complete in East Yorkshire.  Completing all of the roads and villages in the county is high on my list for this year.

Driffield and renders

Today we went out to look at the town of Driffield. It's a market town which is advertised as the Capital to the Wolds, though I'd expect it to be in the Wolds to be the capital, not perched outside the edge. It went well enough, almost all of what we mapped was new.  There is a substantial chunk of the town which looks mapped at first sight, but I suspect will need going over again.  The roads look traced from NPE, they have tags like highway=unclassified when I think they are residential and they have no names. So, we'll check them out on the ground and get the real picture.

I have also added some streams from NPE.  I saw the stream flowing under the road bridge and the only real way to add it was from the old maps. Yahoo is very low-res and there's a cloud in the way.

Driffield will take many visits to complete, but it should be a pleasant place to wander around.

I have been experimenting with rendering my own maps lately.  I started to look at Java 6 (1.6?) over the winter and now it's turned into a project to render maps.  I have started with a tile-based drawing because it is easy to compare it with a Mapnik-rendered tile.  In due course I do want to render a map that is a single png file of a chosen resolution and of a chosen area. It's mainly an exercise to force me to get results from writing Java and, as yet, it is a command-line program.  I've looked at Swing a bit and it could form the GUI for selecting the area to map.

The latest, almost solved, problem is how to draw the coastline.  They are drawn as open-ended ways so the sea is not a closed polygon to simply draw filled.  I have resorted to creating a flood-fill routine (based on the work of others of course) to flood-fill the sea once the coast is drawn.  It may also be useful with drawing multipolygons too.

The next issue to address is how to manage the style sheet. I've looked at MapCSS.  It's interesting, flexible and extendable. It would make a degree of compatibility with Richard's work too. Parsing MapCSS looks to be a real problem to me. I could just create an XML format for the style sheet, but who in their right mind wants to hand-edit an XML style sheet to get a map to render? So I'm still thinking.  Any ideas would be welcome.

Mid winter mapping

This time of year really feels like mid winter. The days are still short, sunset is about 4:30, and every bit of colour seems to have been stripped from the countryside. Even the grass seems dull and gray. There are no hints of new growth, flowers or blossom yet.

After a long break from mapping we set off to take a look at Beverley, a pleasant market town in East Yorkshire. We were looking for detail to add to OSM really - most of the roads and other significant stuff was added some time ago. After a little wander around we went home so I could add what we had found.

On the way home, in fact just a few hundred metres from home, we came across a flock of fieldfares. They have flown here from Scandinavia to escape their winter.  

After I'd finished updating OSM I took a look at Google maps, which I rarely do. I was quite surprised how it has changed, especially adding lots of POIs and businesses. As usual their quality control is poor, with the impressive building of St Mary's Church shown in completely the wrong place. It has been there since the 14th century, so they've had plenty of time to find it. St Mary's is not the biggest church in the town, that has to be Beverley Minster and that's not even named on Google's map.

During my enforced stay at home I've been teaching myself  Java. I looked at it years ago and quickly turned away from it, but now it seems more mature and useful. The Netbeans IDE seems useful too. I've written some OSM stuff as a set of tasks and I'm going to look at some more too. My main gripe is the lack of properties in objects. The fields can be exposed as public, with no control over their use, or hidden as private and then only accessed with set or get methods. I want to see fields encapsulated as properties with setter and getter routines used transparently to validate updates and manage related properties. Always calling obj.getprop() and obj.setprop(x) seems very crude. I could be missing something of course, so add a comment if I am, but no consultancy or training fees are avaiable I'm afraid. I still have to get to grips with the huge library of available classes, but that's what long, dark evenings are for.

Snow, what snow.

All sorts of stuff seemed to get in the way of getting out and gathering some map data.  Most recently the unusual amount of snow has kept us close to home.  Overnight the thaw set in and the roads were largely passable.  I decided not to set off deep into the rural parts of East Yorkshire so we took a look at the bus stops in Bilton, which is on the edge of Hull. 

The drive through Hull was a surprise - there was almost no snow, even in places that would not have been cleared.  On Friday the local news reported that traffic in the city had been badly delayed - a short journey took a few hours.  Today that was hard to believe.

When we got to Bilton the bus stops were quite straight-forward to find. Only one (pictured) had the NaPTAN code on the sign, but that is not unusual.  There was a stretch of road works on the road towards Preston and three stops were wrapped up in that.  They looked as though they had been removed some time ago.  If they were in use there would have been temporary stops beyond the road works, which we did not see. 

The short days restrict photography at the moment so the forays are short and fairly local.

Experimental photograph

My mapping has been almost at a halt for a few weeks now, what with Christmas, snow and other stuff.  A few other distractions have crept in; I tried to take a photo of a bird in the garden using a mobile phone camera through the optics of binoculars, with some success.  Well actually I didn't photograph the bird, a redwing, but the branches of the tree came out well. 

I have tried this sort of thing before using a compact digital camera to take pictures through a telescope, indeed amongst bird watchers it has been popular for some years and a new portmanteau word sprang up: digiscoping. I thought I'd try through a scope to see how easy it is. 

With the telescope on a tripod it leaves your hands free to hold the phone. The position of the camera in relation to the eye piece is the critical thing. It needs to be just the right distance away to maximise the extent of the image used, too close and the image is a small circle surronded by a huge black border, too far away and the image is impossible to keep still.  I'm talking a millimetre or two either way. It's also important to hold the phone perpendicular to the line of sight through the telescope to avoid unbalanced focus.

It worked but not that well.  I snapped a couple of shots of a tree about 50m away, not the most inspiring subject, but its twigs show a good spread to check the focus, which is far from perfect.  The horizontal lines are telephone cables.

My findings: buy a good telephoto lens for your SLR.