Paul Downey
type: channel
description: Whatfettle, marras?
date: 2019-01-02T13:41:49Z
update period: hourly
update frequency: 1
update base: 2000-01-01T12:00+00:00
generator agent: https://wordpress.org/
items: Scope creep
items: Note to self: write more
items: One CSV, thirty stories: 21. Mistakes were made
items: One CSV, thirty stories: 20. Unknown prices
items: One CSV, thirty stories: 19. Bubblepleth
items: One CSV, thirty stories: 18. Choropleth
items: One CSV, thirty stories: 17. Scattermap Calendar
items: One CSV, thirty stories: 16. Mapination
items: One CSV, thirty stories: 15. Hotspots
items: One CSV, thirty stories: 14. Hackday

Scope creep
type: item
creator: Paul Downey
date: 2015-10-13T08:05:53Z
subject: Uncategorized
description: “The problem with agile is scope creepâ€, he said “We run a tight ship here. We’re on a budget, spending public money and can’t afford for things to slip. We need a detailed plan and a fixed contract to hold our suppliers’ feet to the fire when they deviate from it. We have to deliver […]
encoded: “The problem with agile is scope creepâ€, he said “We run a tight ship here. Weâ€ re on a budget, spending public money and canâ€ t afford for things to slip. We need a detailed plan and a fixed contract to hold our suppliersâ€ feet to the fire when they deviate from it. We have to deliver all of the features, and on-time!â€ Iâ€ m used to people challenging agile as if itâ€ s unconventional. As if itâ€ s a new, untried, untested thing. The present is here, itâ€ s just not evenly distributed. But somehow on this day I wasnâ€ t prepared for this challenge. I was flummoxed. Maybe he did run a tight ship and just hadnâ€ t seen the horrors Iâ€ d seen: the service delivered feature complete even though most of the features werenâ€ t needed, the service so complicated it was unusable, feature complete lest the supplier invoked penalty clauses. The system which cost too much to change, which needlessly instructed people to post their passports to an office, needing operational staff post them back unopened. Systems procured with a fixed 10 year contract, and which were already obsolete before they were completed. Maybe he just hadnâ€ t seen agile in action, how well it works even in non-agile environments. Then on the train back it hit me. L'esprit de chemin de fer: The whole point of agile is scope creep! Any worthwhile thing you work on is bound to be a wicked problem: building changes how you think, changes how your users think, changes the world around it. Anyone can envisage something, plan it out, get someone else to make it, but thatâ€ s missing a trick; software is now so trivial to make you can make lots of things and learn from them. Anyone can now make a thing. But making the right thing, thatâ€ s the trick. Start small, discover what your users really need, learn from what you build, and iterate as you learn. Those features? Chances are your users ainâ€ t gonna need ‘em. Agile: make it up as you go along. Waterfall: make it all up before you start, live wi…

Note to self: write more
type: item
creator: Paul Downey
date: 2015-10-11T18:26:45Z
subject: Uncategorized
description: This is me kicking myself up the backside. You need to write more and here's why. You haven't written anything here for almost a year. You had a go at kick-starting this blog but gave yourself a ridiculously hard task which became hard to juggle when the day-job got super-interesting. You also lost heart because […]
encoded: This is me kicking myself up the backside. You need to write more and here's why. You haven't written anything here for almost a year. You had a go at kick-starting this blog but gave yourself a ridiculously hard task which became hard to juggle when the day-job got super-interesting. You also lost heart because a couple of people had a little wobble about some of the stuff you said. That's a shame because you had far more positive than negative comments and you thrive on good feedback (don't we all). You are incredibly lazy but are not being as smart as you like to think. You like meeting people and are happy to explain an idea over and over but then suddenly get bored and need to hide. Rather than turn yourself into one of the "Book People" you need to write stuff down. You work in government and that can make things hard. You were a little spooked by purdah and other noises-off. You know the hardest, most frustrating job in government is communications, but you are surrounded by people who not only can write, do write all day professionally. Don't waste this brilliant opportunity to learn from them; take all the help you can get. Your friend and mentor JP turned you onto the Cluetrain Manifesto and you swallowed it whole. You know it's telling human stories using a human voice that makes the difference. Make sure you get your voice out there. Write! There are times at work when it helps for everybody to sing from the same song-sheet to avoid cacophony, but you should still find ways of giving others the space to tell the story you want to get out there using their own voice, and relax when it isn't exactly what you'd have said. You don't have to do it all yourself but you should be leading by example. So write! The work you do depends on discourse. Writing helps you engage with other people. Fly kites. Learn from others and get stuff out there sooner. Ask better questions sooner rather than holding off until you have all the answers. Being cloaked can sometimes b…

One CSV, thirty stories: 21. Mistakes were made
type: item
creator: Paul Downey
date: 2014-12-02T06:55:04Z
subject: Uncategorized
description: This is day 21 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data I now know quite a few people who work for the Land Registry, regularly hang out […]
encoded: This is day 21 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data I now know quite a few people who work for the Land Registry, regularly hang out with some of the digital team, follow "@LandRegGov":https://twitter.com/LandRegGov on Twitter, and here I am writing these thirty love-notes to their open data. I also have something of an ego. So it was a little disappointing to have to hear third-hand about a recent Land Registry blog post: bq. "Price Paid Data – Improving data quality":http://blog.landregistry.gov.uk/price-paid-data-improving-data-quality/ That's a fairly jaunty title I thought, and entirely relevant to my interests! bq. While we strive to release data of the highest quality we know sometimes that we could do better. This month we are improving our Price Paid Dataset by removing historic transactions that were added in error. My heart sank. bq. Recently a customer reported some duplicate entries in our 2003 and 2004 Price Paid Dataset. After investigation we found there had been an internal error with a process used to cancel applications. Price paid entries were not removed when they should have been. That process changed early in 2005. Weâ€ ve now corrected the data and will be removing around 48,000 transactions from a dataset that contains over 19 million. There were approximately 18,000 duplicates in 2003, 30,000 in 2004 and less than 100 from 2005. Less than 0.25% of records over twenty years doesn't sound too bad, but if 30,000 of the 1,261,448 transactions in 2004 are duplicate, that implies a 2.4% increase in volume and means roughly 1 in every 40 transactions are ones which in wouldn't be present in other years. That coul…

One CSV, thirty stories: 20. Unknown prices
type: item
creator: Paul Downey
date: 2014-11-27T09:42:06Z
subject: Uncategorized
description: This is day 20 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data "Yesterday":http://blog.whatfettle.com/2014/11/25/one-csv-thirty-stories-19-bubblepleth/ was hard work for no avail, so today for a quick-win I've fallen back on […]
encoded: This is day 20 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data "Yesterday":http://blog.whatfettle.com/2014/11/25/one-csv-thirty-stories-19-bubblepleth/ was hard work for no avail, so today for a quick-win I've fallen back on something reasonably obvious, "using d3 to plot":http://psd.github.io/price-paid-data/html/unknown-prices.html the average price-per-postcode to form a well-known album cover: Er, why trot out that old trope, you might ask. Well I was 16 when Joy Division released "Unknown Pleasures":http://en.wikipedia.org/wiki/Unknown_Pleasures and it spoke to me, not least because of Peter Saville's iconic album cover which had no text, but what appeared to be an intriguing, futuristic, computer rendered landscape. I particularly remember trying to recreate the image using 6502 machine code on my friend's dad's "Rockwell AIM 65":http://en.wikipedia.org/wiki/AIM-65 the only computer I'd seen at that point with any kind of output beyond a seven-segment display. The results on ephemeral, thermal printer paper are sadly long-gone. I also remember being thrilled to see the same image appear later the same year, as the lander descended in "Alien":http://en.wikipedia.org/wiki/Alien_%28film%29: Growing up in the north-east of England, we were very aware of "Ridley Scott's":http://en.wikipedia.org/wiki/Ridley_Scott local heritage (he's a "monkey hanger":http://en.wikipedia.org/wiki/Monkey_hanger), and since then I've felt a similar, strange bond with other people who have found ways of paying homage to this image which includes a number of people I know including "Dan Catt":http://revdancatt.com/ whose "Project CAT820":http://revdancatt.com/2013/…

One CSV, thirty stories: 19. Bubblepleth
type: item
creator: Paul Downey
date: 2014-11-25T13:43:32Z
subject: Uncategorized
description: This is day 19 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data "Yesterday":http://blog.whatfettle.com/2014/11/20/one-csv-thirty-stories-18-choropleth/ I made a simple choropleth map of average prices. Today I wanted to iterate on […]
encoded: This is day 19 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data "Yesterday":http://blog.whatfettle.com/2014/11/20/one-csv-thirty-stories-18-choropleth/ I made a simple choropleth map of average prices. Today I wanted to iterate on this hack. Once again this took me longer than expected, this time because I didn't like the results. First-off it was a little remiss of me not to call out one of the design decisions in yesterday's post. The colours are scaled across the entire range of yearly maps, illustrating how house prices have hotted up over twenty years. There is an alternative to scale the prices within each year to show how the distribution of prices have moved over twenty years: I wondered about changing the squares to match the Land Registry's marvellously retro logo: This wasn't too tricky thanks to the "CSS tricks":http://jtauber.github.io/articles/css-hexagon.html outlined by "James Tauber":http://jtauber.com/ which uses adjacent blocks with enlarged boarders to create a mesh of hexagonal divs which tessellate across a plane: Using the grid values with these shapes was a bit of cheat; I really should have recalculated the averages based on the geometry of each hexagon, and worked harder to make them work in any browser beyond Firefox and Chrome, but this experiment was enough to convince me I really didn't like the look of where the hack was heading. Hexagons are just not my bag, unless I'm playing Settlers of Catan:. So I decided to try a different tack and experimented with turning each square div into a circle using a single line of CSS: bc. .circle { border-radius: 50% } I then resized each div to show both the average price and nu…

One CSV, thirty stories: 18. Choropleth
type: item
creator: Paul Downey
date: 2014-11-20T22:39:43Z
subject: Uncategorized
description: This is day 18 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data Following on from "yesterday":http://blog.whatfettle.com/2014/11/17/one-csv-thirty-stories-17-scattermap-calendar/ I wanted to create a "choropleth map":http://en.wikipedia.org/wiki/Choropleth_map to show how prices are […]
encoded: This is day 18 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data Following on from "yesterday":http://blog.whatfettle.com/2014/11/17/one-csv-thirty-stories-17-scattermap-calendar/ I wanted to create a "choropleth map":http://en.wikipedia.org/wiki/Choropleth_map to show how prices are distributed across the country. A number of people have constructed shapefiles for postcodes which can be used in "d3":https://github.com/roblascelles/uk-postcode-map/wiki/Displaying-the-data but as discussed on "day 13":http://blog.whatfettle.com/2014/11/06/one-csv-thirty-stories-13-postcodes/ the licensing of this data isn't clear. So I wrote a small "Perl script":https://github.com/psd/price-paid-data/blob/master/bin/pricegrid.pl to use the points in the "OS OpenData™ Code-Point® Open":http://www.ordnancesurvey.co.uk/business-and-government/products/opendata-products.html dataset to place each price into one of 1024 squares on a 32x32 grid, then a used a small "PHP template":https://github.com/psd/price-paid-data/blob/master/bin/pricegrid.php to present the average price of each square as a coloured grid on an HTML page. Re-running the script for each year also shows how property prices have heated up over time: ["PDF":https://github.com/psd/price-paid-data/blob/master/posters/pricegrid.pdf] A choropleth map made without a map. I've an idea about iterating on this hack for "tomorrow":http://blog.whatfettle.com/2014/11/25/one-csv-thirty-stories-19-bubblepleth/.

One CSV, thirty stories: 17. Scattermap Calendar
type: item
creator: Paul Downey
date: 2014-11-17T16:27:09Z
subject: Uncategorized
description: This is day 17 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data The "previous post":http://blog.whatfettle.com/2014/11/11/one-csv-thirty-stories-16-mapination/ resulted in a scatter map of the property transactions in the price-paid dataset. […]
encoded: This is day 17 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data The "previous post":http://blog.whatfettle.com/2014/11/11/one-csv-thirty-stories-16-mapination/ resulted in a scatter map of the property transactions in the price-paid dataset. Having made the images, I thought it would be a simple matter to string them into a single page to make a calendar of transactions per-postcode for each day. ["75MB A1 PDF":https://github.com/psd/price-paid-data/blob/master/posters/scattermap-calendar.pdf] It turned out the plot took me a lot longer to develop than expected. As much as I like this poster, in particular the techniques used for its production, I am now several days older and a more than a little wiser. I should start by thanking everyone who completed my "short, anonymous survey":http://tinyurl.com/1csv-30stories-survey or "contacted me":http://blog.whatfettle.com/about/ about the series thus far. I'll incorporate people's suggestions into the hacks and give a wrap-up of the survey in the final post. Your feedback is really helping keep me motivated, even when I've been blocked, and guilt-tripping me into not kittening and dropping this project in favour of shiny new ventures, or indeed playing with our new kittens. One actionable suggestion came from my brilliant colleague, "Anna":http://anna.ps, who nudged me to change the scatter maps from plotting the radius to the area. That was a simple matter of using the square-root of the value when scaling the circle for each postcode, and removed some of the splodgy outliers in the daily plots. A question a number of people have asked is what happens when the Land Registry realise more data in a mon…

One CSV, thirty stories: 16. Mapination
type: item
creator: Paul Downey
date: 2014-11-10T23:04:54Z
subject: Uncategorized
description: This is day 16 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data "Yesterday":http://blog.whatfettle.com/2014/11/07/one-csv-thirty-stories-15-hotspots/ we made a map with the total volume of transactions over 20 years. I wanted […]
encoded: This is day 16 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data "Yesterday":http://blog.whatfettle.com/2014/11/07/one-csv-thirty-stories-15-hotspots/ we made a map with the total volume of transactions over 20 years. I wanted to see how that distribution changed over time. A spot of knife-and-forking: bc. cut -d'⋯' -f2,3 data/pp.tsv | sed -e 's/ //' | awk '$2' | sort | uniq -c | sort -rn | sed -e 's/^ *//' -e 's/ */⋯/' -e 's/ *$//' sort -k2 > daily-postcodes.tsv gives a count of the number of transactions for each postcode on each date: bc. 1⋯1995-01-01⋯B297NS 1⋯1995-01-01⋯B315DF 1⋯1995-01-01⋯B458LY 1⋯1995-01-01⋯BB99RQ 1⋯1995-01-01⋯BS110JH 1⋯1995-01-01⋯BS16XF 1⋯1995-01-01⋯BS81BY 1⋯1995-01-01⋯CA119JD 1⋯1995-01-01⋯CO70BZ 1⋯1995-01-01⋯CR35SU ... To make a version of the map, one for each of the days in the price-paid dataset requires an awfully large number of postcode lookups. Rather than sorting and merging individual files 7181 times I elected to write some code to read the OS OpenData™ Code-Point® into a Perl hash table: bc. my $geocodes = "data/codepo_gb.tsv"; my %postcode = (); open my $file, "<", $geocodes or die "unable to open $geocodes"; while (my $line = <$file>) { my ($postcode, $easting, $northing) = split /\t/, $line; $postcode{$postcode} = { easting => $easting, northing => $northing }; } Which we can use to look-up the easting and northing to draw a circle for each postcode: bc. my $c = $postcode{$p->{postcode}}; my $x = $width * $c->{easting} / $max_easting; my $y = $height - ($height * $c->{northing} / $max_northing); my $size = $p->{count}; printf($fp "circle %d,%d,%d,%d\n", $x, $y, …

One CSV, thirty stories: 15. Hotspots
type: item
creator: Paul Downey
date: 2014-11-07T14:20:59Z
subject: Uncategorized
description: This is day 15 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data Following on from "yesterday":http://blog.whatfettle.com/2014/11/07/one-csv-thirty-stories-14-hackday/'s hackday I had a spot of breakfast with "Michael":https://twitter.com/mikiee_t. I tweaked the […]
encoded: This is day 15 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data Following on from "yesterday":http://blog.whatfettle.com/2014/11/07/one-csv-thirty-stories-14-hackday/'s hackday I had a spot of breakfast with "Michael":https://twitter.com/mikiee_t. I tweaked the map from "Day 13":http://blog.whatfettle.com/2014/11/06/one-csv-thirty-stories-13-postcodes/ to "scale the size of each postcode":https://github.com/psd/price-paid-data/commit/38bedba7c6d07d8397380780a1a62d81c48c2236 using the total volume of transactions over 20 years: Michael quickly turned this image into a second poster [PDF] before dashing to the station. I'm cock-a-hoop how this image has turned out. The detail looks like bacteria on a petri-dish, but zoomed out it's apparent not just where people live, but where people buy and sell houses the most, with the coast of East Anglia, Eastbourne and Cornwall darker than you'd expect. I can guess this may be due to turnover of retirement homes or holiday cottages but it's plain to see there are many more interesting stories begging to be discovered from this simple map weighted with other open data from the likes of the "ONS":http://www.ons.gov.uk/. This afternoon I'm on a train travelling back from Plymouth which means several hours "without access to the interwebs":en.wikipedia.org/wiki/Communications_blackout — an ideal opportunity to "iterate this hack":http://blog.whatfettle.com/2014/11/11/one-csv-thirty-stories-16-mapination/.

One CSV, thirty stories: 14. Hackday
type: item
creator: Paul Downey
date: 2014-11-06T23:47:19Z
subject: Uncategorized
description: This is day 14 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data "Yesterday":http://blog.whatfettle.com/2014/11/04/one-csv-thirty-stories-12-stacked/ I started to look into where all the houses listed inside the CSV are located, […]
encoded: This is day 14 of "One CSV, 30 stories":http://blog.whatfettle.com/2014/10/13/one-csv-thirty-stories/ a series of articles exploring "price paid data":https://www.gov.uk/government/statistical-data-sets/price-paid-data-downloads from the Land Registry found on GOV.UK. The code for this and the other articles is available as open source from "GitHub":https://github.com/psd/price-paid-data "Yesterday":http://blog.whatfettle.com/2014/11/04/one-csv-thirty-stories-12-stacked/ I started to look into where all the houses listed inside the CSV are located, but today I was down in Plymouth to help run an internal "Land Registry Hackday":https://github.com/landregistry/hackday I think it was a great event, with some amazing hacks presented. My two favourite hacks had multidisciplinary teams build something, try it with users, realise it was the wrong thing, so built something better as a result. All in a single day! For my part I tried to help one of the hacks consume the "Land Registry INSPIRE index polygons":https://www.gov.uk/inspire-index-polygons-spatial-data by converting the "GML":http://en.wikipedia.org/wiki/Geography_Markup_Language format shapefiles into "TopoJSON":http://en.wikipedia.org/wiki/GeoJSON#TopoJSON files. Unfortunately this took a lot of compute time, and both "topojson":https://github.com/mbostock/topojson and "mapshaper":https://github.com/mbloch/mapshaper kept running out of memory on my huge EC2 machine. You can see the code and some of the polygons in "github.com/psd/landregistry-inspire-data":https://github.com/psd/landregistry-inspire-data Close to the final show and tell I paired up with "Michael":https://twitter.com/mikiee_t and we turned the heatmap from "Day 8":http://blog.whatfettle.com/2014/10/25/one-csv-thirty-stories-8-heatmap-meh/ into a poster which we framed and presented to the Land Registry, to be hung on their canteen wall: [PDF] Tomorrow I hope to make another, similar poster based on the geographical data.