Posts tagged github
Friday Links – January 8th 2010
Jan 9th
It’s time for me to take another stab at occasional link blogging. While I really appreciate those who blog individual links, I seem to keep coming back to ways of packaging links. Here’s a first installment for 2010:
Last month may have been the time for advent calendars—with Drew’s 24ways yet again containing many excellent articles that have me very excited about HTML5—but the jQuery team have decided to follow a similar model in the run up to the release of version 1.4. jQuery14.com kicks off on January 14th, but already has details of their new API website based on the contents of the jQuery Reference Guide.
The folks (downstairs from our new office) at the Really Interesting Group have been keeping busy. Not only can you now see the price list for Newspaper Club (want to make a newspaper like we did at Greenbelt? Newspaper Club should be your first port of call) but you can also read about the fun they had exposing data in Christmas presents. On the newspaper front, I found the idea of the Nashville Retrospect fascinating – it’s a free monthly newspaper devoted to Nashville nostalgia and history.
There’s all sorts of excitement around key-value store Redis. Simon Willison pointed out that the addition of BLPOP and BRPOP support over Christmas means you can use it to drive a queue server without polling. Ezra Zygmuntowicz has been working on an Actor library for ruby based on it. And there’s lots more. It’s good to see that the NoSQL/LessSQL movement is developing beyond simply replacing relational databases and opening up new possibilities and techniques.
For those of us who work with drupal, Project Verity could be quite a boon. Mark Boulton and Leisa Reichelt quite rightly point out that If you’re a company that takes pride in good design and user experience, handing over a Drupal backend can be a bit embarrassing and difficult to reconcile with your company philosophy.
Project Verity is their response and while I’ve not actually seen it, I suspect those two are likely to be onto something. It certainly would be nice to be able to deliver a drupal site with a friendly face for admins. I’ve not really been tracking drupal 7 development, and hopefully it’ll reduce the need for such projects. If not, I imagine a port won’t take long to surface.
Given that I’ve been playing around with a Palm Pre and WebOS lately I’m delighted to see Palm using github. There’s a fair range of sample code that comes with the SDK, but this looks likely to be more comprehensive and it’ll be fascinating to see what develops as people begin to fork the code, throw in comments, etc.
Talking to Wordpress with ActiveRecord
Jul 24th
As mentioned in yesterday’s announcement I’m pulling some content across from this blog (running on wordpress) into the new Ket Lai site (a merb app). I’ve found myself doing similar things a few times lately, such as on Only Connect (on which more, soon) and so have built out a selection of ActiveRecord models to help me talk to a wordpress database from a ruby app.
At Matt’s urging (he’s been using them to move data from a legacy site), I’ve finally put those models up on github. Being a single file they arguably should have been a gist, but I’m reserving the right to reorganise them in future.
They’re far from complete in that there are lots of validations I could have added in, named scopes that would probably be handy, and loads of convenience methods that some might like, but hopefully they’ll be of use to someone and evolve over time.
(on the subject of wordpress, I’m glad to see the addition of changelogs to the plugin directory – being asked to upgrade plugins without any idea what’s changed has long been a bugbear of mine, so hopefully this will resolve some of that)
Rails 2.3 final and theme_support updates
Mar 28th
Somewhere between Ruby on Rails versions 2.3.0 (RC1) and 2.3.2 (final) a change was made to the arguments required for one of the methods the theme_support plugin requires. I must confess I hadn’t spotted it, but github user knapo kindly sent me a message with a patch. That patch is now applied in the main repository, as are a few tweaks to the ActionMailer integration to better handle multipart emails. You can get the new version from github.
Perhaps more significant than those small patches is that I’ve now pushed up a new rails app for use in testing theme_support. The tests are a series of cucumber stories, and the app is very barebones designed entirely to test the various features. I could do with tightening up the way the test app and the main plugin repository are connected, but it’s a start and has been very helpful with the aforementioned patches. Any improvements to the test suite would be much appreciated. You can, of course, find it on github.
Quick update on Heathrow Tower
Feb 13th
While there haven’t been any visible changes to my Heathrow Tower project in the past couple of weeks beyond my throwing in a few greetings in other languages to break things up a bit. Having put some of the statistical plans on hold as the snow last week prevented any data gathered from being anywhere close to representative, I’ve gradually been building up the database behind the scenes so I can start to do some of the more intricate things I’d like to do.
The key data I wanted was the airport codes for the various flights, and geographic data for those airports.
Firstly I found FlightAware.com who will provide all sorts of data from a flight code, but unfortunately the first time I tried making a request to their site using HTTP Client I spotted a comment in the HTML referring visitors to http://flightaware.com/about/termsofuse.rvt which states:
You will only access the FlightAware web site with an interactive web browser and not with any program, collection agent, or “robot” for the purpose of automated retrieval of content.
So I started looking at airline sites. United have a relatively straightforward URL scheme that responds to a GET and returns data that can be scraped. eg:
http://www.ua2go.com/flifo/FlightSummary.do?date=20090201&fltNbr=959
BA and Virgin, on the other hand, both require cookies to be enabled in order to get results from their flight trackers and don’t advertise any other URLs for flight data. Once I’d realised that only one of those three carriers was going to be helpful, I decided not to keep checking airlines.
So, a little frustrated, I tried just typing flight codes into google. And lo and behold… most of them give useful results. It doesn’t get them all, of course, but it’s enough that out of the 838 flight codes in the database, 695 are fully identified. Of those not identified, a number seem to be flights that were diverted to Heathrow but don’t normally go there.
So with some sense of the airports served, I also want to know about the airports themselves. Wikipedia’s pretty good there, with geo data for them all in an easy to capture form. Some, like Heathrow itself, are very easy to find:
http://en.wikipedia.org/wiki/LHR
while others are a bit trickier. But with some manual intervention I was able to get all of that data. The manually produced mappings and the code for pulling in the data can both be found on github.
More updates as time allows…
Tracking Heathrow with twitter
Jan 29th
A few months back—while we were discussing the number of talking objects appearing on twitter—Jenny pointed out to me that all Heathrow airport arrivals and departures data is online. That set my mind racing, as if you know all the flights leaving that currently controversial airport, there are all manner of things you could begin to do. Working out miles travelled and carbon emitted, spotting delays, and so on. But at the time it all came down to a quick note in Things to some day set aside time to explore.
That day arrived this week. The data turned out to be pretty simple to scrape, with a quick wrapper around hpricot, and to throw into an SQLite database using datamapper to give me a little abstraction and a place to throw a variety of methods to make my code simpler. And then it was a small matter of employing John Nunemaker’s twitter gem to set up regular tweets letting followers in on how many flights in and out of Heathrow there have been lately.
The result is a rather pleasing hourly summary, that adds a little rhythm and background awareness into my day. You can follow it at http://twitter.com/heathrowtower.
Perhaps the biggest frustration with the data is that all destinations/origins are given as city names. Given that city names are hardly unique, and even if they were a given city may have several airports connecting with Heathrow, that makes it a bit trickier to do some of the more sophisticated calculations. My hope is that the flight codes (which are given) can soon be transformed into a list of airport codes, which can then open up a route to more useful and interesting data. (if anyone knows of an existing database that does that mapping, please let me know!)
I’m looking forward to that, but I’m also anticipating the ambient awareness that having the bot running will create. Will the hourly ritual of seeing a sentence or two about Heathrow activity reveal any patterns? If they do, maybe I’ll update the code to make more of those. We’ll see.
For now, please do follow the tower on twitter, tell people about it, send it messages if you spot anything interesting, and feel free to take a look at the code over on github.