Jun 10

JangoMail, lackadaisical security, and a workaround

A client recently asked me to integrate their site with the JangoMail mass mailing system. I wanted to keep them happy so agreed to investigate, but was horrified by what I saw in the JangoMail API documentation.

JangoMail appears to be optimised for those with existing databases of email addresses they want to maintain and contact. For those wanting to keep those databases in sync they offer a script you can download and install on your server that they can call with details of various actions (user unsubscribed, user clicked link, job completed, etc) as well as to extract the list of email addresses they should send a given campaign to. So far, so good.

The problem is in the implementation. Once you have downloaded and installed the script on your server, they ask you for your database credentials and then send these, along with an SQL query, over HTTP (or HTTPS) for the script to execute. In the FAQ attached to their blog entry on the topic the obvious question “Is this method of connecting to my data secure?” is asked, with the response:

Yes. It is inherently secure if you opt to have JangoMail connect over https instead of http. It can be additionally secured by restricting the range of IP addresses allowed to connect to the custom script file. JangoMail’s range of IP addresses are: –″

That answer is far from satisfactory. I refuse to give a third party my database credentials, still less to to execute arbitrary SQL received over an HTTP request, even if that is SSL and includes a password.

Surely if they want to keep the steps for the user to a minimum they could still provide an interface that takes credentials (via SSL)–along with appropriate other details like “enforce SSL?”–and generates a PHP script that contains those credentials embedded within it, along with code to generate the SQL from a set of parameters? It’s not a hard thing to do (witness the way wordpress/drupal/etc will generate a config file for you — it’s the same thing). That way JangoMail can take responsibility for making sure that the credentials are only ever sent a small number of times (and via SSL), and the endpoint can contain appropriate validation.

In this case I decided to take matters into my own hands and write a sane script to receive their input. There’s nothing forcing you to put genuine database credentials into their form, so instead I used those fields to provide a username and password I’d use to authenticate their request. In each of the boxes to enter the SQL for your queries I entered some code to generate JSON containing the relevant details.

With that done it’s a relatively trivial matter to parse the JSON, do any validation you may want to do and update your database accordingly. A rough-and-ready (and barely tested) script that does just that can be found in this gist. A perfectly satisfactory (if slightly laborious) solution for any competent web developer.

But of course JangoMail’s target market isn’t competent web developers. They’re clearly trying to target a general audience, and for that audience their lackadaisical approach to security is indefensible.

Mar 09

Selected (belated, extended) Saturday Links

The past two weeks haven’t really left time to compile my selected links, though there have been many. A few days at SxSWi (on which more, later) followed by travelling with the family and the inevitable work backlog moved blogging way down the priority list. So here’s a mammoth selection to get me caught up. Particularly interesting has been the discussion around the future of newspapers (represented here by Clay Shirky, Steven Johnson and Russell Davies), which seem to have finally pushed beyond “how t ind a good business model for papers” to looking at where the real value for society lies and how we can preserve and extend that in a changing landscape.

Mar 09

Selected Saturday Links

Big themes this week have mostly revolved around twitter, facebook, and openness. Some have focussed on facebook redesigning to embrace a more twitter-like “web of flow” approach, and others on the fact that they’re jumping on various open web bandwagons. It’s been interesting to see some tie in with the government transparency thinking going around, as particularly noted by Chris Messina on FactoryCity. Meanwhile there are quite a few nice new tools emerging, and I really must try heroku one of these days.

Dec 08

XML_Feed_Parser: Handing over the reins

For the past few years I’ve been maintaining a PHP package called XML_Feed_Parser. It’s part of PEAR and attempts to offer a unified API for handling RSS and Atom feeds in your PHP code, a little inspired by projects like the universal feed parser. Its parsing and API are pretty comprehensive, but lately I’ve been falling a bit behind in managing it and there are aspects that could definitely do with some attention.

So I’m looking to hand it all over to someone with more time and energy for it than I. Preferably someone who uses it in an active project (being primarily a ruby developer these days, I spend a lot more time with feedtools than with my own package). I’m going to mark the package as ‘unmaintained’ and if you want to take it on, take a look at the appropriate page in the manual.

And if you want the full story of why I’ve chosen now to make this move, it’s made fairly clear on flickr and my other blog.

Nov 08

Testing PHP apps with Ruby tools

As I’ve mentioned here before, when working on web applications built with PHP, whether custom-rolled or drupal-driven, I often find myself missing various tools from the ruby kit. I’ve talked before about using capistrano with non-ruby code, but lately it’s been rspec and its stories that I’ve been craving.

I’m aware of PHPSpec and have played with it from time to time, but the lack of a compelling way to work with mocks/stubs has slowed my adoption, and last time I checked it didn’t offer anything for high level user stories. So this week I set out to harness cucumber and webrat to write some simple stories.

It turns out to be pretty easy. There’s no nice simple support for test environments, fixtures, mocks or stubs, but if you just want to make sure that a few pages load correctly, and have the right elements, or that logging in works as you expected, then it’ll do the job.

I’ve not done any packaging up of the code, mainly because there’s so little to it. My folder structure is:


(click on the links to see sample files)

I simply set up those files, go into the folder and type ‘rake features’ to put your site through its paces.

Sep 08

Book Review: Learning Drupal 6 Module Development

Book Sleeve: Learning Drupal 6 Module DevelopmentLast autumn’s release of Pro Drupal Development was a significant moment in the history of the popular CMS, providing for the first time a relatively comprehensive guide for those wanting to do more than simply manage and skin a drupal site. A number of books have followed it but few have delved as deeply or been such a definitive guide.

Like most of the more recent books, Learning Drupal 6 Module Development focusses on a quite specific area of drupal development, but its a key one for any serious developer and touches every other area of the system. Experienced PHP developers may find that this book (in conjunction with some time for experimentation) will serve as a solid introduction to how they might build applications on top of drupal.

The book focusses on a single project—a website providing biographies of philosophers—and builds the modules it needs, introducing the various available tools and techniques along the way. Much of the time is devoted to generating custom content types, but there’s very solid coverage of the hooks, filters, and actions that let modules really take integrate with the rest of the framework. Theming your output, using AJAX, and working with web services all get some time and illustrate how your app can be part of the wider web and keep up with its prevailing trends. A number of times I found myself reaching for code I’d written over the past few months to make amendments based on examples in the book.

I’ve been quite critical of books from packt lately and some of my criticisms apply here—he book itself feels flimsy and the print quality is poor—but this is also evidence that if an author and editor put the work in they do occasionally produce quality material despite the publisher. There are a number of asides that clarify language which demonstrate an attention to detail too often missing. It would have been good to see clearer signposts as to which features are new in Drupal 6 and which were available previously, but the online API docs can provide most of that.

When I reviewed Pro Drupal Development I noted some disappointment that the book didn’t devote any time to automated testing of drupal code, and that criticism applies here too. Along with staged deployments, automated testing remains one of the least considered aspects of drupal and that’s a serious concern for those looking to build robust well-managed applications on top of it. There are a few good articles online about how to test drupal code, but it would be good to see it taken more seriously as a core part of the module development process.

This book is likely to sit alongside Pro Drupal Development on my desk whenever I’m working on a drupal project and is a worthwhile investment for anyone who spends much time building drupal modules. There are clear areas for improved coverage, but it is as comprehensive an account as you’ll find of how to build modules that take full advantage of the facilities Drupal provides.

Disclaimer: I was sent a copy of this book for review by the publisher. You can find it at amazon US, amazon UK and all sorts of other places.

Apr 08

Project Launch: New Greenbelt website

One of the numerous projects I’ve been juggling over the past few months has been a redesign of the Greenbelt Festival website. That redesign went live late last night.

Working from Wilf‘s designs I initially built new HTML and CSS templates and began to establish some rules for how we’d handle the new image management requirements for a site that is now very photo-heavy. When it came time to apply the new designs to the CMS, however, it became apparent that there was a much bigger job ahead.

That CMS (a bundle of custom PHP that I had inherited) has grown over time and within some quite onerous server configuration constraints to a point where it was due a significant overhaul. Sticking with PHP was a fixed requirement as we’re relying on various APIs and a server architecture that wouldn’t be happy with me shifting to, say, rails, but already having the problem domain mapped out and the opportunity to radically simplify a few things meant I got to enjoy the feeling of stripping out a lot of code without impairing functionality.

One note that Derek Sivers made in his controversial blog entry last year about switching from Rails to PHP was that working with Rails had made him a better PHP developer. I’ve found a similar effect. I have no intention of leaving the world of Rails (I still prefer it by orders of magnitude), but tackling projects like this in PHP are a reminder that working for a while with really good tools is likely to encourage you to seek out best practice in whatever environment you end up in.

Ruby developers who occasionally work on PHP projects as I do may be interested to note that we are using capistrano for deployment, and I intend to use rspec for some testing. I’ll try to write that up once it’s in place.

With a refreshed CMS, new templates, and some standardisation of our many javascript files on top of a jquery foundation, Paul, Greenbelt’s was able to manage the photos and copy to turn that new look into a vibrant and content-rich new site. You can see a few notes he wrote about the redesign on the site.

There’s still a fair way to go. I’ve got a lot of tests to write in order to make it easier for us to make further changes, and various aspects of the site are more than ready for a more fundamental rethink that will let the festival open up its archives better, but all concerned are very pleased to present the fruit of our labours.

Dec 07

Book Review: Object-Oriented Programming with PHP5

book sleeve for Object-Oriented Programming with PHP5With PHP 5.3 looking very close to release, bringing with it significant changes such as namespaces, this is an awkward time to release a PHP book of any generality. Within a few months it will be clearer how well the new features are taking hold and there is almost certainly going to be a clamour for books that show how new features affect development practices and how to make use of them. Even without that, it is hard to recommend this book as it fails to live up to its promise and provides a poor introduction to Object-Oriented programming.

The book’s first problem is the poor quality of the English contained. The author does not appear to be a native English speaker and the publisher has apparently done little to clean up the language. For the most part that simply makes for some curious rhythm and phrasing but there are a number of places where it contributes to inaccuracies, such as constant confusion between objects and classes. Finding such inaccuracies alongside mistakes such as confusing polymorphism and inheritance I’m left wondering how well the authors really know their subject. Packt would do well to employ another reviewer to clean up the syntax and definitions before any reprints.

The structure of the book is curious. Rather than approaching topics through the use of an example project or projects, the author gives disconnected samples, often offering little beyond that which can be found in the online PHP documentation and generally failing to explain how techniques would be useful. While he insists that PHP5’s Reflection API is an important tool, there is little to back up that assertion. There’s a large code sample and a list of the methods and attributes of a reflection object, but no real substance. The database section covers some key examples, such as PDO, ADODB and MDB2, and touches on the ActiveRecord pattern, but uses four pages to list the RDBMSs that ADODB supports and then only spends a page and a half on ActiveRecord, failing to explain either its concepts or advantages in any detail.

The high point of the book is the half chapter on unit testing. A few well chosen examples provide a demonstration of how code can be tested and how that allows for refactoring without fear of regressions. The reasons for unit testing are clear as is basic usage of PHPUnit. I’d have liked to see further examples rather than several pages listing all the default assertions PHPUnit provides, and I disagree with the claim that there should never be more than one assertion per test (individual tests should focus on a single concern, but can use multiple assertions to do so) but by that point in the book it was good just to see a section that made its point clearly.

Fundamentally, I came away from this book feeling it had failed to communicate clearly the core principles or reasons for object oriented programming. By trying to cram in too much reference material, not offering clear contexts for each technique, and not lingering on ideas like encapsulation the book fails to instill OO techniques.

Disclaimer: I was sent a copy of this book for review by the publisher. You can find it at packt, amazon US, amazon UK and all sorts of other places.

Nov 07

Book Review: Pro Drupal Development

It’s surprising given drupal’s popularity that there aren’t more books covering it in detail. Site launches and contributions by the likes of lullabot and bryght have pushed the CMS’ profile and recent releases have emphasised the Web 2.0 potential, but a quick look at amazon reveals only four related titles. Of the four, Pro Drupal Development is definitely the most developer focussed.

This isn’t a book for a drupal newbie. Going in you’ll want to have spent at least a little time setting up a drupal site or two, and while there’s no need to be a PHP guru the authors do presume you’re not going to need help understanding their code samples. They focus on drupal’s internals, with a lot of time spent writing modules, understanding the user, node, menu, theme and related systems, and a little attention for performance optimisation.

There’s a lot of ground to cover and most chapters are short, giving just the essentials on each area. You’ll probably want to pause from time to time to try out the code samples unless you’re already experienced at writing drupal modules. Having written a number of modules and run into various problems I found I was able to focus on the new information and how it would have affected my approach, but if this is new ground the structure of the book may make it rather overwhelming.

The writers are keen to encourage their readers to read the book in order, and some chapters certainly do build on their predecessors, but the real strength of this is likely to be as a reference guide. A quick once-through will help newcomers to module development get a sense of how everything fits together, but chances are you’ll then want to refer back when you actually encounter problems that a given chapter can help with.

I was disappointed not to see more coverage of testing drupal code. As I mentioned yesterday, the lack of attention paid to automated testing in the drupal community frustrates me and it seems that for a book like this to not to provide some coverage of sensible testing regimens is a missed opportunity. Similarly, it would be really good to have some coverage of deployment tips, particularly relating to upgrading active sites. Both of those seem to me like core topics for any book purporting to provide a guide for professionals, but the priorities of the book mirror those of the drupal community where neither topic appears to be a significant concern.

For any experienced developer who needs to get to grips with the insides of drupal and/or write custom extensions, this book will be invaluable. The style won’t suit those looking for a broader scope or lengthier tutorials, but it will help you get to grips with each of the major components quickly and provide enough information to set you on your way. Hopefully it will also trigger further writing about drupal, which may cover more ground and help developers bring some other vital practices to their drupal work.

Disclaimer: I was sent a copy of this book for review by the publisher. You can find it at apress, amazon US, amazon UK and all sorts of other places.

Nov 07

Assessing Drupal as a Rails developer

As I’ve indicated here a few times, when announcing site launches and offering a few hints and tips, I fairly frequently find myself working with Drupal but have long had reservations about doing so. What I’ve so far avoided doing is going into much detail about why that would be, what those reservations are, and so on. But now I’m working on a review of a Drupal book and so it seems appropriate to lay those cards on the table and look at the details on them. It seems easiest to do that by comparing with the framework I do most of my development in: Ruby on Rails.

As ever with technology comparisons, just like any other sort of comparison, a lot of my preference for rails boils down to a combination of taste and experience. I’m also well aware that I’m comparing unlike things. Rails and Drupal set out to solve different problems, and so obviously take quite different approaches. That doesn’t mean, however, that developers aren’t often left choosing between them as the boundaries are blurred and there is significant overlap between their respective domains.

I’ve seen plenty of bad ruby code, but proportionately I’ve seen a lot more bad PHP, and as a PHP-based solution Drupal inherits some of that baggage. Drupal has pretty good support for a wide range of cutting-edge ideas and tools, but by and large the Rails community has seemed much more clued-in to what’s happening at the forefront of web development. That may of course change as the Rails community continues to transition from being chiefly early adopters to a much more mainstream crowd, but it marked my early impressions of the framework.

A case in point is the integration of automated testing in Rails. There is some community support for automated testing in Drupal (primarily through the simpletest module) and there are plenty of Rails projects that lack tests, but there is a strong focus on Test-Driven and Behaviour-Driven development in the Rails community, and the combination of Rails’ object orientation, simple rake tasks and fixtures/mocks makes it very simple to write a solid test kit.

Testing is complicated by the hooks system in Drupal. Hooks are a powerful feature, allowing any module to broadcast and listen for a variety of events and respond appropriately, but it can be difficult to know what order modules will receive those events and so you end up performing a variety of contortions to make sure your responses are given the appropriate weight. Arguably Rails’ layers of filters and plugins can introduce similar complications and I’ve certainly spent quite a bit of time tracking down performance hits coming from rogue filters in plugins and engines, but so long as inheritance is properly handled it’s not all that difficult to do a little introspection to make sure everything’s in its right place.

The fact that Drupal relies heavily on web-based configuration makes it easy to put a lot of power in the hands of admin users, reducing the reliance on a developer, but it also makes upgrades and deployments a more complex business. When I add a new module to an existing Drupal project I usually find myself noting down the steps I take to configure it in my development and/or staging environments so that I can repeat that process in production. In a Rails project I’d write a migration and set up capistrano to run that migration when I push my new version live, significantly reducing duplication. The Drupal approach may be simpler for non-developers, but as a developer I frequently get frustrated at the repetition of tasks that could be so much easier.

That issue is one of the key things pushing me away from the nice-in-concept CCK. Adding new content types to a site through a web interface is an appealing idea, but it’s a pain to have to work out how that changed things in my database to produce a script, or to have to repeat a manual process. Since I know how, I’m much more likely to produce a module for my new content type, simply because I know that deploying that is a much quicker process.

One reason it’s been fascinating reading the Drupal book I’m currently reviewing is that books on Drupal are so thin on the ground. That’s a real shame as while there’s a lot of content on Drupal online, it’s not really been well organised for experienced developers wanting to build serious solutions who aren’t already intimately involved with Drupal. By contrast, while the Rails books are mixed in quality the best ones not only get you up and running fairly quickly, but provide a good entrance to exploring the internals and writing effective extensions. That’s not necessarily a reflection on the core projects themselves, but both online documentation (where, admittedly, Rails is still lacking) and books make a significant difference to the developer eco-system and quality of the solutions that are likely to be built.

I don’t mean to sound too negative, and I’d love to receive comments showing me how to address the shortcomings I perceive. I’ve built sites I’m proud of on top of Drupal, and where their focus is a good fit for Drupal’s core features or mainstream modules it’s been a very good solution, providing a suite of content management tools at very little cost. But as soon as there’s serious custom development to do, those advantages tend to dissipate pretty quickly when set alongside the advantages of Rails for agile, regularly upgraded and well tested solutions.