James Smith - Main

My EcoMo09 Google Maps Hack

2009-12-17T14:17:00Z

In early September, Betavine organised a 24-hour hack event called EcoMo09, with an environmental theme. Unfortunately I couldn't stay the whole time, but I went along as an AMEE representative to support other people using our API during the event. In the quiet periods, I managed to come up with my own hack, which used GreaseMonkey and AMEE to add carbon emissions to Google Maps.

AMEE-Google Mashup Greasemonkey script by James Smith from Carbon Tippy Toes on Vimeo.

You can even get the code and try it yourself. It's very basic, and actually most of the work was dealing with Google Maps and making Greasemonkey behave properly - the AMEE bit, getting the carbon value, was dead easy.

The Activist Geek Philosophy

2009-11-10T09:37:00Z

On my old blog, I used to write about more political subjects; war, ID cards, etc. This one has, to date, been mostly all about technical subjects - most of my political comment goes on Twitter instead. However, after a conversation recently with my colleague James H, there are some questions I'd like to present and invite comment on. They aren't about particular issues or party politics, but instead about something I feel is above that. The question is:

Is there a unifying philosophy within the community I work in (a certain subset of the "geek" community), and if so, what are its core tenets?

I'm sure this has been written about many times before, but I want to clarify it in my own head (rather than simply reading what others have written), and invite help from whoever may read this.

So, first, let's have a stab at identifying the community I'm talking about - this in itself is a pretty hard problem. I guess I'm talking about "activist geeks", in a way. The sort of people who are involved in the work of MySociety, or Social Innovation Camp. The people who go along to events like geeKyoto, or perhaps Interesting. They are the people that go out and create innovative things that make the world better. I can't put any names on a list here, though there are some obvious ones of course, but you will probably know if this is your sort of community or not.

So having made a (very) vague stab at who I'm talking about, what are the core philosophies that come out of that? One caveat: obviously this is my own personal viewpoint, and implies absolutely nothing on behalf of those examples linked above. I could be completely wrong on all of this (though I'd invite comment). I think I am what would traditionally be called "left wing", so my please note that my views may be coloured by that.

Firstly, I think that while there are certain things that do unite the community, they often don't fit along traditional political lines. You cannot say that this community is left-wing for instance; there are certainly members of all political parties involved. We'll come back to some of the things that don't unite us later on, but let's find some common ground first.

Privacy

Even though many of us share vast amounts of information on various social networks, we have a strong belief in privacy of personal data. Perhaps this is better phrased as belief in control of personal data. I can delete my social network accounts at any time, and can decide who can see them, so I remain in control (as long as I trust the site). On the other side, I will not be able to remove my record from the ID card database or decide who accesses it, so I am not in control of that. This is part of a more fundamental belief in universal human rights.

Transparency

While personal information should be kept tightly controlled by its owner, information about (and generated by) government and public services should be fully transparent and open. The MP expenses scandal was an example of further erosion of trust in government because of an opaque system that was open to abuse. As they say (I think), sunlight is the best disinfectant. Government operations should be open to public scrutiny, and assets produced by that system (OS maps anyone?) should be freely available to the taxpayers who have paid for them.

Tolerance

I don't think we give a monkeys where someone is from, what their background is, or about their religion, sexuality, or favourite colour. It's just not relevant. The rest of you, get over it.

Anti-authoritarian

It always confuses me that a criticism of the left is that it wants to control people. Maybe that's true, in which case I need to recalibrate my open personal compass a little, but my belief is that the community is very much about freedom of the individual. There is a strong anti-authoritarian streak - none of us want to end up in a police state.

Social Conscience

However, the above freedom of the individual does not lead to blind self-interest. I believe that the community is not one of self-interested people with no care for others, but of a group of empowered individuals who actively engage in and make society better.

Engagement

My final point is that the above lead to a community that is engaged with the world around them. We engage on issues of importance, such as my own personal focus, climate change, and they are interested in the way their society is governed. I believe that those qualities are fundamental for a functioning democracy, and we see that currently most of society does not measure up to that standard (a situation which we try to improve of course).

On the other hand, I seriously doubt we are all united on things like economics (I love that picture), the role of corporations, how to deal with climate change, foreign policy, and so on.

I'm sure there is more that can be said, but I will let it play out in the comments (if anyone wants to add to this). My final question is - if there is a central theme emerging, does that mean we should do something with it in these days when so many of the above viewpoints are rather missing from general society?

Discuss.

IBM DeveloperWorks articles

2009-10-28T17:58:00Z

Over the summer, I was asked to write an article for the IBM DeveloperWorks site about energy monitoring and so on. Eventually, that turned into a series of articles centered around AMEE and how you can track energy consumption and calculate carbon emissions with it.

So, now that they're all published, here's a bit of a linkdump to the various articles and other things that came out of them.

Introduction to AMEE

This is a technical introduction to the AMEE platform, how it works, and what you can do with it. It covers things like signing up, creating profiles, storing data, and getting results. It was also adapted into a screencast for the AMEE developer site.

Calculate your computer's carbon footprint using AMEE

This one is an introduction to the wide variety of ways in which you can track energy data using AMEE, using the monitoring of computer energy use as a case study. It also shows some code examples for the AMEE ruby gem.

Monitor home energy with AMEE

This is my favourite of the three - a long tutorial, with lots of code, which shows a complete implementation of an online energy-monitoring system, using a CurrentCost display, AMEE, and a little Rails code. This uses a lot of the open source software I've been writing over the last year, which regular readers will have seen in previous posts, but it's really nice to tie it all together and tell the world how it works!

James Smith on embedding environmental intelligence

Scott Laningham was nice enough to interview me for the DeveloperWorks podcast when the first article went out, so if you want to hear me blathering on about it in person, you can do so!

Reboot Britain

2009-07-08T09:17:00Z

I was lucky enough to be invited to talk at the Reboot Britain conference on Monday. It was a fascinating day, with lots of inspiring moments. It left me with a feeling that we can make the world a better place if we all get out there and make things. It's something I already believed, but it's great to have a recharge every now and again.

My session was on hacking energy data, kind of a Homecamp intro for the uninitiated. My slides are up on Slideshare, and embedded here for your viewing pleasure. Hopefully at some point I'll get to re-record the audio and add it in.

If you can see it, you can change it

View more documents from James Smith.

The Carbon Diet goes open source

2009-05-21T20:38:00Z

Well, I finally got round to it. I started developing The Carbon Diet 2 years ago (had the idea 3 years ago!), and it's finally spread it's wings and joined the world of open source.

After I'd been developing it for a few months, it became obvious that it was never going to be a commercial proposition, so it's been my intention to open it up for a very long time now. However, I was putting it off until "after I get it finished", or "after I tidy up the code", or any number of other reasons. Having a small child puts paid to all those plans, so I've finally bitten the bullet and released my dodgy code to the world as it is, warts and all.

The site is built in Rails, but it's pretty creaky. It was originally written in version 1.2 as I was learning, and a lot of the old-style controller and route code is still hanging around, waiting to be improved. I've started, but there is a long way to go to bring it up to a decent standard.

Anyway, if you use the site (or not!) and fancy improving it, now you can. Head on over to github to grab the code and join in. I hope to see you there!

Publishing CurrentCost data to the world

2009-03-28T16:18:00Z

A while ago, I started hacking around with a CurrentCost real-time energy monitor. This is a very nice little device that measures your electricity use, but more importantly has a serial output on the bottom so you can get data out of it.

Well, eventually I decided that while being able to get data off the meter was nice, it would be better if I could publish it somewhere. So, I wrote a program in Ruby to do exactly that. It reads data from the meter, and sticks it on the web. I'll go into a bit more detail in a minute, but first, you can download it (and get the source code) from github.

The program operates as a system daemon, which runs in the background on your Linux box (Mac should work as well) and uploads to various places. Adding new publishers is dead easy, so if you want to publish your data somewhere else, you can easily add it.

The Carbon Diet

First up, my own site, The Carbon Diet. This uploads your daily usage history from the meter into your carbon diet account so that you don't have to take so many meter readings to get an accurate graph. This is still pretty experimental and needs more work, but it's pretty useful already. You can see how it looks on my profile.

AMEE

Next, what I think of as the important one. AMEE, if you're not aware, AMEE is a neutral aggregation platform for sharing energy data and carbon calculations (disclaimer: I work for them these days). Think OpenID for your energy identity. Anyway, we have a nice "smart meter" demo which uses my currentcost app as the data source. Every minute, it uploads into an AMEE profile, and then another app makes a nice graph of the CO2 produced. In theory, the Carbon Diet could pull that data from AMEE instead of me publishing it directly, but that's still to come.

Pachube

Another energy data sharing service is Pachube (pronounced "patch bay"). As far as I can tell, this is more geared at art and design than rigorous data, but it's fun to play with. They've done a bunch of stuff with the CurrentCost, and now my app joins the throng. My pachube feed updates every 6 seconds - every time the meter sends data out.

Twitter

Finally, what would be the point of a web energy publisher if it couldn't tweet? If you really want to, you can see my minute-by-minute energy usage on Twitter by visiting @james_energy.

Running Mephisto on Ruby 1.8.7

2008-11-04T22:52:00Z

My server runs Ubuntu, and seeing as 8.10 (Intrepid Ibex) is out, I thought "woo, let's upgrade". Bad idea. Well, partly. Most of the upgrade went fine, but there was one problem. Intrepid comes with Ruby 1,8.7, which works fine with Rails >=2.1, but doesn't work with anything lower. Mephisto, on the other hand, doesn't work with Rails 2.1 yet, and is only usable with Rails <=2.0.2. So, you get an error starting Mephisto that looks like this:

So, I was faced with having to downgrade to Ruby 1.8.6 (a royal pain as Intrepid doesn't provide packages for it), or "fix" Mephisto to work with 2.1 (oh my god, don't even get me started). Fortunately, another solution presented itself.

I came across this blog post which describes the problem I was having. The gist (it's in Spanish) is that the main problem with Rails 2.0.2 and Ruby 1.8.7 is that Ruby 1.8.7 has added a String#chars function to the core language. Rails also adds this function, but the two have different return types, hence causing much breakage. The solution? Remove Ruby's String#chars and let Rails do its thing. Add this code to an initializer in your Rails 2.0.2 app, and it will work (better, at least) with Ruby 1.8.7.

begin
  String.class_eval { remove_method :chars }
rescue NameError
end

As you can tell by the fact that you're reading this blog, it works a treat. Hurrah!

EDIT: Another solution, which I only just worked out, would be to install the very badly-named Ruby Enterprise Edition, which sits alongside the system's Ruby and is call-compatible with 1.8.6. This makes a lot of sense if you're using passenger (mod_rails), as well.

Gems, Bookmarks, and Sandwiches

2008-10-13T20:31:00Z

I've been crazy busy for a while, but what little spare time I have grabbed here and there has been used investigating some areas of Rails I hadn't played with before, like route globbing and running without a database. The results are a couple of tools, one useful, one most definitely not.

The first (and most useful) is Has My Gem Built Yet?. I have a bunch of ruby gems on GitHub and get annoyed that I have to keep running "gem update" while waiting for GitHub to build the gem once I've pushed the code. Well, no more. Just put in the details, and the page checks for you and pops up a message when the gem is done. Nice. It got picked up on the GitHub blog, and I've even had some patches sent in. It's a nice feeling :)

The second app is somewhat less useful (though still entertaining). After a suggestion on Twitter by Tom Taylor, I built a website which lets you bookmark sandwich combinations. There are a couple of interesting bits in this though - first is route globbing, which is the Rails way of using arbitrary bits of the URL as parameters to an action. Not something you use in every project, but still. More importantly, I've used it to test out rubymarks, a Rails plugin for generating links for social bookmarking services.

I love Rails for this sort of thing. I can't think of other system I've used that would let me create such fun things in the tiny snippets of spare time I have at the moment. And of course, everything above is open source and available on Github.

Mocking Kernel#require

2008-08-31T10:15:00Z

The other day, I remembered to add coverage analysis to my AMEE-Ruby tests, using rcov. The results were pretty good - most of the code was already tested, except a few failure cases, and I quickly wrote some new tests to make sure those were working properly. One little bit of code stood out though. Because AMEE talks XML and JSON, my gem can use JSON, but only if the JSON gem is available on the system (XML support is built into core Ruby). Problem is, require calls throw errors if they fail, so I had to write some code to handle the load failure and carry on regardless. This is the code inside amee.rb:

begin
  require 'json'
rescue LoadError
  nil
end

Problem with this is that the gem is installed on my system, so the require never fails, and the rescue is never used. Rcov notices, and I can't get to 100% coverage, which is annoying.

So, mocking to the rescue. We have to make Kernel#rescue throw an error if we try to require 'json', even if the gem is installed. At first, I tried to use flexmock to do this, but couldn't make it work - if anyone knows how, please tell me. My final approach was to monkeypatch Kernel#require inside my test so that require 'json' (and only json) would fail:

it "should cope if json gem isn't available" do
  # Monkeypatch Kernel#require to make sure that require 'json'
  # raises a LoadError
  module Kernel
    def require_with_mock(string)
      raise LoadError.new if string == 'json' 
      require_without_mock(string)
    end
    alias_method :require_without_mock, :require 
    alias_method :require, :require_with_mock
  end
  # Remove amee.rb from required file list so we can load it again
  $".delete_if{|x| x.include? 'amee.rb'}
  # Require file - require 'json' should throw a LoadError,
  # but we should cope with it OK.
  lambda {
    require 'amee'
  }.should_not raise_error
end

We have to make a new require function, then use alias_method to rename the old one and install our new one in it's place. Rails includes alias_method_chain to make this easier, but that's not available in pure Ruby. Never mind. Once we've done the monkeypatch, we take amee.rb out of the $" array, which lists all the currently-required files, to make sure we can require it again, then simply run the require and make sure it catches the error that is thrown by our patched require. And the most satisfying part? The rescue is executed, the test works, and we get to 100% coverage. I've got a nice warm fuzzy feeling inside... mmmmmm.

CurrentCost data live on Pachube

2008-08-29T09:56:00Z

So, the other day I got a nice little tray icon working for my CurrentCost power monitor. That's great, but data is only really gets fun when it's mashable, so the next step was to get it online somehow.

Pachube is a site which aggregates data feeds from real-world (and virtual-world) devices, shows them on a map, makes graphs, things like that, so it seemed like a good first attempt at putting my power data online. My first thought was to get my app to post data at regular intervals to the service, but unfortunately Pachube doesn't work like that. Instead, it acts more like a news reader, not a publishing platform - Google Reader instead of Blogger, if that makes sense. So, I had to publish my feed live on the web and point Pachube at the URL.

First step: EEML. This is an XML-based format which Pachube reads which can contain not only multiple data feeds, but tags and other metadata. So (as seems to be the fashion), I wrapped it up in a Ruby gem, available from GitHub as ever. The gem simply provides utility classes to build an EEML feed and convert it to the XML-based format for delivery over the web.

Then, the final step was to publish the data on the web. For that, we need a web server. However, having a full web server for just one feed seemed overkill, and I didn't want to publish to yet another intermediate server, so we need to serve the data directly. Ruby to the rescue once again. WEBrick is a simple web server which is part of the core Ruby libraries. You create a server, write simple servlet classes, and mount them at particular locations. For instance:

# Create WEBrick server
s = WEBrick::HTTPServer.new( :Port => 50000 )
# A simple "hello world" servlet 
class HelloServlet < WEBrick::HTTPServlet::AbstractServlet
  def do_GET(request, response)
    response.status = 200
    response['Content-Type'] = "text/xml"
    response.body = "hello world"
  end
end
s.mount("/", HelloServlet)

Now, http://localhost:50000/ will say "hello world". From here it's a simple modification to publish the EEML feed. EEML-Ruby includes a simple EEML server script as an example. So, after building this into the tray app, now whenever my CurrentCost is connected and the app is running, it serves up EEML data to the web. You can see the data feed (fairly intermittent, as obviously the meter isn't always connected to my PC at the moment) live on Pachube.

Some successful CurrentCost hacking

2008-08-22T09:25:00Z

After a bit of work, I've finally got my CurrentCost meter working in Ruby, and I now have a power monitor sitting in my system tray! There were a few stages involved...

Serial comms: The ruby-serialport library that already existed for Ruby was no good to me. Firstly, it didn't seem to be in a working state, but more importantly, the license it is under (GPL) is no good to me. So, I had to write my own. I've created a nice simple serial library (including a gem) for Ruby called RB232, which is available on GitHub. It only supports reading at the moment, and only works on Linux systems, but it's a start. Next!

Reading CurrentCost data: Once RB232 was in place, this was pretty easy. Just create a couple of classes to wrap up the process of getting data from the meter, and away you go. Easy. Also released as a Ruby gem on Github.

User interface: Last step was to make a simple user interface for the meter, which you can see in the picture above. It's a simple tray icon that changes colour based on power usage. It's based heavily on another very useful tool called cctrayrb, so many thanks to Daniel Parnell for doing the heavy GUI lifting there. The app is included as part of the currentcost-ruby gem mentioned above.

Anyway, it's all freely available, so if you have a CurrentCost meter and a serial cable for it, you can grab the code and get going. Enjoy :)

Twitter support in JabberStatus

2008-08-07T21:10:00Z

JabberStatus was originally inspired by Twitter's facility to update your status via XMPP/Jabber. Unfortunately, Twitter's Jabber interface has been down for months now, which is rather sucky.

The solution to this? Extend JabberStatus to work for Twitter as well as Facebook. It turned out to be very easy, so it's online now. Just add twitterstatus@jabber.org to your contact list and it will talk you through the rest.

The JabberStatus code has moved to GitHub now, so you can grab it from there. The code for the CO2Updates app has also moved there.

One Hundred Months

2008-08-06T09:56:00Z

I just saw the One Hundred Months campaign, and decided it was ripe for a bit of automated Twittering. So, 5 minutes hacking and we have One Hundred Months on Twitter. Code (as ever these days) is available from GitHub.

I'm amazed by what computers can do sometimes. This one seriously took me longer to publish to the world than to write.

Hacking your energy usage with the CurrentCost

2008-08-06T00:09:00Z

The other day, I managed to get hold of a CurrentCost energy monitor (available to buy from here, or maybe from your electricity supplier). Now, the nice thing about this particular monitor (apart from the ton of information on-screen) is the fact that it has a serial output on the bottom, which you can (with a bit of hacking) plug into your PC, and - bingo - lovely XML data!

However, once you have it connected and spewing XML at you, you really need something to do with all that data. I don't have anything big written yet, but my first step towards making something useful is a Ruby gem, which is available from GitHub. So far it can only parse the XML data from the meter - direct access to the serial port is hopefully coming soon.

In other news, the AMEE gem I started a while ago is still coming on. It can now use the XML and JSON interfaces, parse the whole Data API, and retrieve a list of Profiles. Not a bad start.

AMEE for Ruby

2008-07-10T21:29:00Z

In a fit of why-the-hell-not, I've just started writing my first gem for Ruby, which is going to be a wrapper around the AMEE carbon calculation engine. It's still in a very early incarnation, but more will be forthcoming soon. At the moment it can authenticate, and parse DataCategory nodes. DataItems and DataItemValues will be following after a short (UK-based) holiday. Source code and installation instructions are available from GitHub.