Open Source and Social Cloud Computing

Tim O'Reilly has written an excellent review post on Open Source and Cloud Computing which says, among other things:

The interoperable internet should be the platform, not any one vendor's private preserve.

So here's my first piece of advice: if you care about open source for the cloud, build on services that are designed to be federated rather than centralized. Architecture trumps licensing any time.

But peer-to-peer architectures aren't as important as open standards and protocols. If services are required to interoperate, competition is preserved. Despite all Microsoft and Netscape's efforts to "own" the web during the browser wars, they failed because Apache held the line on open standards. This is why the Open Web Foundation, announced last week at OScon, is putting an important stake in the ground. It's not just open source software for the web that we need, but open standards that will ensure that dominant players still have to play nice.

The "internet operating system" that I'm hoping to see evolve over the next few years will require developers to move away from thinking of their applications as endpoints, and more as re-usable components. For example, why does every application have to try to recreate its own social network? Shouldn't social networking be a system service?

This isn't just a "moral" appeal, but strategic advice.[...]

A key test of whether an API is open is whether it is used to enable services that are not hosted by the API provider, and are distributed across the web.

I think this API openness test is not strong enough. As I wrote in An API is a bespoke suit, a standard is a t-shirt, for me the key test is that implementations can interoperate without knowing of each others' existence, let alone having to have a business relationship. That's when you have an open spec.

The other thing I resist in the idea of an internet operating system is that that the net is composable, not monolithic. You can swap in and implementations of different pieces, and combine different specs that solve one piece of the problem without having to be connected to everything else.

The original point of the cloud was a solved piece of the problem that means you don't have to worry about the internal implementation.

Thus, the answer to "shouldn't social networking be a system service?" is yes, it should be a Social Cloud. That's exactly what we are working on in OpenSocial.

Here Comes Everybody - Tummlers, Geishas, Animateurs and Chief Conversation Officers help us listen

Bob Garfield's de haut en bas attack on web commenters upset two very skilled conversational catalysts, Ira Glass, and Derek Powazek. The false dichotomy of 'we choose who you get to hear' and 'total anarchic mob noise' was dismissed by Jack Lail too. At the same time, Ben Laurie explained how the IETF's open-to-all mailing lists can be hijacked by time-rich fools, talking about the Open Web Foundation.

At Supernova last month, listening to Clay Shirky talk about the problems of collective action reminded me of a small nit I have with his excellent book Here Comes Everybody (which you should all read). He talks about the deep changes that ridiculously easy group forming online has wrought, but he also explains that most of these groups fail, in various ways.

The key to this is finding people who play the role of conversational catalyst within a group, to welcome newcomers, rein in old hands and set the tone of the conversation so that it can become a community. Clay referred to Teresa Nielsen-Hayden, who is a great example of this, and I have had the privilege to discuss this with Teresa, Amy Muller,Christy Canida and others at the Troll Whispering session at Web2Open, and heard very similar stories from Gina Trapini, Annalee Newitz, Jessamyn West and Jeska Dzwigalski at The Users Are Revolting at SXSW.

The communities that fail, whether dying out from apathy or being overwhelmed by noise, are the ones that don't have someone there cherishing the conversation, setting the tone, creating a space to speak, and rapidly segregating those intent on damage. The big problem with have is that we don't have a English name for this role; they get called 'Moderators' (as Tom Coates thoroughly described) or 'Community Managers', and because when they're doing it right you see everyone's conversation, not their carefully crafted atmosphere, their role is often ignored.

In other languages there are words closer to this role - Suw and I thought of geisha a while back, whereas Teresa suggested the Yiddish Tummler - both Deb Schultz and Heather Gold liked that one. In French animateur has the broader connotations of discussion, leadership and guidance needed, but in English we are stuck with enervated latinate words like facilitator. Even an eloquent and charismatic presidential candidatehad a difficult time explaining what a 'Community Organizer' does, around the same time that Bartlett was resorting to card tricks.

Which brings me back to Clay's book - in it he gives an account of the #joiito chatroom that completely misses the rôle that JeannieCool played there, making her sound like a n00b. The software tool, jibot, that has helped keep that conversation going for 5 years, was built to support Jeannie's role as conversational catalyst. I do hope he gets a chance to correct this in the next edition.

The broader issue is one that we are still working on - building rules for who gets to speak where and when, re-imagining the historic model of a single hegemonic public record that print Journalism still aspires to, from its roots in the coffeeshops of London into the many parallel publics we see on the web, and how legal precedents designed for a monopoly of speech make no sense here.

In the meantime, if your newspaper, social media initiative or website isn't working right, you need to find your tummler, geisha, animateur or conversational catalyst, but you should consider giving them a big name title like 'Chief Conversation Officer'.

Shortening URLs, or getting inbetween?

With the rise of short message systems like Twitter, there is a growth in URL shorteners (as each one's namespace gets full, others get shorter). Today bit.ly launched to big fanfare in the blogosphere.

I took a closer look. What I noticed is that the older generation of these - tinyurl.com and xrl.us use a 301 Moved Permanently redirect, whereas bit.ly and is.gd use a 302 Found redirect, which means 'don't cache the redirected URL, keep checking the original'.

In other words, these services are saying in their HTTP responses that they may change what the short URLs point to in future, putting browsers, indexers and caches on notice that this may happen.

I also noticed that bit.ly, like tinyurl.com, allows you to pick a custom label from their namespace, but if you do it returns two 302 redirects in sequence (once to a more cryptic bit.ly url, then to the external one you chose). I pointed bit.ly/k at this blog, so you can check it yourself with curl:

$ curl --head http://bit.ly/k
HTTP/1.1 302 Found
Location: http://bit.ly/fwNKA

$ curl --head http://bit.ly/fwNKA
HTTP/1.1 302 Found
Location: http://epeus.blogspot.com

Apart from the extra delay this introduces, this is also telling your browser and web crawlers not to cache this, as they may change it in future. Compare tinyurl.com:

$ curl --head http://tinyurl.com/kevinm
HTTP/1.1 301 Moved Permanently
Location: http://epeus.blogspot.com

Google's advice for webmasters is to use 301 for redirects, as this signals the preferred URL.

Google as a restaurant? Watch Gordon Ramsay

Jeff Jarvis says he's writing a metaphorical application of Google principles to running a restaurant. Over the last few weekends, while sorting out stuff at home, I've been watching Gordon Ramsay's Kitchen Nightmares which BBC America seems to be playing continuously at weekends. If you haven't seen it, do watch some - each episode, Ramsay spends a week at a failing restaurant in the UK and tries to help them turn it around.

After seeing a few, there are recurrent themes that Ramsay comes up with: simple menus, built on good ingredients that local people understand, served promptly. Which fits well with Google's ten things - simple frontend, low latency results, user-focused. How he tries these out involve analogues for user testing, A/B experiments, and profiling under high load.

Of course, Google does run restaurants - so Jeff can read how they get built and tested directly.

I'm with the stupid network

I'm with the Stupid network
Originally uploaded by Kevin Marks

I'm looking forward to the Supernova conference next week, because Kevin Werbach always brings together an interesting group of people who care about the Internet and its future. We don't all agree on everything, which makes for some interesting debates, but we do tend to back the Open Web and the Stupid Network. It was the tenth anniversary of David Isenberg's 'Rise of the Stupid Network' paper this week, so I came up with this t-shirt design idea.

How not to be viral

Graphing Social Patterns East is on tomorrow, and I'm sorry not to be there, though m'colleague Patrick Chanezon will be. However, reading the schedule I notice the word 'viral' is still much in evidence.

If you behave like a disease, people develop an immune system

At the Facebook developer Garage last week, I heard a developer say: when I hear 'viral' applied to software I replace it with 'cancerous' to clarify. A few months back I wrote that social Apps should be Organic, not Viral, and at Google I/O last week I expanded on this with m'colleagues Vivian Li and Chris Schalk. Here's an overview of the alternative reproductive strategies to being a virus that we came up with:

r-Strategy - scatter lots of seeds

Break free
Originally uploaded by aussiegall

Some plants and animals, like dandelions and frogs, rely on having huge numbers of offspring, with the hope that a few of them will survive - this is known as an r strategy. In application terms this is like wildly sending out invitations, or forcing users to invite their friends before showing them useful information. It may help you spread your seed, but most of them will die off rapidly.

K-Strategy - nurture your young

Proud mama
Originally uploaded by debschultz

Mammals take the opposite strategy; they have a few young, and nurture them carefully, expecting most of them to grow up to adulthood and reproduce themselves. This is known as a K strategy. This translates into software by following Kathy Sierra's principles to create passionate users who will share your application through word of mouth. Another way to nurture your users is to encourage them to use your application before they have to install it, as Jonathan Terleski describes.

Fruiting - delicious with a seed in

help yourself
Originally uploaded by *madalena-pestana*

Many plants encourage their seeds to be spread more widely by wrapping them in fruit, so that animals or birds will carry them further, eating the fruit and helping the seed to propagate. The analogy here is in making sure your invitations aren't just bald come-ons for your application "a friend said something - click here to find out what" - with a forced install on the way, but instead are clearly bearing gifts to the receiving user, so they will want to click on the link after seeing what is in store. This is one of Jyri Engström's principles for Web 2.0 success with Social Objects.

Rhizomatic - grow from the roots up

Sweetness / Dolcezza
Originally uploaded by WTL photos

Another reproductive strategy that many plants, including strawberries and ginger use is to send out runners or shoots from the roots, so that they spread out sideways, from the bottom up, known as rhizomes or stolons. The analogy here is for social applications that spread through appearing in users activity streams and via entries in application directories, growing outwards through the 'grass roots' runners that they send out as part of their normal usage.

Being dumb gets low CPMs

A lot of the debate around viral applications reminds me of a David Foster Wallace quotation:

TV is not vulgar and prurient and dumb because the people who compose the audience are vulgar and dumb. Television is the way it is simply because people tend to be extremely similar in their vulgar and prurient and dumb interests and wildly different in their refined and aesthetic and noble interests.

Social networks aren't like TV - everyone sees something different in them. If you want to gather engaged, inspired, interested and indeed valued users, write an application that speaks to their refined and aesthetic and noble interests, and see how they will spread it through their social networks to find the others who share their interests.

It was interesting to see Slide redirecting away from virality today. GSP West was on at the same time and place as eTech, and I heard some eTechies refer to it as 'Grasping Social Parasites'; I hope that the growing realisation that a disease is not a good model to base your business on means that tomorrows conference will spread a better reputation for GSP East.

Miasma theory - wrong in the 1840s, wrong now

A couple of years ago I wrote:

My generation draws the Internet as a cloud that connects everyone; the younger generation experiences it as oxygen that supports their digital lives. The old generation sees this as a poisonous gas that has leaked out of their pipes, and they want to seal it up again.

Bill Thompson and Nick Carr are worried about governments interfering too:

In the real world national borders, commercial rivalries and political imperatives all come into play, turning the cloud into a miasma as heavy with menace as the fog over the Grimpen Mire that concealed the Hound of the Baskervilles in Arthur Conan Doyle's story.

Except, if you have read or listened to Steven Johnson's excellent The Ghost Map, you'll know that the miasma theory of disease was a fatal error for urban England in the 1840s - the real problem was not the bad smells in the air, but the diseases in the water. The fault, dear governments, lies not in our clouds but in your pipes.

An API is a bespoke suit, a standard is a t-shirt

Brad is calling for APIs, and even the NYT is proposing one, but there is a problem with APIs that goes beyond Dave's concern about availability.

When a site designs an API, what they usually do is take their internal data model and expose every nook and cranny in it in great detail. Obviously, this fits their view of the world, or they wouldn't have built it that way, so they want to share this with everyone. In one way this is like the form-fitting lycra that weekend cyclists are so enamoured of, but working with such APIs is like being a bespoke tailor - you have to measure them carefully, and cut your code exactly right to fit in with their shapes, and the effort is the same for every site you have to deal with (you get more skilled at it over time, but it is a craft nonetheless).

Conversely, when a site adopts a standard format for expressing their data, or how to interact with it, you can put your code together once, try it out on some conformance tests, and be sure it will work across a wide range of different sites - it's like designing a t-shirt for threadless instead.

Putting together such standards, like HTML5, OpenID, OAuth or OpenSocial or, for Dave's example of reviews, hReview, takes more thought and reflection than just replicating your own internal data structures, but the payoff is that implementations can interoperate without knowing of each others' existence, let alone having to have a business relationship.

I had this experience at work recently, when the developers of the Korean Social network idtail visited. I was expecting to talk to them about implementing OpenSocial on their site, but they said they had already implemented an OpenSocial container and apps using OpenID login, and built their own developer site for Korean OpenSocial developers from reading the specification docs.

I'm looking forward to more 'aha' moments like that this week at I/O.

Talking about OpenSocial all over the place

I've been travelling a lot to conferences in recent months, and been interviewed by a lot of different journalists too. Here are a few links to them.

• Cloud computing with Joyent at Web 2.0(video)
• Chris Vallance of BBC Pods and Blogs (audio)
• Jemima Kiss of The Guardian (audio)
• Data Portability podcast (audio)
• Kimberley Dykeman of web2.0 TV (video)
• Christina Warren of Download Squad (video)
• Caroline McCarthy of CNET (text)

For more in-depth details on OpenSocial, come along to Google I/O on May 28th-29th in San Francisco

Portable Apps, not data?

Brad Templeton has a post on Data Hosting not Data Portability that fits in neatly with the VRM proposal I discussed yesterday. In fact, what he describes is a great fit for OpenSocial.

He says:

Your data host’s job is to perform actions on your data. Rather than giving copies of your data out to a thousand companies (the Facebook and Data Portability approach) you host the data and perform actions on it, programmed by those companies who are developing useful social applications.

Which is exactly what an OpenSocial container does - mediate access to personal and friend data for 3rd party applications.

This environment has complete access to the data, and can do anything with it that you want to authorize. The developers provide little applets which run on your data host and provide the functionality. Inside the virtual machine is a Capability-based security environment which precisely controls what the applets can see and do with it.

This maps exactly on to Caja, the capability-based Javascript security model that is being used in OpenSocial.

Your database would store your own personal data, and the data your connections have decided to reveal to you. In addition, you would subscribe to a feed of changes from all friends on their data. This allows applications that just run on your immediate social network to run entirely in the data hosting server.

Again, a good match for OpenSocial's Activity Streams (and don't forget persistent app data on the server).

Currently, everybody is copying your data, just as a matter of course. That’s the default. They would have to work very hard not to keep a copy. In the data hosting model, they would have to work extra hard, and maliciously, and in violation of contract, to make a copy of your data. Changing it from implicit to overt act can make all the difference.

The situation is worse than that; asking people for their logins to other sites is widespread and dangerous. I'd hope Brad would support OAuth as a step along the way to his more secure model - especially combined with the REST APIs that are part of OpenSocial 0.8

If you're interested in these aspects of OpenSocial, do join in the linked mailing lists, and come along to the OpenSocial Summit on May 14th (just down the road from IIW).

Mixing degrees of publicness in HTTP

At the Data Sharing Workshop the other day, we had a discussion about how to combine OAuth and Feeds, which I was reminded of by Tim Bray's discussion of Adriana and Alec's VRM proposal today.
The session was tersely summarized here, but let me recap the problem.

When you are browsing the web, you often encounter pages that show different things depending on who you are, such as blog, wikis, webmail or even banking sites. They do this by getting you to log in, and then using a client-side cookie to save you the bother of doing that every time. When you want to give a site access to another one's data (for example when letting Flickr check your Google Contacts for friends), you need to give it a URL to look things up at.

The easy case is public data - then the site can just fetch it, or use a service that caches public data from several places, like the Social Graph API. This is like a normal webpage, which is the same for everyone, returning a HTTP 200 response with the data.

The other common case is where the data is private. OAuth is a great way for you to delegate access to a web service for someone else, which is done by returning an HTTP 401 response with a WWW-Authenticate: OAuth header showing that authentication is needed. If the fetching site sends a valid Authorization header, it can have access to the data.

The tricky case is where there is useful data that can be returned to anyone with a 200, but additional information could be supplied to a caller with authentication (think of this like the social network case, where friends get to see your home phone number and address, but strangers just get your hometown). In this case, returning a 401 would be incorrect,as there is useful data there.

What struck me was that in this case, the server could return a 200, but include a WWW-Authenticate: OAuth header to indicate that more information is available if you authenticate correctly. This seems the minimal change that could support this duality, and much easier than requiring and signalling separate authenticated and unauthenticated endpoints through a HTML-level discovery model, or, worse, adding a new response to HTTP. What I'd like to know from people with deeper HTTP experience than me is whether this is viable, and is it likely to be benign for existing clients — will they choke on a 200 with a WWW-Authenticate header?

HTTP does have a 203 response meaning Non-Authoritative Data, but I suspect returning that is more likely to have side effects.

Digital publics, Conversations and Twitter

Last week, I left the Web 2.0 conference to listen to Mimi Ito, danah boyd and their colleagues talk about their research on Digital Publics.

Now if you haven't been paying attention, that plural of 'public' there may throw you. Surely things are either 'public' or 'private'? As danah explains:

Just as context is destabilized through networked publics, so is the meaning of public and private. What I learned from talked to teens is that they are living in a world where things are "public by default, private when necessary." Teens see public acts amongst peers as being key to status. Writing a public message to someone on their wall is a way of validating them amongst their peers. Likewise, teens make choices to go private to avoid humiliating one of their friends.

Yet, their idea of public is not about all people across all space and all time. They want publics of peers, not publics where creeps and parents lurk.

Bly Lauritano-Werner (17, Maine):

My mom always uses the excuse about the internet being 'public' when she defends herself. It's not like I do anything to be ashamed of, but a girl needs her privacy. I do online journals so I can communicate with my friends. Not so my mother could catch up on the latest gossip of my life.

Properties of technology have complicated what it means to be in public. We are all used to being in publics that don't include all people across all space and all time. Many of us grew up gossiping with friends out in public and stopping the moment that an adult walks over. This isn't possible when things are persistent. And it's really hard to be public to all peers and just keep certain people out. So teens are learning how to negotiate a world where the very meaning of public and private have changed. Again, this is a good thing. They're going to need these skills in the future.

The day before, at Web2Open, I had heard something similar in the Troll Whispering session. Christy Canida explained that when someone posts something trollish or otherwise dubious on her site, they get put in a state where only they can see their posts, but no-one else can (except Christy and the other conversation monitors). This damps down the flame responses until Christy and co have time to review, and maybe release them, but in their view the post is on the site, but no-one is responding.

This varying view of the web, depending on who you are, seems odd at first, but it is in fact a recognition in code of what actually exists in human attention. We don't all read the same web, we see our own reflections in what we seek through searches or filtered by our homophily-led reading.

Which is where Twitter comes in. Like Jeff, I've been twittering more than blogging recently, and while immediacy is part of it, a far stronger thing is that I have a sense of public there - a public of people I choose to follow and who chose to follow me. Everyone who uses Twitter sees a different, semi-overlapping public, which maps closer to our individual idea of the digital public we are speaking to, and listening to; one that maps more closely what the socialogist and theorists have been describing for a while.

Comcast's Bialystock and Bloom Business Model?

Tomorrow, the FCC is holding a public hearing at Stanford on Broadband network management practices. With striking timing, Comcast today managed to announce a 'Internet Bill of Rights' without inviting any users, and simultaneously cut off Dave Winer's net connection for exceeding their secret usage limits. I can't link to Comcast's policy because their website mungs the text in via javascript - here's what they say:

Excessive use means data usage that is not characteristic of a typical residential user of the service as determined by Comcast.[...]Comcast currently identifies well less than 1% of Comcast High-Speed Internet customers as excessive users each month. [...]Many excessive users consume more data than a business-class T1 line running at full capacity in a month. [T1 is 1.5 Mbit/sec - Comcast claims to offer 12 Mbit/sec for PowerBoost, and 6/8 Mbit/sec standard] [...] Currently, each month Comcast identifies the top bandwidth users of its High-Speed Internet service by determining aggregate data usage across its entire customer base nationwide.

What they are saying is that they use a crude averaging model, and penalize you if you don't fit, for example by using the connection capacity they promise more than 10% of the time. Now, this could be called Procrustean, but it reminds me of The Producers, where Bialystock and Bloom sold a hundred people 10% shares of the show, assuming it would fail. Sadly for Comcast, people like Dave are finding new uses for the net's bandwidth, and not just checking email sporadically any more.

Conventional internet service user models are based on users downloading more then they upload, from common big media sites that can be easily cached. However, as Odlyzko pointed out, citing Lesk's now decade-old work, the dominant form of data creation is photographs. Now all these photographs are actually digital, and we want to share them so others can see them. Because we aren't allowed to run our own servers by the likes of Comcast, we have to upload them to Flickr or Photobucket or Picasa to share them. This gives us an 'upload more than you download' network flow, as we send them up at full multi-megapixel resolution, but browse a few of each others' at thumbnail or reduced size. And that's before we even consider video uploading (which I've noticed Comcast throttles at 0.4 Mbit/sec for me).

Comcast hit the news before by sabotaging Bittorrent transfers by faking reset packets, but what Bittorrent is really doing is arbitraging around the asymmetric network bandwidth delivered by these outdated user models.

Bob Briscoe recently wrote an interesting proposal on handling congestion by TCP signalling to reveal the costs of congestion. This was spun by George Ou as an attack on P2P protocols, but the underlying principle of penalising those who cause congestion is an interesting one. The question I'd like answered is that if I have a gigabit network at home, and the internet backbone is multi-terabit, when Comcast throttles my uploads to 400 kilobits, aren't they the ones causing the congestion?

Be Organic, not Viral

I just got back from the VLAB Multi-platform Social Networking event, which I thought was very interesting overall. Jeremiah Owyang did a great moderating job, and Jia Shen, Sourabh Niyogi, Ken Gullicksen and Steve Cohen brought lots of different viewpoints to the discussion. Growing and deriving value from Apps within Social Networks is still full of lots of unknowns, but it was good to hear some basic shared principles come through - my summary of one point was 'before you think about a Business Model, make sure you have a Pleasure Model'.

Another point well made by Steve Cohen of Bebo was something I've been thinking for a while too - the hunger for 'Viral' growth is a mistake - what you really need is 'Organic' growth. Just as we distinguish between Organic search results and bought or spammed ones, social network sites and their users are distinguishing between the viral apps that are essentially parasitic, using their hosts as a means to their propagation, and the ones that organically become part of the social ecology, making both the site and the users richer by their presence.
I spent the last weekend fighting off a flu virus, partly by eating lots of organic fruit. I expect social networks and their users will continue to do the same.

The Social Cloud

My talk from LIFT is here for you to watch below (20mins, needs flash):


The others are up at the LIFT Video site