Audio, Video, HTML5 and standards

The chaps at Mozilla, Christopher Blizzard and Robert O'Callahan reopened the HTML5 <audio> and <video> debate yesterday, with a spirited defence of their decision to support only the patent-unencumbered* Ogg format and Vorbis and Theora codecs in Firefox releases as part of their HTML5 support.

Now, I understand their motives here - back when I was at Apple, I spent a big chunk of time trying get permission to add support for Vorbis to QuickTime, but didn't manage to get it past Apple management's fears. However, all the browsers I use now claim to support HTML5 <audio> and <video>, so I thought I'd try it out. I made some simple test pages using mp3, .au and WAV files, to see how they were supported.

What I found was a bit disappointing - it seems that the way that the spec is written, you can support <audio> but no file formats or codecs at all (my Droid does this), and if you can't play the file you're not supposed to show the fallback HTML contents

This means that Firefox, Droid won't show the link to the audio file below:

though browsers that don't support <audio> at all will. Here's the markup:
<audio src="http://homepage.mac.com/kevinmarks/dystopia.mp3" controls><a href="http://homepage.mac.com/kevinmarks/dystopia.mp3" type="audio/mpeg"> Looking Up From Dystopia </a></audio>

However, if I use a direct link or an embedded <iframe>, Firefox will use available plugins to play the file (both Flash and QuickTime happily play mp3's). Thus using <audio> give me less compatibility with current browsers.

On phone browsers, odder things happen - iPhone gives a clickable button for the <audio>, but auto-loads an <iframe>; Droid ignroes iFrames, Palm Pre doesn't have <audio> but <iframe> behaves like the iPhone.

Smarter behaviour with declarative audio would be nice here.

*Submarine patent trolls keeping periscopes down may exist.

Storytelling and performance

Alice Mathias in the NYT points out that public profiles are a kind of performance:

Facebook did not become popular because it was a functional tool — after all, most college students live in close quarters with the majority of their Facebook friends and have no need for social networking. Instead, we log into the Web site because it’s entertaining to watch a constantly evolving narrative starring the other people in the library.

I’ve always thought of Facebook as online community theater. In costumes we customize in a backstage makeup room — the Edit Profile page, where we can add a few Favorite Books or touch up our About Me section — we deliver our lines on the very public stage of friends’ walls or photo albums. And because every time we join a network, post a link or make another friend it’s immediately made visible to others via the News Feed, every Facebook act is a soliloquy to our anonymous audience.

It’s all comedy: making one another laugh matters more than providing useful updates about ourselves, which is why entirely phony profiles were all the rage before the grown-ups signed in. One friend announced her status as In a Relationship with Chinese Food, whose profile picture was a carry-out box and whose personal information personified the cuisine of China.

This is of course what danah has been saying for years:

While some early adopters viewed Friendster as a serious tool for networking, others were more interested in creating non-biographical characters for playful purposes. Referred to as Fakesters, these Profiles represented everything from famous people (e.g., Angelina Jolie) and fictional characters (Homer Simpson) to food (Lucky Charms), concepts (Pure Evil), and affiliations (Brown University). Some Fakesters were created to connect people with common affiliations, geography, or interests. The most active and visible Fakesters, however, were primarily crafted for play. [...]

Fakesters had a significant impact on the cultural context of Friendster. In their resistance, their primary goal was to remind users that, “none of this is real.” They saw purportedly serious Profiles as fantastical representations of self, while the Testimonials and popularity aspect of the Friend network signified the eternal struggle to make up for being alienated in high school. Through play, Fakesters escaped the awkward issues around approving Friends and dealing with collapsed contexts, mocking the popularity contest. Their play motivated other participants to engage in creative performance, but at the same time, their gaming created a schism in the network resulting in a separation between playful participants and serious networkers.

Meanwhile, Jonathan Sanderson is explaining the art of video storytelling, and how to stop short of lèse majesté:

I want to write about fakery in television, because there's something odd going on. None of these 'scandals', from naming Socks the cat to having someone stand in for competition winners when the phone line goes dead in the full glare of live transmission, is particularly shocking to anyone who's made videos. Not worked in broadcast, note -- made videos. When I get a bunch of 14 year-olds to make their first short film, they'll frequently assume they can fake stuff, cheat, and generally bend the resulting video to their will.

Now, all it takes is for me to stare at them for a few moments. The light will go off in their heads and they'll say 'Oh, right. OK, yes. Fine. We'll do it for real.' But the natural human affiliation with cheating is sufficiently powerful, it's often the first assumption.

Later in the day, when the same group is putting together their sequence, they'll find me and say 'If we change the order like this, the film makes more sense. But... that's faking, isn't it?'

...which is, of course, the crux of the matter, because all video is faked to some extent or other. Everything you do up to the point where you start editing is just collecting raw material -- your film is made, crafted, shaped, in the edit suite, not in front of the camera.

It has to be this way, because real life plays out excruciatingly slowly. The responsibility and skill in making films, then, lies in telling stories more quickly, and more engagingly, than real time. Which requires that you leave bits out, which in turn requires judgement about which parts are important.

Telling stories honestly is an aspiration, but not a requirement -- the temptation to cheat and edit the material in order to tell an even better story even more quickly is always there. If the story's better, and more people watch, that's a success, right? If teenagers hacking away in iMovie in a school lab face these sorts of dilemmas and compromises, you can imagine the discussions that happen in chic Avid suites in Soho.

We all live in the minds of others through telling them stories about ourselves, but we also live in our own minds that way too - the research on memory shows how we readily confabulate extra detail to flesh out a story, and that every time we remember somethign we reify it further, combining it with new experiences both real and confabulated, so that the tale grows in the telling.

As danah notes, the persistence, searchability and replicablity of these digital environments belie our self-constructed memories with awkwardly concrete virtual histories. Massively Multiplayer Online Truth is an interesting game, and one where we are still working out the rules.

Caliban's Mirror, YouTube edition

The UK Professional Association of Teachers has called for YouTube to be banned because it has been used to show people being bullied. Meanwhile, Oxford police are pursuing a man who pinched a news reporters' bum on air:

I trust the PAT will call for Channel 4 to be banned too.

Perhaps the net can fulfill Bentham's dream of the panopticon, where prisoners are always watched. In the Philippines they are showing the way. With Bentham, Busby Berkeley and Michael Jackson as guiding spirits, 1500 prisoners perform "Radio Gaga", "Thriller", the Algorithm march, and, yes, "YMCA" for the camera. If they're taking requests, I nominate Pink Floyd's "In the Flesh".

YouTube, like the rest of the web is a mirror to life. If you don't like what you see, look for something else, like this elegy for Concorde:

Or remember Antonioni

Or Syd Barrett

Or make a video to tell your own story.

B-roll is the new a-hole

Editing shows respect. Steve Gillmor has put up his new video podcast Bad Sinatra which is him unedited wandering around and chatting to people. Now, I like Steve, and enjoy having rambling conversations with him about things, but I don't think it works as video. With video, as with audio, editing shows respect for your audience. To do a truly live or as-live show, like the splendid In Our Time, you need to plan it out in advance and choreograph it. Otherwise, you need to edit. Carefully.

If you want to hear a perfectly edited podcast, listen to Radiolab. I spend 90 minutes a day cycling and listening to my iPod, which is exactly the use case for podcasting I explained to the BBC a few years back. Now that shows of this quality are there to be downloaded regularly, my tolerance for self-indulgent rambling podcasts like bad voicemail messages is way down.

I made this point way back at the dawn of podcasting , when Chris Lydon's well-edited interviews inspired us to download them to our iPods. In the online world we are each others' media, as we mediate what is worth reading for each other through our blogs and link streams. This too, is a form of editing, and doing it shows respect for each other. Steve, tighten it up into something worth watching. My children know how to do it.

Microformats and media

A few weeks ago at Web2Open, Mary Hodder and I gave an audience-led talk on Microformats and Media Info. Fumi Yamazuki videoed it and posted it on YouTube. The web connection was a little unreliable, so I started out typing up examples by hand...

iPhone's great step back from iChat

I did watch the iPhone Macworld Keynote last week - here's my chapter-list version of the stream so you can skip through it to the bits you find interesting:


(I'd do the same for the download except 1. Apple obfuscated the url enough that I can't be arsed to packet sniff it out and 2. they still don't have HTTP 1.1 seeking support working right, despite me building and demoing it about 5 years ago).

After watching it, my biggest surprise was how much of a step back it was from the rich interaction that iChat supports. With iChat you get presence info, chat, sending documents and integrated audio-video chat (when the other user's computer and connection supports it). Instead, iPhone had a legacy telco worldview baked in, with calls not conveying any further context (watch the combined demo near the end, where Jobs has to retype Schiller's email when talking to him on the phone). The iPhone has the camera on the wrong side to be a videophone, and Jobs did not mention any ability to make calls over Wifi rather than the Cingular network, or anything about IM (as opposed to SMS).

My hope is that this is just Jobs not mentioning the features that don't demo well yet, but my sidekick's AIM integration is the reason I am so hooked on it; it buffers chats server-side so thatintermittent phone connections on the train don't interrupt conversation flow.

Patent Trolls

I broadly agree with Paul Graham's essay on Software Patents, but I do think he underestimates the damage from patent trolls, and from what he calls the mafia-like behaviour of some patent holders.

Paul has been lucky in the field he has worked in, but in the Audio and Video area there are many patent thickets. Perhaps it is the history of Farnsworth's victory over RCA that makes video engineers patent hungry.

My first startup, The MultiMedia Corporation, was a spin-out from the BBC in 1990. One of our products was a program called MediaMaker that combined video from tape or videodisc, CD Audio, Pictures, digitised audio and Director animations into picture icons on a timeline for making presentations. It was demoed on stage at Macworld by the CEO of Apple, and we got Macromind to publish it.

Then the patent troll showed up. A company called Montage had made a video editing system that included several video monitors showing edit points from tape. The company had gone out of business but a lawyer had bought up the patents, including one on using a still image to represent a video sequence. The troll was working his way round the video companies, and he caused enough trouble to stop work on the product while we worked on a legal defence instead.

Later, while I was at Apple on QuickTime, there was a steady stream of patent trolls claiming that Apple should pay them royalties; enough to keep several lawyers busy, and a lot of engineers spending time working on prior art evidence demonstrations.

Several potential features were excluded from QuickTime due to patent thickets. The obvious one was the Unisys LZW patent that encumbered GIF, but there were other more subtle pressures that meant adopting open source codecs was discouraged. Working on the patent license agreements for MPEG meant that technology ready to ship was deferred pending legal agreement on more than one occasion.

So I'm much lass sanguine than Paul about this. I think software patents should not be granted, and the European Union's banning of them is the right decision. I hope the Gowers Review in the UK makes this UK law as well.