Lunch at the Internet Archive

Rosie & I went up to the Presidio for lunch with the Archive team.
Brewster manages to get a great deal done with a very small staff and a huge penumbra of friends and volunteers. Lots of fascinating stuff about the Bookmobile abroad in Africa and Asia, and met Rick Prelinger and Lisa Rein too.

Afterwards we drove back to the Santa Cruz mountains for a great Mars watching Party, and had to go via Highway 9 to avoid the complete blockage of 17, Rosie winning the skilled driver award again.

Lunch with Dave Winer

Over the famous spicy noodles at Jing Jings, I signed up with Dave to speak at BloggerCon.
I'll be on the 'Tech' Panel, where blog users talk back to vendors about technical needs, and I'm putting together a talk on the Sunday about the need for a new model for video and audio to make them more blog-like and less like voicemail.
I also think I persuaded Dave to include live IRC at the conference - more on that later.

Lunch with Lessig

Had a good chat with Larry about the similarities and differences between Creative Commons and mediAgora - the goals are broadly aligned, but the methods slightly different.

iTunes parody make serious point

iTunes iSbogus attacks iTunes store for funneling money to the record labels instead of artists.

MailFrontier

Had a nice long chat with the MailFrontier folks about the detailed workings of their spam reduction tools Matador and Anti-spam gateway- a clever and systematic approach. I mentioned my swarming class action anti-spam idea.

Coffee with SocialText

Met up with Ross Mayfield & Pete Kaminski of SocialText, who blend Wiki's and blogs for Enterprise customers, and build nice tools for conference communications.
I think there is potential in extending the 'third place' of an online conference chatspace and combining it with a real-world cafe/bar at the conference, but you'd need a friendly host like Jeannie to make it work.

Lunch with Peter Hoddie

Good to catch up with Peter again; had a great long conversation about digital media, the net, philosophy and more.
he's doing interesting things with mobile devices and net standards at Kinoma.

Postel's Dilemma

Aaron, Dave and Brent have been discussing Postel's law and XML.
Brent says:
Some people have said that the issue with not well-formed feeds and aggregators is a kind of prisoner�s dilemma.

In other words, if all aggregator developers pledged to reject not well-formed feeds, then that would be incentive for people to fix their feeds.

It is a kind of Prisoner's Dilemma, but it is an iterated one. If the aggregators are relatively forgiving, users will trust them, but poorly formed feeds will end up somewhat mangled in different aggregators, leading to less trust in the feeds. Over time, the forgiving aggregators and clean feeds win out.
The power of forgiveness has been demonstrated experimentally.

BBC opens archives

Greg Dyke is going to put the BBC archives online
15 years ago, I joined the BBC as a video engineer with Television Film Services, an odd department that had dominion over (among other odd sites) Ealing Studios and the film & video archive at Windmill Road.
I spent a couple of weeks at Windmill Road, working on the 2-inch Videotape machines which you could edit videotape like film on - a razorblade to cut the tape and splice with sticky tape. You needed to cut at a frame boundary, which you found by looking for the sync pulses on the tape by holding a plastic box full of iron filings over the tape, making the tracks visible.
I walked through the archives, seeing rack after rack of old editions of Nationwide, film canisters of a forgotten adaptation of Waugh's Sword of Honour trilogy, and many more fascinating looking can's and tapes.
At the time the BBC did not have a rolling re-copying program, and many programs on tape were lost to oxide decay, not recorded at all, or had the tapes re-used. Some missing Dr Who and Top of the Pops and Dad's Armyhave been recovered, and Rosie's cousin Rob Greenwood spent a great deal of time restoring video for Dr Who and other programs at BBC Enterprises in the 90s, in many cases from 'illicit' off-air recordings by hobbyists.

I hope the BBC can cut through the tangle of copyright restrictions on music in the programs and performance fees for all the old programs; if not the release may be very spotty.

The issue of non-commercial re-use only is also tricky; when purchasing archive footage from the BBC for MMC museum and CD-ROM projects, we often got a tape full of useful material, only to be told that rights on the bits we wanted were unknown or unavailable, and we had to re-edit.

I've heard odd tales about BBC trying to constrain rights geographically before - when they launched the BBC Choice satellite channel, they webcast it for a bit, but tried to only allow UK people to see it, which is a nonsense on the internet.

Danny has more discusion

LOAF hits

Finally an aggregation standard that can handle the nuances of forked Mornington Crescent games and WeightWatchers calculators.

Heavy Weather

That'll teach me to post weather forecasts... it rained today. (It just doesn't do that in the Bay Area in August).

I'll leave the prognostications to Rosie - she called the stock peak for Apple within 1%

Bay Area weather - perfect, with earthquakes

I read this article by Virginia Postrel before moving here, nearly 6 years ago now; it seems appropriate today.

Silicon Valley's perfect weather means you don't need backup plans, just in case it rains. It means you don't resent spending a beautiful day inside at work, because tomorrow and tomorrow and tomorrow will be just as gorgeous. It means you have more energy, sapped neither by sleep-inducing clouds nor enervating heat and humidity. It means fewer days dragging into the office with a brain dulled by allergies and winter colds. It means you have more life.

BUT AGAINST THE BEAUTIFUL blue skies of the valley sprawl its tawny hills, their curves clearly visible beneath a bare wisp of foliage. In the dry landscape of the West, the earth is not camouflaged by trees and vines and underbrush. In the valley, the ground itself is omnipresent.

And, as everyone knows, it is also unstable.
[...]
Good weather plus earthquakes creates an utterly different environment. On a day-to-day basis, you can concentrate on your goals, with no need for contingency plans. Your softball game, your picnic, your wedding won't be rained out. But everything could change in an instant. You can't anticipate earthquakes, can't plan for them, can't even predict when and where they'll strike. Instead of providing the certainty of seasons, nature promises a future of random shocks. All you can do is develop general coping skills and resources. There is nothing familiar about the aftermath of an earthquake, and no one survives it alone.
[...]
Once they hit the light, no one can anticipate just where innovations will lead--or whether they will in fact succeed. It is by trusting the search, permitting experiments whose results no one can know, that we allow advances to occur. In a 1979 paper, Wildavsky prefigured his discussion of anticipation and resilience with a meditation on the sources of progress. It depends, he suggested, on spontaneity and serendipity, on discoveries no one can predict or foresee: "Incessant search by many minds...produces more (and more valuable) knowledge than the attempt to program the paths to discovery by a single one....Not only markets rely on spontaneity; science and democracy do as well....Looking back over past performance, adherents of free science, politics, and markets argue that on average their results are better than alternatives, but they cannot say what these will be....The strength of spontaneity, its ability to seek out serendipity, is also its shortcoming--exactly what it will do, as well as precisely how it will do it, cannot be specified in advance."

Nowadays it seems that every place wants to be like Silicon Valley--to discover its secrets and copy them. Here, then, is a secret that can be copied, even in places with lousy weather and stable ground: Don't ask for answers in advance. Don't try to create a life without surprises. Trust serendipity.

The opposite of mediAgora?

How instant IM criticism tanks over-hyped movies:
The problem, they say, is teenagers who instant message their friends with their verdict on new films - sometimes while they are still in the cinema watching - and so scuppering carefully crafted marketing campaigns designed to lure audiences out to a big movie on its opening weekend.

"In the old days, there used to be a term, 'buying your gross,' " Rick Sands, chief operating officer at Miramax, told the Los Angeles Times. "You could buy your gross for the weekend and overcome bad word of mouth, because it took time to filter out into the general audience."

How I emailed myself into a job and blogged my way out of it

Joi and Ross have been talking about finding work through 'weak ties'.

I came to Apple in 1998 through just such a route - I had been a regular contributor to the QuickTime-API and QuickTime-Talk mailing lists, so when I was interviewed and hired, many of my new colleagues were people I'd been sharing software hacks, hints and jokes with for many years. The trust I had built up in them meant that I was ready to leave the UK and my successful software publishing management career to become an engineer in the US.
I had been hired with the same kind of trust, rather than with any specific job in mind, and so I always had a broader set of contacts than the org chart implied, and we would help one another out.

I carried on as before, talking about my work on mailing lists, and joining ones relevant to the areas I worked in - video editing and streaming media, among others. I had found another group of people with common interests to talk with. Gradually, however, constraints became tighter.

When you are a hardware company, you really need to keep new hardware secret to avoid the Osborne effect - that customers won't buy today because they are waiting for your newer model. Apple's secrecy and PR policies are shaped with this in mind, so when I started arguing in the abstract against DRM, I was warned not to do so using my Apple account, as it might be taken as a corporate position, or foreshadow future products.

I stopped using my Apple email address in public, and began to avoid talking about the technology I worked on - after all there are plenty of other fascinating subjects, and I didn't want to inadvertently reveal something confidential.

Gradually, almost insensibly, my interests changed. While following up on my mediAgora ideas, I got more and more involved in the details of small world networks and catastrophe theory, and different kinds of social software.

I brought some of these ideas back to work - using a wiki to describe and discuss my ideas for how to improve QuickTime. My manager liked the idea of the wiki, subsuming it into corporate processes to the point where we had meetings to discuss wiki pages, but the ideas were deferred from release to release. My work was ongoing maintenance of the existing code to keep pace with the OS and hardware changes below it, and minor feature requests for the Apps above it.

Now, when compiling on my ageing hardware took too long, or reinstalling the OS every 2 days ate into development time, I spent some downtime on my new interests, as well as helping other Apple employees as before. The work continued, but the passion was gone.

My corporate manager fired me on Friday for not meeting his goals.

Rather than fight it, I am going to take this as a cue to follow a new direction. I'll look to my new 'weak ties' for inspiration.

And I'll still be using hardware and software from Apple and QuickTime - there are a lot of good people still inside.

Old Hollywood looking for rebirth

Frank Pierson's commencement address to USC Film School is stirring, but has more rhetoric than answers:
We have to remind ourselves that this viewer is only another aspect of ourselves, that we have also in us-as he does-a better part, that needs to be cultivated and to express itself. There is no single audience with a single personality. There is the larger audience-currently under-served-that has vast variety of appetites that we can, we must, satisfy. [...]

Our defense is the farmers' market, the yard sale, the auctions. We had hopes for the Internet, but that's being turned into a marketing tool. In the field of entertainment and the arts our last defense may be Tivo and the remote control.

Liberal critics have raised the alarm over corporate censorship, the exclusion from theaters and TV of anything except what seems marketable and the eliminations of anything that might offend somebody anywhere. But the danger of censorship in America is less from business or the religious right or the self righteous left, than to self-censorship by artists themselves, who simply give up. If we can't see a way to get our story told, what is the point of trying? I wonder how many fine, inspiring ideas in every walk of life are strangled in the womb of the imagination because there's no way past the gates of commerce? [...]

You are now our future, and this is the challenge you face. It is a bigger challenge than it seems because you cannot recapture something you never knew. It is your gargantuan task to create this spirit out of thin air, in the face of resistance and lack of interest, in your own style and out of your own imagination. Something new and as yet unknown.

I'm ready when you are

Email experiments confirm six degrees, Milgram didn't

Joi points out a new scientist story about Duncan Watts' experiment that shows 6 degrees of separation, even though Milgram didn't - most of his messages didn't get through.

As well as getting this backwards, the article says:
The researchers were surprised to discover that message chains did not rely on a few highly connected individuals, so-called "hubs". Previous research by Watts and fellow Cornell mathematician Steven Strogatz had suggested that such "hubs" were important to all social chains.

This is a common misunderstanding - the hubs are a simplifications, and don't really exist; with enough elements, power law distributions are fairly smooth, and the variety of scales of connectedness is important.

Beware implicit gaussians

I've discussed Power law distributions in the context of music, box office and other charts and the danger of modelling them with gaussians - you over weight the 'top ten' hits, and ignore the 'long tail' of many smaller creators.

The stock market has a complementary problem. Options pricing is done using a model known as Black-Scholes, that has as a key parameter historical volatility, and the variance of price fluctuation is used for this. Now it has long been noticed that Stock market price fluctuations are not distributed as a gaussian - Mandelbrot pointed out they follow a scale-free Power law distribution decades ago..

However, gaussians are still used all the time in quant analysis. In this case it is the other end of the curve that fails - they see the small fluctuations and model them, but the big disturbances are far more common than is predicted. It is essentially this error that caused the LTCM collapse. The quant's dismiss this effect as acts of god, and unmodellable, but these are exactly the events that destroy your portfolio.

Malcolm Gladwell wrote a profile of Nassim Taleb, one trader who does use power laws rather than gaussians, and does well out of it.

Social Fisking

Xeni picks up on it

Fisking is normally more one-sided than this - I call the QuickTopic twist 'Social Fisking', as it can become a conversation.

The best classic Fisking I've seen was Lawmeme vs Francis.

Fisking Xeni

Xeni Jardin points to an article she wrote for Grammy magazine on compulsory licensing.

I've fisked it using QuickTopic.

If you want to join in, please do