Epeus' epigone

Edifying exquisite equine entrapments

Sunday, 20 October 2002

Congestion control for RSS

Dave Winer, Joel Spolsky and Phil Ringnalda are discussing the problem of RSS aggregators that check for updates by polling becoming an effective Distributed Denial of Service attack.

As others have said, adopting HTTP's "If-modified-since" timestamp fetch can help here, by only doing a full-page fetch when the RSS has changed. In addition, adopting RFC 3299's way of only sending changes will help reduce the bandwidth of the RSS fetches (I mentioned this back in January when it first came out).

However, this doesn't reduce the number of HTTP setup/teardowns. To do this, the aggregators need to get smarter. They can do this by estimating an update frequency for each feed - something modelled on TCP's congestion control (exponential back-off, with 'no change' treated as congestion) would probably suit well.
If the aggregator polls the feed, and finds no changes, it doubles the polling interval. If it polls and finds changes, it decrements the polling interval by the number of changes found multiplied by the overall polling frequency. The lower bound is the maximum polling frequency set by the user (once an hour is common). You could set an upper bound, or let it establish itself which blogs are moribund.
Posted by Kevin Marks at 12:02

No comments:

Post a Comment

Newer Post Older Post Home
Subscribe to: Post Comments (Atom)

This is my personal blog. Any views you read here are mine, and not my employers'.

Atom Feed

Support the Open Rights Group
My photoKevin Marks Me on Twitter
Me on G+

People's thoughts I read:

Daily

Rosie
San Jose Young People's Theatre
Dave Weinberger
Doc Searls
Gonzo Engaged
AKMA
Cory & friends
Denise Howell
Charles Wiltgen
Shelley Powers
James Lileks
Suw Charman
Halley Suitt

Weekly

Andrew Marks
Blogsisters
Arts & Letters Daily
Bricklin, Frankston & Reed
Steve Yost
Jeneane Sessum
Brian Micklethwait et al
Tom Matrullo
Gary Turner

Sporadically

Small Pieces
Stuart Cheshire
RageBoy
Nonzero
Neil Gaiman
Thomas Vincent
Brad deLong
Andrew Odlyzko
ProSUA

No to Mickey Mouse Computers

powered by blogger

Blog Archive

  • ►  2017 (2)
    • ►  May (1)
    • ►  April (1)
  • ►  2015 (7)
    • ►  November (2)
    • ►  May (3)
    • ►  April (1)
    • ►  January (1)
  • ►  2014 (3)
    • ►  October (1)
    • ►  April (2)
  • ►  2013 (5)
    • ►  June (1)
    • ►  May (1)
    • ►  April (2)
    • ►  March (1)
  • ►  2012 (8)
    • ►  December (1)
    • ►  May (1)
    • ►  April (1)
    • ►  March (1)
    • ►  January (4)
  • ►  2011 (11)
    • ►  December (1)
    • ►  November (1)
    • ►  September (2)
    • ►  August (2)
    • ►  July (1)
    • ►  April (2)
    • ►  January (2)
  • ►  2010 (16)
    • ►  November (1)
    • ►  October (1)
    • ►  September (3)
    • ►  June (1)
    • ►  May (2)
    • ►  April (2)
    • ►  March (2)
    • ►  February (2)
    • ►  January (2)
  • ►  2009 (22)
    • ►  November (2)
    • ►  October (2)
    • ►  September (2)
    • ►  August (3)
    • ►  July (2)
    • ►  June (2)
    • ►  May (2)
    • ►  April (1)
    • ►  February (2)
    • ►  January (4)
  • ►  2008 (28)
    • ►  December (2)
    • ►  November (3)
    • ►  August (1)
    • ►  July (2)
    • ►  June (3)
    • ►  May (5)
    • ►  April (2)
    • ►  February (3)
    • ►  January (7)
  • ►  2007 (45)
    • ►  November (3)
    • ►  October (4)
    • ►  September (4)
    • ►  August (10)
    • ►  July (3)
    • ►  June (8)
    • ►  April (2)
    • ►  March (6)
    • ►  February (3)
    • ►  January (2)
  • ►  2006 (119)
    • ►  December (13)
    • ►  November (8)
    • ►  October (16)
    • ►  September (10)
    • ►  August (3)
    • ►  July (6)
    • ►  June (24)
    • ►  May (3)
    • ►  April (10)
    • ►  March (7)
    • ►  February (8)
    • ►  January (11)
  • ►  2005 (101)
    • ►  December (10)
    • ►  November (13)
    • ►  October (9)
    • ►  September (8)
    • ►  August (7)
    • ►  July (7)
    • ►  June (8)
    • ►  May (12)
    • ►  April (7)
    • ►  March (6)
    • ►  February (1)
    • ►  January (13)
  • ►  2004 (53)
    • ►  December (8)
    • ►  November (5)
    • ►  October (6)
    • ►  September (7)
    • ►  July (5)
    • ►  June (3)
    • ►  May (2)
    • ►  March (3)
    • ►  February (7)
    • ►  January (7)
  • ►  2003 (196)
    • ►  December (12)
    • ►  November (14)
    • ►  October (21)
    • ►  September (23)
    • ►  August (19)
    • ►  July (11)
    • ►  June (14)
    • ►  May (9)
    • ►  April (22)
    • ►  March (20)
    • ►  February (16)
    • ►  January (15)
  • ▼  2002 (224)
    • ►  December (15)
    • ►  November (21)
    • ▼  October (22)
      • AlienAid - Guy Fawkes and Halloween - London, UK ...
      • AlienAid - Telephone Numbers - London, UK to Bay A...
      • Has Google stopped spidering?
      • AlienAid - Bacon - London, UK to Bay Area, CA, US
      • AlienAid - Tea - London, UK to Bay Area, CA, US
      • AlienAid
      • Oops - Daylight saved
      • Congestion control for RSS
      • The Fallacy of the Almost-General-Purpose Computer
      • Neal Stephenson: Jipi and the Paranoid Chip
      • Sorry Eric!
      • Digital Identity through tone of voice
      • Trust people, not computers!
      • Gary Loveman Harrahs COO
      • The Strategy that Works - Larry Downes
      • Dan'l Lewin on .NET
      • Panel Andy Grove and Clayton Christensen
      • Clayton Christenson - The Innovators Solution
      • interactive conference?
      • Digital Choice and Freedom
      • Conferring today
      • Quote of the day
    • ►  September (12)
    • ►  August (11)
    • ►  July (28)
    • ►  June (19)
    • ►  May (29)
    • ►  April (18)
    • ►  March (19)
    • ►  February (16)
    • ►  January (14)
  • ►  2001 (13)
    • ►  December (2)
    • ►  November (11)

About Me

My photo
Kevin Marks
Kevin Marks works on IndieWeb and open web tech. From 2011 to 2013 he was VP of Open Cloud Standards at Salesforce. From 2009 to 2010 he was VP of Web Services at BT. From 2007 to 2009, he worked at Google on OpenSocial. From 2003 to 2007 he was Principal Engineer at Technorati responsible for the spiders that make sense of the web and track millions of blogs daily. He has been inventing and innovating for over 25 years in emerging technologies where people, media and computers meet. Before joining Technorati, Kevin spent 5 years in the QuickTime Engineering team at Apple, building video capture and live streaming into OS X. He was a founder of The Multimedia Corporation in the UK, where he served as Production Manager and Executive Producer, shipping million-selling products and winning International awards. He has a Masters degree in Physics from Cambridge University and is a BBC-qualified Video Engineer. One of the driving forces behind microformats.org, he regularly speaks at conferences and symposia on emergent net technologies and their cultural impact.
View my complete profile