July 22, 2004

Fixing InfoWorld's RSS growing pains

As InfoWorld's CTO, Chad Dickerson, posted here, InfoWorld has been noticing a lot of congestion at the top of the hour when thousands of RSS clients all hit our servers simultaneously to check for updated feeds. We've done a number of things to alleviate this, but thanks to excellent feedback from the RSS community we've done a lot more today that will hopefully resolve some of these issues.

Some of the links from Chad's blog that helped in our debugging:


  • Slashdot discussion about Chad's column and RSS traffic patterns in general, as well as comments from Brooks Talley and John Allspaw (both former InfoWorlders) and myself
  • Dare Obasanjo suggests gzip encoding and conditional GET, but notes: "The one thing that HTTP doesn't provide is a way for clients to deal with numerous connections being made to the site at once." There are methods to deal with that, of course (more servers, CDN services like Akamai and Speedera, etc.) but those solutions smack of mindlessly adding more lanes to the freeway instead of doing the hard work of analyzing the traffic problem and working on the fundamental issues.
  • Sam Ruby points back to Dare Obasanjo's suggestions and notes: The functionality is clearly there in HTTP.  The word is clearly not getting out to everywhere it should be.
  • Nick Bradbury of FeedDemon fame add his voice to the chorus, and also notes that FeedDemon only checks feeds every three hours by default
  • Phil Windley: None of these problems are unsolvable and frankly, its nice to have scalability problems. It's a sign of success. (Agreed!)

    • The solutions we implemented today:

  • We are now sending proper Expires info, so that caching servers etc can follow them.
  • The missing ETags was actually an issue related to our less than spectacular previous content management system, Dynabase, that generated all of our old articles in HTML, but used a .XML file extension.
  • Those XML files had Includes, and you can't do both ETags and Includes together. We renamed the old .XMLs to .HTML, so we now don't need to run .XML files through the Includes processor, so the ETags are back working.

    It's definitely been a productive day. We'll see over the next few weeks how this impacts our requests load at the top of the hour. =)

    Posted by Kevin Railsback at July 22, 2004 05:16 PM | TrackBack
    • Comments

    Just a thought, probably already suggested, but why not set up a page for each of the popular aggregators with screenshots of how to change the update time. Then pump some items down your RSS feeds saying "important info for users of aggregator X".

    Nagging the developers to make the time more random might help eventually too...

    Posted by: Danny at July 24, 2004 12:55 AM