Scoble has been writing about RSS bandwidth concerns lately, so I thought I’d once again post on this topic. I’ve posted before about using conditional HTTP Get (If-Modified-Since) to decrease RSS bandwidth consumption, but here’s a simple recap of how this works:
Almost all aggregators store the date/time that a feed was last updated, and they pass this to the HTTP server via the If-Modified-Since HTTP header the next time they request the feed. If the feed hasn’t changed since that date/time, the server returns an HTTP status code 304 to let the aggregator know the feed hasn’t changed. So, the feed isn’t re-downloaded when it hasn’t changed, resulting in very little unnecessary bandwidth usage.
This sounds simple enough, but there’s a big problem here: many high-traffic RSS feeds are created dynamically through server-side code, and the HTTP server won’t automatically support conditional HTTP get for dynamic feeds. So, all too often the feed is rebuilt each and every time it’s requested – which is obviously a huge waste of both bandwidth and CPU time. One solution is to write your own code to return a 304 based on the If-Modified-Since header, but in many cases it makes more sense to use a static feed that’s rebuilt only when new information needs to be added to it. For example, my FeedDemon FAQ feed is a static RSS file that’s rebuilt whenever I add a new entry to the FeedDemon FAQ. This way, my HTTP server takes care of the If-Modified-Since comparison, and there’s no unnecessary regeneration of the feed.
However, while this works well for feeds that don’t require many updates, it’s not the best approach for feeds that need to be updated more frequently. This is the problem I faced with my support forum feeds, which are created dynamically from information stored in a SQL Server database. Since new forums posts are often made every few minutes, I decided to use server-side code to limit how often aggregators can download the feeds. Almost all aggregators support conditional HTTP get, so I simply check the If-Modified-Since date/time, and if it’s within the last 15 minutes I return a 304 to tell the aggregator the feed hasn’t changed – even if it has. This prevents aggregators from downloading the entire feed more often than once every 15 minutes.
Here’s a snippet of the ASP.NET code I use to do this:
Dim dtNowUnc As DateTime = DateTime.Now().ToUniversalTime
Dim sDtModHdr = Request.Headers.Get("If-Modified-Since")
' does header contain If-Modified-Since?
If (sDtModHdr "") And IsDate(sDtModHdr) Then
' convert to UNC date
Dim dtModHdrUnc As DateTime = Convert.ToDateTime(sDtModHdr).ToUniversalTime
' if it was within the last 15 minutes, return 304 and exit
If DateTime.Compare(dtModHdrUnc, dtNowUnc.AddMinutes(-15)) > 0 Then
Response.StatusCode = 304
Response.End()
Exit Sub
End If
End If
' add Last-modified to header - FeedDemon stores this with cached feed so it's
' passed to the server the next time the feed is updated
Response.AddHeader("Last-modified", dtNowUnc.ToString("r"))
Now, I’ll be the first to admit it’s not the most elegant hack, but so far it has worked very well for me. I considered checking the date/time of the most recent forum post and using that for the If-Modified-Since comparison, but that would’ve required a database hit each time the feed was requested, so I opted for the less precise but more CPU-friendly solution.