#SMOOC Day 5: The Filter

As web companies strive to tailor their services (including news and search results) to our personal tastes, there's a dangerous unintended consequence: We get trapped in a "filter bubble" and don't get exposed to information that could challenge or broaden our worldview. Eli Pariser argues powerfully that this will ultimately prove to be bad for us and bad for democracy.

Pioneering online organizer Eli Pariser is the author of "The Filter Bubble," about how personalized search might be narrowing our worldview.

I've recently stumbled upon a TED talk by Eli Pariser titled "Beware inline filter bubbles". He talks about how modern content delivery services try to deliver personalized relevant information, and filter out the rest, thus, limiting the diversity of content we consume. He calls it a "filter bubble."

I find this topic very interesting because I'm currently developing a similar service myself - http://6feeds.com. It's a service that will recommend you news based on you recent activity: what you've read, shared etc; and help you discover new content. I've started working on it, because I've noticed, that the amount of stories I have to skim through to find something interesting, is just too large. I thought that this process could be optimized with the help of modern technologies.

NYU new-media professor Clay Shirky accurately describes the problem of information overload in Internet economics by focusing on the real problem: filter failure.

If you haven't watched it, you must. It does more to explain the dearth of effective information filters that we wade through today. It has application to open source (180,000-plus projects on SourceForge, but which are useful?), but far broader implications.
Here is where he lays out the crux of the problem, which I've transcribed:

This is not a real-time reverse-chronological news feed. There is no concept of followers. Rather, imagine a set of posts with metadata. This set is continuously increasing. Each user has liked/disliked these posts in different ways. There is a black box ranking algorithm which ranks these posts to generate a personalized feed for each user. The ranking algorithm is designed in a way such that each user's action affects the ranking of all other users, sometimes in a big way, sometimes in a small way. So I am guessing a fan-out-on-read approach will be most feasible. I was wondering whether there are other design issues I should be aware of and other strategies that can be used. Obviously, I want to optimize for generating the top X posts in a user's personalized feed.

Friday, January 14, 2011

The slides is for an internal presentation Dan McKinley did about building Etsy's activity feed.

Publishing Methods

"Push" Model, or Fan-out-on-write

This method involves denormalizing the user's activity data and pushing the metadata to all the user's friends at the time it occurs. You store only one copy of the data as in the schema above, then push pointers to friends with the metadata. The problem with this method is that if you have a large fan-out (a large number of followers), you run the risk of this breaking while your feed accumulates a backlog. If you go with this strategy, you also risk a large number of disk seeks and random writes. You'll want some sort of write-optimized data store such as Cassandra, HBase, or BigTable.

"Pull" Model, or Fan-out-on-load

This method involves keeping all recent activity data in memory and pulling in (or fanning out) that data at the time a user loads their home page. Data doesn't need to be pushed out to all subscribers as soon as it happens, so no back-log and no disk seeks. The problem with this method is that you may fail to generate a user's news feed altogether. To mitigate this risk, you should have a fallback mechanism in place that approximates the user's feed or serves as a good alternative.