Newsmap as a Model for Smart Aggregation
Information overload. It’s the next big issue in publishing, and technology in general. The day you have 400 e-mails in your inbox, 900 new items in your RSS aggregator, and 8 Instant Messenger windows on your screen will come. For some people, it’s already here.
With the internet still growing and changing at such a rapid rate, the raw amount of information your brain processes will see a huge increase in this decade. There’s probably even a Moore’s Law-esque equation for it. So if we are finding that the flow of information into our lives is only going up and our free time is only going down, how do we deal with this increasing imbalance?
The answer is agentry. A smart agent of sorts that sits on your desktop and acts as mediator between you and the world. If you want something from the world, your agent goes out and finds it for you. If the world wants something from you, it needs to talk to your agent first. We may have various agents in our lives already but none accomplishes what the smart agent of the future will accomplish.
The easiest part of the agent to envision is how it will handle e-mail. When you send an e-mail out, it will vouch for your reputation as a non-spammer when it talks to its agent cronies on the receiving side of the e-mail. It will also reject pieces of mail on your behalf if and when it detects there is a certain predetermined probability it is unwanted. The smart agent can learn a lot of things by examining word cluster patterns from all of your incoming e-mail. After a few weeks of training, his success rate is well into the 90th percentile. Apple’s mail program already uses adaptive learning technology to combat spam and I would say its success rate with me is about 95% with no false positives. By analyzing the proximity of words across many documents, the Apple junk mail filter is effectively assigning meaning to documents, albeit in a limited way. It is said that the Apple junk mail filter can successfully tell the difference between a spammer e-mailing you about Viagra and your grandmother e-mailing you about Viagra. Now if only if could detect what might gross you out as well…
The second major function of this agent would be to gate your instant messaging abilities. Without admitting anything, let me just say that it is very possible to waste a lot of time on instant messenger every day. Some days you feel like starting up conversations and chatting. Other times you feel like there is too much work to be done for that. And the same is true for anyone you might want to communicate with, although they may be on the opposite end of the spectrum than you on any given day. So how can a smart agent deal with this problem? By learning from you. If you start up a lot of conversations on any given day, it can infer that you’re more available. If you type a lot of keystrokes in Word or create a lot of objects in Illustrator, perhaps you’re a bit too busy to talk. So how does the agent decide who to let through and when? By learning who your most important contacts are and applying sliding scale logic depending on how busy you are and how important they are. The same thing a sports agent does when fielding requests for his client’s time and/or endorsement.
If you’re too busy to talk to someone near the middle of your “importance scale”, they will receive either a polite automated message from you, or nothing at all if you prefer. Their opening request made on instant messenger will instead be cued up in your inbox or aggregator, and you can get to it whenever you have time. If you slack, your agent will remind you to get off your ass and take care of your correspondence.
So now that your agent can handle e-mail and instant messages for you, what will it do for you on the web? That’s where Newsmap comes in.
In case you haven’t seen it yet, Newsmap is a visual representation of what’s going on in the world as aggregated by Google News and visualized by Marcos Weskamp. It may appear confusing at first, because it is. It’s clearly not smart enough to derive meaning and importance from news based on our own preferences, but it’s a step in the right direction. It illustrates the fluidity with which will can manipulate information on a page. It demonstrates how what will eventually be web services from Google can be displayed in the most non-Googlelike manner possible. Sure, right now Newsmap does all sorts of weird and counterproductive things to headlines like rotate them 90 degrees and squeeze them into an unreadable space, but what if this was a sane layout which metamorphosed productively as news arrived and your viewing habits were keening observed? What if, knowing I’m a huge Survivor fan, Newsmap always bumped Survivor-related news above other, less relevant news? What if Newsmap wasn’t a webpage at all and acted as my screensaver instead? It would be gangbusters to run Newsmap run as a screensaver and then be able to activate it by simply moving my mouse to a certain corner of the screen. Hello Macromedia… are you there? Central? Why aren’t you doing this already?
The key to our information gathering lives is all about smart aggregation. The days of media companies deciding what’s on your “front page” are numbered. Within five years, I believe customizable newsreader technology (whether client-side like Net News Wire, or server-side like Bloglines), will be as prevalent as the web is right now. The web will still be there for viewing entire bodies of content like full stories and video, but the web will not be the notification source that this content is available. Instead, it will be simple aggregators like we have today, and then eventually, creative ones like Newsmap… albeit in a much more effective form.
Whoever creates the first and/or best one of these smart agents will make a lot of money. You can bet some of the big boys are already working on it: Microsoft, Yahoo, Google, and possibly Apple or Macromedia. If I had to bet who would release something first, it would be Microsoft, but if it’s anything like Clippy The Anthropomorphized paper clip, I’ll pass. I’d really like it to be Apple because they are the gold standard for simplicity and usability, but I’m just not seeing that kind of talk coming out of the company lately. You want to sell me your .Mac service for $99 a year?
Tell you what. Be my agent. I’ll pay you triple that.
Out of curiosity, how would Newsmap be improved? Would articles that you are interested be clicked on and then shrink and disappear, or be read by your agent and digested for you? Is your agent involved with the whole process, or is it just like sifting through aggregator entries?
Hey Mike,
Great comments.
“It would be gangbusters to run Newsmap run as a screensaver and then be able to activate it by simply moving my mouse to a certain corner of the screen. Hello Macromedia… are you there? Central? Why aren’t you doing this already?”
So are you saying that we should support some type of screensaver support? I am honestly curious as to what features you would like to see in Central that support this vision of the future that you are talking about here.
After the 3rd paragraph I finally realized this was a wish list and not a review of something that had already existed.
By that point I already had taken my wallet out of my pocket and had my credit card in my hand!
Enticing…
There could also be a wi-fi motion sensor to realize when you stood up and walked away so then it would place your phone and IM on away mode.
web –
The last part of your wish has been granted (sortof). You can use a Bluetooth-enabled cellphone, a Mac running OS X, and Salling Clicker to do just that (maybe not the phone part for your desk phone, but I’m not sure). I use this setup at home and at work. It also locks my screen when I leave, too. It used to also say, “Elvis has left the building!” until my officemate starting throwing things at me.
Hi Dan,
Thanks for stopping by. I would say that yes, I would love to see screensaver support in Central. But I would also go even further than that in saying that Central’s main implementation should actually be as a screensaver.
I don’t use screensavers right now… it’s all well and good to have pictures of Anna Kournikova fading across my screen when I come back to my computer, but it really offers me no utility other than reminding me that I’m not dating Anna Kournikova.
Here is the scenario I would like to see (and one we’d definitely develop for, were it to materialize):
You install Central as a replacement for whatever screensaver you may be using, if any. The default settings would be pretty basic. Activate after 30 minutes of inactivity, and just show me some common stuff like local weather and possibly headlines. As a user, at this point, I’m already feeling like I’m getting more utility out of it than any other screensaver I’ve used. In other words, I already like Central.
Unlike other screensavers though, Central should not go away when you move your mouse. There should be a clearly labeled “Return to Desktop” button which makes it go away. What this does is let the user interact with it and add features very easily.
Pretty soon, a user may have movie times, RSS feeds, sports scores and all sorts of things displayed with the crispness of Flash and the timeliness of dynamic data feeds. So once I have my Central screen all customized to my liking, then I may want to invoke it without waiting for 30 minutes of inactivity.
This is where you begin to draw the user into making Central part of their daily routine. By offering “hot-corner” functionality where I can just slide my mouse to a corner of the screen and invoke Central, you’d instantly make Central easier to use than a browser. Why force users into this weird paradigm where you have to launch Central from the Dock or the Taskbar only to create another floating environment to deal with? I don’t want to have to worry about another application at this point… just make it my screensaver and I’ll be gently coerced into using it.
A good example for you guys at Macromedia to look at is how Apple does Fast User Switching. With one keystroke, I can completely escape whatever desktop environment I’m in, and move over to another desktop. I’d like to think of Central as another desktop (as I’m sure Macromedia would as well), so just make it easy for me to do so.
We have sent suggestions to Macromedia in the past along these lines and the response has been positive, but someone actually once suggested that we use one of the screensaver wrapper programs like ScreenTime to accomplish this. That is really missing the point. I know you guys would love to own the desktop environment, and I just think the “screensaver agent” is your silver bullet.
Excellent article – something that I’ve been thinking about for a while now. You’re right on when you say that whoever does this first will make a lot of money.
One thing that always comes to mind for me is the incredible west of bandwidth and network resources that comes from polling (or pulling even) content from the web – be it RSS feeds or other news sources. Ultimately, one day every news source needs to move to a push model where aggregation services are notified via a ping (which is done in many places already), and these aggregators in turn notify you – or in this case, your agent – of updated news. Then the agent can treat and filter incoming news stories just as it treats and filters incoming instant messages.
The guys over at PubSub are realizing this, and are preparing to do this push delivery via XMPP. Of course, that’s still only one half of the equation.
I’ve thought about the same lines myself, which is why I build memigo. Memigo is a bizarre news aggregator: it doesn’t actually hit the RSS feeds of 1000s of sites. Instead, it hits only a few dozen sites that I’ve determined to be “leading edge”, i.e. ones that pick up stories early on. And then, magic happens: the bot tries to figure out if it can get a story (right now, based on the originating site of the story), and if it can it ranks it based on the reputation of the authoring site and the referring site(s). Plus, there are social net features and the like, plus collaborative filtering and contextual alerts and all of that (hand-waving major features away). But the interesting thing is that the live feed of the bot is open: you can mail memigo, send a link via the web, have it parse your blog, whatever. In effect the memigo ‘bot’ is an agent.
And then, there is the next step, which I am probably never going to get around to writing (I got another startup to work on): have memigo on your desktop: have the memigo agent rate your news/content sources and re-dice and slice that content for you.
Information overload already was the next big thing five years ago. I guess these fashion trends just repeat themselves every couple years.
Matthias: You are right. And, people were predicting nifty little pieces of software to keep information overload in check. Hell, even Apple was dreaming out loud about this with their Knowledge Navigator (QT movie). That was 1987.
Information overload is a real problem. But instead of whiz-bang software tools we got simple, elegant solutions — news readers, weblogs, good spam filters, email search tools à la Gmail — that tackle individual problems instead of doing it all at once.
People like to anticipate revolutions in the way we use technology, but it’s easy to forget that revolution is happening all around us.
From an article by Clay Shirky, discuss…
Clay is a very insightful writer and I love a lot of what he has to say. I don’t always agree with him, but I always find his writings provocative.
I don’t see the up-to-now “failure” of smart agents that Clay sees. I see technology just now getting to the point where it can support such a concept. Don’t forget that Clay’s article was written in June of 1999… before the advent of a lot of the agents we are already starting to see. His opinion back then was probably based on false starts by Microsoft, PointCast, and many other companies who tried to rush things out before smart technology was mature. As I mentioned in the original post, Apple’s junk mail filter and all other filters which act as it does, are already smart agents. I train it to accept or reject mail in the future on my behalf by helping it understand what I consider junk. It comes out of the box “pre-educated” to a certain degree, but it only gets better as it begins to know me. This sort of functionality wasn’t around in 1999 because spam was not even 1/100th of the problem it is now.
Another example of an agent already living on my machine is LaunchBar. If anyone out there is using OS X and hasn’t tried LaunchBar yet, please stop reading this comment and install it. LaunchBar provides keystroke access to every single application, document, MP3, video file, URL, or e-mail address which is stored on my computer. Instead of me traversing through my directories to find a paper I wrote on advertising, I just type in “command-space ADV” and the paper comes up. The word-matching algorithms are impressive, but what makes this program truly smart is that it learns from my behavior. If I have Adobe Photoshop and Adobe Illustrator on my machine, I may type in “ADO”, to which LaunchBar will ask which Adobe program I want to launch. After I make my choice, LaunchBar learns and defaults me to that program upon future “ADO” keystrokes. In other words, it intercepts my keystrokes and acts on my behalf, knowing what I’m probably trying to accomplish.
I think Clay is right in that there are certain scenarios where I really don’t want an agent doing my thinking for me. These are typically the types of things I already don’t use the web for: dating women, picking out clothes, etc. There are certain things in life which are so subjective that it is really difficult to take the human element out of them. That said, I’m sure some people wouldn’t mind having their agent set up blind dates for them.
Clay’s other scenario of buying plane tickets actually makes perfect sense to me. Here is the permanent set of rules I’d give my agent for buying tickets:
These are all things I currently do manually on the web, and I don’t enjoy doing them. They are also all things which can be expressed programatically and dealt with appropriately by an agent.
So I guess what it comes down to is that I agree with Clay in that I don’t necessarily want an agent to do all of my “thinking” for me… but much of my day is spent not really “thinking” at all but rather performing tasks a computer could easily do.
The more of my time that is freed up for either critical thought or recreational activities, the happier and more productive I will be.
Thanks for your detailed and informative response. I’ve let my thoughts simmer a bit, and this is what I’ve cooked up.
First, your definitions of smart agents differ:
ClayWeb crawling agents as opposed to stored preferences in a databaseMikeSits on your desktop and acts as a mediator between you and the world
With this in mind, it can be seen how both your examples of smart agentry sidestep Clay’s constraints:
Spam filters mediate between the user and an essentially local resource (their mail) so Clay doesn’t really apply at all.
Newsmap would appear to work by piggy backing off a BFW as described by Clay.
Could buying plane tickets be made to work? Rule 1 probably isn’t too much of an issue, as I think the number of physical flights are not growing that fast. Rules 2 and 3, however, would suggest that the only viable solution is another BFW acting as an aggregator. Who ever runs this service would have to deal with the problems that Clay describes, but at least they can do this from a business context, building up relationships and service levels with the providers. I guess this is already happening to a large extent, there is just no super-aggregator that covers all airlines.
p.s. happier, more productive, a Radiohead fan perchance?
Hey, I someone’s told me how to run NewsMap as a screensaver (or any web page for that matter): use this free utility: http://leong.com.au/WebFlash/
Have you seen Findory? It’s a first step toward your information agent.
Findory is a personalized newspaper and weblog reader. It learns from your interests, searches thousands of sources, and builds a front page just for you.
In a sea of information, Findory provides focus. It helps you find the news you would otherwise miss.
I’d be curious to hear your thoughts on it, Mike.
You may find this site interesting.
http://news-bot.net/
Smart Aggregation, Electronic Agents, Online Secretaries
Mike Davidson’s article Newsmap as a Model for Smart Aggregation provides an insightful view of the future of information retrieval, presentation and use. I readily admit to being an information junkie – I’m subscribed to several lists (evolt, WebDes…
Smart Aggregation
Is the era of Smart Aggregation upon us?…
Davidson on agents
Davidson on agents: Mike Davidson (welcome! 8) writes here of having the computer tailor its results to you, and ties it in with the Newsmap tool. Some agents live on a remote machine (Findory, eg), while others live on the…
PubSub Gets Ready For Push
News delivery and instant messaging are going to merge – it’s only a matter of time.
newsmap
Very interesting: newsmap. Check it out and read Mike's blog post. I need to read a bit closer before discussing further though (I thought it was so cool that I'm posting it right away!). A bit of the discussion is about a push model for ne
Newsmap
I love this graphical representation of what’s hot in the news: newsmap. [from newsmap description] Newsmap is an application that visually reflects the constantly changing landscape of the Google News news aggregator. A treemap visualization algorithm…
Great RSS Quotes from My Aggregator
RSS: a Shift, from What…to What? “He also neatly sort of answers his own question – with greater precision than I can ever muster – by saying: ‘If I visit houses of content, as I seem to do on…
The geometry of the news
Newsmap is a graphic representation by Marcos Weskamp of the shape of the news of the moment and of the day.
Smart Aggregation – The Next Big Step in Desktop Info Management?
Mike Davidson who works in media product development for Disney initiates an interesting discussion on information overload and how a smart aggregator could be the answer to the enormous amounts of data we will (and some are) processi