How To Keep Widgets From Slowing Down Sites: WEDJE
The whole world is going to widgets. This overused, overhyped term refers to third-party code one places on their website or blog in order to display such things as Flickr photos, Twitter status, or iTunes playlists. Everybody and their mom is putting out widgets these days, and although only about 1% of them are useful or interesting, they are an important new distribution mechanism that is changing the way companies think about syndication.
But there is a big problem with widgets: they slow down the sites that use them. In the best case, you have a company like Flickr whose servers are almost always snappy, and in the worst case, you have a young startup that is constantly struggling against increasing demand and occasionally can’t serve up any code at all.
The problem is that in either of these cases, the completion of your site’s loading and rendering depends on someone else’s code living on someone else’s server. Including a fast and reliable Flickr widget still slows your site down by at least a split second and including a less stable one can leave your site hanging indefinitely.
We’ve been developing what we think is going to be a gangbusters widget at Newsvine over the last few months but just as we were getting ready to deploy it, Intern Rob and I hacked together what we think is a method of deploying widgets in such a fashion that they don’t affect the load times of their parent sites whatsoever.
Following is a breakdown:
Root Causes
Most widgets are deployed using one of two methods: raw embedding or javascript embedding.
Raw embedding usually involves providing a chunk of object/embed code to display a Flash widget inline. The advantage of this method is that it works in environments like MySpace which don’t allow the use of javascript. The disadvantages are that a) many people don’t like placing raw Flash code in their pages, b) some widgets aren’t Flash-based, and c) the widget author can’t change much after the fact since the widget’s code is more or less set in stone.
Javascript embedding involves using a script include to remotely deploy code into the user’s site. The advantage is that it’s much cleaner and more flexible. The disadvantage is that it slows sites down a little every time and a lot when there are problems.
The Solution: WEDJE
Providing a raw embed method is still a necessity if your target environment is MySpace, but if you’re going after people with their own sites/blogs, javascript embedding is still a better ideal. We need a catchy acronym for our method in order for people to use it so we’re going with: WEDJE. Or, “Widget Enabled DOM Javascript Embedding”… which is an acronym inside of an acronym, so it’s extra cool.
In a nutshell, here are what the two existing javascript methods do versus what WEDJE does:
Standard document.write
Most javascript embedding simply uses document.write to write code into your site inline. The problem is that if the widget server hangs, the rest of your site can’t draw until the script is done executing. This could be a split second or it could be forever. Many people mistakenly think you can use the defer attribute in your script tag to keep this from happening, but defer effectively kills the document.write function entirely so this is not a solution.
Deferred innerHTML code
The other method of javascript embedding is to write out a div with an ID, load the javascript with the defer parameter added, and then use innerHTML to write to the div whenever the script has loaded. This seems great in theory, but in reality, Internet Explorer is the only browser which seems to properly render the rest of the page before the script is done executing.
WEDJE
WEDJE is similar to the innerHTML method above except it creates what is effectively a cross-platform, cross-browser defer, enabling your script to load and execute asynchronously across all environments. We write out a div with javascript, then we create a script element with javascript, and then we append the script element to the div, again with javascript.
By linking elements together in this way, browsers appear to completely decouple the loading and execution of our attached javascript from the loading and execution of the original document, which is exactly what we’re looking for.
An example of WEDJE in action is below. Reload this page and watch the area below. There is a 7 second delay in the execution of the javascript file in order to simulate a slow-loading script:
This text is before the widget code.
This text is after the widget code.
Notice how everything loads regardless of the state of the widget? Voila.
Caveats
Intern Rob and I are under no illusions that this is rocket science, nor that it hasn’t already been tried by someone else, nor that it works perfectly. To the last point, we’d love your feedback as to what could be improved or where it may not be working. So far, tested browsers are:
- Safari: Mac and PC
- Firefox: Mac and PC
- Internet Explorer: PC
- Opera: Mac and PC
1. Here is the sample code which goes in the js file:
2. Here is the sample code to embed in your page:
Interesting. I’ve always wondered what effects those fancy little widgets had on my loading times, I guess this confirms it. WEDJE sounds good.
Thank god someone’s thinking about this. Anyone from Technorati reading this? (hint hint: link count)
Hahah. Leave it to Mike D to coin a term :) If I use this method to embed an Atom feed widget, does that make it an Atomic WEDJE?
Another caveat — the external script cannot itself use document.write to deliver it’s code. This shouldn’t be a problem because innerHTML’ing the widgetholder anchor div inserts the code exactly where document.write would. But it does make it difficult to adapt already existing scripts that use document.write to use the WEDJE insert method. And it also means that the external script has to specifically reference the div that’s been written by the inline script on the target site.
I spent about an hour last night trying to adapt the WEDJE method to work with javascript Flickr badges to no avail — you just can’t use document.write in the external script as far as I can tell.
Very cool. In Sandvox we call these things “pagelets” since “widget”, to a Mac user, is something that goes into Dashboard.
I like the idea here but there are some issues with document.write in XHTML… A google search reveals some explanation of the issue.
So I’m wondering if there is a way to avoid the document.write and do the operation completley with DOM operations….
Dan: Yet another reason why serving pages up with an XML mimetype seems pointless to me. So much downside, so little upside… for now at least. :)
The original version of WEDJE (which is to say, the version we cooked up last night), wrote the container div out as part of the standard HTML, so you could always do that I suppose.
Dan — you can avoid the document.write by simple including the widgetholder div right above the script block — That was how we originally hacked this thing together. Mike had the good idea to include the div inside the script itself to make the whole thing a bite more self-contained, but the document.write part of the script isn’t essential if you want to avoid it.
I’m pretty sure you need some sort of tag with a unique id on the target site to reference with the external script — that DOM hook is what allows the browser to go back and deposit the external script’s embedded content after it’s moved on to render the rest of the page.
You could create the div wrapper with DOM methods as well. To do so, I would do the following:
Here it is in code:
wrapper = document.createElement(“div”);
wrapper.id = wrapperID;
widgetLocation = document.getlementById(“scriptid”)
widgetLocation.parentNode.insertBefore(wrapper, widgetLocation);
Scott: Awesome! I will test that across the browsers listed above and then merge it into the code if it’s all good. Good call.
I like more Scott’s method to create the widget div, since it doesn’t use document.write(), which can’t be used with XHTML served as application/xhtml+xml :)
I think Scott’s method is fantastic. It also means less code has to be copied and pasted onto the target site.
Cool concept, guys!
I’m glad you got the document.write out of there with Scott’s help — but doesn’t innerHTML *also* choke on XHTML pages served as XML? I thought so, but I could be wrong.
I’m with Mike: this is just one more reason not to do XHTML, and especially XHTML served as XML. But the fact is that some people do, so if WEDJE could work with XML-served pages, that would be awesome.
Jeff — I created a few test pages to look at serving XHTML as XML. There aren’t any problems serving XHTML with the xml declaration plus the application/xhtml+xml content type — as long as the document has a .html extension. If you use a .xml or .xhtml extension then the external innerHTML doesn’t work.
Here’s one of my test pages that demonstrates WEDJE + an external innerHTML script not working in an XHTML as XML doc with a .xhtml extension
Here’s the exact same code working with a .html extension
Fortunately, the code thats sitting on the end user’s site doesn’t use innerHTML. So if you were concerned about putting your widget on sites created by XHTML as XML claquers, you could use a different (and slower) insertion method in the external script.
I’ll do some more tests to see if I can get WEDJE working on XHTML as XML by not using innerHTML in the remote script. Thankfully I think there are few sites out there that serve up pages with .xml or .xhtml extensions.
How about a browser compatibility chart to go with all this?
Heya, page seems to stall in Opera 9/Mac while the code executes – it gets to “This text is before the widget code.” then there’s a long pause before both the rest of the page and the widget show up.
Opera 9.20, Build 3669 on MacOS X (10.4.9).
@Rob: Nice. I stand corrected. :)
So I may merge in Scott’s method, but it made me think of something: If you’re going to commit to doing that, you pretty much also have to commit to getting rid of innerHTML in the js file and always using DOM manipulation to insert every single part of the widget. It’s certainly not impossible, but I don’t know if I like the cost/benefit there. Regardless, I suppose it doesn’t hurt to at least set the inline stuff up like Scott suggests and then let widget authors choose to use DOM manipulation or not on their own.
Geoff: Good idea. I’ll work a chart up.
Ben: Damnit. Leave it to Opera to act completely unlike every other browser and ruin the party. I’m sorry but as a developer, I *hate* that browser. Hate, hate, hate. Anyway, at least WEDJE doesn’t work any *worse* in Opera than standard embedding methods. It just doesn’t have the advantages it has in other browsers.
Great idea and I definitely hope this can be perfected to work with all modern browsers. Unfortunately it’s not working on Firefox 2.0.0.4 for Windows XP (at least on my laptop). I tested it in Safe Mode to be sure none of my extensions were causing the problem, but I’m still seeing your page initially load up to the last comment, and then the rest of the page loads after the widget loads.
John: I think (but am not sure) that this has more to do with the fact that I have a bunch of onload stuff going on on my site (e.g. sIFR). Content after the widget is indeed loading, but certain stuff isn’t rendering. I’ll investigate further, but it would still appear to be an improvement over normal embedding.
Ah yes Mike you’re exactly right… the page is completely loading quickly as it should, it’s just not rendering completely initially. Definitely an improvement over normal embedding.
During a reload on this page, Camino (1.5) doesn’t render the right sidebar until the 7-second widget delay expires.
One thing you could try is using a setTimeout to get the javascript in the WEDJE to run asynchronously, eg:
setTimeout(function(){
// wedje here
}, 0);
I found this great technique on Isaac Schlueter’s blog, you can read more about it here: http://isaacschlueter.com/2006/10/settimeout-for-fat-inits/
You could also use an iframe. I find this technically superior because it causes no delay and doesnt involve javascript. If done properly preserves your pages layout if the widget changes.
Peter: Good call! I just merged the setTimeout in and now it works in Opera. Firefox/Camino/et-al now also render more fully, while Opera renders as Firefox used to before the setTimeout improvement. Getting better!
Somebody want to rewrite these two blocks of code as object oriented JS? It would look nicer and variable names could be shortened because it would lessen the chance for conflicts.
John: Yeah, iframes are nice but they require you to set the width and height and they are also not as accessible from the parent document’s DOM. Still, there are definitely good uses for them, and setting width and height can be seen as an advantage in some cases.
If you formalized the way it all works, you could use dean edwards packer to get one final speed boost out of it in the end (depending on if its size is worth it) http://dean.edwards.name/packer/
Seems to work fine in on Linux: tested Firefox 2.0.0.3, Konqueror 3.5.6, and Opera 9.2.0.
In theory, this could be used for stats scripts as well, like Google Analytics, crazyegg, etc. Has anyone tried this? To defer the loading of the script with setTimeout?
Not rocket science, if you’re into Javascript, but like you said, more and more people are using it that don’t care about learning javascript only to include a single widget. Like always, you’re able to provide a step by step plan how to help them. Well done.
The target javascript at step 1. will not work with patched IE7 (not sure about IE6 but I’d assume a fully patched IE6 does the same?). All you get is the very helpful and informative “Unknown runtime error” message. This is because you can’t write to an .innerHTML off a scripted element.
To get things to work with IE7 and not break FF2 use:
var myimg=document.createElement(‘img’);
myimg.src = “widgetexample.gif”;
myimg.width = 160;
myimg.height = 380;
document.getElementById(‘wedje_div_yourcompanyname’).appendChild(myimg);
I was just fooling around, try this out:
// we want to encapsulate this wedje within its own scope so that
// if we have multiple on one page, we don’t redefine their variables.
// also, global variables are evil!
(function() {
var wedje_id = ‘wedje_’ + Math.random();
document.write(”);
// a lambda is used because passing a string is essentially doing
// an eval, and given a wedje, there *could* be a possibility of xss..
// that is unlikely, but even so, eval’s are evil!
setTimeout(function(){
try {
var wedje = document.createElement(‘script’);
wedje.src = ‘https://mikeindustries.com/blog/scripts/sleepywidget.php’;
document.getElementById(wedje_id).appendChild(wedje);
} catch(e) {
// error loading wedje
}
},1);
})();
With comments it’s a bit longer than yours, but there are a few things (some mentioned in the comments) that I wanted to draw attention to..
– reusability: Using a random wedge div id lets one have more than one wedje on their page
– scoping: I put everything within a self-calling function so that assuming multiple wedjes are on a single page, their variable definitions are not passed to global scope and also don’t overwrite each other.
One thing I forgot:
Pass the div id as a query argument to your .php script, eg:
wedje.src = ‘https://mikeindustries.com/blog/scripts/sleepywidget.php?id=’+wedje_id;
Why not just use an Iframe to display the widget? The iframe will not slow the load of the parent page, even if their script fails to load it will be restricted to the iframe versus the full page. I will keep any 3rd party scripts from executing on you page/site (which creates a security risk – like cookie reading/setting etc).
Mike,
In your Application/XML Mime-Type Version example, you’ve got the ” before the ) right near the end.
yourcompanyname “),1);
(Editor’s Note: Thanks. Fixed!)
i sorta understand what you are saying… i no doubt feel that widgets are teh sole reason to my site loading slower than html2.0. but i really dont understand how to impliment this system on my site.
i installed a wordpress client on my webspace hosted by powweb, and am not real savvy on this sorta stuff. any idea where i would put the code(s) and what codes? becaues i would love to be able to delay all this stuff to a degree so my site loads super fast again.
Medhi — its up to the creators of your specific widgets to provide embed codes that use WEDJE. The external scripts you’re loading in probably haven’t been written in this fashion, so you’re likely out of luck for the time being :(
document.getElementById(“wedje_div_yourcompanyname”) has no properties
https://mikeindustries.com/blog/scripts/sleepywidget.php
Line 1
I’m getting a javascript error in ffox winxp. Did someone break it?
I’m getting the same Javascript error (FF/Mac)
Yeah, me too, now I don’t believe that it really does work unti I see it.
One thing to note, I did not create a div in my example code using your current company name scheme (so it won’t work until sleepywidget.php) is changed. I did this because it means one less thing to change. If anything, I think it would be best if everything but the url were generally standard across all wedjes, what do you think?
This looks like a good solution – and thanks to Peter for mentioning Packer – I’ve been thinking I would have to write something like it for a project I’m working on, now I don’t.
I am, however, getting the same error as George, Thomas, and Julian, using FF 2.0.0.4 on XP Home SP2, and I can’t see the widget at all.
Sorry everyone. I broke it last night because I was too lazy to test on a PC! Everything is all fixed now including the weird IE error. All good in the hood. Works in all reasonable browsers to my knowledge.
Well, until this is world standard, is anyone working on blocking the things, whatever you call them?
I am an older guy, just retired, and just as soon as I get DSL on a G4/Safari, every website gets so busy with these things that they slow my system to a crawl.
It’s really aggravating.
Sorry.
Mouse is double-clicking sometimes.
Lou, do you use Greasemonkey?
No. That’s something that works with Firefox, right?
I have been waiting for Leopard to come out so that I can load Firefox 2.0. I still have Panther 10.3.9 so I am waiting, but it is aggravating because I about half expect Craig’s List to ask me to update my browser.
It looks like Greasemonkdy could help me out. Is that true?
Thank you! This worked like a charm and actually got rid of the IE flash alert problem too.
Ok, I just rewrote the embedded code as an anonymous function now so it’s a bit more compact and eliminates the chance for variable conflicts.
So as of now, WEDJE appears to be pretty optimized. The last remaining problem is that the XML version seems to be throwing an error in IE. If anyone has any ideas about that, let me know.
http://www.matthom.com/archive/2007/03/22/javascript-badges-should-reside-on-the-bottom-of-the-html-page
I was going to say this earlier, but i forgot…
if the person puts the information at the bottom of their page wouldn’t it not effect the loadtime?
Great article! Love your site!
Sorry to say, but this doesn’t work for me (Win XP, FF 2.0.0.4).
I experience a 7 second delay upon reload with a partially rendered page before the page completes and scrolls back to the widget’s position.
I’ve posted a screenshot of Firebug’s network timing output.
Not a very good alternative to server-side includes, since JavaScript inclusion excludes a great portion of web users, including search robots like Google. Also, the ‘language’ attribute on the ‘script’ element has been deprecated for almost 10 years. About time to get rid of it, don’t you think?
This is pretty awesome, google should use this with their js such as adsense and analytics.
Did anyone try multiple widgets on one page?
I can’t get it to work.
Jick
Asbjorn: Uhhhh, an alternative to server-side includes? What? Who said this has anything to do with those? Widgets are *third-party code* which cannot be placed on your server.
Jick: To use multiple ones, you just change the name of the main div in both the inline JS and the linked JS file. The idea is that this div will be named something unique each time, based on what widget is embedded.
This is completely awesome that you’re actually developing something with your readers. I know it started with Mike and Rob, but seeing the progression of such a functional THING with readers is pretty cool!
Deja vu… we came up with essentially the same approach for NewsGator Widgets. We’ve been using it for about six months now and it works quite well, although there are a few gotchas in IE6. For instance, your widget may not load if there’s something else on the page that takes a very long time to load. Also we’ve run into some very odd problems with appending script blocks when someone else on the page is doing document.write()’s. But by and large it’s a great approach.
If you want to check out our implementation there are number of pages around that have NG widgets on them. A couple good examples are the Discovery Everest Blog(second box down in the right column) and the photo rotators on my blog (top two in the right column).
We’re giving away free accounts, you can sign up here if you want to (click the “add buzz to your site” link in the upper right)
Brian: Maybe I’m missing something but I checked out the widget code on your blog and I am not seeing the similarities at all. From what I can tell, you are calling a script normally, using document.write, and not guarding against slowness in the script returning. I know I’m missing something, so if you could be so kind, please enlighten me…
There is the one write in there, but it’s just loading static JS so it’s fast. Still, you’re right that it’s not good, so I’ll be taking it out in the next version.
If you watch the requests we actually do a number of other requests after the page loads (or during, depends on your browser).
So I just read all these comments and I’m wondering what the finished script looks like. Is it the one at the top? Sounds like it been though a few versions all within the life of these comments.
tc: Yeah, I’ve merged all suggestions/improvements into the above code. I wouldn’t call it final (because nothing ever is these days) but it’s lookin’ pretty good.
For this, I use the object element and a small satellite page with the widget script in
<object classid=”clsid:25336920-03F9-11CF-8FD0-00AA00686F13″ data=”your URL here” type=”text/html”>
Mike and Rob
Excellent work as I know I’ve had the same headaches with embedding widgets as well. I also really like your approach to the solution. Can’t wait to run tests with it in the wild.
On a side note when I see WEDJE I think of the all to feared wedgy. So I’m off to give my web sites a Wedgy. :-D
Steve — you may have something there… I made a test using your far-more-straightforward object embed method, completely sans-WEDJE:
http://robgoodlatte.com/dev/WEDJE/objecttest.html
The only downside I see is that you have to set the height of the object. I also haven’t tested this across all browsers, only Safari. If you’re right this could be a real “D’oh!” moment..
Steve: Yeah, the object method seems like a good alternative to iframes, but that’s basically all it is, right? Just XHTML 1.1’s alternative to the IFRAME tag?
Steve — I took a really close look at embedding using the object tag, and I actually think it’s pretty useful — but there are a handful of limitations:
Embedding via the object tag is a great option for flash widgets that will always have the same height. But depositing HTML elements doesn’t really work with an object embed due to the styling issues. So count Flickr/del.icio.us/magnolia/newsvine embedded JSS links out. WEDJE still looks like the best option for embedding HTML.
Just to note, the loading times that this script seems to (mostly) solve is not restricted to widgets, but any third-party content. The biggest site-killer I’ve found so far is actually ads. I don’t know how many times I’ve gone to a site like Ars Technica, Wired, or even Digg and had to wait anywhere from 2 to 20 seconds while a banner ad that I don’t even want to see is loaded from some ad server somewhere – and that server is overloaded for some reason.
So thanks for solving the widget load-times, but we really need those ads to stop breaking sites! :)
I’m a little bit confused about where I’m supposed to place the code. Does it go in the tags of the header, or in a .js file? And which .js file? (I’m using WordPress and want this code for my sidebars.)
How do I insert codes for Flash content? I want to put a video on my website, but it asks for a widget code and I don’t know how to write it.
This is a very intuitive technique. Iframes could be used as well, and then implement a script that would auto size the iframe to match its contents.
I agree with Rust, ads are really bad for this sort of thing. Also those tracking gifs that are insisted on by marketing (Hit Box, Revenue Science, etc.). There needs to be some standards set.
May be we should lobby doubleclick?
It appears Javascript variables are not available to the widget when using WEDJE in IE6 or IE7.
<script type=”text/javascript”>
<!–
var widget_background_color = “black”;
var widget_text_color = “white”;
//–>
</script>
<script type=”text/javascript”><!– WEDJE… //–></script>
Works in Firefox and Opera, but in IE6 or IE7 the widget will appear without styling. Passing style parameters via the widget URL is a workaround, but personally I find it awkward.
Also, I think using Math.random() in the container name is good practice so users can embed multiple of the same widget on a page (with different parameters for example).
Has anyone figured out how to insert and execute javascript in DIV tag ?
Like below but of course this doesn’t work :
myDiv.innerHTML+=’..script tag with src..’;
One thing you could try is:
div.innerHTML = ”;
Whoops.. seems that the javascript filter block it, here’s a spaced out version that goes inside the above single quotes:
If you don’t see that either then the trick is that you put some sort break to the “script” part of the script tag so that html doesn’t see either part as an opening or closing tag.
eg: scr’+’ipt
Hmm… I posted a comment here, and its gone :S
I tested this and less than 5 minutes in IE5 had crashed twice.
I did a little more testing, and it works, and you can refresh the page many times and it works. but, when you click in the address bar and simply press enter (ie. go to same page without refreshing…) it crashes.
Any fix for this?
Joe. Thanks for discovering this. I had noticed some IE crashing as well and was never able to pin it down to a certain behavior. The fact that it doesn’t happen upon a full reload makes me think it’s some sort of timing issue. I’ll look into it, but these things are pretty hard to debug sometimes…
The widget is not loading in the example?
Are some “famous” widgets using this technique? Or do the random crashing behaviour happen too often to be used in a real environment?
thanks!
Hi Matt. Yeah, that’s probably just an artifact of me switching to WordPress… but yeah, I wouldn’t use WEDJE at this point because I never got around to fully sussing out the IE6 issue. It’s unpredictable.
I like the method to isolate widget load from the page load. Thanks.
We are looking to distribute widgets to clients. Not every page view will use our widget, but only when the consumer directly interacts with the widget. Therefore, to limit each trip to our server for the basic widget draw, we want to have the HTML live in the distributed .js file. This way only when a consumer interacts with the widget will AJAX API calls come to our servers; otherwise it is all on our clients servers?
So, deployment is providing the snippet _AND_ the .js file which we ask them to host on their system.
Again, the .js file contains the widget display, defaults, etc and the API calls to our systems.
Is this how other people are doing things? What are some alternatives to think about.
Wow, I have been looking for this forever. This REALLY helped me out with the horrible twitter badge hanging my site everytime.
Thank you :)
In its latest version, it crashes on IE7 every single time when I press Enter in the address bar.
Fixing the IE crash:
It seems like if the script src is added to the script element AFTER the script element is appended to the div, IE doesn’t crash.
So, the setTimeout first argument should be: “document.getElementById(‘wedje_div_yourcompanyname’).appendChild(s);s.src=’https://mikeindustries.com/blog/scripts/sleepywidget.php’;”.
Don’t forget to remove the original s.src=… before the setTimeout call.
Sorry to bring an old post back to life, but I have a question I don’t see an obvious answer for…
Why not just put the tag above the script tag? This way you wouldn’t have to worry out the differences between text/html and application/xml.
Is there something obvious I’m missing here?
Thanks for the great informative post/thread btw.
Sorry, the html was filtered out of that last post… It should say “Why not just put the ‘div’ tag above the script tag.