We’ve recently been running into a recurring issue with AWS’ Elastic Load Balancing service. The problem manifests itself in vastly increased latency or eventual dropout of requests against the load balancer.

We verified the health of our instances in each availability zone, and concluded that it was not a problem at the instance level. Second, we confirmed that our DNS queries were being routed to the ELB correctly. They were. So the issue was at the ELB level.

As it turns out, there appears to have been/still is a bug where ELB will suddenly decide that there are no available instances in a particular availability zone, or at least it will have trouble connecting to instances in that particular zone. Even if the instances are perfectly fine, and pass all health checks.

The solution was to try to identify the suspect zone, and de-register that zone and all of its instances from the ELB. Once re-registering the zone & instances completed, the ELB was back up to full functionality again.

Hope this helps next time you run into any ELB issues!

P.S. We’re big fans of Amazon Web Services – if you haven’t already, check out our recent feature on the AWS Blog.

We’re often told that that our real-time data is analytical gold for the decision makers behind all kinds of online establishments; from personal portfolio sites right up to the busiest e-commerce enterprises. We pour all our passion, blood, sweat and tears into making our applications as useful, intuitive, and beautiful as possible, so you benefit from our continued efforts to make our data as insightful and actionable as possible.

But it gets better. We’ve opened the gates to our analytical gold mine, empowering developers with access to the same raw data that fuels our very own applications. With 3.2, we’ve announced the immediate availability of the new real-time GoSquared Live API.

Calling all developers!

We’re very excited about the possibilities. Using the API, you can write adaptive applications that display your sites’ traffic in any number of interesting ways. These could range from epic visualisations that boldly showcase your Ãœber development talent, to adaptive modules on your website that respond to traffic changes in real-time. The best part is, it’s all platform and language agnostic, so you can use the languages and tools you love.

Head on over to the developer documentation to check out what this baby can do.

Read on for details about our competition – win $150 of analytics by playing with our new API!

Example apps we’ve built with the GoSquared API.

(What we built in 2 days)

We couldn’t wait to have a play ourselves, so we’ve whipped up a couple of example applications to help inspire your creative flair and show you what’s possible with the API. We’ve created two applications:

Radar

The radar shows you a dynamic real-time representation of your users’ geographic location. Now you can view your websites’ traffic through the eyes of a James Bond villain. Sound effects not included.

Try Radar now →

Top Content

The top content widget tells you what content on your site is the most popular, right now and over time, enabling you to effectively and adaptively funnel your audience to your hottest content without even lifting a finger.

Try Top Content now →

Competition time.

The GoSquared API also facilitates seamless integration with 3rd party products and services, so platform developers can start integrating GoSquared analytics with their own systems.

I think this calls for a little competition. We’ll give away 6 months of the GoSquared Plus plan to the 3 most impressive applications built using the GoSquared API. One entry per person please (but of course you can make as many apps as you like!) To enter the competition, start working on your apps and give us a shout (and a link) via Twitter (@GoSquared) or leave a comment on this post.

So, what are you waiting for? Get to grips with the GoSquared API documentation now!

We can’t wait to see what you come up with.

We’ve noticed some odd things happening to some of our stats, and it’s been brought to our attention that some of you, our customers, are experiencing similar issues. The problem pertains to pages with Facebook’s “Like” button or any of their other social plugins.

Quietly duplicating requests.

Under certain circumstances, a page with a Like button can register one extra pageview for every button you have on the page. This is undesirable, because if you run a site with, say, ten like buttons on the homepage, you’ll see eleven pageviews for each and every time someone actually views the page, not one. Obviously this is going to completely throw off all your tracked stats and analytics. Not good.

Blame IE.

It turns out this is a quirk of Facebook’s methods. For security reasons, AJAX requests typically aren’t allowed to be sent across domains (so, for example, a facebook plugin running on yourdomain.com wouldn’t be able to send an AJAX request to facebook.com). Most sensible browsers now implement a feature where a specific header (Access-Control-Allow-Origin) can be set to allow this behaviour. With one exception. Internet Explorer, being Internet Explorer, shuns all compatible web standards, so facebook have had to make a workaround. And for reasons we can’t quite fathom, they’ve decided to do that by loading your whole page again in an invisible iframe.

How this may have been affecting you.

This has a negative impact both server- and client-side. Not all sites have the tens of thousands of servers facebook has, and this method will multiply load levels by however many Like buttons you have on the page. And because of the way it’s implemented, it can slow the client-side browser down by a massive amount too. And for those of you who have auto-playing audio or video on your pages, you may have noticed you can’t put a like button on the page, because you sometimes end up heading a second audio stream coming apparently from nowhere. That’s all caused by this Facebook’s issue.

What can you do about this?

Firstly, we’ll be rolling out a new version of our tracker this week which won’t track these fake hits from Facebook. So you won’t get crazily distorted stats. But there are a couple of other things you can do to improve the situation further. Firstly, if you are using any other analytics services that allow you to filter out certain URLs, tell them to ignore any hits with the parameters “fb_xd_bust” or “fb_xd_fragment” in the URL. For example in Google Analytics you can set a custom filter to exclude request URIs matching the pattern “fb_xd_bust|fb_xd_fragment”.

We’d also recommend you take a look at this page on Facebook’s developer site. They don’t draw much attention to it, but the “Custom Channel URL” section is extremely important to anyone who doesn’t want their stats to be completely thrown off by Facebook’s somewhat strange methods.

To sum up.

• Facebook use some strange method for their cross-domain ajax, which distorts all your stats and puts extra load on your servers, and they could do with drawing more attention to it.
• We’re rolling out a new tracker that takes care of the fake hits on our end so your stats don’t get broken.
• Check out ways to work around this so you don’t end up putting extra strain on your server, and ways to stop these fake hits showing up in your other analytics services.

What’s this? A new GoSquared Help Sheet? Hell yeah!

We’re pleased to bring you a brand new GoSquared Help Sheet for your Node.JS needs.

What’s Node.JS?

If you’ve read this far, you probably already know, but for those intrigued, Node.JS is an evented I/O framework for the V8 JavaScript engine. It’s ideal for writing scalable network programs such as web servers.

We’ve been working on some exciting things with Node.JS, and we felt it was only fair to share our knowledge in the form of an easy-to-read Help Sheet.

Feel free to print it out, stick it on your wall, and get developing!

As always, if you find any errors or typos or anything like that, please drop us a note in the comments.

Get the Node.JS Help Sheet PDF (530KB)

Want more?

If you’re stuck, we also have some other helpful Help Sheets – for CSS2 and CSS3, HTML, PHP, and VI, as well as handy OS X Character Reference sheets. All available for FREE download now!

There are a lot of new database systems popping up that can be interpreted by the newbie as “memcache, but with persistence”: MemcacheDB, Membase, redis, Project Voldemort etc. I’ve seen the term “persistent Memcache” being thrown around a lot, which is an oxymoron in itself, as the term “persistent” undermines the implied ephemeral nature of a “memory cache”. We must be careful not to mislead those impressionable by buzzwords into adopting the wrong technology for the wrong uses.

It’s not uncommon to hear the comparison “like a persistent version of memcache” being thrown about, and it is understandable why it is used. While brave new ground is being covered by these new DBMs, this simile acts as a straightforward introduction to these new technologies and how they work. If you’ve worked with web development or MySQL for any tangible amount of time, chances are you’ve had experience with Memcache and its simple key/value data model. So it’s a great technique to help tame these wild and mysterious new technologies so that we understand them better in terms of what we already know.

However, we must be careful not to let this comparison transcend the fundamental differences between these systems. I fear that many regard “persistent Memcache” as a drop-in replacement for the original Memcache so that they don’t need to worry about cache timeouts or its ephemeral nature. Whereas the important distinction to make here is that these systems are databases, and Memcache is a cache, and nothing else. For best performance, it is important that both are still used in a way that complements their design and intended usage. I am worried this likeness will foster poor data model design, as developers feel more secure with interchangeably using their caches as databases, and vice versa, adopting the wrong technologies and software design paradigms on this premise. Sure, many of these “NoSQL” persistent databases can be used as a cache, but to engineers building high performance systems, it is important to keep the distinction in mind to avoid unnecessary overheads.

Today we’re releasing Map View (beta) into GoSquared’s Labs. Check it out.

What is Map View?

Map View is the same beautiful map that you will instantly recognise if you’ve been using GoSquared LiveStats. We’ve cut out everything else to leave just the map – allowing you to show off the locations of all your visitors to anyone you want without the worries of showing email addresses, user IDs and other personally identifiable information.

How does this fit in with my business?

Map View is ideal for large screens and projectors. We received a ton of feedback from customers who wanted to run just the map from LiveStats on a big screen in their office or their reception area. We also realised it’d be beneficial for everyone if we could offer Map View to those who wanted it, without the extra burden of all the extra information around visitors, pages, and referrers.

Do I need a GoSquared Account to try out Map View?

You sure will, if you don’t have one already. You can sign up for GoSquared from just $9.99 per month, or have a play on the free account right now.

A few months ago, AWS introduced an extremely useful little feature to their consistently developing EC2 platform: Instance Tagging. Before tags, it was relatively tricky to define a clear instance nomenclature, forcing you to implement a vague approach using security groups if you wanted something close to the same functionality. Since there is also a full API surrounding tags, they can be put to work in many useful ways.

Naming instances

Besides attaching up to ten custom nuggets of arbitrary data to instances, tags can be extremely handy when it comes to naming your EC2 instances, and more importantly, using these names to differentiate between instances and their roles. For example, you have three instances. One holds your master database and website, the second is a slave database in replication, and the third is a staging/test server. You’ll want to make pretty damn sure that when you’re SSH’d into all three of your instances, the commands you send are received by the intended recipient server.

In the EC2 management console, you can define a tag key for an instance which is then mapped to a value. There is a default tag key called “Name”. In the value for that “Name” key, you can type in the name you want to assign to that instance. For our servers in this example we can call the master server master-0, the slave server slave-0, and the staging server staging-0. I like this naming system because on the left of the hyphen you can define the type or classification of that server representing the role it performs, and the number following the hyphen can be incremented for each server of the same type, therefore acting as a unique ID for each server.

Conveniently, you can define which of these tags become a column in the table listing all of your instances. If you have a large number of instances, having this custom column for tags present can make it far quicker to find the instance you’re looking for at a glance, rather than trying to derive the right one from all of the other parameters displayed.

Doing some magic with .bashrc

I am a big fan of customising the bash prompt so that it displays only useful contextual information I need to know when I send commands. One of the problems I came across whilst working simultaneously on multiple instances is making sure I am sending commands to the correct one. Sure, most bash prompts include the server’s hostname in the prompt, but beware – there are flaws with this. For example, if you reboot an EBS-backed instance, the hostname it shows is not necessarily the one it is operating from. I needed something more definitive than that.

Here’s some example code you can place in your .bashrc file to display the instance ID and server name in the command prompt:

INSTANCEID=`curl -s http://169.254.169.254/latest/meta-data/instance-id`
SERVER_IDENTIFIER=$INSTANCEID
SERVER_TAG=0
if [ -f /var/opt/server-tag ]; then
SERVER_TAG=`cat /var/opt/server-tag`
SERVER_IDENTIFIER="$SERVER_TAG:$SERVER_IDENTIFIER"
fi
if [ "$color_prompt" = yes ]; then
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@${SERVER_IDENTIFIER}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
else
PS1='${debian_chroot:+($debian_chroot)}\u@${SERVER_IDENTIFIER}:\w\$ '
fi
unset color_prompt force_color_prompt

The great thing about this bashrc code is that if you have a file saved at /var/opt/server-tag which contains the server’s tagged name, the prompt will look something like this:

That prompt clearly defines that I am working as user Ubuntu on a server called Node#1 with instance ID of i-b92990d5, so I don’t need to think twice about that.

Using tags as hostnames

Now that we have named our instances using tags, we can turn them into hostnames so we can address them directly in our scripts, avoiding the need for expensive lookups against Amazon’s API to find server internal IPs. I’ve written a simple script to add entries to the /etc/hosts file for each of your instances that have a tag called “Name”. The value in this “Name” tag will become that server’s hostname, which then forwards to its internal IP address. While we’re at it, we also create the aforementioned /var/opt/server-tag file so our .bashrc script can display the server name in the command prompt. Note this script requires the AWS SDK for PHP and because we’re modifying the /etc/hosts file, you need to run it as root.

< ?php
// Install the AWS PHP SDK from http://docs.amazonwebservices.com/AWSSDKforPHP/latest/ and follow the setup instructions on that page.
require_once "aws-php-sdk/sdk.class.php";
 
define('SERVER_TAG', '/var/opt/server-tag');
 
try{
	// Parse hosts file into an array
	$hosts_file = file_get_contents('/etc/hosts');
	$hosts_lines = explode(PHP_EOL, $hosts_file);
	foreach($hosts_lines as $hlkey => $hline){
		if(empty($hline)) continue;
		$host_lines[$hlkey] = preg_split('/\s/', $hline);
	}
	$hosts_updated = false;
	$this_instance_id = file_get_contents('http://169.254.169.254/latest/meta-data/instance-id');
	// Check server tag directory
	$server_tag_dir = dirname(SERVER_TAG);
	if(!file_exists($server_tag_dir)) mkdir($server_tag_dir, 0777, true);
	// Get the server tag
	$server_tag = file_get_contents(SERVER_TAG);
 
	$EC2 = new AmazonEC2();
	$result = $EC2->describe_instances();
	if(!$result->isOK()) throw new Exception("ec2->describe_instances failed");
	// Each "reservationSet" is essentially an instance or compute "reservation"
	foreach($result->body->reservationSet->item as $rset){
		$iset = $rset->instancesSet->item;
		if($iset->instanceState->code != 16) continue; // Instance is not in "running" state, so ignore
		if(isset($iset->tagSet)){
			// Server is tagged. We should note it down in the /etc/hosts file
			$tset = $iset->tagSet->item;
			$server_identifier = false;
			if(is_array($tset) && count($tset) > 1){
				foreach($tset as $tag){
					if($tag->key == 'Name') $server_identifier = (string)$tag->value;
				}
			}
			else{
				if($tset->key == 'Name') $server_identifier = (string)$tset->value;
			}
			if($server_identifier){
				// This server is tagged with an identifier. Make sure we've got it recorded in the server-tag file if this is our server
				if((string)$iset->instanceId == $this_instance_id && $server_tag != $server_identifier) file_put_contents (SERVER_TAG, $server_identifier);
				// Now me ensure the correct host entry is written or up to date
				$nice_si = str_replace('#', '-', $server_identifier); // Replace # with - to avoid conflicts
				$hosts_match = false;
				foreach($host_lines as $hlkey => $hline){
					if(strstr($hline[0], '#')) continue; // Skip comment lines
					while($hostname = next($hline)){ // Loop through the hostnames to see if this server exists in the host file
						if($hostname == $nice_si){ // Found a hostname match, so check the mapped IP address
							$hosts_match = true;
							$ip_addr = (string)$iset->privateIpAddress;
							if($hline[0] != $ip_addr){
								// Private IP address has changed. Update it after a quick sanity check on IP address
								if(!empty($ip_addr) && strlen($ip_addr) >= 7) $hline[0] = $ip_addr;
								$host_lines[$hlkey] = $hline;
								$hosts_updated = true;
							}
						}
					}
				}
				if(!$hosts_match){
					// This server does not exist in the hosts list. Add it.
					$ip_addr = (string)$iset->privateIpAddress;
					if(!empty($ip_addr) && strlen($ip_addr) >= 7){
						$host_lines[] = array($ip_addr, $nice_si);
						$hosts_updated = true;
					}
				}
			}
		}
	}
	if($hosts_updated){
		foreach($host_lines as $hlkey => $hline){
			$host_lines[$hlkey] = implode(" ", $hline);
		}
		if(file_put_contents('/etc/hosts', implode("\n", $host_lines))){
			echo "/etc/hosts has been updated successfully\n";
		}
		else echo "Could not update /etc/hosts file\n";
	}
	else echo "No changes to make\n";
}
catch(Exception $e){
	die($e->getMessage());
}
 
?>

After running this script, we will have entries in our /etc/hosts file for all of our tagged servers, and because these are now defined as operating system hostnames, you can use these hostnames for connections between your servers (for example, if you were connecting to mysql from slave-0 to master-0 you could define the hostname for the connection as master-0:3306 and the connection will be made via Amazon’s internal DNS network. If you set this script to a regular cron job, you can ensure that the hostnames are kept updated and new ones are added as you tag new instances.

Today, we’re very excited to announce the latest version of GoSquared LiveStats – version 3.

We’ve rewritten LiveStats from the ground up to focus around seeing which content is popular on your site at a glance. The centre of the interface shows a list of all the pages people are viewing on your site. The left side shows where your traffic is coming from – what people are tweeting, which sites are linking to you, and what people are searching for.

Learn more about the great new features of LiveStats 3, and sign up now!

Today we’re announcing two huge improvements to GoSquared. We’re introducing a new structure for user accounts and plans to make GoSquared more affordable, easier to join and simpler to upgrade. We’re also releasing an overhaul of the Site Manager application and renaming it Settings.

The new Settings app

Introducing a completely redesigned application for managing everything in your GoSquared Account, along with all the websites you’ve added to GoSquared.

Settings will look familiar if you’ve been using GoSquared for any period of time – it’s replacing what was previously known as Site Manager for managing the sites in your GoSquared Account. All the settings that were there before – editing of website name, URL, updating your site image preview, sharing options – are all still available.

Along with managing your websites, you can also edit your personal details, change your password, and upgrade your GoSquared Account (more on that shortly).

Updates to GoSquared Accounts

The most notable change introduced with the Settings app is that there is a new option in the sidebar called “My Account” – head here to upgrade your GoSquared Account.

Until today, when you signed up for a plan (see plans and pricing), that plan would apply to a website on GoSquared. From now on, plans will apply to user accounts rather than to websites.

What does that mean? Before – if you signed up for a premium plan, that plan would only apply for one website, so even if you had 3 websites in your GoSquared account, you would only gain the increased pageview limit and additional features of the premium account for one website. We didn’t think that was fair – and a lot of people requested we change things so that the pageviews they were paying for could be shared across all sites in their account.

From today onwards when you sign up for a plan on GoSquared, the features and the number of pageviews on that plan will apply to all sites in your GoSquared account.

If you’re only on the free plan and you currently have multiple websites in your account then not too much will change except the pageview limit on the free plan – 10,000 pageviews monthly, will now be shared across all of your websites.

If you have any queries about the latest changes to GoSquared accounts, have a look at this support document.

Behind the scenes

We’ve improved the functionality of the Settings app to be easier to use, and even faster than before by utilising the latest in HTML5, CSS3, and jQuery goodness. You won’t have to reload the screen at all – everything refreshes immediately, and tabs switch as soon as they’re clicked without any waiting.

Settings is there to be quick and easy – people want to jump in, change a detail, and get back to looking at their stats. That’s why we designed the URL structure to allow you to bookmark any screen in Settings and return to it at a later date. It’s also why you can jump straight to Settings from the sign in screen by clicking the Settings icon. Small things, but hopefully they’ll save you some valuable time.

I don’t understand! Help me!

We’ve put together a support document outlining some of the top questions you may have regarding the changes to GoSquared accounts. Have a read of the support page, and if you still have questions please feel free to email us, tweet us, facebook us, or leave a comment on Get Satisfaction.

Still not using GoSquared for real-time web analytics? Sign up now

This has been a huge project behind the scenes for the whole team, and we hope you like the changes. It’s very easy to keep adding new features when building software, but this is one of those occasions where we dedicated considerable time to simplifying, clarifying, and improving a core part of our offering. We look forward to your feedback!

Enjoy,

James Gill, Geoff Wagstaff, James Taylor
A.K.A. The GoSquared Team

A few days ago we released our new sign in screen. If you’re already a member of GoSquared you’ll probably have noticed it’s a whole lot better than what it replaces.

Try it on iPhone

There’s a few handy little things you may not have noticed, though. For example, if you’re on a desktop computer, or anything with a fairly large screen size, you’ll see the default sign in screen – with icons for each of our main apps, some handy links at the bottom of the screen, and of course the all important email and password fields. However, if you load the same page on your iPhone or any other small screen mobile device running a modern browser, you’ll get a specifically designed screen that cuts out everything you don’t need to see on a mobile device. That’s not all – we’ve gone an extra step and implemented an optimised design for when your mobile device in in landscape or portrait. We hope this will make it easier for people to keep up to date with the status of their site while on the go.

Responsive Web Design

For those web designers among you, we optimised for these different screen sizes by utilising some of the modern technologies available in CSS3. This is an increasingly popular method called “responsive” web design – pages adapt to suit the device they’re being shown on. There’s a great article about responsive web design on A List Apart.

Form + Function

Another little feature you may not have noticed – the app icons (for LiveStats, Trends and Settings) you see when using the sign in screen on the desktop are not just for looks. Simply click any of them while entering your email and password and you’ll be taken straight to the chosen app. It’s only a small thing but we hope it’ll save a second or two of your valuable time that could be spent watching your site’s traffic.

If you have any feedback, please drop us a line in the comments, get in touch on Twitter, or give your feedback on Get Satisfaction.

Enjoy.