Working as part of a large search engine marketing company, we see a number of websites “freshly born”, that have only thought about search engine optimisation (SEO) after the launch.
In an ideal world, an SEO agency or consultancy will be brought in to a project prior to a launch, but that isn’t always the case. Due to the number of newly launched websites we see, we have a number of things that we look out for straight away to determine what sorts of initial changes are going to be required to get a project going in the direction the client needs.
Most SEO’s will have come across situations where there are real technical obstacles to overcome before a website is going to be able to rank to its full potential. I’ll touch on a few basic, but fundamental things to get right over the course of this post.
First of all are the Robots.txt and Meta Robots tags. These are bits of text placed behind the scenes of a website in their code that allow a webmaster to control access to a website from search engine crawlers.
You can use a robots.txt file to block a website in its entirety, or prevent specific areas of a website from being accessed. Sometimes prospective clients will come to you and say;
“I am not getting any traffic from search engines to my websites at all, and I have no idea why?”
One of the first things to look at is the websites robots.txt file for the following:
User-agent: *
Disallow: /
This prevents a website crawler from being able to visit any page on that website – you are telling the crawler than anything after the / part of the URL (which comes after your domain name) is completely off limits to them.
Without wanting to complicate matters too much – it is possible to get blocked pages indexed by search engines, but it’s never going to be the same as if you don’t have this in place on your website.
Similarly, if you have the following tag in the <head> part of your webpage, you are going to find it really difficult to start getting website traffic from search engines.
<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>
If anything, this is a stronger directive – if enough pages link to a blocked file with robots, it can still get indexed (although it is unlikely to rank for anything), but with this tag in the code of a web page, you completely stop a search engine from being able to index that page in any way shape or form.
I have seen this a hell of a lot of times on newly launched websites. Commonly, this is an oversight, from a rightly intentioned web designer who doesn’t want a search engine to be able to index this content from a development server – absolutely the right thing to be doing!
The problem is that they are all too often forgotten about when it comes to making a website live for the first time. People willy nilly start thinking about moving the website from the test server to the main domain and consider their job done, forgetting the handy things they put in place to aid it whilst it was in development, and its only later, months down the line that the client starts to worry about why their fantastic newly launched website, isn’t bringing them in any money.
Screaming Frog SEO Spider Tool
There are some easy ways to check this – you can load up the robots.txt file and look in the webpage source code to see if these pesky hangovers are still in place, but lately my first port of call has been to fire up a new tool, the Screaming Frog SEO Spider, and run a scan of the website.
If it doesn’t find any pages on its scan, I then look at the robots.txt, and 99% of the time, I’ll find that the problematic version of this file is in place. If it manages to find lots of pages, there is a tab that allows you to look at “Meta and Canonical”, and it will tell you if any pages have the pesky robots meta tag in place (there is a very slightly different version of it that is not a problem – make sure if looking for a problem that it has “noindex” or “nofollow” as part of it, else its likely not the problem.
This tool is also really good at spotting another common launch issue with new websites – duplicate Titles and Meta Descriptions. Once you have run a scan, you can switch the tab to “Page Titles” or “Meta Description”, and filter these by duplicates – if you see lots of pages (and in many cases all pages) that have exactly the same of these across a website, this likely indicates some potential problems.
It’s probably not as potentially serious as the robots issues mentioned first, but it can seriously hamper a websites ability to rank for the terms it wants to be found for.
Think of it this way – these items on a website tell search engines what a page is all about – they clearly look at a lot of other things as well, but you have a quick title and description of what to expect on that page. You have say, 1000 pages on this website – and they all have the same title and description.
Search Engines will look at these and be really confused:
Huh? Every single document/page on this website is about exactly the same thing? Well, I don’t want to look and read 100s of pages that are the same – instead, unless lots of links point to specific pages, I am just going to ignore them.
This may be overstating it to a degree – but I would be very confident that a website that previously had 1000 pages all with the same titles and descriptions moving to a website with a unique title and descriptions, gets a considerable traffic lift as a result of that change.
A lot of the initial stages on SEO projects that I have worked on (or we have looked at generally), require ironing out these sorts of issues as the very start of the campaign.
If you manage to avoid all of these issues when a website launches, then you are likely to give yourself a good leg up in search engine visibility!
Peter Handley
A guest post from Peter Handley, aka ismepete on Twitter, who heads up the SEO Department, as SEO Services Manager at Vertical Leap, a search engine marketing company based in Portsmouth, Hampshire.
[...] check out Daniel Bianchini’s blog with Top Questions to Ask Your Potential SEO Agency, and Common Technical Mistakes Made When a Website Launches where Dean Cruddace has kindly let me post on SEO Begin.I’ll be looking to be doing some more [...]