If you’ve ever installed WordPress to a website, you’ve run through the very quick and painless “famous five-minute” installation questionnaire.
There’s a question at the bottom of the page, just before the submit button, that can seem somewhat innocuous. Most of us will not enable the box that states “Discourage search engines from indexing this site” — we would answer “no”, to this question – we want our sites to appear in the search engines.
But what if we’re developing the site — say on a development server, or in a sub-folder or sub-domain of the production location — and you don’t want the search engines to index your information at that location. Well, then, you would make sure you enable this option. Makes sense, right?
However, you would have to remember that you’ve set this option the way you have, when you go live. It can be difficult to remember anything as mundane as having to disable a checkbox on some squirreled away configuration page. It’s understandable that this setting could get overlooked when it’s “go time”.
When the [Settings] Hit the Fan
Last month, I had a client contact us with praise for what a fantastic looking website they now have, thanks to our designs. And he was extremely happy with how his website had come together. He just had one small problem. Suddenly, after going live, all of his pages were now missing from Google.
I read his message and my heart stopped for a couple of beats. This is the last thing anyone wants have happen to a website.
What [on earth] happened?
There are only a couple of reasons for being de-indexed. Pretty much either you are getting penalized, or you’ve told the search engines to de-index your site.
It took just a few seconds for us to confirm that his website still had the “robots.txt” instruction that tells search engines to not spider the site, that was initially set up when he started building his site. Not wanting the unfinished development site to be spidered, he had answered “yes” to the question of “discourage search engines”.
We instructed him to log in to the WordPress dashboard and revise the setting on the “Settings” -> “Reading” control panel. Make sure you are not discouraging search engines from spidering YOUR site you need to be in Google!
This seems like a no-brainer, but it can, and does get overlooked. It is a fantastic feature to have in WordPress, to give the power to discourage the search engines, but, with great power, you can really really, really, pooch your site.
Thankfully, after revising his setting, he was re-indexed within in a couple of days and was fully back in business within a week.