Monday, May 9, 2011

How to Avoid Stumbling Blocks for Search Engine Spiders and Get Indexed in search engines?

Leave a Comment
I have often been asked the same question by various website owners and other Expert SEO Analyst and I received that same old question once again in an email from one of my blog readers yesterday. To summarize, his problem is that his website has been there for 3 months now but all its pages are not indexed. While I replied back to his email, I thought it would make a good blog post to put the tips on how to get your page indexed in Google and other search engines.

Search engine spiders move from one website to another website by following links, so if you are creating a new website and want it indexed by search engines, all you will need is to establish some inbound links from already indexed pages to your website and you will be well on your way to the Google index.
Now the question is, if it is that easy even then why websites are often not indexed or often partially indexed ( just the home page or may be a few pages and most pages remains un - indexed ) ?

We need to understand that Google bot or any search engine spider for that matter is nothing but an executable program and they are not human beings. Webmasters often create stumbling blocks for search engine spiders that prevents the bots from indexing a website properly. It is important that we keep away from them. So what are the stumbling blocks for search engine spiders ?

First let’s talk about issues that would simply STOP the search engine spiders.

    * Pages that require log in. A search engine spider cannot sign up and submit the log in information – they are not human :)
    * Pages accessible via select form and submit button ( again the same logic applies)
    * Pages that require a drop down menu to be accessed.
    * Web pages that are accessible only via a search box
    * Documents that are blocked by robots Meta tag or a robots.txt file
    * Bait and Switch pages. Pages that redirect to some other page before displaying the content. This is basically cloaking technique and search engines can ban a website for using this technique.

While the above factors are absolute no-no s if you plan to get your website indexed in search engines, there are other websites that still face indexing problem though they do not have any of the above factors. Why?

    * Dynamic URL s with more than two parameters – Spiders find it very difficult to crawl this type of pages and they are pretty reluctant on these pages because these pages often result in errors when accessed by non-human visitors.
    * Pages that are buried below the third level on your website. To put in simple words, try to make all your pages accessible from the home page within a max limit of 3 clicks. Pages below the third level can also be indexed only if there are good numbers of external back links to those pages.
    * Pages requiring a session id or cookie. For human visitors the browser can retain the session id or cookies; search engine spiders would often fail to retain these cookies or session id.
    * Pages that contains frames often makes the page inaccessible to search engine spiders.

More Info At SEO Tips And SEO Guide
   

0 comments:

Post a Comment

Download 10000+ Submission List

Download 10000+ Submission List
New Bookmarking,Directory,articles,forums,Blogs,Video,Rss Submission List

Popular Posts