Google fills out your search forms

Occasionally I have found sites that on their home page you would have to select a company or product from a drop down box, and then enter the site from that information. Until now, Google could not access those pages without being provided direct links. Sites with these types of pages were often called the “Deep Web” or the “Invisible Web”, because search engines could not access them. Google has said in the past that they believe that 80% or more of web pages are “hidden” to them because they require a form to fill out to find them.

Now, if you had hired a good Search Engine Optimizer, this would not be an issue as they would know how to provide links to those pages so all search engines could access them appropriately. However, sometimes the advice of your SEO expert is ignored, or you didn’t include one on your team, and thus search engines can’t access those pages.

In April Google announced that it could begin search pages which required a user to fill out a form. This has all types of interesting applications, both good and bad. You need to understand what this means, as well as what this can do. So without further ado, I present The Good, The Bad, and The Truth.

• The Good:
  • Now more pages will be accessible.
  • Simple “categorical” search forms will no longer cause Google to stumble. For example I recently built a simple movie web application. In it people could search by genre. I had to devise ways to not use a drop down when possible so Search Engines could find the reviews.
• The Bad:
  • If you tried to “hide” pages, you need to rethink your method. Consider the use of the robot.txt file or robots meta tag to properly ask search engines to not process certain files.
  • Some people fear that this means Google will explore or try to hack restricted access sections of your site. (Remember your robots.txt file in these instances.)
• Some Truth:
  • Only Google has announced this feature. While other search engines will probably have to follow suite, as this time they don’t and they still account for 35-45% of all search traffic.
  • Only simple forms are filled out. Google is not (currently) entering information into text boxes, so many forms cannot be processed.