Getting our listings spidered

[crdesign]

I am currently working on a new project web site which will list professionals in a niche market for them to advertise. Think along the lines of yellow pages, but the web site is completely tailored to a niche market.

My question is how google and other search engines will be able to see our listings. Since the listing will be search based. A user will come to the site and enter in their zip code to find the professionals that service their area. How will google ever see those listing? Or is it even a concern?

Obviously the site will have more than just the listings. Since it is a niche market the plan is to have blogs, forums, ecommerce, and as many things pertaining to the niche as possible. I am developing ways for the professionals to use the blogs as extended marketing to showcase their expertise, which will link to their company specific information page and their web site. So any professional who takes active part in the web site will obviously reap the benefits of this.

So back to the main issue though. Does google follow the urls in the action parameter of a form? If so would it be wise to have a drill down page that the search engines could spider based on that to find all the pages on our site. Since the content for listings and company information pages are dynamically created from data created by the professional, with search engine friendly urls (no query parameters) I am hoping to capitalize as much as possible on those pages.

I am stumped as to what might be the best practice here for this situation. I have been fairly successful in the last year creating a website that places #1 in natural google results for 5 of 6 keyword phrases we target (the 6th flipping between low page 1 and high page 2 from week to week, in a highly competetive keyword phrase). So I'm not 100% foriegn to what it takes to get good results from google. But I am a little foggy about what to do in this situation.

I have valued the advise on these forums for about a year now also and hope someone can give me a little insight. Thanks in advance to everyone here!

[crankydave]

IMO you'd be helping yourself by providing geographic drill down pages that could be easily indexed.

If your parameters are prefilled, they can, to the best of my knowledge, follow those. If it calls for an action to be taken, ie enter zipcode-submit, they've no way of knowing what action to take and can't follow.

Dave

[wige]

I would recommend using mod_rewrite or another similar method to give each professional's listing a URL similar to yourdomain.com/zip/listing-name.html, that way you could pull a list of available zip codes from the database and create an index page. This will also facilitate the creation of a sitemap. This information could then be divided by state as your site grows, and the zip code part of the url may be eventually replaced by /state/nearest-city/ which will give your site a structure users will be able to easily understand and navigate. I have an entertainment site that is built on a similar concept-it is designed to be searched but I use mod rewrite to categorize the thousands of events and generate index pages for the different categories and as a result I find many users browse the site rather than searching.

[crdesign]

Great ideas, thanks.

I did think of doing a geographic drill down. I figured if anything that would be the safest way to optimize for SEO without using any blackhat tricks like hidden links. That would give the users the option of searching via zip code or drilling through the site to their specific location.

I've never played around with mod-rewrite (even after 10 years of web development). But I use a similiar type of scripting which creates very much the same effect as mod-rewrite with PHP. My only problem or I guess question is would I include each zip code in the sitemap for the search engines? Not a big deal since that could be dynamically generated everytime we ad a listing to a zip code, just add that zip to the sitemap. To me though it seems like the search engines might question the pages since it would never land on a page featuring links to those zip pages. The search result pages obviously would be those zip pages, so the site map would essentially be a hardcoded link to those search results.

I actually like both ideas, maybe a combination of both would work the best?

[crankydave]

I see 2 potential problems with the zip codes.

First, unless a search returns all results within "x" miles it's not going to be very user friendly. For example, look at the number of zip codes in Los Angeles.

Second, I don't know how many general searchers use a zip code as a means of finding something local. I would tend to believe they'd be more likely to use city or state.

If your search by zip is going to return all the results within "x" miles, great. Personally, as a navigatable and crawlable site map, I'd prefer using a state/city URL rewrite wige suggested rather than the zip code. Makes the site user friendly and search friendly.

Dave

[crdesign]

The 2 problems you see with zip codes I see as pluses. First everyone knows the zip code they live in, where as the city you live in can be left up to interruptation (I currently live in an area that can be postmarked as 3 different cities without any effect on the mail getting to me and that is due to the zip code not being left to interruptation). Secondly when I do a search for something locally in google I always put the zip code not the city name. Why? faster to type on my computer and cellphone. Which is another reason, instead of someone looking for a professional that is "close" to them. The search results actually are determined by the professional choosing the zip codes they want to service.

I don't want to get into a great debate as to why I think the zip search works best for the purposes I intend, because no matter what it will be one persons preference and opinion versus anothers. I don't think either solution is wrong personally. I just think the solution we have devised is the simplest and fastest for the user, which of course is an opinion that could change overnight. Me personally, I hate going to a site to look up information for something local and finding that instead of giving them one small piece of information I have to either spend time looking for the link for my state, then my metro area, then my city, when in 5 keystrokes and a click (or 6 keystrokes) I could have the same information tailored for my needs.

Now back to the original purpose of the post.

I'm not concerned obviously with the navigation of the site. I think it is simple, easy to use, and intuitive (always my main goal). My problem is how google and other search engines will be finding the listing pages based on zip codes. I may in fact do a drill down navigational page in the future (once we have enough listings to warrant it). But until then I just need a solution that will let the search engines "land" on the listing pages since it seems they won't follow the search forms action url without default data (which just can't be done for obvious reasons). I could very easily return a drill down page when no zip code is submitted or when invalid data is submitted, but that seems useless to work on if the search engines will not follow the action url anyhow.

I also don't want a huge link page with every zip code listed on it linking to the search results, because it really serves no purpose to the users of the site. But if I add a search result zip code link to the sites sitemap will google or other search engines take offense to not seeing that link anywhere else on the site, since it is basically a search result page. Or should I even be concerned about that?

Now that I think about that also, it is also very possible that 100 zip code search results could result in 100 pages that are exactly the same. Would google penalize us for duplicate content based on that? My first and only assumption is it would. Would I be better off just looking for a solution that gets google to each professionals unique page on our site, and not worry about having google reach the search results lists?

Sorry for being so long winded, just trying to be as thorough as possible. Thanks again!

[bhartzer]

I would definitely provide "normal" or regular "a href" links to the pages so that the spiders can crawl the site. But, keep in mind that using zip codes or another type of geography might not be appropriate for the site.

You need to think of your users first, and how they would typically look for the site to be categorized. If they would be concerned in finding a listing near them, then using geography would make sense. But, if that's really not an issue (in some industries, the users are looking for the type of product or service and don't care where it's located), then you need to categorize it in some other way.

I would first make sure you are familiar with the "theme pyramid" type of internal linking. I would then use a breadcrumb trail to break it all out by categories.

[kula]

Cranky missed the other side of the problem with zipcodes! In the larger, more geographically spread out states (i.e., Montana, North Dakota) the same zipcode covers locations hundreds of miles apart. Here in Hawaii, the same zipcode covers locations on different Islands!

There are free zipcode data files that include lists of cities, and even gis data for each city. You could, in addition to having each advertiser pick zipcodes, offer them lists of cities for each zipcode. Additionally, using the gis data, you can construct a pretty good "within x miles" search! (by returning results with long, lat meeting a computed test) All the data is available for free.

[carpediem]

Originally Posted by crdesign

I don't want to get into a great debate as to why I think the zip search works best for the purposes I intend, because no matter what it will be one persons preference and opinion versus anothers. I don't think either solution is wrong personally.

Hi crdesign,

I don't believe that Cranydave is basing his suggestion(s) to you on a "personal belief/preference".

Originally Posted by crankydave

Second, I don't know how many general searchers use a zip code as a means of finding something local. I would tend to believe they'd be more likely to use city or state.

It has been our experience and research also suggests that when people search locally, they tend to have their subject in conjunction with their location + city and/or state.

The No. 1 Local Search Query is…? » Small Business SEM

SearchEngineWatch: Study: Search Driving Offline Conversions for Local Service Businesses

However, as more mobile users search on their phone, this may change, as it is easier to search via zipcode. Something to consider.

How about combining both? Have the city, state (zipcode). Besides, why restrict your ability to convert more visitors?

Danielle

[SemAdvance]

I would link to a sitemap page from the home page.

On that site map page link to all fifty states.

Each of the states can have the zip codes as a link to a information page, with the professionals listed and link to each one.

Feed a high PR link to the main site map page, and indexing will be a breeze.

Peace

[crankydave]

Thank you kula... forgot about that as well.

Sorry crdesign, wasn't my intent to get into a debate over the use of zips. Just an observation on my part.

If you provide a static link to the search results, same results served to the user, you'll be fine. I'm assuming those pages have links to the actual "product" pages on them which will then in turn, need to be followed.

Dave

[crdesign]

Yah kinda turned into a zip code debate unintentionally... but that is good since it is very much at the center of this problem. I guess after reading through this I neglected to mention these would be in house services only. In other words the provider will be traveling to the user only.

I am going to keep this very short so hopefully no one over looks this question again and continues focusing on the zip codes.

If I add pages to the sitemap that are not visible to google anywhere else on the site (that googlebot can see anyway) will this have an adverse effect or does it matter?

[crankydave]

Originally Posted by crdesign

Yah kinda turned into a zip code debate unintentionally... but that is good since it is very much at the center of this problem. I guess after reading through this I neglected to mention these would be in house services only. In other words the provider will be traveling to the user only.

I am going to keep this very short so hopefully no one over looks this question again and continues focusing on the zip codes.

If I add pages to the sitemap that are not visible to google anywhere else on the site (that googlebot can see anyway) will this have an adverse effect or does it matter?

As long as those pages are linked and as long as the visitor can see them, I don't see a problem. You'd run a risk if you'd be delivering pages to Google the visitor can't see.

Dave

[crdesign]

Thank you dave! that was the answer I was looking for. Got it in a round about way but still got it. I'd be delivering a sitemap of pages that google could see everything the user could see in a targeted format. The targeted format (the search) was the part I couldn't get google to see easily.

I do appreciate everyone's feedback, even about the zip codes. When we get closer to launching the site I'll make a followup so everyone can see how it turned out and maybe some of you will see that the zip code based search is a good idea. Then again, you may not. I am going to take into consideration some things that I had not prior to this (sparsely populated areas, etc).

I have given it more thought and I think from what I have gathered I will be doing a drill down by state right from the initial release, which will list all information pages for providers in that state as a way for users to see if their favorite provider is listed or provides the niche service we are targeting. Then I may look into more radial searches (still based on zip codes) since I already have that programming in place for the providers to find the zip codes they want to target. I could go into great details about why one suggestion or another is not optimal for the site, but that wasn't the real subject anyhow.

Thanks again everyone!

[crankydave]

I'd be delivering a sitemap of pages that google could see everything the user could see in a targeted format. The targeted format (the search) was the part I couldn't get google to see easily.

DO NOT deliver different pages to the user than you deliver to Google. Cloaking.

Dave

[jeffposaka]

Use text link breadcrumbs on each page. The SE can see how you drilled down to the listings page also it improves usability on a bigger site.

Once there is a static link to every page to the site, it will be easy to submit an xml sitemap to Google and Yahoo.

[crdesign]

Originally Posted by crankydave

DO NOT deliver different pages to the user than you deliver to Google. Cloaking.

Dave

Yah I got that... not different pages they would be the same page, just 2 different ways of getting to them.

Maybe a visual.....

The user
------------
search->listing->information page

Google
-----------
sitemap->information page

Google would never see the listing page, which is fine since it is just a list with no real use for the search engines anyhow and could cause problems with duplicate pages possibly.

Additionally I will work on a way for the user (even though it's not the most effiecient or even accurate result) can drill through the site to the information page.

The user
----------
home page->state->region/city->provider listings->information page

Without the search the user does not see all the providers available to them so the search is really my main focus.

[crankydave]

Perfect.

As you develope a drill down for your visitors, you *might* find yourself pleasantly surprised at how many visitors use and prefer it AND give you added leverage for your advertisers/clients.

Dave