How To Get Dynamic Pages Indexed By Google?

[Nick-UK]

Dear All,

Bit of a puzzler this one, I'm much in need of your advice and opinions!

I look after the SEO for a large, dynamically driven database site.

This site ranks well for selected terms in Google/Yahoo/MSN.

However, in Google about 90% of the pages indexed are only shown as supplemental results.

The problem I have is that to get to the "Used products" section of the site you have to use a form to specify which type of products you want to view. Then, after you have narrowed down your search, a list of products are displayed, each one with a quite awful, un-user-friendly URL with about 10 parameters (the site is .asp based).

I know we can implement some URL re-writing techniques to make the pages more spiderable but I have two questions:

1. How can I get these dynamic pages spidered by Google when you have to go through a form first to narrow down your search for a product?

2. What happens if Google was to index these pages and then a few days later the product is sold and (I'm assuming) the URL no longer exists? I don't want to spam Google with a lot of product pages that lead to a 404. Any ideas?

Thank you in advance for your time,

Best wishes,

Nick (UK)

[stymiee]

1. How can I get these dynamic pages spidered by Google when you have to go through a form first to narrow down your search for a product?

If the other search engines can find those pages Google can, too. Anyway, make a Google sitemap and tell them where those pages are.

2. What happens if Google was to index these pages and then a few days later the product is sold and (I'm assuming) the URL no longer exists? I don't want to spam Google with a lot of product pages that lead to a 404. Any ideas?

404 pages aren't spam and won't cause you problems.

[Nick-UK]

If the other search engines can find those pages Google can, too. Anyway, make a Google sitemap and tell them where those pages are.

But if the pages are dynamically generated how do I create a Google sitemap? I do not know what pages will be in the database day-to-day so not sure how I could list the pages in a sitemap. How would I name the pages? How could I differentiate one similar product to another?

[southplatte]

Originally Posted by Nick-UK

But if the pages are dynamically generated how do I create a Google sitemap? I do not know what pages will be in the database day-to-day so not sure how I could list the pages in a sitemap. How would I name the pages? How could I differentiate one similar product to another?

Generally you point Google to a sitemap file in html or xml format - why could you not just pull the from the database in real-time? eg. perform a query for the links that are there when the sitemap is loaded so that way it is always current?

[bj]

There are scripts that create sitemaps of dynamic sites. I have one running on my site, which has a few static pages and two different scripts generating the rest of it. My sitemap script is automatically updated weekly by running the sitemap script as a cron job. It automatically updates my sitemaps and sends me an email to let me know it's been done.

I do admit though that the sitemap spiders might find the forms a challenge. You might need to have an asp guy craft a sitemap script for you.

[computergenius]

Originally Posted by Nick-UK

But if the pages are dynamically generated how do I create a Google sitemap?

I wrote a program that updates my sitemaps by going through the tables in the database once a day, figuring out which pages have changed, and creating the file.

I decided not to create the xml page on demand as the page is quite big, and I was concerned about it timing out whilst Google was reading it. Anyway, if I create it, I can tell Google if I have updated it - most days there are some changes.

[satorigirl]

Depending on the e-commerce software you're using, you may be able to install a mod, or create a cron file that creates your Google file at specific intervals. I know Virtuemart and Zen Cart have mods to do it.

I'm going to be optimizing a large Miva shop, so I'm dealing with the same issues.....

[imvain2]

Since you currently have listings in google, you could always follow the "KISS" principle.

Create your own site-map page (not for submitting to google), pull the results out from the database and list the items as direct links to the products.

Then as part of your navigation, even if its part of your footer navigation, put a link to your new site-map page. So once the search engines spider your site and see the new site-map link they will spider that page and access all of your products.

Instead of deleting product pages once they are sold, it would a better idea to simply disable the online selling process for that item (if its ecommerce) and put in red letters "SOLD". This way visitors can see that at one time you did sell the item they are looking for and it will prompt them to search your website for similar items.

[jrb@w3f.com]

Originally Posted by bj

I do admit though that the sitemap spiders might find the forms a challenge. You might need to have an asp guy craft a sitemap script for you.

That is what I found too.

You have to be careful, or Google will list "includes"
as if they were whole pages too. (They are greedy, want INPUT!)

Yeah, you need a good script that will read your database and
generate the links, outputting into sitemap format.
Spydering thru forms would be a bear.

Making it only run when asked for, real-time, is genious. - Thanks bj.

[bj]

No problem. Just had to go through something similar on my site. I had working sitemaps, installed a comment spam plugin on my blog, since I was getting over a hundred spams a day, and the sitemap thing stopped working the minute I installed the comment spam plugin.

These things tend to be a bit . . . delicate. But it beats generating a sitemap for a site with hundreds of pages by hand.

[EditFast]

Originally Posted by Nick-UK

The problem I have is that to get to the "Used products" section of the site you have to use a form to specify which type of products you want to view.

The pages are there. They exist. Creating a page with links to each of the product pages and a link from your main page to this page will bring the SE bots.

URL rewrite on Apache is a good way to make them easier for the bots.

If you create a sitemap.XML page listing all the pages on your site and submit it to Google you will save a lot of time. I use Brian Pautsch's application (http://www.brianpautsch.com/Blog/200...emapGenerator/) to do this and it works great on dynamic pages but the resulting sitemap will need some editing. If you purchase the app you can enter exclusion parameters that make the editing unnecessary.

2. What happens if Google was to index these pages and then a few days later the product is sold and (I'm assuming) the URL no longer exists? I don't want to spam Google with a lot of product pages that lead to a 404. Any ideas?

Create a script that tracks such pages and adds a redirect to you .htaccess file so the user is directed to a page with similar products (presuming the user might want to purchase them) or send them to the main page or your search page, but be sure to give them a notice that the item they were looking for is no longer available.

[EPageBrad]

There is another way around this. You say that you have to fill out a form to get a listing of products. Can you set up a script that takes your 100 or so most popular "searches" that will list the actual products available? What we do at EPage Classifieds (where we have a similar issue with transitory content and a database driven site) is we created an area that can be indexed that replicates popular searches.

Essentially, you will have a map to pages with current results for popular terms. See http://epage.com/classifieds/B.html for an example of one such page we created. You can follow any link to a page that has 10 results that match this. Because the second page is from the database, there are no missing results.

Finally, create a custom 404 page that makes sense and has content for your users. If the product is not there any more, make a simple page that has links to a search, popular categories, info about your site, etc.

[subsystems]

I strugled with discontinued products myself and eventually came up with a smooth way of handling discontinued products.

If you are using ASP then use the following. I'm sure the statements for the language you are using can be found pretty easily.

Discontinued products are best handled by making a small change to your product display page.
Determine if the product is discontinued and if so, add the following line before the <html> tag...

<%response.status="404 Not Found"%>

Then instead of a cold 404 page. Simply leave all the usual navigation, headers, footers etc in place and only replace the area where the product would be shown. Use something like the following...

"We were unable to locate the product you requested. This may have been a link from a search engine or a bookmarked page that is out of date."

Lastly, in the case of a product that has been replaced by a newer model, you need to have the replacement product id stored in the database with the listing for the discontinued product.

You should use the following asp code...
<%
newaddress="[URL for newer product]"
response.status="301 Moved Permanently"
response.addheader "Location", newaddress
%>
Then use the following text...

"The product you requested has been replaced by a newer model. Please visit the new page at [URL]."

[bj]

URL rewrite on Apache is a good way to make them easier for the bots.

To the best of my knowledge he'll need to use something else since his site is .asp and most likely on a windoze server, unless I've missed something. There is an analogous function (IIS?) on an .asp server, but I understand it's not as easy to use or as flexible as mod rewrite on Apache.

There is another way around this. You say that you have to fill out a form to get a listing of products. Can you set up a script that takes your 100 or so most popular "searches" that will list the actual products available? What we do at EPage Classifieds (where we have a similar issue with transitory content and a database driven site) is we created an area that can be indexed that replicates popular searches.

This might be your best bet. You only need to create the sitemap once and it will always pull up the most recent items for those searches. Pretty clever.

[blitzen]

Beware if you use the unique user information to id the cart is in the URL. My dynamic cart software did that with "usr=12345678".

G indexed unique pages with the "usr" parameter. And, each time G visited my site, "new" pages showed up because a different "usr" parameter was assigned. The pages were the same, but the "usr=" parameter was different.

Duplicate page penalty? I had a hard time getting indexed in G. I thought it was Big Daddy, but now I think it was several reasons, the innocent duplicate pages being one reason why.

If someone clicked that link in the G search results, then they would see that usr's cart! Not good.

G recommends to use "id=" for the unique cart parameter. That meant I would have to hack the software to comply and change "usr=" to "id=". What a pita.

What if the other search engines don't follow the "id=" convention?

I don't know what the other SE's do and was getting REALLY frustrated having my site design and configuration controlled by the SE and discovering the right way via trial and error and loss of sales.

I eventually changed the whole software to use search-engine friendly URLs using mod_rewrite. A real pita.

When I first converted from static to dynamic site, the site PR dropped from 4 to 0 in G but apparently fares okay in the other SE's. It's still at zero for no apparent reason. Sigh.

Just be careful and get alot of expert and experienced advice. If you change something then wait to see results, you're losing money.

Good luck! Dynamic sites are the way to go if the SE's can keep up with technology and stop penalizing the innocents.

[AjiNIMC]

Htaccess can be used for it, rewriting the urls. http://www.acomputerportal.com/htaccess.html has some helpful links.

[bj]

Htaccess can be used for it, rewriting the urls.

Not on a Windows server.

[AjiNIMC]

oops I missed the asp part. I always worked on Apache.

[Peter (IMC)]

I get the impression that what you consider a dynamic website is not the same thing what we technical people consider a dynamic website.

Your site works with a lot of products that are only temporary available and then the product doesn't exist anymore, and won't exist again, or at least not in the way it was presented before.

Your products can be found only through a search form, which indeed means that search engine spiders can't find your products. Spiders don't fill out forms.

One strategy I would consider is to create a directory structure where a visitor can click through to the desired (sub) category. In all these category pages you can present the products with links to the product pages them selves. Do try to use static urls for the category pages. Do also make sure that you don't create very deep levels. A product page shouldn't be more than 4 or 5 clicks away from the homepage.

Also do create "most popular" areas and other ways to highlight products or categories to create the necessary cross linking in the whole site.

[Peter (IMC)]

Originally Posted by bj

Not on a Windows server.

I believe it is called ISAPI_REWRITE on a windows server and it works pretty much the same as htaccess.

[mnsandy]

Hey, problem of this dude is supplemental pages!!! How you don't understand that!!!

If these pages werent' supplemented then google could proceed deeper and crawl all other 10% left.

To get out of sup thats an challange - basically unique content and strong links can help. But if u have hundreds of pages then listings in dmoz, yahoo, other uber high PR and trusted sites like about.com etc could be necesary.

[edhan]

Originally Posted by Nick-UK

2. What happens if Google was to index these pages and then a few days later the product is sold and (I'm assuming) the URL no longer exists? I don't want to spam Google with a lot of product pages that lead to a 404. Any ideas?

Well, 404 will not be regarded as spam to search engines. I like to suggest you make use of 404 page to promote additional products when they are shown this page. You can constant change this page content once the item is sold. Just one page of 404 is simple and easy to maintain for additional profit.

[TrafficProducer]

Google with a lot of product pages that lead to a 404. Any ideas?

Create own 404 page.

This can have links to a menu of links that work, or just one link that works. ( maybe your sitemap.html of main index page)

Or use code such as this in your header to auto link:

<META HTTP-EQUIV="Refresh" CONTENT="0; URL=http://www.trafficseeker.com/cgi-bin/reseller/clickthru.cgi?id=billytail">

The CONTENT="0 gives the time in seconds so you could display an error message "Page not found linking you to our main page", etc.. before autolinking.

[Conficio]

Originally Posted by blitzen

If someone clicked that link in the G search results, then they would see that usr's cart! Not good.

I would dump this shopping cart software like a hot potato. No one should be able to access someone else's shopping cart. You need a software that implements proper session tracking. For example, the usr= parameter is also linked to the IP it is initiated from. The moment another IP accesses it it is dumped.

Originally Posted by blitzen

G recommends to use "id=" for the unique cart parameter. That meant I would have to hack the software to comply and change "usr=" to "id=". What a pita.

Google is not recommending such thing. It recommends to not index pages with such shopping cart identifiers at all. This recommendation is independent of what the name of the parameter is. This example says id=. As mentioned before dump this crappy software and use something that is save and professional.

K<o>

[Conficio]

Hi there,
here is what I would recommend to do:

• First generate on each product page the list of categories it is assigned to as a list of links which fires of the search form with the category query. This improves your internal link structure and allows the customer better navigation. It also allows SEs to reach any product page from another with two hops.
• Build a Google Sitemap.xml every day querying your database directly, and use the appropriate update frequency for the product pages (daily, as they can be sold every day). You can actually submit more than one sitemap.xml to Google one for the static low frequency change content and one for the dayly generated stuff. Saves you some work in not having to combine these two.
• Use the same script to also submit your entries to Groogle and Google base. Gee this will do double duty and bring you more visitors from Googles services.
• Also use a simple conversion script to extract a urllist.txt file for Yahoo!
• May be there are more dynamic price comparison sites and selling boosters (EBay, Amazon) you can tab into with the scripts?
• I'd reinforce, keep the dynamic pages for sold items around (marked as sold) for a while but serve them with a 404 status. That way the SE will drop them soon and the end use gets a menu of products in similar categories to explore your offerings (remember my first recommendation?).

Good luck
K<o>

[Pascal77]

Originally Posted by Nick-UK

Dear All,

Bit of a puzzler this one, I'm much in need of your advice and opinions!

I look after the SEO for a large, dynamically driven database site.

This site ranks well for selected terms in Google/Yahoo/MSN.

However, in Google about 90% of the pages indexed are only shown as supplemental results.

The problem I have is that to get to the "Used products" section of the site you have to use a form to specify which type of products you want to view. Then, after you have narrowed down your search, a list of products are displayed, each one with a quite awful, un-user-friendly URL with about 10 parameters (the site is .asp based).

I know we can implement some URL re-writing techniques to make the pages more spiderable but I have two questions:

1. How can I get these dynamic pages spidered by Google when you have to go through a form first to narrow down your search for a product?

2. What happens if Google was to index these pages and then a few days later the product is sold and (I'm assuming) the URL no longer exists? I don't want to spam Google with a lot of product pages that lead to a 404. Any ideas?

Thank you in advance for your time,

Best wishes,

Nick (UK)

same problem overhere.
A sitemap wil only cost you time with no result.
Also url rewriting wil not make that much difference.

suggestion :
More OBL and IB
Things like most populair etc are a good way to start.
Perhaps you can create a RSS feed with always the latest items(automatic) and submit those feeds to the right places.

But as long as your site is not competing with the big ones or has a good age(older then 3 years) it's gonna be difficult to get those 90% out off supplemental results. Just build and work on your site, that's al you can do.

[kgun]

1. By dynamic URL's you mean pages pulled from a database. You have to use embedded SQL statements to produce those pages / URL's. You can also use embedded SQL statements to produce a XML sitemap.

2. Se how Google produces a XML sitemap.

3. It should be possible to produce an analogous sitemap by ASP at the same time you update the site. You may be able to find a class that does the job for you by browsing an ASP class library.

4. It is said that 99 % of the problems you meet in PHP has already been solved. So by making the right KW search, you may find a plugin or class that does the job for you. Or is this one difference between PHP's open community and ASP's more closed?

5. ISAPI_REWRITE is IIS analogy to Apache's mod_rewrite as far as I remember.

Related links:
http://evolvedcode.net/content/code_...e-rewrites.asp

http://www.websitepublisher.net/foru...read.php?t=944

http://boards.phpbb-seo.com/phpbb-mo...ons-vt241.html