View Full Version : Inner pages are not crawl ?
divengrabber
08-18-2009, 08:33 AM
Hello Everyone ,
My site working good in Google, yahoo, MSN but major Problem is that my inner pages are crawl by any search engine is very late like resource page ,about us page and other .My home page is crawl with in 3 or 4 days but inner pages are last crawl in 30 dec. It is major problem .Directory summation is continue and updation is also .Please suggest me exact reason and proper ans.
clonfran
08-18-2009, 09:35 AM
You need to get links to those deep pages. I wouldn't worry about the resource page and about us page. Actually, think about having those pages as nofollow in order to concentrate pagerank on your more important pages.
Cigtina23
08-19-2009, 11:59 PM
Just continue the promotion for your site and make sure that it was
quality. SEs crawlers has their way to crawl your inner pages so
don't be bother if your pages is not index or what.. be patient..
that is one of the quality of an SEO.
sitesires
08-22-2009, 04:27 PM
Huge factor to have you inner pages crawled, have a site map somewhere on your home page, this gives the crawlers access to your other pages.
<add this to your signature please>
crankydave
08-22-2009, 05:43 PM
Huge factor to have you inner pages crawled, have a site map somewhere on your home page, this gives the crawlers access to your other pages.
<add this to your signature please>
Disagree.
Site map matters little. What matters is proper navigation.
Dave
williamc
08-22-2009, 05:46 PM
Huge factor to have you inner pages crawled, have a site map somewhere on your home page, this gives the crawlers access to your other pages.
That is what a proper menu system is for.
forumer12
09-02-2009, 02:42 PM
Make sure you have links to all your inner pages on your main page. Also when you are working on backlinks make some backlinks to your inner pages as well.
Try adding fresh content, getting links for those pages and have proper navigational menu so that these pages are directly linked to home page.
i think, creating a comprehensive sitemap of your website is the key..
Atiga
09-28-2009, 02:45 PM
Install a sitemap and submit it to google webmaster. That would help.
It seems that we still have a lot of people that believe that the search engines cannot find inside pages, without a sitemap.
WRONG, people! If your navigation is properly constructed, no sitemap is necessary. If you have a very large site, with several thousand pages, and your navigation is NOT properly constructed, a sitemap may help. Build your navigation right, and you'll have no need of a sitemap. :rolleyes:
williamc
09-28-2009, 10:47 PM
You can't fix stupid, Doc :)
You can't fix stupid, Doc :)
Well, you can....
but it's illegal in all 50 states. :rolleyes:
full house
09-29-2009, 12:55 AM
maybe there is no follow that permitted the bots to inter the inner pages or there are errors on the page that bots think its under construction so they don't continue crawling your site. Use webmaster tool in checking your site.
DOC said ok,Inner pages must be crawled, only the inner pages are'nt included yet.
It can take time for all your pages to get crawled OR indexed, reubin. I have a site up since the beginning of April, and the home page is the only one that's been touched so far.
Patience is the watchword. ;)
LoanShark
10-05-2009, 01:11 PM
Why does it take longer for some sites to get indexed and other sites et indexed in hours? I want to get a better handle on ow big g works.
Why does it take longer for some sites to get indexed and other sites et indexed in hours? I want to get a better handle on ow big g works.
As I understand it (and I'm sure there are others here that can offer a more in-depth explanation), there are a number of factors that enter into it. Two in particular, come to mind for me.
Google crawls and indexes a webpage based upon its perception of the page's popularity. That translates into your inbound links, and their characteristics. If you're the only one that likes your page, Google isn't likely to place it in the SERPs. ;)
Also, any new page is going to start out in the sandbox. That is basically a "wait and see" if it's going to establish any credibility/popularity period. This might be two or three weeks, in very fortunate cases, or over a year in some less fortunate cases. I can't tell you how Google determines that timeframe.
Google might crawl the site, determine that it's brand new, and then return two or three months later to crawl it again. If nothing has changed, Google's "opinion" probably won't change either. Whereas, if there is new, updated material, and/or new IBLs, it might decide to return more often, which could result in a change of their outlook
I think that quality IBLs and fresh content are two very important aspects of getting your pages indexed. You haven't indicated what your IBL status is, so that's just an open-ended comment.
I want to get a better handle on ow big g works. Wouldn't we ALL? ;)
miccharlys
10-06-2009, 01:16 PM
the best way to crawl your inner pages is creating a site map...Once google bot enter in the site map it will find all the pages and thus crawl it
williamc
10-06-2009, 01:19 PM
miccharlys: Yet another person who fails at reading the thread.....
Canonical
10-08-2009, 04:03 PM
As Doc pointed out, submitting a sitemap is not really necessary to get inner pages crawled if your site has a good interlinking structure which is crawlable and the engines can use the navigations/linking to determine which pages on your site are most important and which are not.. And contrary to popular belief, they do not increase the frequency in which your site is crawled.
Pretty much all a sitemap is good for is telling the search engines:
"When you DO get around to crawling me again, if you're only going to index X pages from my site then...
1) index these pages first because they are the most important (highest priority)...
2) if you still have room for more pages from my site in your index then index these pages next as they are the next most important pages (next highest priority)...
3) if you still have room for more pages from my site in your index then index these pages next.
and so on.
So basically, it lets you prioritize the order that the engines pick your pages to be indexed if they are not going to index them all. The engines use it to "assist" with the crawl.
If your site is crawl-friendly, you should really only need sitemaps if you have a huge number of pages on your site like some ecommerce sites or forum sites that have hundreds of thousands or millions of pages.
If you want them to crawl you more frequently, continue to generate new content (i.e new pages) and get more inbound links. If you want them to crawl your inner pages then get inbound links to inner pages.
If each time Google visits your site for a scheduled crawl they find a bunch of new pages of content, they will start coming back more and more often. They try to find a balance between your content generation rate and their crawl rate.
The reason inbound links affect the frequency is that each time they are crawling another site that links to you, they will crawl the link from the other site to your site to make sure the link is still good... discover 404s, 301s, etc. While they are on your site crawling your page that the other site links to, they will often crawl a small number of pages on your site. So the more links, the more of these partial crawls you get.
Excellent explanation, Canonical.
seoconsult1
10-12-2009, 07:40 AM
As I know that you need to short out these things then your inner pages also crawled frequently but never think that if your home page cache in every 3 days then inner pages also. Its quite impossible.
Do these thing if you need to cache you inner pages frequently.
1. Make relevant Meta and Title Tag for each page.
2. Place alt, title, heading and all tags also in these pages.
3. Make some quality back link for all those pages.
4. Improve your internal linking structure.
5. Remove orphan pages.
6. Place unique and informative content on these pages.
7. Make a sitemap.xml and set priority for each pages.
After done these work you see that you inner pages also start caching.
Denit
11-06-2009, 02:00 PM
did any change in robot.txt???
please check it,,,
Impossible related to robot.txt,if yes the spider will not crawl it forever .