Got more questions? Find advice on: SQL | XML | Regular Expressions | Windows
Welcome to AspAdvice Sign in | Join | Help

Re: Don't neglect your robots.txt file

  •  12-03-2007, 8:29 AM

    Re: Don't neglect your robots.txt file

    Sorry for any confusion there.

    My goal was to get Google to crawl my sitemap default.aspx?feed=googlesitemap (or default.aspx?feed=sitemap both work) but inform Google to not index my rss and atom feeds default.aspx?feed=rss20 and default.aspx?feed=atom10.

    When I put this line into my robots.txt file

    default.aspx?feed=

    Google did not index my rss or atom feeds, but it also refused to crawl my sitemap because the sitemap url begins with feed=.  I want Google to crawl my sitemap (that's why I created them) so I changed the above definition to

    default.aspx?feed=rss

    default.aspx?feed=atom

    and now Google will not index my rss and atom feeds, but will crawl my sitemap since my sitemap url is not blocked by my robots.txt entry anymore.

    Hope this helps.

     


    Sincerely,

    Carl
    -----
    vine type - content management with standards in mind - vinetype.com
    -----
    Filed under: , ,
View Complete Thread