seo - How can I prevent Googlebot from crawling my Underscore client-side templates? -


in google webmaster tools, under crawl errors/other, we're seeing 400 error urls this:

/family-tree/<%=tree.user_url_slug%>/<%=tree.url_slug%> 

enter image description here

this not real url, or url intended crawled. underscore/backbone template:

<script type="text/template" class="template" id="template-trees-list">   <% _.each(trees, function(tree) { %>     <a href="/family-tree/<%=tree.user_url_slug%>/<%=tree.url_slug%>" rel="nofollow">       <%= tree.title %>     </a>   <% }); %> </script> 
  1. why google crawling inside of script block?
  2. why google ignoring rel="nofollow" attribute?
  3. is there else can keep googlebot away our underscore templates?

update: i'm open using robots.txt if can find right pattern keep pages , block bad pages. example, want keep /surnames/jones/queries while blocking /surnames/jones/queries/<%=url_slug%>. have thousands this. looks googlebot may support basic patterns, though not full regexes.

update 2: well, doesn't @ root cause, , seems bit brittle long-term solution, tested in gwt following robots.txt patterns work:

user-agent: googlebot disallow: /*url_slug%%3e$ disallow: /*url_slug%%3e/$ 

just block these via robots.txt , you'll fine


Comments

Popular posts from this blog

java - Intellij Synchronizing output directories .. -

git - Initial Commit: "fatal: could not create leading directories of ..." -