I put an .htaccess file in my webroot with the following contents

RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^.*(Googlebot|Googlebot|Mediapartners|Adsbot|Feedfetcher)-?(Google|Image)? [NC]
RewriteRule .* /var/www/503.html

This website is in maintenance mode, and I don't want anything indexed yet. I tested the code with a firefox User-Agent switcher plugin, and looking at the access log it shows this at the end of each log entry, but watching in TamperData or Firebug, it still returns a 200 server response instead of a 503. What am I doing wrong?

"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

contents of /var/www/503.html

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html>
<head>
<title>503 - Service temporary unavailable</title>
</head>
<body>
<h1>503 - Service temporary unavailable</h1>
<p>Sorry, this website is currently down for maintainance please
retry later</p>
</body>
</html>

:::EDIT::: Added small snippet from the rewrite.log Here is a small sample, I went thru the entire log, and everything is referring to a javascript or image file:

172.16.173.26 - - [15/Jun/2010:15:03:31 --0500] [qa-test.com/sid#2b6c1c8ba938][rid#2b6c24cfdd18/initial] (4) [perdir /var/www/qa-test.com/web/] RewriteCond: input='' pattern='^.*(Googlebot|Googlebot|Mediapartners|Adsbot|Feedfetcher)-?(Google|Image)?' [NC] => not-matched

Comments

Can you post the contents of 503.php? Maybe something is preventing the correct header from being output.

Written by zildjohn01

I updated my 503 page, the path, and the original post to show the error I am getting. I will put LogLevel debug to see what happens now.

Written by Hallik

Google only indexes content that is linked to from another site (and from there it crawls the links within your site) or manually submitted to their index.

Written by Josh Stodola

Accepted Answer

You have an infinite loop. The result of your rewrite is being rewritten again.

RewriteBase /
RewriteCond %{HTTP_USER_AGENT} (?:Googlebot|Googlebot|Mediapartners|Adsbot|Feedfetcher)-?(?:Google|Image)? [NC]
RewriteCond $0 !(?:^|/)503\.html$
RewriteRule .* /var/www/503.html [R=503]

As @BalusC pointed out, you also need the R flag.

Written by Artefacto
This page was build to provide you fast access to the question and the direct accepted answer.
The content is written by members of the stackoverflow.com community.
It is licensed under cc-wiki