If you specify in your robots.txt file that such-and-so robot should
not look at anything on your site, will it note that and never return?

I'm assuming they'd rather not burn the disk space, and will come back
as often as before.

IOW if you want to temporarily make your site off-limits to all robots
because of testing/whatever, then later you open it up again, have you
forever disappeared yo'sef?

--
http://www.ren-prod-inc.com/hug_soft...action=contact

Fleeing from the madness of the . jungle
hug <contact_info@sig_line.clickit> stumbled into news:alt.www.webmaster
and said:

The algo behind every bot is different, some won't even ack that you have
a robots.txt to start with.

--
William Tasso - I was looking back to see, if she was looking back to see,
if I was looking back at her.

How To Usenet: http://williamtasso.com/usenet/ - prove I'm wrong.

The best way is to block the ip range serverwide

--
Bil

*Kind Regards
*Bill

'www.ukwebmasterforums.com
(http://www.ukwebmasterforums.com

Supported By 'Google Adsense
(http://adsense.ukwebmasterforums.com/), Drive Ads To Your Sit
-----------------------------------------------------------------------
Bill's Profile: http://www.ukwebmasterforums.com/member.php?userid=
View this thread: http://www.ukwebmasterforums.com/showthread.php?t=737

"William Tasso" <SpamBlocked@tbdata.com> wrote:

Obvious.

Assumed.

I'm concerned about the major search-bots; googlebot, msnbot, slurp,
etc. Do they continue checking robots.txt to see if they're allowed
in? I don't want to "temporarily" shut them out then find it's
permanent, that's my real concern.

Thanks. btw, good morning William.

--
http://www.ren-prod-inc.com/hug_soft...action=contact

On Sun, 29 Jan 2006 05:27:06 -0700, hug put finger to keyboard and
typed:

Google will recheck every now and then, so removing robots.txt (or
removing an exclusion from it) will result in Google indexing the
areas that were previously forbidden to it.

I don't know about the other majors, but I'd guess that they're the
same as Google in this respect.

Mark
--
http://www.MotorwayServices.info - read and share comments and opinons
"Look at the stars; look how they shine for you"

hug <contact_info@sig_line.clickit> wrote:

A lot of unwelcome bots don't fetch that file, or just fetch it and ignore
it.

No. Google for one follows links. It seems to be very hard to get pages
out of Google :-D

--
John Experienced (web) developer: http://castleamber.com/
Perl SEO tools: http://johnbokma.com/perl/
NEW ----> Textpad reference card (pdf): http://johnbokma.com/textpad/

hug <contact_info@sig_line.clickit> wrote:

Yes, but they cache the file, so a change to this file doesn't show up
immediatly. Also, removing pages from Google seems to take ages.

Doubt if that's an easy thing if you have incoming links.

--
John Experienced (web) developer: http://castleamber.com/
Perl SEO tools: http://johnbokma.com/perl/
NEW ----> Textpad reference card (pdf): http://johnbokma.com/textpad/

Bill <Bill.22ebu2@noreply.ukwebmasterforums.com> wrote:

Preferable on the firewall. But yup, that works for bots using a range and
sticking to it.

--
John Experienced (web) developer: http://castleamber.com/
Perl SEO tools: http://johnbokma.com/perl/
NEW ----> Textpad reference card (pdf): http://johnbokma.com/textpad/

On 29 Jan 2006 John Bokma wrote in alt.www.webmaster

Google still lists stuff from when Charles was my host.

--
D?
http://yorkshirepete.com/

Thanks all who replied.

--
http://www.ren-prod-inc.com/hug_soft...action=contact