Your spider won't leave my site alone
Hi.. What's your spidering policy? I had an RSS feed translated yesterday (thank you) and something at your IP address has been fetching the RSS ever since. The RSS file is actually static and simply there to inform people that the project is now closed. Today I've temporarily renamed the file so that it returns a 404 which is so far being ignored.
[Tue Nov 18 14:12:11 2008] [error] [client 207.7.120.226] File does not exist: /usr/local/www/data/plogger/plog-rss.rss
[Tue Nov 18 14:12:12 2008] [error] [client 207.7.120.226] File does not exist: /usr/local/www/data/plogger/plog-rss.rss
[Tue Nov 18 14:19:19 2008] [error] [client 207.7.120.226] File does not exist: /usr/local/www/data/plogger/plog-rss.rss
[Tue Nov 18 14:19:20 2008] [error] [client 207.7.120.226] File does not exist: /usr/local/www/data/plogger/plog-rss.rss
[Tue Nov 18 14:12:11 2008] [error] [client 207.7.120.226] File does not exist: /usr/local/www/data/plogger/plog-rss.rss
[Tue Nov 18 14:12:12 2008] [error] [client 207.7.120.226] File does not exist: /usr/local/www/data/plogger/plog-rss.rss
[Tue Nov 18 14:19:19 2008] [error] [client 207.7.120.226] File does not exist: /usr/local/www/data/plogger/plog-rss.rss
[Tue Nov 18 14:19:20 2008] [error] [client 207.7.120.226] File does not exist: /usr/local/www/data/plogger/plog-rss.rss
1
person has this question
I have this question, too!
Tell me when someone answers.
The more people who ask this question, the more it gets noticed.
The more people who ask this question, the more it gets noticed.
Create a customer community for your own organization
Plans starting at $19/month
-
Inappropriate?Hi Peter,
First of all apologies for the problem this has caused, I am working on a fix as we speak. I will be implementing a much better cacheing system later this afternoon.
We have actually been too successful with our SEO so lots of search engines from around the world will be finding the feeds, however this has meant that each time a spider visits a feed on our site we need to contact your server to pull in your feed.
We need to do this in some way so that we can tell when new content has been added , but the current way we do it is too intensive. The new way will just ping your server when we need to check for new content rather than pulling in the whole feed.
As I say I will be turning on the new cacheing system this afternoon (just going through final testing) so I will let you know when it is back up and running, you could then turn your feed back on and hopefully you won't get the same problem.
Can you send me the full feed url as well so I can check.
Thanks
Mike
1 person says
this answers the question
-
Inappropriate?Hi Mike,
No harm done :-)
The static feed is at http://www.mildewhall.com/plogger/plo... and this redirects automatically to http://www.mildewhall.com/plogger/plo...
My live feed is at http://www.mildewhall.com/TMP/ffront.rss and this is updated regularly.
Thanks for the speedy reply!
Loading Profile...



EMPLOYEE