Hi,
My hosting provider (IXWEBHOSTING) has been hacked several times in the last month.
On the 5th of november all my domains had a .htaccess redirect set up so any search results would go to a antivirus 2009 webpage.

I found this quickly and corrected it almost as it happened - so no problems there.
But ever since google cant crawl my pages, in webmaster tools i get this error:

Network unreachable: robots.txt unreachable
We were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.

My robots.txt is fine, it has correct permissions etc and can be viewed in a browser.
Ive checked the http headers and they all check out (200).

Does any one have any ideas what could be stopping google???

Ive asked my hosts about firewalls etc and they respond with please let us know which ips so we can look into this. -how can i know the ips if they are blocking them!!

PLEASE ANY HELP IS MUCH NEEDED!!

thanks
adam

ps main site is www dot arc-net.co.uk

Create a google account, add webmaster tools and send a message to google
to review/reconsider your site.

Its not that google have banned me, just they are unable to crawl my sites for some reason.

Ive requested anyway - it cant hurt i spose


thanks for the reply

Do you have anything in the .htaccess that may be blocking them?

What about just removing the robots.txt entirely since you are not using it to block anything?

ive tried removing both htaccess and robots.txt

still google thinks its there and tries to download it but cant - same error message.

Ive checked my sitemap and resubmitted 100 times

can i check if they are blocking google at a firewall level - are there any logs i should be looking for?

You could try looking in the apache logs to see what is happening with the robots.txt. To see who is accessing it:

egrep robots.log /usr/local/apache/domlogs/arc-net.co.uk

Then to check the error log

egrep robots.log /usr/local/apache/logs/error_log |grep USER-NAME


That may give you a better idea of what is happening.

Im on a shared host IXWEBHOSTING so i dont get direct access to the logs - i get a log for each domains access but its not much help - i can see other bots ie msn, yahoo slurp, getting my robots.txt but never any google access:

67.195.37.94 - - [21/Nov/2008:05:29:36 -0500] "GET /robots.txt HTTP/1.0" 200 30 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

Have any of your pages been cached since you have been having problems? If so Google probably hasn't been blocked. Has the amount of pages indexed dropped when using the site: command?

You could try temporarily putting a regular link to your robots.txt in your footer sitewide and see when one of the Googlebots follows it. Google doesn't request a robots.txt every time it crawls so doing this can speed it up as it will follow the link.