[meaghannf]

Hi there, so I was going over all my log files and of course I wasn't surprised to see bots crawling through my page but I've noticed that almost every time, they try to enter my page through a file at /robots.txt which doesn't exist on my site. Should I have something here? Is this just how they enter the site or are they actually looking for something at that location?

Basically, I'm wondering how to optimize the work these google and yahoo bots are doing on my site. Thanks in advance for any feedback!

[Cameron]

You need to ask someone on DigitalPoint forums. They know all about stopping these bots and they also know the best methods to get them to crawl your website. They are really helpful over there.

[meaghannf]

I've just taken a look at the forums and you're right, incredibly helpful on the technical side. Thanks for the suggestion!!