How do Webcrawlers work?

[billtrasolini]

Can someone please explain how Webcrawlers work?

Can Google, for example, see every directory and file in your Web Root?

Sincerely,


Bill

[veikoh]

Yes, if you do not set deny *.* in your robots.txt file

[adventity]

they simply keep track of your keywords. and keep vary their algorithms for crawling but one thing is clear that crawlers crawl the websites through their backlinks........