well what is the limit of crawler if after using some crawling function we give some site link which how much pages or how much data we can get and did crawler has any limit of crawling the site ?
infact going to make very simple search engine that can cache keywords title & description of site automatic when people give their Link of site:)
please give me some guaidness and details on this