Should AOFrozen City Crawler be sleath?

A

AOFrozenCity

New Member
Joined
May 9, 2004
Messages
203
Reaction score
0
I have own AOFrozen City crawler program. It can make a fake agent user and others. Should AOFrozen City crawler look like normal visitor?

Google uses crawler bot. It's user agent is googlebot.
 
AOFrozen City Crawler program can download images, movies, flash files and other, write them in database, calculate ranks, re-index in database, collect sites for cache and collect links for AOFrozen City 2.x/3.x. It will make AOFrozen City site expanded bigger and more information.
 
AOFrozenCity said:
AOFrozen City Crawler program can download images, movies, flash files and other, write them in database, calculate ranks, re-index in database, collect sites for cache and collect links for AOFrozen City 2.x/3.x. It will make AOFrozen City site expanded bigger and more information.
Click to expand...
How would you know if the images or files aren't repeats?
 
If I want to build search engine then it will have file and image repeats. If I want to make gallery, flash games and other then I will build template files for crawler to identify files if they are already or not. Only I will crawl on sites which have archives and same files not change because of the templates.
 
Back
Top