Most of today search engines are crawler-based which have three major elements. As it has been mentioned above, search engines are using software called crawlers or spider to travel the web page in concurrent. As a web robot, Crawler is responsible to go and search in a specific page, create a copy of all the pages content which have been visited and find and track the particular links which associates the current page to other pages. By repeating this method again and again crawlers crawled billions of pages on web every month.
The spiders returns to the site on regular bases, for instance every two weeks or even daily in some cases, to look for changes.
Subsequently search engines will process each page and create an index for second part of search engines. The index looks like an enormous book, holding a copy of each page that the crawler finds. Besides, the index is including the keywords and where they are to be found, plus information about the links form those pages. The index is updated when a web page changes.
The last part is search engine software. When users search for a keyword in search engine they are not actually searching the web, instead they are searching search engines index. As soon as users send the query into search box, search engines software searches the index and finds all the pages including those keywords. As a result, search engines will find thousands of pages. After that search engine identifies the most relevant matches by it algorithm (more than 200 different factors different for each search engine) for example how many time a keyword appear in each page? Dose this keyword appear in the title, heading or URL of the page. It will also check each page importance (PageRank) to understand the quality of each page. Combing all this factors search engines will send back the search result in less than split second with most relevant matches. Search engines will also display relevant ads on the top of the page or on the right hand side.
The common part of all crawler-based search engines has been already mentioned and explained above. However not all of them work in a same way. There are various ways in how these parts are adjusted. As a result, numerous outcomes can be obtained from the same search on several search engines.
The below table shows the number of searches performed each day by US in December 2010 and January 2011, base on comScore figures:
|Core Search Entity||Dec-10 Percentage||Jan-11 Percentage||Dec-10 Actual Num in Billion||Jan-11 Actual Num in Billion|
|Total Explicit Core Search||100%||100%||16,443||16,950|
All in all, nearly 17 billion searches were carried out just in one month in US, Based on the huge number of searches made every year all around the world. Every company tries to get into the first page of results in order to attract more customers.