What are the essential tools that every IT person should have in the modern IT world? That is the idea behind a new Kingston University module called IT Toolbox. Over a 12 week semester, first year students will be guided through a series of activities such as blogging, running a server, client and server side scripting, search, social networking and problem solving. Each of those activities will be published here and anyone is welcome to join in.
This is a lecture for IT Toolbox and as I mentioned last week I am trying to get students to ask themselves hard questions about common technology systems in order to prepare them for analysis, design and build projects of their own in the future.
Key questions for this session
What was search like before Google and how did Google become so powerful?
What sorts of technologies are involved?
How does Google decide the order for search results?
How does Google make money?
Why does Google do all the other things it does?
What will happen in the future?
Terminology you should know about search and Google
What can we learn from looking at Google that will be useful in the future?
Google (1998) introduced reputation based ranking (PageRank) which was less prone to manipulation. Google’s interface was kept very simple and very fast.
Specialist servers/software used for crawling, indexing, load balancing, ranking, proxy serving, image searching, image caching, advertising matching, ad serving, spelling, suggestions, tracking, reporting, data mining, spam filtering, click fraud checking
crawlers (spiders) collect data from web sites and pages by visiting sites and returning data
indexers take data and create indices about their content
front end matches visitor queries to indexed data and returns content
A crawl of the web probably takes 6-10 weeks.
Crawler starts with a list of URLs, ‘visits’ each, returns the contents of each document including additional URLs (internal and external links) which are added to the list to be crawled - recursively!
Interesting things about crawlers
Web analytics packages allow you to see when crawlers have visited your site
Google, Bing and yahoo! provide tools to control how often the spider visits
Data collected includes the relative links between sites as well as the data on the pages - this is only one of many types of meta data that are recorded
Sites that are updated more frequently are crawled more often
PageRank - how Google decides on the order of its results
The original algorithm was produced as part of a computer science project at Stanford University and is published. The current algorithm is a commercial secret.
The algorithm ranks pages according to the popularity of the page as measured by the number (and quality) of in-bound links to that page.
Pages with more links rank higher than pages with fewer links but…
Pages with better links ranker higher than pages with poorer links and…
Better links come from pages with higher ranks - recursion again!
The home page of this site has about 774 links pointing at it including 94 links from other sites.
A link from the Guardian is worth more than a link from this site because the Guardian has a higher ranking than this site
It is very hard for spammers to create artificial reputation without getting real links from other high ranking sites (and Google works hard to detect these and modifies its algorithm as necessary)
BUT Relevance is still one of the most important factors in selecting which result is shown first. Within the most relevant pages, “PageRank (latest version)” is used to define the order.
You cannot pay to be listed higher in the results
Trying to trick Google’s algorithms can get you banned from the index completely
PageRank is not a simple algorithm these days and some sites are more trusted than others
How does Google make money?
Google is an advertising company and auctions the space on the right hand side of the search results as well as displaying ads on other sites.
Google made $21 billion from AdWords in 2008
Adwords is successful because it presents relevant advertising just as a potential customer has expressed a need or desire (via a search query)
Why does Google do all the other things it does?
Sometimes these other things create additional advertising opportunities such as Gmail
Sometimes they create additional data insights that help improve search
Sometimes they create tools to help advertisers reach their potential customers
Often they create additional opportunities for Google to keep talking to its audience
Always they are thinking about how they enlarge their business for the benefit of their shareholders
What will happen in the future?
Google will continue to improve its algorithm to try and provide the best possible search experience
Google will continue to provide tools to encourage advertisers to spend money
Bing and yahoo! will try to create better search experiences for customers
Location based, thematic and ‘meaning based’ searches are very much in fashion: cinema times, flight bookings, product search, image and video
“Semantic Web” ideas provide a fertile ground for future developments
Someone may come up with a new idea (and make billions!)
Some terminology you should know about search and Google
crawlers, spiders
paid and sponsored results versus natural or organic results
snippets - the little descriptions under each search result
PageRank and trust
Revelance and reputation
On page and off page search engine optimisation - things you can do to improve your relevance and reputation
What can we learn from looking at Google that will be useful in the future?
Simple looking systems can be very complex behind the scenes
Understanding the basic principles can help us understand some of the complexity
Very big systems require lots of hardware, software, support and power
New ideas can disrupt traditional businesses
Data about data can be key to creating a business
Computer science projects can turn into billion dollar projects
Recent comments:
On October 5, 2009 at 2:02 PM, Kunal Ramchandani wrote:
A good intro into all things Google! An interesting thing to note for your section on the future is that Google is working on Google Caffeine ("the new Google"). Mashable's take here: http://mashable.com/2009/08/10/google-caffeine/ Where you can test it: http://www2.sandbox.google.com/
Sound installation at Vauxhall Ragged School as part of London Open House [Sep 20, 2009]
Photo blog
Search
About this site
This site is an educational resource, a research project and a publication channel written by Jonathan Briggs, Professor of eCommerce at Kingston University. As well as being Professor at Kingston, Jonathan...
On October 5, 2009 at 2:02 PM, Kunal Ramchandani wrote:
A good intro into all things Google! An interesting thing to note for your section on the future is that Google is working on Google Caffeine ("the new Google").
Mashable's take here: http://mashable.com/2009/08/10/google-caffeine/
Where you can test it: http://www2.sandbox.google.com/
www.kunalramchandani.com
Jonathan replies: Thanks Kunal (works with me at the OTHER media on all things Google)