A web crawler is a program that scours the Internet moving from website to website.
Web crawlers have many different purposes, such as sending out junk mail, finding dead links
within a domain, and searching websites and databases for relevant information like that of
Google. This project focuses on using a web crawler to map the hierarchy of links within a
particular domain. Starting at the St. Lawrence University’s home page, the web crawler gathers
all the links that are found while crawling the St. Lawrence domain.