WebABLE's Web Crawler
If you're here, you probably noticed the WebABLE crawler show up in your server logs, and you want to know who we are and why we're crawling your site. Here's some information for you!
About the Crawler
The WebABLE Crawler is a project to identify information on accessibility topics around the web. The crawler analyzes individual web pages to assess whether the topic is about web site accessibility and whether the web site is accessible to people with disabilities. The goal is to provide access to this information at the WebABLE web site so that people with disabilities can easily find topics of interest on pages that are accessible.
How do you display my content?
The title of your content and an excerpt of the content will be displayed on our site, to allow users to get some information about the article topic. The full content is stored in the back-end of the site for searching and indexing purposes. A direct link to your content will always be available.
Opting Out
The WebABLE crawler respects the robots.txt standards to control access. If you don't want us accessing your content, add the following to your site's robots.txt file:
User-agent: webable-crawler Disallow: /
Contact Us
Believe it or not, it's not our intent to cause your site any trouble. If we did - we're sorry! Please let us know, so we can help fine-tune our process to avoid causing trouble. Or let us know if you have any other questions!