HTTP status, robots.txt, meta tags
HTTP status code lookup
Generate robots.txt files
Generate HTML meta tags
Check HTTP response headers
Parse and analyze user agent strings