TombaPublicWebCrawler
Our web crawler indexes publicly available business contact information from websites across the internet.
Technical Details
Version: 3.0 Obeys Robots.txt: Yes User-Agent: Mozilla/5.0 (compatible;
TombaPublicWebCrawler/3.0; +https://tomba.io)
What Is TombaPublicWebCrawler?
TombaPublicWebCrawler is an indexing robot for our business contact search engine. Similar to how Google indexes web pages, our crawler scans publicly available online sources to discover professional contact information.
Our technology processes:
- Corporate websites
- Press releases
- Electronic news services
- Public business directories
- Professional profiles
Using advanced natural language processing, we build a comprehensive database of business professionals and their contact information.
What Does the Crawler Do?
The crawler:
- Visits publicly accessible web pages only
- Extracts business contact information
- Indexes professional email addresses
- Respects all access restrictions
Important: We only analyze public web pages. No private or authenticated content is accessed.
Robots.txt Compliance
Yes, we strictly respect robots.txt.
We honor both Disallow and Allow directives. Our crawler reads the robots.txt file before accessing any page on your website.
Controlling the Crawler
Adjust Crawl Frequency
To set a minimum delay between requests, add to your robots.txt:
User-agent: TombaPublicWebCrawler
Crawl-Delay: [seconds]
Replace [seconds] with your preferred delay time.
Block the Crawler
To prevent TombaPublicWebCrawler from visiting your site entirely:
User-agent: TombaPublicWebCrawler
Disallow: /
Important Notes
- Changes to robots.txt may take time to be detected (before the next scheduled crawl)
- Syntax errors in robots.txt may prevent proper parsing
- The crawler will continue previous behavior if directives are unrecognizable
More about robots.txt
For more information about robots.txt format and usage:
Questions or Concerns?
If you believe TombaPublicWebCrawler is misbehaving on your website, or if you have questions:
Email: support@tomba.io
We take all reports seriously and will investigate promptly.