UnGovr Crawler Information
User-Agent: Mozilla/5.0 (compatible; UnGovrBot/0.2.1; +https://ungovr.org/crawler)
Contact: crawl@ungovr.org
What is UnGovr?
UnGovr is a nonprofit civic infrastructure platform dedicated to making government information accessible, searchable, and machine-readable. We believe that public information should be easy to find and use by residents, journalists, researchers, and civic organizations.
What We Crawl
Our crawler (UnGovrBot) automatically discovers and indexes:
- Government websites: City, county, state, and federal agency websites
- Open records: Meeting agendas, minutes, budgets, contracts, and reports
- Laws and codes: Municipal codes and ordinances (including those hosted on commercial platforms)
- Public data: Any information that is legally considered an open record under state and federal law
Our Commitment to Responsible Crawling
We take website performance and server load seriously:
- Respectful crawling: We follow a conservative crawl delay (typically 1-2 seconds between requests)
- Off-peak hours: When possible, we crawl during low-traffic periods
- Bandwidth consideration: We limit concurrent requests to avoid overwhelming servers
- Error handling: We back off immediately if we detect server issues
Request Authentication
UnGovrBot supports Web Bot Authentication per RFC 9421 HTTP Message Signatures. This allows CDNs and web servers to cryptographically verify that requests originate from our legitimate crawler.
Our public key is available at:
https://ungovr.org/.well-known/http-message-signatures-directory
The directory contains our Ed25519 public key in JWK format, which can be used to verify request signatures.
While less secure, you can also validate UnGovrBot requests in two additional ways:
- HTTP request source IP will reverse lookup in DNS to
bot.ungovr.org - The User-Agent HTTP header will contain
UnGovrBot
Contact Us
If you have questions, concerns, or need to report an issue: crawl@ungovr.org
Common reasons to contact us:
- You believe that our crawler is causing a performance issue on your server(s)
- You believe that the crawler is accessing content that should not be public
- You'd like to provide a data feed or API instead of crawling
- You have questions about what we're collecting
For Webmasters
Preferred: Provide a Data Feed
Rather than crawl, we'd prefer to work with you directly! If you manage a government website or a platform hosting government data, we can:
- Consume structured data feeds (JSON, XML, APIs)
- Work with you to schedule crawls during off-peak times
- Receive notifications when content is updated (webhooks)
- Collaborate on data standards and formats
Please contact us at crawl@ungovr.org to discuss options.
Legal Framework
Our crawling is conducted under the understanding that:
- Open records laws (FOIA, state open records acts) require government information to be publicly accessible
- Courts have generally upheld the right to access publicly available web data
- We operate as a US based 501(c)(3) nonprofit serving the civic good
We are committed to operating within applicable laws and welcome dialogue with webmasters, government agencies, and platform providers.
Technical Details
Crawl Rate: 1-2 seconds between requests (conservative)
Concurrent Requests: 1-2 per domain
Respect for Cache: We use If-Modified-Since headers to avoid re-downloading unchanged content
Peak Hour Avoidance: Scheduled crawls typically run during low-traffic periods