UnGovr Crawler Information
User-Agent: Mozilla/5.0 (compatible; UnGovrBot/0.3.53; +https://ungovr.org/crawler)
Contact: crawler@ungovr.org
What is UnGovr?
UnGovr is a nonprofit civic infrastructure platform dedicated to making government information accessible, searchable, and machine-readable. We believe that public information should be easy to find and use by residents, journalists, researchers, and civic organizations.
What We Crawl
Our crawler (UnGovrBot) automatically discovers and indexes:
- Government websites: City, county, state, and federal agency websites
- Open records: Meeting agendas, minutes, budgets, contracts, and reports
- Laws and codes: Municipal codes and ordinances (including those hosted on commercial platforms)
- Public data: Any information that is legally considered an open record under state and federal law
UnGovrBot also reads a company's own public website when that company asks us to, for a LexLint Preview, read by LexSite. It never scans a company's site on its own initiative.
Our Commitment to Responsible Crawling
We take website performance and server load seriously:
- Respectful crawling: We follow a conservative crawl delay (typically 1-2 seconds between requests)
- Off-peak hours: When possible, we crawl during low-traffic periods
- Bandwidth consideration: We limit concurrent requests to avoid overwhelming servers
- Error handling: We back off immediately if we detect server issues
Request Authentication
UnGovrBot supports Web Bot Authentication per RFC 9421 HTTP Message Signatures. This allows CDNs and web servers to cryptographically verify that requests originate from our legitimate crawler.
Our public key is available at:
https://www.ungovr.org/.well-known/http-message-signatures-directory
The directory contains our Ed25519 public key in JWK format, which can be used to verify request signatures.
While less secure, you can also validate UnGovrBot requests in three additional ways:
- HTTP request source IP will reverse lookup in DNS to
bot.ungovr.org, andbot.ungovr.orgresolves back to that IP - The User-Agent HTTP header will contain
UnGovrBot - The source IP is one of the addresses listed under IP Addresses below
IP Addresses
UnGovrBot crawls from a small, fixed set of addresses. You can allowlist them directly:
45.33.54.25346.225.226.27173.255.221.1822600:3c01::2000:2dff:feac:c4af2600:3c01::2000:bdff:fe44:e1bc2a01:4f8:1c16:3936::1
This list is generated from our live server inventory each time this page is built, so it changes only when the servers do. Each address reverses in DNS to bot.ungovr.org. The signature check above remains the strongest verification, since it does not depend on any address.
LexSite
User-Agent: Mozilla/5.0 (compatible; UnGovrBot/0.3.53; +https://ungovr.org/crawler) LexSite/1
LexSite is LexLint's website reader. It reads a company's public pages when that company asks, for a LexLint Preview: a preview of what a full LexLint would show, made only from the company's public web pages and nothing app-specific. LexLint is UnGovr's service for the law that applies to software. When a company asks for a LexLint Preview, LexSite reads that company's public web pages as UnGovrBot, and LexLint lists the software laws that may apply to what those pages describe. The result goes to the person who asked for it. UnGovr does not publish it.
When a scan runs
- Only with the company's permission. A person at the company asks for the scan and gives permission, on the company's behalf, to read its domain. UnGovr keeps a record of that permission, and without one no scan starts.
- Only that company's site. A scan reads the domain it has permission for, plus any other domain of the company's that the person named when asking. If the site's privacy policy is on another domain, the scan reads that one page there and follows no link from it.
- Only public pages. A scan reads pages anyone can open. It does not sign in and submits no forms.
How to recognize a scan
- User-Agent: UnGovrBot's own, with
LexSite/1after it, so your logs can tell a scan you asked for from UnGovrBot's other reading. Before November 13, 2026 the token wasLexPeek/1, and before October 12, 2026 it wasLexScan/1; a rule that names either should nameLexSite/1now. - Signature: every request carries a Web Bot Authentication signature made with UnGovrBot's key, published in the key directory above.
- Addresses: a scan comes only from the UnGovrBot addresses listed under IP Addresses, each of which reverses in DNS to
bot.ungovr.org.
To let a scan you asked for through a firewall or a bot-protection service, allowlist those addresses. The signature remains the strongest check, since it does not depend on any address.
How fast a scan reads
- One request at a time, at least 2 seconds apart, or further apart when your robots.txt sets a longer Crawl-delay for UnGovrBot
- At most 40 pages, 20 MB and 20 minutes of reading per scan, the privacy-policy page on another domain included
- A scan reads your site once. It does not return unless someone asks for a new scan
How to stop a scan
- robots.txt. A scan reads robots.txt as UnGovrBot before every request. It follows the group for
UnGovrBot, or the*group when no group names UnGovrBot. On the host the scan was asked to read, a robots.txt that disallows the home page, cannot be reached, or sets a Crawl-delay longer than 60 seconds ends the scan before it reads anything else. Elsewhere, a page those rules disallow is skipped, and so is every page on another host whose robots.txt cannot be reached or asks for a longer delay. - Refuse it. If the host the scan was asked to read refuses its first request (HTTP 401, 403, 429 or 503), or your site refuses three pages in a row, the scan ends. An HTTP 429 on any later page stops the reading there. A CAPTCHA on any page of your site ends the scan as well, because a scan never answers one.
- Write to us. Send the domain to crawler@ungovr.org and we stop the scan. A stopped scan sends no further request.
When a scan ends this way, the person who asked for it is told why, so they can decide with whoever runs the site whether to allow it.
The privacy-policy page on another domain is the one exception: there, a robots.txt rule, a refusal or a CAPTCHA skips that page, and the scan goes on with the company's own site.
UnGovr Agent
User-Agent: Mozilla/5.0 (compatible; UnGovrAgent/0.3.53; +https://ungovr.org/crawler)
UnGovr Agent is not a crawler. It is the name UnGovr uses when it takes an action on a government website instead of reading one: filling in and submitting a form the site itself offers, such as signing up for a public meeting's notice list. Each action is either one a person asked UnGovr to carry out through our own products, or one UnGovr takes in its own name for its public-records work.
UnGovr Agent always identifies itself. You can verify it the same ways as UnGovrBot:
- Each page it opens carries a Web Bot Authentication signature made with the same key, published in the same directory as above
- The User-Agent HTTP header will contain
UnGovrAgent - HTTP request source IP will reverse lookup in DNS to
agent.ungovr.org, andagent.ungovr.orgresolves back to that IP
Questions about UnGovr Agent go to the same address as crawler questions: crawler@ungovr.org.
Contact Us
If you have questions, concerns, or need to report an issue: crawler@ungovr.org
Common reasons to contact us:
- You believe that our crawler is causing a performance issue on your server(s)
- You believe that the crawler is accessing content that should not be public
- You'd like to provide a data feed or API instead of crawling
- You have questions about what we're collecting
For Webmasters
Preferred: Provide a Data Feed
Rather than crawl, we'd prefer to work with you directly! If you manage a government website or a platform hosting government data, we can:
- Consume structured data feeds (JSON, XML, APIs)
- Work with you to schedule crawls during off-peak times
- Receive notifications when content is updated (webhooks)
- Collaborate on data standards and formats
Please contact us at crawler@ungovr.org to discuss options.
Legal Framework
Our crawling is conducted under the understanding that:
- Open records laws (FOIA, state open records acts) require government information to be publicly accessible
- Courts have generally upheld the right to access publicly available web data
- We operate as a nonprofit organization serving the civic good
We are committed to operating within applicable laws and welcome dialogue with webmasters, government agencies, and platform providers.
Technical Details
Crawl Rate: 1-2 seconds between requests (conservative)
Concurrent Requests: 1-2 per domain
Respect for Cache: We use If-Modified-Since headers to avoid re-downloading unchanged content
Peak Hour Avoidance: Scheduled crawls typically run during low-traffic periods