Reference

Crawler FAQ

Technical details for site operators about the Panscient crawler, including user-agent strings, opt-out instructions, and request behavior.

Q.01

What is the user-agent of the Panscient crawler?

The production crawler identifies itself with the user-agent panscient.com. A separate testing crawler uses pantest.
Q.02

How do I stop the crawler from visiting my site?

The crawler obeys the Robot Exclusion Standard. Add the following to your robots.txt to block all Panscient crawlers:

User-agent: panscient.com
Disallow: /

User-agent: pantest
Disallow: /
Q.03

How frequently does the crawler visit my site?

The crawler makes at most one request per second to any single domain, and typically far fewer. The full corpus is refreshed quarterly. Defined datasets can be collected and delivered on schedules as frequent as daily.
Q.04

How does the crawler find sites to visit?

The crawler discovers sites primarily from the public list of registered .com domains published by Verisign, and by following links between pages on the public web.
Q.05

Why is the crawler requesting pages that don't exist on my site?

The crawler follows links it finds elsewhere on the web. If another site links to a URL on your domain that no longer exists (or never did), the crawler will attempt to fetch it and record the response. This is normal and can be ignored.
Q.06

Who do I contact with questions about the crawler?

Email crawler@panscient.com for any crawler-related question.

See also the Privacy Notice.


Legal

California Residents

California Consumer Privacy Act Notice

  1. About this Privacy Notice

    Pursuant to the California Consumer Privacy Act of 2018 ("CCPA"), Panscient Inc. ("Panscient," "we," "us" or "our") are providing this California Consumer Privacy Act Notice ("CCPA Notice"). This CCPA Notice supplements our Privacy Policy and applies solely to California consumers, effective January 1, 2025. Terms used but not defined shall have the meaning ascribed to them in the CCPA.

  2. Personal Information Collected, Disclosed, or Sold

    We collect the following categories of personal information, as indicated below:

    CategoryExamplesCollected, sold and disclosed
    A. Identifiers.A real name, alias, postal address, unique personal identifier, online identifier, Internet Protocol address, email address, account name, Social Security number, driver's license number, passport number, or other similar identifiers.We may have collected some of these data elements on California Residents.
    B. Personal information categories listed in the California Customer Records statute (Cal. Civ. Code § 1798.80(e)).A name, signature, Social Security number, physical characteristics or description, address, telephone number, passport number, driver's license or state identification card number, insurance policy number, education, employment, employment history, bank account number, credit card number, debit card number, or any other financial information, medical information, or health insurance information. Some personal information included in this category may overlap with other categories.We may have collected some of these data elements on California Residents.
    C. Protected classification characteristics under California or federal law.Age (40 years or older), race, color, ancestry, national origin, citizenship, religion or creed, marital status, medical condition, physical or mental disability, sex (including gender, gender identity, gender expression, pregnancy or childbirth and related medical conditions), sexual orientation, veteran or military status, genetic information (including familial genetic information).We do not collect this type of information.
    D. Commercial information.Records of personal property, products or services purchased, obtained, or considered, or other purchasing or consuming histories or tendencies.We do not collect this type of information.
    E. Biometric information.Genetic, physiological, behavioral, and biological characteristics, or activity patterns used to extract a template or other identifier or identifying information, such as, fingerprints, faceprints, and voiceprints, iris or retina scans, keystroke, gait, or other physical patterns, and sleep, health, or exercise data.We do not collect this type of information.
    F. Internet or other similar network activity.Browsing history, search history, information on a consumer's interaction with a website, application, or advertisement.We do not collect this type of information.
    G. Geolocation data.Physical location or movements.We do not collect this type of information.
    H. Sensory data.Audio, electronic, visual, thermal, olfactory, or similar information.We do not collect this type of information.
    I. Professional or employment-related information.Current or past job history or performance evaluations.We may have collected some of these data elements on California Residents.
    J. Non-public education information (per the Family Educational Rights and Privacy Act (20 U.S.C. Section 1232g, 34 C.F.R. Part 99)).Education records directly related to a student maintained by an educational institution or party acting on its behalf, such as grades, transcripts, class lists, student schedules, student identification codes, student financial information, or student disciplinary records.We do not collect this type of information.
    K. Inferences drawn from other personal information.Profile reflecting a person's preferences, characteristics, psychological trends, predispositions, behavior, attitudes, intelligence, abilities, and aptitudes.We do not collect this type of information.

    We collect this personal information from publicly available web sources.

    We share this personal information with: our affiliates; our customers; our vendors, service providers, suppliers, agents and representatives; joint venture partners; resellers, distributors, referral partners and integrators; and other parties where required by law or to protect our rights.

    We may also sell this personal information to our affiliates; our customers; our vendors; joint venture partners; resellers, distributors, referral partners and integrators.

    We may use this personal information to operate, manage, and maintain our business, to provide our products and services, to communicate with you, for our vendor management purposes, and to accomplish our business purposes and objectives, including, for example, using personal information to: develop, improve, repair, and maintain our products and services; process or fulfill a request or other transactions submitted to us; personalize, advertise, and market our products and services; conduct research, analytics, and data analysis; maintain our facilities and infrastructure; undertake quality and safety assurance measures; conduct risk and security control and monitoring; detect and prevent fraud; perform identity verification; perform accounting, audit, and other internal functions; comply with law, legal process, and internal policies; maintain records; exercise and defend legal claims; and fulfill legal obligations.

  3. Consumer Rights

    You have the right to request that we disclose to you

    1. the categories of personal information we collected about you and the categories of sources from which we collected such information;
    2. the specific pieces of personal information we collected about you;
    3. the business or commercial purpose for collecting personal information about you; and
    4. the categories of personal information about you that we shared or disclosed and the categories of third parties with whom we shared or to whom we disclosed such information in the preceding 12 months.

    You also have the right to request that we delete personal information we collected from you subject to certain exceptions.

    You also have the right to not be discriminated against in pricing and services because you exercise any of your rights under the CCPA. Panscient does not offer financial incentives or price or service differences to consumers in exchange for the retention or sale of a consumer's personal information.

  4. How to Make a Request

    You may make a request for the disclosures or deletion described above by emailing ccpa@panscient.com or via this form.

    You may be required to submit proof of your identity for these requests to be processed as a verifiable consumer request. We may not be able to comply with your request if we are unable to confirm your identity or to connect the information you submit in your request with personal information in our possession. You may designate an authorized agent to make a request on your behalf subject to proof of identity and authorization.

    We will respond to your request consistent with the CCPA, which does not apply to certain information, such as information made available from government records, certain data subject to the Fair Credit Reporting Act (FCRA), Gramm-Leach-Bliley Act (GLBA) and certain other laws, and where its application is preempted by, or in conflict with, federal law or the United States or California Constitution.

  5. Changes

    We will update this CCPA Notice from time to time. Any changes will be posted on this page with an updated revision date.

Last updated: June 3, 2025