Evaluating Privacy Policies under Modern Privacy Laws At Scale: An LLM-Based Automated Approach

Qinge Xie from Georgia Tech presents a large-scale evaluation of modern website privacy policies using a novel LLM-based framework. The research systematizes privacy practices from 10 major US and EU regulations into 34 clauses and analyzes over 100,000 websites to reveal current trends in data collection, sharing, and consumer rights disclosure.

October 7, 2025 4 min read

Evaluating Privacy Policies under Modern Privacy Laws At Scale: An LLM-Based Automated Approach — source: https://www.youtube.com/watch?v=Q10SqEyUUlo

Website privacy policies offer a critical, albeit complex, window into the data handling practices of online services. Motivated by the proliferation of privacy laws post-GDPR, such as the CCPA and other US state-level regulations, this work undertakes a systematic, comprehensive, and large-scale evaluation of modern privacy policies to understand what they disclose about current privacy practices.

Systematizing Modern Privacy Regulations

To ground the analysis in legal requirements, the researchers first investigated 10 prominent privacy laws, including GDPR and nine US state laws. From these legal texts, they identified and systematized 34 distinct clauses on privacy practices, grouping them into four overarching themes:

Personal Information Practices: Details on the categories of personal information collected, the purposes for collection, etc.
Explanation of Consumer Rights: The right to know what data is collected, the right to opt-out of data sharing, etc.
Methods for Exercising Consumer Rights: Instructions on how to submit a consumer request and relevant contact information.
Required Specific Disclosures: Unique requirements from certain laws, such as the CCPA’s mandate for large businesses to publish metrics on the number of consumer privacy requests received annually.

An LLM-Based Analysis Framework

Given the lengthy and unstructured nature of privacy policies, the team developed a pipeline leveraging a Large Language Model (LLM) to automate the analysis.

Model and Preprocessing: They chose the open-source Llama 2 model, allowing them to run it on their own infrastructure, which is crucial for processing long documents. To manage long contexts and improve performance, policies were segmented into 1,000-token chunks.
LLM Tasks: Two primary tasks were designed:
- Coverage Task: A binary classification to determine if a policy segment covers a specific legal clause.
- Inspection Task: A multi-class classification to assess in-depth practices, such as the specific categories of personal information collected and shared, the purposes, and the types of third-party recipients.
Evaluation: The framework was evaluated against three existing academic datasets and a new manually annotated dataset created to cover all 34 identified clauses. The model achieved high performance, with an average F1-score of 0.94 for coverage tasks and 0.88 for inspection tasks, outperforming prior methods.

Large-Scale Measurement and Key Findings

The validated LLM framework was then applied to a dataset of over 18,000 high-confidence privacy policies from a set of ~4,800 distinct domains, including top US and EU companies and a random sample from the top 100,000 websites.

Clause Coverage Insights

Focus on Data Practices: Policies heavily focus on detailing personal information practices but provide significantly less coverage for consumer rights and specific disclosures.
Rank Matters: Higher-ranked domains are more likely to have comprehensive policies that cover more clauses, likely because they are more often subject to privacy laws.
Regional Differences: There are clear differences between US and EU domains. For example, over 61% of top US domains mention the CCPA-specific “right to non-discrimination,” compared to only 11% of EU domains.

Vague Categories: The most commonly collected and shared category of data is “Identifiers.” However, this is an extremely broad category that can include everything from a name to a driver’s license number, offering little concrete understanding to the user.
Sensitive Data: A non-trivial percentage of websites disclose the collection and sharing of sensitive data types, including biometric, sensory, and inferred data.
Collection Implies Sharing: For nearly all data categories, the majority of domains that collect a specific type of personal information also share it with third parties, highlighting the privacy implications of first-party collection.

Third-Party Recipients

Google’s Dominance: Google services are overwhelmingly the most common named third-party recipients of user data.
Advertising Ecosystem: The categories of third parties (e.g., “Advertising Partners,” “Data Analytics”) align closely with the named entities, painting a clear picture of the data-sharing ecosystem.
Government Sharing: A notable portion of websites disclose that they share data with government and law enforcement agencies.

Recommendations and Future Work

The study concludes with several key recommendations for stakeholders:

For Service Providers: Create simplified versions of lengthy policies and provide greater visibility into consumer rights.
For Consumers: Engage more actively in exercising their privacy rights, as the data shows a relatively low number of requests being made.
For Regulators: The impact of regulations is visible in policies. However, regulators should consider narrowing the scope of broad definitions like “personal information” to provide more clarity for users.
For Researchers: Future privacy studies should account for the requirements of multiple different laws and incorporate newer definitions of personal data.

The project’s code, data, and LLM framework have been open-sourced to support broader investigations into web privacy practices.