Summarize, Judge, Refine: Decoupled Content Understanding and Policy Learning for Multimodal Content Moderation

# Decoupled Content Understanding and Policy Learning for Multimodal Content Moderation
Source: [https://arxiv.org/html/2609.22094](https://arxiv.org/html/2609.22094)
###### Abstract

Content moderation systems traditionally entangle multimodal understanding with policy\-specific classification, requiring full pipeline retraining for every policy change and suffering from label scarcity since multimedia cannot be meaningfully augmented\. We proposeSummarize\-Judge\-Refine \(SJR\), a two\-model architecture that decouples these concerns via a natural language interface: a multimodal Content Model produces structured text summaries, and a text\-only Policy Model classifies them against policy definitions\. An iterative co\-training loop refines the Content Model via GRPO to produce policy\-relevant summaries, while text\-space augmentation generates adversarial summary variants—an augmentation pathway impossible on raw multimedia—enabling few\-shot policy bootstrap\. Every decision is grounded in a human\-readable summary, providing interpretability as a structural byproduct\. On misleading advertisement detection, SJR achieves \+23\.6% relative non\-misleading F1 over a zero\-shot chain\-of\-thought baseline, outperforming end\-to\-end SFT, STaR/RFT, and RLFT\. Notably, a variant trained on*zero real violating examples*—with all positive\-class data synthetically generated—matches the full\-data model within 0\.2% relative on violating F1, demonstrating that new policies can launch without any real violation data\.

Summarize, Judge, Refine: Decoupled Content Understanding and Policy Learning for Multimodal Content Moderation

Zeeshan Ahmed, Yang Qin and Hanqing HuangMeta AI\{ahzee, yangqin, hanqinghuangy\}@meta\.com

## 1Introduction

Online platforms host billions of pieces of multimedia content—videos, images, and text—that must be evaluated against hundreds of evolving content policies\. The scale and diversity of this challenge demands automated content moderation systems that are accurate, interpretable, and rapidly adaptable to new policies\. For example, an advertisement might show a trustworthy product page, but its landing page delivers a deceptive phishing site; detecting this requires understanding both modalities and reasoning about their consistency against a “misleading advertising” policy\.

The dominant paradigm trains end\-to\-end multimodal classifiers that map raw content directly to violation decisions\(Kielaet al\.,[2020](https://arxiv.org/html/2609.22094#bib.bib10); Chatzakouet al\.,[2019](https://arxiv.org/html/2609.22094#bib.bib8)\)\. While effective in high\-resource settings, this approach suffers from two fundamental limitations:

1. 1\.Tight coupling\.Content understanding and policy logic are entangled in model weights\. Every policy change—from definition updates to entirely new categories—requires retraining the full multimodal pipeline, a process that typically takes weeks of data collection, labeling, and training\. This creates an operational bottleneck: platforms maintain hundreds of policies that evolve continuously\.
2. 2\.Cold\-start problem\.New policies launch with scarce labeled data\. Multimedia data is*fundamentally resistant to augmentation*: rotating an image or adding Gaussian noise does not alter its semantic content, cropping risks removing the violating region, and generating realistic violating videos raises both technical and ethical challenges\. This asymmetry—text is easy to augment, multimedia is not—is the central observation that motivates our approach\.

#### Our insight\.

Natural language can serve as a*complete information bridge*between content understanding and policy classification\. Projecting multimodal content into structured text summaries yields modularity \(independent training\), augmentability \(NLP toolkit for policy learning\), interpretability \(summaries as rationales\), multilingual coverage \(any\-language content normalized to a common language\), and training efficiency \(policy iteration requires SFT on a text\-only model rather than the full multimodal pipeline\)\.

#### Key challenge\.

A generic summarizer produces*policy\-agnostic*descriptions that omit the details needed for correct classification\. We address this through*iterative co\-training*: a reinforcement learning loop refines the Content Model so that its summaries progressively emphasize policy\-relevant signals, while the Policy Model learns from increasingly informative summaries—a cooperative dynamic where both models teach each other through the text bridge\.

#### Contributions\.

Our contributions are as follows:

1. 1\.Atwo\-model architecture\(Content Model→\\rightarrowText Summary→\\rightarrowPolicy Model\) that formally decouples multimodal content understanding from policy classification via a natural language interface, enabling independent training and modular policy updates, with interpretability as a structural byproduct \(§[3](https://arxiv.org/html/2609.22094#S3)\)\.
2. 2\.Aniterative co\-training algorithmin which*both*the Content Model and Policy Model evolve across rounds: the trained Policy Model serves as an autorater whose classification accuracy provides the reward signal for GRPO\-based\(Shaoet al\.,[2024](https://arxiv.org/html/2609.22094#bib.bib20)\)refinement of the Content Model, which in turn produces improved summaries that the Policy Model retrains on—a cooperative dynamic structurally distinct from single\-model self\-training \(STaR/RFT\), fixed\-reward RL, or frozen\-teacher distillation \(§[3](https://arxiv.org/html/2609.22094#S3)\)\.
3. 3\.Atext\-space augmentation frameworkthat generates adversarial and counterfactual summary variants from an augmentation model, enabling few\-shot policy bootstrap from as little as non\-violating examples only—an augmentation pathway that is impossible on raw multimedia\. We show that a Policy Model trained on*zero real violating examples*matches a full\-data model within 0\.2% relative on violating F1 \(§[3\.4](https://arxiv.org/html/2609.22094#S3.SS4)\)\.
4. 4\.Comprehensive experiments on misleading advertisement detection, comparing SJR against zero\-shot prompting, chain\-of\-thought reasoning\(Weiet al\.,[2022](https://arxiv.org/html/2609.22094#bib.bib26)\), end\-to\-end SFT classifiers, self\-training \(STaR/RFT;Zelikmanet al\.,[2022](https://arxiv.org/html/2609.22094#bib.bib25); Yuanet al\.,[2023](https://arxiv.org/html/2609.22094#bib.bib23)\), and reinforcement learning from task feedback \(RLFT\), demonstrating that SJR with text\-space augmentation achieves superior accuracy while providing interpretable decisions \(§[5](https://arxiv.org/html/2609.22094#S5)\)\.

## 2Related Work

### 2\.1End\-to\-End Content Moderation

Traditional content moderation trains classifiers directly on multimodal inputs with policy\-specific labels\. Early work used CNNs for image\-based detection\(Chatzakouet al\.,[2019](https://arxiv.org/html/2609.22094#bib.bib8)\), while recent approaches leverage multimodal transformers\(Kielaet al\.,[2020](https://arxiv.org/html/2609.22094#bib.bib10)\)and causal debiasing methods for multimodal clickbait detection\(Yuet al\.,[2024](https://arxiv.org/html/2609.22094#bib.bib28)\)\. Multi\-task approaches share a backbone across policies but still couple the learned representation to the joint policy distribution\(Schusteret al\.,[2021](https://arxiv.org/html/2609.22094#bib.bib18)\)\. End\-to\-end supervised fine\-tuning \(SFT\) of large multimodal models has become a strong baseline: the entire model is trained to map raw inputs directly to policy labels, achieving high accuracy when sufficient labeled data is available\. However, every new policy or label\-distribution shift requires retraining the full pipeline\.

More recently, LLMs have been deployed for content moderation at industrial scale\.Qiaoet al\.\([2024](https://arxiv.org/html/2609.22094#bib.bib3)\)describes a system for scaling LLM\-based reviews for advertising policy violations in online advertising—a setting closely related to ours\.Oaket al\.\([2025](https://arxiv.org/html/2609.22094#bib.bib4)\)use LLMs for re\-ranking social media content to reduce exposure to harmful material at platform scale\. These systems achieve strong performance but remain end\-to\-end: content perception and policy logic are entangled in the same model weights, they leave little to no room for multimedia data augmentation, and extending to new policies requires retraining\. SJR explicitly separates content summarization from policy classification and optimizes both iteratively\.

### 2\.2Policy\-as\-Prompt and LLM\-Based Moderation

Recent work explores encoding policies as natural language prompts to LLMs\.Pallaet al\.\([2025](https://arxiv.org/html/2609.22094#bib.bib16)\)formalizes the “Policy\-as\-Prompt” framework, showing LLMs can moderate content given policy descriptions as context\.Bonagiriet al\.\([2025](https://arxiv.org/html/2609.22094#bib.bib6)\)demonstrates scalable few\-shot moderation using LLMs\. Chain\-of\-thought \(CoT\) prompting\(Weiet al\.,[2022](https://arxiv.org/html/2609.22094#bib.bib26)\)further improves LLM\-based classification by eliciting intermediate reasoning steps before a final judgment\. Relatedly, NLI\-based approaches frame violation detection as textual entailment, where the content serves as a premise and the policy description as a hypothesis\(Yinet al\.,[2019](https://arxiv.org/html/2609.22094#bib.bib22); Calabreseet al\.,[2022](https://arxiv.org/html/2609.22094#bib.bib7)\), naturally supporting zero\-shot transfer to new policies\. These approaches achieve implicit decoupling through the prompt or entailment interface\.

### 2\.3Self\-Training, Iterative Refinement, and RL Fine\-Tuning

Self\-training methods iteratively improve a model by training on its own high\-confidence predictions\. STaR \(Self\-Taught Reasoner;Zelikmanet al\.,[2022](https://arxiv.org/html/2609.22094#bib.bib25)\) bootstraps chain\-of\-thought reasoning by filtering for correct\-answer traces and retraining, progressively improving reasoning quality\. RFT \(Rejection Fine\-Tuning;Yuanet al\.,[2023](https://arxiv.org/html/2609.22094#bib.bib23)\) generalizes this idea by sampling multiple candidate solutions and retraining on correct ones\. RAFT \(Reward rAnked FineTuning;Donget al\.,[2023](https://arxiv.org/html/2609.22094#bib.bib24)\) bridges SFT and RL by ranking model\-generated samples with a reward model and fine\-tuning on top\-ranked outputs\. Reinforcement learning from human feedback \(RLHF;Ouyanget al\.,[2022](https://arxiv.org/html/2609.22094#bib.bib27)\) and its AI\-feedback variant RLAIF\(Baiet al\.,[2022](https://arxiv.org/html/2609.22094#bib.bib5); Leeet al\.,[2024](https://arxiv.org/html/2609.22094#bib.bib12)\)train language models using preference\-based reward signals, typically with a frozen reward model\. Reinforcement learning from task feedback \(RLFT\) directly optimizes model outputs against task\-specific reward functions \(e\.g\., string\-match accuracy\), avoiding the need for preference data altogether\.

SJR’s co\-training loop relates to these paradigms but differs in a structural way:*both*the generator \(Content Model\) and the evaluator \(Policy Model\) are iteratively retrained\. In RLHF/RLAIF, the reward model is fixed; in STaR/RFT, there is a single model that self\-improves\. In SJR, the Policy Model provides a reward signal to the Content Model, is then retrained on the Content Model’s improved summaries, and produces an updated reward signal—creating a*cooperative co\-training*dynamic\. Unlike knowledge distillation\(Hintonet al\.,[2015](https://arxiv.org/html/2609.22094#bib.bib2)\), where the teacher is frozen, or GANs\(Goodfellowet al\.,[2014](https://arxiv.org/html/2609.22094#bib.bib1)\), where the generator and discriminator compete adversarially, SJR’s two models cooperate: the Policy Model rewards the Content Model for useful summaries while itself learning from them\. We adopt GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2609.22094#bib.bib20)\), which eliminates the need for a separate critic network by computing advantages relative to a group of sampled outputs\.

### 2\.4Data Augmentation for Low\-Resource Classification

Text augmentation techniques—synonym replacement and simple edits\(Wei and Zou,[2019](https://arxiv.org/html/2609.22094#bib.bib21)\), back\-translation\(Sennrichet al\.,[2016](https://arxiv.org/html/2609.22094#bib.bib19)\), LLM\-based synthetic generation\(Schick and Schütze,[2021](https://arxiv.org/html/2609.22094#bib.bib17)\), and counterfactual augmentation\(Kaushiket al\.,[2020](https://arxiv.org/html/2609.22094#bib.bib9)\)—are effective for low\-resource NLP tasks\.Zenget al\.\([2024](https://arxiv.org/html/2609.22094#bib.bib29)\)recently showed that multimodal LLMs can generate synthetic training data for misinformation detection, validating the broader idea of LLM\-generated augmentation for content moderation\. However, these techniques operate on text and cannot be directly applied to raw multimedia: rotating a video frame does not change its semantic content, and generating realistic violating images raises technical and ethical challenges\.

SJR’s text\-space augmentationresolves this asymmetry by introducing an Augmentation Model that takes raw multimodal content and a policy description as input, and generates adversarial summary variants with specific misleading elements injected—producing training pairs whose*outputs*are text summaries even though their*inputs*are real multimedia\. This mechanism helps enable few\-shot policy bootstrap: multimodal examples—even non\-violating ones only—are fed to the Augmentation Model, which produces thousands of diverse text\-space examples for the Policy Model\.

MultimodalContentxxContent Modelfθf\_\{\\theta\}TextSummaryssStage 1: SUMMARIZEMultimodalContentx′x^\{\\prime\}AugmentModelhψh\_\{\\psi\}AdversarialVariantsStage 1\.5: AUGMENTPolicy Modelgϕg\_\{\\phi\}Decisiony^\\hat\{y\}Policy DescriptionppStage 2: JUDGEStage 3: REFINE — RL reward signal

Figure 1:The Summarize\-Judge\-Refine \(SJR\) architecture\.Stage 1: a multimodal Content Model projects raw multimedia into a structured text summary\.Stage 1\.5\(optional\): an Augment Model generates adversarial and counterfactual summary variants\.Stage 2: a text\-only Policy Model classifies summaries against a policy description\.Stage 3: the Policy Model’s accuracy serves as a reward to refine the Content Model via GRPO, with a KL penalty anchoring it to pretrained behavior\. Dashed arrows indicate optional augmentation\. Stages repeat iteratively\.

## 3Method: Summarize\-Judge\-Refine \(SJR\)

### 3\.1Architecture

Given multimodal contentx∈𝒳x\\in\\mathcal\{X\}, a policy descriptionpp, and a violation labely∈\{0,1\}y\\in\\\{0,1\\\}, the standard approach learns a monolithic functionh:𝒳×𝒫→\{0,1\}h:\\mathcal\{X\}\\times\\mathcal\{P\}\\rightarrow\\\{0,1\\\}that entangles content perception with policy reasoning\. SJR instead decomposes this into two models connected exclusively through natural language text \(Figure[1](https://arxiv.org/html/2609.22094#S2.F1)\)\.

#### Content Modelfθf\_\{\\theta\}\.

A multimodal language model that maps raw multimedia content to a tuple of structured textual summaries:s=\(s1,s2,…,sn\)=fθ​\(x\)s=\(s\_\{1\},s\_\{2\},\\ldots,s\_\{n\}\)=f\_\{\\theta\}\(x\), where each component is a natural language string describing one facet of the content \(e\.g\., the on\-platform and off\-platform sides\)\. Initialized from a pretrained multimodal LLM,fθf\_\{\\theta\}ingests raw visual, auditory, and textual signals and produces summaries in sufficient detail for downstream policy classification\. In the initial round, these summaries are generic—a general\-purpose captioning of what the model observes\. Through iterative co\-training \(§[3\.2](https://arxiv.org/html/2609.22094#S3.SS2)\), the summaries progressively emphasize policy\-relevant signals\.

#### Policy Modelgϕg\_\{\\phi\}\.

A text\-only classifier that takes a summaryssand policy descriptionppas input, outputting a binary verdict:

y^=gϕ​\(s,p\)\\hat\{y\}=g\_\{\\phi\}\(s,p\)\(1\)Designed to be a lightweight text\-only LLM \(see §[4](https://arxiv.org/html/2609.22094#S4)\),gϕg\_\{\\phi\}operates entirely in text space\. Its input is a pair of structured summaries \(e\.g\., an on\-platform content summary and an off\-platform landing\-page summary\) together with a policy definition prompt\.

#### Natural Language Interface\.

The summaryssis the*only*information channel between the two models\. The Policy Model never observes raw pixels, audio, or video frames\. This strict separation enforces five properties: \(i\)modularity—the Content Model and Policy Model can be updated independently; \(ii\)augmentability—the Policy Model’s training data \(text summaries\) can be augmented even when the underlying multimedia cannot; \(iii\)interpretability—the summary constitutes a human\-readable rationale for every decision; \(iv\)multilinguality—the Content Model absorbs language diversity by summarizing any\-language content into a common language, so a single Policy Model handles all languages without per\-language labeled data; and \(v\)training efficiency—policy iteration requires fine\-tuning only a text\-only model instead of the full multimodal pipeline\.

### 3\.2Training Algorithm: Iterative Co\-Training

We train SJR through an iterative loop of three stages plus an optional augmentation step \(see Algorithm[1](https://arxiv.org/html/2609.22094#alg1)in Appendix[A](https://arxiv.org/html/2609.22094#A1)for full pseudocode\)\. Each roundkkgenerates summaries with the current Content Model, trains the Policy Model on those summaries \(plus augmented data\), and then refines the Content Model via GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2609.22094#bib.bib20)\)using the Policy Model’s accuracy as reward\. For each input,GGcandidate summaries are sampled, scored by the Policy Model, and group\-relative advantages are computed\. The Content Model is then updated via a clipped policy gradient with a KL penalty anchoring it to the pretrained distribution\. The cooperative dynamic is important: the Content Model learns to describe content in a way that makes the Policy Model’s job easier, and the Policy Model learns to classify from increasingly informative summaries\.

#### Computational design\.

Each stage involves only light fine\-tuning \(one or two epochs\), not full retraining\. In practice, 2–3 rounds are needed for the co\-training loop to stabilize, as the summaries quickly become policy\-relevant and subsequent rounds yield diminishing returns\.

### 3\.3Reward Design

The reward for each generated summary is based on whether the trained Policy Model, acting as an autorater, correctly classifies the summary:

R​\(s,p,y\)=𝟙​\[y^=y\]R\(s,p,y\)=\\mathbb\{1\}\\left\[\\hat\{y\}=y\\right\]\(2\)wherey^=gϕ​\(s,p\)\\hat\{y\}=g\_\{\\phi\}\(s,p\)is the Policy Model’s predicted label \(extracted via string matching from its generated output\) andyyis the gold label\. The reward is binary: 1 if the prediction matches the label, 0 otherwise\. This rule\-based reward uses the same string\-match mechanism as standalone RLFT, but with a structural difference: the reward signal flows through the*trained Policy Model*operating on text summaries, not through the Content Model itself\. The Policy Model thus serves as an autorater—its classification accuracy on the generated summary determines whether that summary is useful for downstream decision\-making\.

### 3\.4Text\-Space Augmentation for Few\-Shot Policy Bootstrap

The natural language interface enables a powerful data augmentation strategy that is*structurally impossible*on raw multimedia\. For a new policypnewp\_\{\\text\{new\}\}with limited—or even exclusively non\-violating—multimodal examples, we generate diverse text\-space training data for the Policy Model using an Augmentation Model that takes raw multimodal content as input\.

#### Adversarial summary generation\.

The primary augmentation mechanism uses anAugmentation Modelhψh\_\{\\psi\}\(a capable multimodal LLM\) to generate adversarial summary variants\. Given the raw multimedia contentxxand the policy descriptionpp,hψh\_\{\\psi\}produces multiple counterfactual summaries that represent plausible variations of the content:

1. 1\.Misleading variants:hψh\_\{\\psi\}modifies the summary to introduce specific deceptive elements—contradictory claims between the on\-platform and off\-platform descriptions, exaggerated promises, or misdirected calls\-to\-action—producing labeled positive \(violating\) examples anchored on real content patterns\.
2. 2\.Benign variants:hψh\_\{\\psi\}generates paraphrases that preserve the non\-violating semantics while varying surface form, producing diverse negative examples\.
3. 3\.Borderline variants:hψh\_\{\\psi\}generates edge cases with subtle discrepancies, forcing the Policy Model to learn fine\-grained decision boundaries instead of superficial cues\.

Each augmented example carries a label assigned by the Augmentation Model based on whether the generated variant constitutes a violation under the policy definition\. This produces a44–6×6\\timesexpansion per input example, yielding55–7​n7ntotal training examples fromnnmultimodal seeds \(which need not include any violating examples\)—all without additional data collection or labeling\.

#### Zero\-positive policy bootstrap\.

A particularly powerful consequence of text\-space augmentation is that new policies can launch with*zero real positive examples*\. The Augmentation Model takes real non\-violating content and generates synthetic violating variants, producing a training set where all negative examples are real and all positive examples are synthetically generated\. This eliminates the positive\-label collection bottleneck: a new policy can ship as soon as a small set of verified legitimate examples is available, with the Policy Model bootstrapping its understanding of violations entirely from synthetic contrast\. We evaluate this extreme few\-shot scenario in §[5](https://arxiv.org/html/2609.22094#S5)\.

#### Why this is impossible on multimedia\.

Generating a “misleading variant” of a real video advertisement—modifying the landing page to contain contradictory claims while keeping visual fidelity—is technically intractable\. Text summaries, by contrast, can be freely modified, paraphrased, and counterfactually edited while maintaining coherence\. The natural language interface transforms an impossible multimedia augmentation problem into a straightforward text generation task\.

## 4Experimental Setup

### 4\.1Task: Misleading Advertisement Detection

We evaluate SJR onmisleading advertisement detection, a multimodal content moderation task that requires comparing what an advertisement promises \(*on\-platform content*\) against what its destination actually delivers \(*off\-platform landing page*\)\. An advertisement is*misleading*\(positive\) if the landing page contradicts, misrepresents, or fails to deliver on the claims made in the ad creative; it is*non\-misleading*\(negative\) if the landing page is consistent with the advertisement\. The exact decision boundary varies across applicable policies, making this a setting where SJR’s modular policy updates are particularly valuable\.

This task is a natural fit for SJR because it inherently requires*cross\-modal comparison*: the on\-platform ad may contain images, video, and overlay text, while the off\-platform landing page is a separate web document with its own text and visual elements\. An end\-to\-end classifier must jointly encode both modalities and learn the comparison logic; SJR instead summarizes each side into text and delegates the comparison to the Policy Model\.

### 4\.2Dataset

We use a dataset of paid advertisements collected from a major online advertising platform\. Each example consists of:

- •On\-platform signals: advertisement text and associated images/videos, OCR\-extracted text from visual creatives, and the destination URL linking to the off\-platform landing page\.
- •Off\-platform signals: scraped landing\-page text content and landing\-page screenshots\.
- •Label: binary decision \(misleading vs\. non\-misleading\)\.

#### Training set\.

Approximately 120K examples with knowledge\-distilled labels \(55% non\-misleading, 45% misleading\) obtained from a larger teacher model, supplemented with a small set of human\-annotated examples to anchor label quality\. Distilled labels are expected to contain inherent noise, which particularly affects RL\-based methods\.

#### Evaluation set\.

A separate set of∼\{\\sim\}2K examples with expert\-corrected labels from domain specialists\. Each example carries an importance weight reflecting estimated real\-world prevalence, enabling both unweighted and weighted metric reporting\. Multiple landing\-page scrapes per advertisement support per\-entity OR\-aggregated evaluation \(an advertisement is flagged if*any*of its scrapes is classified as misleading\)\.

#### Evaluation metrics\.

We report precision, recall, and F1 for both the positive class \(misleading\) and the negative class \(non\-misleading\)\. All metrics are weighted by estimated real\-world prevalence and per\-entity OR\-aggregated\. The primary metric isweighted per\-entity non\-misleading F1, which measures the system’s ability to correctly identify legitimate advertisements—the operationally significant metric, since false positives would result in legitimate ads being incorrectly blocked\.

### 4\.3Baselines

We compare SJR against four families of baselines \(Table[1](https://arxiv.org/html/2609.22094#S4.T1)\), all evaluated on the same evaluation set for direct comparability\.

Table 1:Baseline methods\. All methods use the same base LLM family and are evaluated on the same evaluation set\.#### SJR variants\.

We evaluate several configurations of SJR to isolate the contribution of each component:

- •Two\-step \(no training\): Content Model summarizes, Policy Model classifies—both using pretrained weights with no fine\-tuning\. Tests the architecture alone\.
- •SJR \(JUDGE only\): Policy Model is fine\-tuned on real summaries from the pretrained Content Model \(no augmentation, no RL on the Content Model\)\. Tests the value of training the Policy Model alone\.
- •SJR \+ Augmentation: Same as above, but the Policy Model is trained on real summaries*plus*adversarial variants from the Augmentation Model\. The Content Model remains pretrained \(no RL\)\.
- •SJR \+ Augmentation \(multi\-round\): The full SJR loop with multiple co\-training rounds: the Content Model is refined via GRPO \(REFINE stage\) using the Policy Model as autorater, and the Augmentation Model regenerates fresh adversarial summaries each round\.
- •SJR \+ Augmentation \(non\-misleading only\): Same as multi\-round, but the seed data contains*only non\-misleading*examples—zero real violating examples\. All positive\-class data is synthetically generated via the zero\-positive bootstrap mechanism \(§[3\.4](https://arxiv.org/html/2609.22094#S3.SS4)\)\. Tests whether SJR can learn to detect violations without ever seeing a real one\.

### 4\.4Implementation Details

All three models—Content, Augmentation, and Policy—use the same multimodal LLM capable of processing images, video, and text\. The SJR architecture is designed to use a smaller, text\-only model for the Policy Model, which would reduce inference cost and enable faster policy iteration; in our experiments, we use the same model for all three roles to isolate the architectural contribution from model\-capacity effects\. We leave the cost benefits of a smaller Policy Model to future work\.

All fine\-tuning—SFT, STaR/RFT, RLFT, and SJR—uses LoRA adapters \(rank 8\) for parameter\-efficient training\. For theend\-to\-end SFTbaseline, the multimodal LLM is fine\-tuned on the full training set, mapping raw content directly to a binary label\.STaR/RFTruns 3–5 iterative rounds: in each round, the model generates multiple CoT traces per example at non\-zero temperature, retains only traces that produce the correct final answer, and retrains on the filtered set\.RLFToptimizes a single model via GRPO with binary string\-match accuracy as reward over multiple rounds\.

ForSJR, GRPO uses a group size ofG=16G=16candidate summaries per input\. The Policy Model is trained for 3 epochs in the first round and 1 epoch in subsequent rounds\. Content Model RLFT \(when enabled\) uses 1 epoch per round with a KL penalty coefficientλ\\lambdato prevent drift\. All baselines and SJR variants use temperatureT=0T=0\(greedy decoding\) at evaluation for reproducibility\.

## 5Results

Table[2](https://arxiv.org/html/2609.22094#S5.T2)presents the main results\. SJR with augmentation and multi\-round co\-training achieves the highest F1 on both classes: 79\.25% non\-misleading F1 \(\+23\.6% relative over the zero\-shot CoT baseline\) and 93\.73% misleading F1 \(\+2\.1% relative\), outperforming all baselines by a wide margin\.

Table 2:Main results on misleading advertisement detection \(%\)\. All metrics are weighted and per\-entity OR\-aggregated\. Top group: baselines; bottom group: SJR variants, §[4](https://arxiv.org/html/2609.22094#S4)\. “nm\. only” = trained with non\-misleading examples only \(zero real violating examples\)\. “multi\-round” = three rounds of co\-training\. Best value per column inbold\.#### Baselines\.

We report all relative improvements against the zero\-shot CoT reference \(non\-misleading F1 64\.14%, misleading F1 91\.78%\), a precision\-heavy but recall\-limited baseline \(non\-misleading recall 52\.41%\)\. End\-to\-end SFT and STaR/RFT are the strongest single\-model baselines, lifting non\-misleading F1 by \+17\.5% and \+18\.9% relative, respectively\. The two methods achieve these gains through different precision–recall trade\-offs: STaR/RFT sharpens precision \(80\.66%\) at the cost of recall \(72\.28%\), while SFT improves both more evenly\. On the misleading side, SFT lifts F1 by \+1\.2% relative, while STaR/RFT and RLFT dip slightly below the reference \(−\-0\.8% and−\-1\.2% relative\)\. RLFT shows a more pronounced trade\-off: it achieves the highest non\-misleading recall of any method \(82\.08%\) but the lowest precision \(64\.83%\)\.

#### SJR component analysis\.

The SJR rows isolate each component’s contribution\. Two\-step inference without training \(64\.94% non\-misleading F1\) is roughly on par with the zero\-shot CoT baseline, confirming that the architectural decomposition is not inherently beneficial—the gains come from training\. Training the Policy Model on real summaries \(JUDGE only\) raises non\-misleading F1 to 74\.21%, approaching end\-to\-end SFT and demonstrating that a text\-only classifier over summaries can nearly match a full multimodal model\. Single\-round augmentation adds a modest gain \(74\.75%\)\. The decisive improvement comes from multi\-round co\-training: the REFINE stage pushes non\-misleading F1 to 79\.25%—a 4\.5 percentage\-point gain that accounts for roughly 30% of SJR’s total lift above the zero\-shot CoT baseline\.

#### Cold\-start\.

The non\-misleading\-only variant trains with*zero real violating examples*—all positive\-class data is synthetically generated via text\-space augmentation \(§[3\.4](https://arxiv.org/html/2609.22094#S3.SS4)\)\. After three co\-training rounds, it achieves 78\.21% non\-misleading F1 \(\+21\.9% relative over the zero\-shot CoT baseline\) and 93\.58% misleading F1 \(\+2\.0% relative\), within 1\.3% and 0\.2% relative of the full\-data variant\. New policies can therefore launch as soon as a non\-violating seed set is collected, removing the positive\-label bottleneck entirely\.

## 6Analysis

We analyze three aspects of SJR’s behavior: the dynamics of multi\-round co\-training, the role of augmentation, and the convergence of the cold\-start variant\.

Table 3:Per\-round F1 \(%\) for SJR \+ Augmentation with full training data vs\. non\-misleading\-only \(cold\-start\) data\. NM = non\-misleading, M = misleading\. Best per column inbold\.#### Co\-training dynamics\.

Table[3](https://arxiv.org/html/2609.22094#S6.T3)shows a non\-monotonic trajectory: non\-misleading F1 dips from 74\.75% \(R1\) to 72\.25% \(R2\) before climbing to 79\.25% \(R3\)\. The R2 dip reflects a recall drop \(73\.61%→\\to69\.11%\) as the refined Content Model’s summaries temporarily misalign with the Policy Model\. By R3, the models co\-adapt: recall recovers to 80\.54% while precision improves to 78\.00%, yielding a balanced profile unique among all methods\. At least two REFINE iterations are needed for this balance to emerge\.

#### Role of augmentation\.

Single\-round augmentation adds only 0\.54 percentage points over JUDGE\-only \(74\.75% vs\. 74\.21%\), but it enables effective multi\-round co\-training by providing diverse adversarial variants that sharpen the Policy Model’s boundary and yield a more informative reward signal for Content Model refinement\. The cumulative contribution grows from \+0\.54 percentage points in Round 1 to \+5\.04 percentage points by Round 3\.

#### Cold\-start convergence\.

The non\-misleading\-only variant converges toward full\-data SJR across rounds, and unlike the full\-data variant, its trajectory is monotonically increasing \(no R2 dip\)\. On misleading F1, it actually leads in R2 \(93\.85% vs\. 92\.25%\), suggesting synthetic violations provide complementary diversity\. By R3, both variants converge to within 0\.15 percentage points on misleading F1\.

## 7Conclusion

We presented Summarize\-Judge\-Refine \(SJR\), a framework that decouples multimodal content understanding from policy learning through a natural language interface\. On misleading advertisement detection, SJR outperforms all baselines on both the non\-misleading and misleading F1 metrics, achieving \+23\.6% and \+2\.1% relative improvement over a zero\-shot CoT baseline—gains that exceed end\-to\-end SFT, STaR/RFT, and RLFT by a substantial margin\. The multi\-round co\-training loop is the decisive contributor, accounting for roughly 30% of SJR’s total lift\.

SJR’s most practically significant result is its cold\-start capability: a variant trained with zero real violating examples achieves near\-parity with the full\-data system, removing the positive\-label bottleneck that has historically delayed new policy launches\. While SJR’s two\-model inference pipeline is more computationally expensive than a single end\-to\-end model, it can serve as a high\-quality teacher for knowledge distillation into a smaller student model, combining SJR’s accuracy and interpretability at training time with efficient single\-model inference at serving time\.

## Limitations

SJR introduces a two\-model inference pipeline, increasing the number of LLM calls relative to a single end\-to\-end classifier\. The architecture is designed to offset this through a smaller, text\-only Policy Model; in our experiments, we use the same model for all three roles to isolate the architectural contribution, deferring the inference cost reduction to future work\. The co\-training loop exhibits non\-monotonic dynamics—performance can dip in intermediate rounds before recovering—though in our experiments three rounds consistently yielded the best results\.

The cold\-start mechanism’s effectiveness depends on the Augmentation Model’s generation quality; we expect that more capable Augmentation Models will produce higher\-quality synthetic violations, further improving cold\-start performance\.

Our evaluation covers a single content moderation task \(misleading advertisement detection\) on an internal dataset\. While the architecture is domain\-agnostic, generalization to other multimodal policy tasks remains to be demonstrated\.

## References

- Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon,et al\.\(2022\)Constitutional ai: harmlessness from ai feedback\.arXiv preprint arXiv:2212\.08073\.Note:Introduces Constitutional AI \(CAI\), a method for training harmless AI assistants using AI\-generated feedback rather than human labels, establishing the RLAIF paradigm\.Cited by:[§2\.3](https://arxiv.org/html/2609.22094#S2.SS3.p1.1)\.
- A\. Bonagiri, L\. Li, R\. Oak, Z\. Babar, M\. Wojcieszak, and A\. Chhabra \(2025\)Towards safer social media platforms: scalable and performant few\-shot harmful content moderation using large language models\.arXiv preprint arXiv:2501\.13976\.Note:Demonstrates that LLMs can perform few\-shot content moderation by conditioning on policy descriptions and a handful of examples, achieving competitive performance with dedicated classifiers\.Cited by:[§2\.2](https://arxiv.org/html/2609.22094#S2.SS2.p1.1)\.
- A\. Calabrese, B\. Ross, and M\. Lapata \(2022\)Explainable abuse detection as intent classification and slot filling\.Transactions of the Association for Computational Linguistics \(TACL\)10,pp\. 1440–1454\.Note:Proposes framing abuse detection as NLI, where policy descriptions serve as hypotheses and content as premises, enabling explainable and policy\-aware classification\.Cited by:[§2\.2](https://arxiv.org/html/2609.22094#S2.SS2.p1.1)\.
- D\. Chatzakou, I\. Leontiadis, J\. Blackburn, E\. De Cristofaro, G\. Stringhini, A\. Vakali, and N\. Kourtellis \(2019\)Detecting cyberbullying and cyberaggression in social media\.ACM Transactions on the Web \(TWEB\)13\(3\),pp\. 1–51\.Note:Presents methods for detecting cyberbullying and cyberaggression in social media using multimodal features from posts including text and network signals\.Cited by:[§1](https://arxiv.org/html/2609.22094#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.22094#S2.SS1.p1.1)\.
- H\. Dong, W\. Xiong, D\. Goyal, Y\. Zhang, W\. Chow, R\. Pan, S\. Diao, J\. Zhang, K\. Shum, and T\. Zhang \(2023\)RAFT: reward ranked finetuning for generative foundation model alignment\.Transactions on Machine Learning Research \(TMLR\)\.Note:Proposes Reward rAnked FineTuning \(RAFT\), which ranks model\-generated samples by reward score and fine\-tunes on the top\-ranked outputs, bridging SFT and RLHF\.Cited by:[§2\.3](https://arxiv.org/html/2609.22094#S2.SS3.p1.1)\.
- I\. Goodfellow, J\. Pouget\-Abadie, M\. Mirza, B\. Xu, D\. Warde\-Farley, S\. Ozair, A\. Courville, and Y\. Bengio \(2014\)Generative adversarial nets\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.27,pp\. 2672–2680\.Cited by:[§2\.3](https://arxiv.org/html/2609.22094#S2.SS3.p2.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.Cited by:[§2\.3](https://arxiv.org/html/2609.22094#S2.SS3.p2.1)\.
- D\. Kaushik, E\. Hovy, and Z\. C\. Lipton \(2020\)Learning the difference that makes a difference with counterfactually\-augmented data\.InInternational Conference on Learning Representations \(ICLR\),Note:Introduces counterfactual data augmentation where human annotators minimally edit examples to flip labels, creating training pairs that isolate causal features from spurious correlations\.Cited by:[§2\.4](https://arxiv.org/html/2609.22094#S2.SS4.p1.1)\.
- D\. Kiela, H\. Firooz, A\. Mohan, V\. Goswami, A\. Singh, P\. Ringshia, and D\. Testuggine \(2020\)The hateful memes challenge: detecting hate speech in multimodal memes\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.33,pp\. 2611–2624\.Note:Introduces a benchmark of 10K multimodal memes requiring joint reasoning over image and text to detect hate speech, demonstrating limitations of unimodal approaches\.Cited by:[§1](https://arxiv.org/html/2609.22094#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.22094#S2.SS1.p1.1)\.
- H\. Lee, S\. Phatale, H\. Mansoor, T\. Mesnard, J\. Ferret, K\. Lu, C\. Bishop, E\. Hall, V\. Carbune, A\. Rastogi, and S\. Prakash \(2024\)RLAIF vs\. rlhf: scaling reinforcement learning from human feedback with ai feedback\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),PMLR, Vol\.235,pp\. 26874–26901\.Note:Shows that RL from AI feedback \(using an LLM as reward model\) can match or exceed RLHF performance, enabling scalable alignment without costly human annotations\.Cited by:[§2\.3](https://arxiv.org/html/2609.22094#S2.SS3.p1.1)\.
- R\. Oak, M\. Haroon, C\. Jo, M\. Wojcieszak, and A\. Chhabra \(2025\)Re\-ranking using large language models for mitigating exposure to harmful content on social media platforms\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(ACL\),Note:Proposes using LLMs for re\-ranking social media content to reduce exposure to harmful material, demonstrating LLM\-based content integrity enforcement at platform scale\.Cited by:[§2\.1](https://arxiv.org/html/2609.22094#S2.SS1.p2.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in Neural Information Processing Systems \(NeurIPS\)35,pp\. 27730–27744\.Note:Introduces InstructGPT, demonstrating reinforcement learning from human feedback \(RLHF\) for aligning language models to follow instructions, establishing the RLHF training paradigm\.Cited by:[§2\.3](https://arxiv.org/html/2609.22094#S2.SS3.p1.1)\.
- K\. Palla, J\. L\. Redondo García, C\. Hauff, F\. Fabbri, H\. Lindström, D\. R\. Taber, A\. Damianou, and M\. Lalmas \(2025\)Policy\-as\-prompt: rethinking content moderation in the age of large language models\.InACM Conference on Fairness, Accountability, and Transparency \(FAccT\),pp\. 840–854\.Note:Formalizes the policy\-as\-prompt paradigm where content moderation policies are encoded as natural language prompts to LLMs, eliminating the need for policy\-specific model training\.Cited by:[§2\.2](https://arxiv.org/html/2609.22094#S2.SS2.p1.1)\.
- W\. Qiao, T\. Dogra, O\. Stretcu, Y\. Lyu, T\. Fang, D\. Kwon, C\. Lu, E\. Luo, Y\. Wang, C\. Chia, A\. Fuxman, F\. Wang, R\. Krishna, and M\. Tek \(2024\)Scaling up llm reviews for google ads content moderation\.InProceedings of the 17th ACM International Conference on Web Search and Data Mining \(WSDM\),Note:Describes Google’s system for scaling LLM\-based content moderation for Google Ads, addressing policy violation detection in advertisements at production scale\.Cited by:[§2\.1](https://arxiv.org/html/2609.22094#S2.SS1.p2.1)\.
- T\. Schick and H\. Schütze \(2021\)Generating datasets with pretrained language models\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 6943–6951\.Note:Demonstrates that pretrained LMs can generate high\-quality labeled training data from task descriptions, enabling effective data augmentation for low\-resource text classification\.Cited by:[§2\.4](https://arxiv.org/html/2609.22094#S2.SS4.p1.1)\.
- T\. Schuster, A\. Fisch, and R\. Barzilay \(2021\)Get your vitamin c\! robust fact verification with contrastive evidence\.InProceedings of the Conference of the North American Chapter of the ACL \(NAACL\),pp\. 624–643\.Note:Proposes contrastive evidence\-based fact verification with multi\-task learning across related NLI and fact\-checking objectives\.Cited by:[§2\.1](https://arxiv.org/html/2609.22094#S2.SS1.p1.1)\.
- R\. Sennrich, B\. Haddow, and A\. Birch \(2016\)Improving neural machine translation models with monolingual data\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 86–96\.Note:Introduces back\-translation as a data augmentation technique for NMT, generating synthetic parallel data from monolingual corpora\.Cited by:[§2\.4](https://arxiv.org/html/2609.22094#S2.SS4.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\.K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Note:Introduces Group Relative Policy Optimization \(GRPO\), a variant of PPO that eliminates the critic network by using group\-relative advantages computed from multiple sampled outputs per input\.Cited by:[item 2](https://arxiv.org/html/2609.22094#S1.I2.i2.p1.1),[§2\.3](https://arxiv.org/html/2609.22094#S2.SS3.p2.1),[§3\.2](https://arxiv.org/html/2609.22094#S3.SS2.p1.2)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35,pp\. 24824–24837\.Note:Demonstrates that prompting LLMs to produce intermediate reasoning steps \(chain\-of\-thought\) before the final answer significantly improves performance on arithmetic, commonsense, and symbolic reasoning tasks\.Cited by:[item 4](https://arxiv.org/html/2609.22094#S1.I2.i4.p1.1),[§2\.2](https://arxiv.org/html/2609.22094#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2609.22094#S4.T1.1.2.1.2.1.1)\.
- J\. Wei and K\. Zou \(2019\)EDA: easy data augmentation techniques for boosting performance on text classification tasks\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 6382–6388\.Note:Proposes four simple text augmentation operations \(synonym replacement, random insertion, random swap, random deletion\) that significantly improve text classification in low\-data regimes\.Cited by:[§2\.4](https://arxiv.org/html/2609.22094#S2.SS4.p1.1)\.
- W\. Yin, J\. Hay, and D\. Roth \(2019\)Benchmarking zero\-shot text classification: datasets, evaluation and entailment approach\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 3914–3923\.Note:Proposes zero\-shot text classification via NLI where class labels are converted to hypotheses, enabling classification without any task\-specific training data\.Cited by:[§2\.2](https://arxiv.org/html/2609.22094#S2.SS2.p1.1)\.
- J\. Yu, S\. Wang, H\. Yin, Z\. Sun, R\. Xie, B\. Zhang, and Y\. Rao \(2024\)Multimodal clickbait detection by de\-confounding biases using causal representation inference\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 10128–10139\.Cited by:[§2\.1](https://arxiv.org/html/2609.22094#S2.SS1.p1.1)\.
- Z\. Yuan, H\. Yuan, C\. Li, G\. Dong, K\. Lu, C\. Tan, C\. Zhou, and J\. Zhou \(2023\)Scaling relationship on learning mathematical reasoning with large language models\.arXiv preprint arXiv:2308\.01825\.Note:Introduces Rejection Fine\-Tuning \(RFT\), which generates multiple candidate solutions per problem, rejects incorrect ones, and fine\-tunes on the remaining correct solutions to improve mathematical reasoning\.Cited by:[item 4](https://arxiv.org/html/2609.22094#S1.I2.i4.p1.1),[§2\.3](https://arxiv.org/html/2609.22094#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2609.22094#S4.T1.1.4.3.2.1.1)\.
- E\. Zelikman, Y\. Wu, J\. Mu, and N\. D\. Goodman \(2022\)STaR: bootstrapping reasoning with reasoning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35,pp\. 15476–15488\.Note:Introduces Self\-Taught Reasoner \(STaR\), an iterative self\-training method where a model generates chain\-of\-thought rationales, filters for those that produce correct answers, and retrains on the successful traces to bootstrap reasoning ability\.Cited by:[item 4](https://arxiv.org/html/2609.22094#S1.I2.i4.p1.1),[§2\.3](https://arxiv.org/html/2609.22094#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2609.22094#S4.T1.1.4.3.2.1.1)\.
- F\. Zeng, W\. Li, W\. Gao, and Y\. Pang \(2024\)Multimodal misinformation detection by learning from synthetic data with multimodal large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 10467–10483\.Cited by:[§2\.4](https://arxiv.org/html/2609.22094#S2.SS4.p1.1)\.

## Appendix ATraining Algorithm

Algorithm 1Summarize\-Judge\-Refine \(SJR\)0:Dataset

𝒟=\{\(xi,pi,yi\)\}\\mathcal\{D\}=\\\{\(x\_\{i\},p\_\{i\},y\_\{i\}\)\\\}, Content Model

fθf\_\{\\theta\}, Policy Model

gϕg\_\{\\phi\}, Group size

GG, Rounds

KK, Clip range

ϵ\\epsilon, KL coefficient

λ\\lambda
1:Initialize

fθ←f\_\{\\theta\}\\leftarrowpretrained multimodal LLM

2:forround

k=1k=1to

KKdo

3:// Stage 1: SUMMARIZE

4:foreach

\(xi,pi,yi\)∈𝒟\(x\_\{i\},p\_\{i\},y\_\{i\}\)\\in\\mathcal\{D\}do

5:

si\(k\)←fθ​\(xi\)s\_\{i\}^\{\(k\)\}\\leftarrow f\_\{\\theta\}\(x\_\{i\}\)\{Generate text summary\}

6:endfor

7:// Stage 1\.5: AUGMENT \(optional\)

8:

𝒟aug\(k\)←AdversarialAugment​\(\{xi,pi,yi\}\)\\mathcal\{D\}\_\{\\text\{aug\}\}^\{\(k\)\}\\leftarrow\\text\{AdversarialAugment\}\(\\\{x\_\{i\},p\_\{i\},y\_\{i\}\\\}\)
9:// Stage 2: JUDGE

10:Fine\-tune

gϕg\_\{\\phi\}on

\{\(si\(k\),pi,yi\)\}∪𝒟aug\(k\)\\\{\(s\_\{i\}^\{\(k\)\},p\_\{i\},y\_\{i\}\)\\\}\\cup\\mathcal\{D\}\_\{\\text\{aug\}\}^\{\(k\)\}
11:Evaluate

gϕg\_\{\\phi\}on validation set

→\\rightarrowmetrick

12:// Stage 3: REFINE \(GRPO\)

13:foreach

\(xi,pi,yi\)∈𝒟\(x\_\{i\},p\_\{i\},y\_\{i\}\)\\in\\mathcal\{D\}do

14:Sample

GGsummaries:

\{si1,…,siG\}∼fθ\(⋅\|xi\)\\\{s\_\{i\}^\{1\},\\ldots,s\_\{i\}^\{G\}\\\}\\sim f\_\{\\theta\}\(\\cdot\|x\_\{i\}\)
15:Compute rewards:

rig=R​\(sig,pi,yi\)r\_\{i\}^\{g\}=R\(s\_\{i\}^\{g\},p\_\{i\},y\_\{i\}\)for

g=1,…,Gg=1,\\ldots,G
16:Group\-relative advantages:

a^ig=rig−μ​\(\{rig\}g=1G\)σ​\(\{rig\}g=1G\)\\hat\{a\}\_\{i\}^\{g\}=\\frac\{r\_\{i\}^\{g\}\-\\mu\(\\\{r\_\{i\}^\{g\}\\\}\_\{g=1\}^\{G\}\)\}\{\\sigma\(\\\{r\_\{i\}^\{g\}\\\}\_\{g=1\}^\{G\}\)\}
17:endfor

18:Update

fθf\_\{\\theta\}via clipped policy gradient:

19:

ℒ=−𝔼​\[min⁡\(ρ​a^,clip​\(ρ,1±ϵ\)​a^\)\]\+λ⋅KL​\(fθ∥fθ0\)\\mathcal\{L\}=\-\\mathbb\{E\}\\left\[\\min\\left\(\\rho\\hat\{a\},\\text\{clip\}\(\\rho,1\{\\pm\}\\epsilon\)\\hat\{a\}\\right\)\\right\]\+\\lambda\\cdot\\text\{KL\}\(f\_\{\\theta\}\\\|f\_\{\\theta\_\{0\}\}\)
20:where

ρ=πθ​\(s\|x\)/πθold​\(s\|x\)\\rho=\\pi\_\{\\theta\}\(s\|x\)/\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(s\|x\)
21:ifmetrickconvergesthen

22:break

23:endif

24:endfor

25:return

fθ,gϕf\_\{\\theta\},g\_\{\\phi\}

## Appendix BReward Function Extensions

The binary rewardRRused in our experiments \(§[3](https://arxiv.org/html/2609.22094#S3)\) can be extended with additional terms for settings where faithfulness or fluency constraints are needed:

R​\(s,p,y\)=α⋅𝟙​\[y^=y\]\+β⋅rfaithful​\(s,x\)\+γ⋅rfluency​\(s\)R\(s,p,y\)=\\alpha\\cdot\\mathbb\{1\}\\left\[\\hat\{y\}=y\\right\]\\\\ \+\\beta\\cdot r\_\{\\text\{faithful\}\}\(s,x\)\+\\gamma\\cdot r\_\{\\text\{fluency\}\}\(s\)\(3\)
- •Faithfulness rewardrfaithfulr\_\{\\text\{faithful\}\}: measures whether the summary accurately reflects the actual content, preventing the Content Model from hallucinating policy\-relevant details that do not exist in the input\.
- •Fluency rewardrfluencyr\_\{\\text\{fluency\}\}: ensures generated summaries remain coherent and human\-readable, measured by perplexity under a reference language model\.

In our experiments, we found that the binary accuracy reward combined with KL regularization was sufficient to maintain both faithfulness and fluency, so we setβ=γ=0\\beta=\\gamma=0\. These extensions are included for generality and may be valuable in settings where the Content Model exhibits hallucination or degenerate outputs\.

## Appendix CPer\-Round Precision, Recall, and F1

Table[4](https://arxiv.org/html/2609.22094#A3.T4)expands the per\-round F1 results \(Table 2 in the main text\) with precision and recall for both classes, providing deeper insight into the co\-training dynamics discussed in §[6](https://arxiv.org/html/2609.22094#S6)\.

Table 4:Full P/R/F1 \(%\) per round for SJR \+ Augmentation\. The Round 2 recall drop in the full\-data variant \(73\.61%→\\to69\.11%\) and recovery in Round 3 \(80\.54%\) are visible\. The NM\-only variant shows a contrasting pattern: precision spikes in Round 2 \(85\.55%\) while recall drops \(68\.87%\), but both converge by Round 3\.
## Appendix DPrompt Templates

We provide the core instructions from each model’s prompt template below, anonymized to remove platform\-specific details\.

#### Content Model \(Stage 1: SUMMARIZE\)\.

The Content Model receives both on\-platform and off\-platform signals and produces two independent summaries:

> You are an expert content analyst\. You are shown two sides of a sponsored ad\-and\-landing\-page pair: \(A\) the on\-platform content the user sees before clicking, and \(B\) the off\-platform landing page reached after clicking\. Write TWO independent factual summaries—one of each side—AND classify each side’s content category, so a downstream policy classifier can decide whether the landing page matches the on\-platform promise\. You must NOT make the misleading\-link decision yourself\. ON\-PLATFORM SUMMARY \(200–400 words\): Write a cohesive factual preview of what the user is led to expect on the landing page after clicking\. OFF\-PLATFORM SUMMARY \(200–400 words\): Write a cohesive factual summary of what the page actually communicates\.

#### Policy Model \(Stage 2: JUDGE\)\.

The Policy Model receives the summary pair and a policy description, outputting a binary verdict:

> You are an expert content moderator\. Determine whether a URL is MISLEADING or NOT MISLEADING, given a JSON block describing both sides of the click: on\_platform\_summary and off\_platform\_summary\. A link is MISLEADING only if the landing page is NOT semantically related to what the on\-platform content promises\. “Semantically related” means the same central topic, offer, or intent\.

#### Augmentation Model \(Stage 1\.5: AUGMENT\)\.

The Augmentation Model takes raw multimodal content and generates adversarial summary variants from a catalog of scenarios:

> You are an expert adversarial data generator\. Given a real landing page and on\-platform content, generate ONLY the adversarial off\-platform summaries that are PLAUSIBLE for this specific input\. Each summary should be either: \(1\) HARD NEGATIVE—not misleading, but easily confused as misleading \(causes false positives\), or \(2\) HARD POSITIVE—is misleading, but easily confused as not\-misleading \(causes false negatives\)\. Pick only scenarios that genuinely fit this LP’s brand, vertical, format, and content\. Quality over quantity\.

The adversarial scenario catalog spans hard\-negative scenarios \(cases that are not misleading but are easily confused as misleading, causing false positives\) and hard\-positive scenarios \(genuinely misleading cases that are easily confused as not misleading, causing false negatives\), covering offer\-level mismatches, brand\-confusion cases, and category\- or redirect\-based substitutions, among others\.