Channel Decomposition of Political Ads

0. Read first: four findings, two questions

Solid In a mean-pooled omni embedder the joint representation follows the visual channel: in cross-party composites the joint-condition party classifier followed the visual donor in 19/19 distinct pairs against prosody and 19/19 against text (17/19 and 17/19 when the two donor ads are excluded from the prototype); prosody against text is 10/19, which establishes no dominance either way. Shapley shares of absolute contribution are V 0.43, P 0.28, T 0.29, stable across two scoring methods (section 1). These are properties of this embedder on these ads; the identity confound of 6.1 applies to all of them.
Hypothesis The dominance may be a token-count effect: the embedding is a mean over tokens, and video contributes far more tokens than audio or text at our settings (rates in 6.2; realised counts were not logged). Sampling at 0.5 and 0.25 fps would show whether the shares move; settling the mechanism needs a content-preserving control, such as frame duplication or per-modality reweighted pooling (6.2).
Solid The modality gap is measurable inside an omni model: the distance between condition means is largest for V–T (0.70) and smallest for A–P (0.17). It does not, however, explain why zero-shot anchor scores sit at about 0.70 in every condition: centering each condition on its own mean and renormalising leaves those AUCs unchanged (V 0.73 → 0.74, P and T unchanged), because a constant offset within a condition does not change rankings. Why the anchors separate the parties only weakly is open (1.4).
Heuristic Your exposure pattern reproduces qualitatively at n = 22 with CLIP: image AUC is flat from 3 s on, text AUC rises from chance to 0.65 by 45 s (section 2). And the caveat that governs everything above: with one candidate on the R side and two on the D side, identity leaks into every channel, so absolute AUCs measure who is on screen, not partisan style (6.1).

Two questions. Your Appendix A.2 shows the image accumulation curve is essentially invariant to chunk length; was the fitted 40.38-word equivalent re-estimated under those alternatives, and how much did it move? And could a filtered-audio arm be added to a survey design that currently standardises delivery with synthetic voices? Details in section 9.

1. Omni embedder, channel subsets, Shapley decomposition

Model: nvidia/omni-embed-nemotron-3b, a retrieval fine-tune of the Qwen2.5-Omni thinker (4.70 billion parameters by the safetensors index; embedding = mean of the last hidden states over all tokens). Each ad was split into three channel conditions: V muted video (visual content, including any on-screen text), P filtered audio (low-passed at 400 Hz and loudness-normalised, intended to keep pitch and rhythm and to remove intelligible words; intelligibility, residual words and voice identity were not tested), T transcript. All seven non-empty subsets were encoded as one input each, plus two leakage references: A raw audio with words, VA the original ad. One model embeds every condition, which is what makes the subsets comparable and the decomposition possible.

1.1 Partisan signal per condition Solid

AUC, leave-one-out	V	P	T	VP	VT	PT	VPT	A	VA
Zero-shot anchors (cosine to R minus D text anchors; cross-modal, CLIP-style)	0.73	0.71	0.72	0.71	0.70	0.68	0.70	0.68	0.71
Prototype, within condition (cosine to mean_R−mean_D of the other ads)	0.99	0.93	0.90	1.00	1.00	0.93	1.00	0.93	1.00
Logistic regression on the embedding	0.87	0.66	0.67	0.88	0.88	0.68	0.89	0.67	0.88

The three rows are different instruments and should be read down a column, not across rows. The near-perfect prototype values in V-containing conditions are the identity leak of 6.1, not partisan style.

1.2 Shapley values of the three channels Solid Heuristic

Exact three-player Shapley values on per-condition scores standardized within condition, with the empty coalition at zero. The values sum to the score of the full VPT input by construction (asserted numerically). Share of absolute contribution, prototype scoring:

Scoring	V share	P share	T share	AUC of each channel’s φ against party (V / P / T)
Prototype (within condition)	0.43	0.28	0.29	0.89 / 0.88 / 0.89
Zero-shot anchors	0.44	0.29	0.27	0.64 / 0.66 / 0.65

The shares (solid: a property of this model on these ads) agree across scoring methods. Whether they say anything about partisan communication is heuristic, for the reasons in 6.1 and 6.2.

1.3 Conflict test Solid

Cross-party composites: the first channel from an ad of one party, the second from an ad of the other (10 draws per direction, seeded, drawn with replacement, which yielded 19 distinct pairs per test). Each composite is scored by the sign of its cosine to the party prototype of the corresponding joint condition (VP, VT or PT); “the first source wins” means the composite is classified with the first donor’s party. With prototypes from all 22 ads: 19/19 for V against P, 19/19 for V against T, 10/19 for P against T. With the two donor ads removed from the prototype (the conservative variant): 17/19, 17/19, 9/19. In every sampled composite with a visual donor, the classifier followed the visual donor; 10/19 for prosody against text establishes no dominance in either direction.

1.4 Modality gap inside the omni model Solid

Euclidean distance between the means of unit embeddings per condition: V–T 0.70, V–P 0.62, P–T 0.39, A–P 0.17. The gap you center away in Section 2.2 is present in a natively multimodal model as well. Two things it does not show. First, it is not what limits the zero-shot row in 1.1: rank-based scores within one condition are invariant to a constant offset, and centering plus renormalising moves those AUCs by at most 0.01. The gap matters where scores from different modalities are pooled or compared in absolute terms, which is what a pooled model does. Second, A–P at 0.17 is a distance between centroids; per ad, the distance between the raw-audio and filtered-audio embeddings averages 0.41 (range 0.29–0.63), so the filter changed individual embeddings substantially even though the two condition means stay close.

2. CLIP mini-replication of Sections 2.1–2.3, 3 and 4.1

Frame every 3 s and the transcript cut into as many chunks; CLIP ViT-B/32 (the 512-dimensional space of your Section 2.2); mean per ad; unit-sphere mean-centering per modality with the mean over all 22 ads, as in your step (ii), so the held-out ad contributes 1/22 to the centre (centering inside each training fold instead gives image AUC 0.64, text unchanged); leave-one-out logistic classifier per modality; a stacked logistic on the two predicted probabilities that is not nested (its inputs were produced with the held-out ad’s label in the base fits; a nested version gives 0.52); exposure curve on prefixes; concept decomposition by non-negative sparse projection on 50 concept words. Not attempted: the transfer of ad-trained classifiers to TV news (your Section 3.1–3.2), the conversion-rate estimate, the concept-removal effects of Section 4 and the survey experiment.

Measure	Image	Text	Stacked
AUC, leave-one-out Heuristic	0.66	0.68	0.46
Correlation of image and text predictions	−0.04
Modality gap, norm of mean difference, before → after centering Solid	1.11 → 0.10

The stacked value is not interpretable: the evaluation is not nested (0.46 at C = 10; nested 0.52) and both are at chance. The near-zero correlation is in the direction of your “far from perfectly correlated”, but is not evidence at this n. The gap figure is one you do not report; before centering the two modality mean directions sit about 72° apart (the means are not unit vectors, so the angle is computed from their normalised dot product, not from the chord).

Exposure curve Heuristic: AUC of the classifier trained on full ads, tested on the first k seconds (leave-one-out). Image 0.62 / 0.64 / 0.58 / 0.67 / 0.62, text 0.48 / 0.43 / 0.52 / 0.60 / 0.65 at 3 / 6 / 12 / 24 / 45 s.

Concept decomposition Heuristic Substituted

Mean concept weight, R minus D. Images: rally, crowd, freedom, patriotism, economy against healthcare, hospital, countryside, love, education. Text: unity, strength, pride, freedom, patriotism against abortion, healthcare, military, hospital, police. Reconstruction cosine is only 0.14 (image) and 0.25 (text), for two reasons: fifty concepts against your 160, and a sparsity search that aimed at about eight active concepts but stopped after a fixed number of steps, reaching 4.8 (image) and 4.4 (text) on average, with 25 of 44 decompositions below the accepted band. The weights are indicative at best.

3. Data and pipeline

Solid 22 ads from official 2024 candidate channels on YouTube: 10 from Donald J. Trump (R), 12 from Kamala Harris and Joe Biden (D). Label = party of the sponsoring candidate, as in your Section 2.1. Length 28–80 s, 360p.
Solid Transcripts from faster-whisper large-v3 with word timestamps (53–257 words per ad); frames at one per 3 s for section 2 and at 1 fps, at most 64 frames, for section 1.
Solid Part A ran in under 90 s (observed, not logged); Part B took about 6.5 min including one out-of-memory restart (212 s after the resume). Ad-level and condition-level embeddings are cached, so sections 1.1–1.4 and the Part A AUCs and gap regenerate from the files; the exposure curve and the concept weights need the frame-level embeddings, which were not kept.

4. Research protocol, step by step

Condensed from the session log. About one hour of wall-clock time from locating the paper to the last result, then the write-up.

Source. Located the job market paper (version of 16 June 2026, 76 pages) and the ETH lecture deck on the author’s site; extracted the pipeline of Sections 2.1–2.3, 3 and 4.1 and the modality-gap treatment of 2.2. No code or replication package accompanies the paper (the author’s GitHub holds a personal site and a math-camp repo), so the pipeline was re-implemented from the text. Solid
Scope. A same-day proof of concept in two parts: (A) mini-replication with CLIP, (B) an omni-embedder extension. Hardware inventory before any install: RTX 5080 16 GB, torch 2.8 cu128, transformers 4.57.3, scikit-learn 1.8, faster-whisper 1.2.1, ffmpeg. Nothing had to be installed for (A).
Sample. Listed the official YouTube channels of the 2024 candidates (Trump; Harris; Biden), took the first 30–40 uploads of each feed, kept those between 25 and 95 s, and removed four R-feed items that were not campaign ads by title (a Christmas message, a cabinet-confirmation clip, a weekly recap, an endorsement clip); no D-feed item needed a title exclusion. Result: 12 D, 10 R, 28–80 s. Downloaded at 360p with audio. Auto-captions were missing for most R ads and were replaced by Whisper transcripts for all ads, which is also closer to the paper’s input. Substituted
Part A, one script, configuration first (frame step, exposure grid, concept list at the top): frames every 3 s; transcript words assigned to frame windows; CLIP ViT-B/32; per-ad means; unit-sphere centering per modality; leave-one-out logistic per modality and stacked; prefix exposure curve; non-negative Lasso projection on centered concept vectors. Deviations from the paper are enumerated in the script. Run time under 90 s. Solid
Part B environment. A virtual environment on top of the system packages; qwen-omni-utils and torchvision 0.23 (cu128) installed with the operator’s approval; nvidia/omni-embed-nemotron-3b (9.4 GB) downloaded while (A) ran. The sentence-transformers route in the model card failed on module paths from an older library version; the transformers route worked with 4.57.3, without the preview branch the card recommends.
Channel preparation. Per ad: muted video (V), audio low-passed at 400 Hz twice plus loudness normalization (P), video re-muxed with the filtered audio (VP), raw audio (A), Whisper word transcript (T). 66 variant files and 22 transcripts. Substituted
Encoding. 9 conditions × 22 ads, 8 text anchors and 60 composite evaluations (57 distinct cross-party pairs; one pair per test was drawn twice because the draw was with replacement) through the same model, every embedding cached to disk. Two failures fixed on the way: the transformers 4.57 processor loader tripping over the snapshot’s additional_chat_templates folder (loaded from a symlinked copy without it), and one CUDA out-of-memory on the longest ad (cache clearing plus expandable segments, resumed from the cache). 212 s after the resume.
Analysis. Three scores per condition; exact three-player Shapley values on standardized scores with the sum identity asserted; conflict test against full-sample prototypes; distances between condition means. Solid
Reading rules fixed before the numbers were read: AUC standard error about 0.12; agreement between anchor and prototype scoring is the robustness signal; A vs P and VA vs VPT are the leakage checks. The interaction term was dropped after inspection as a scale artifact (7). Heuristic
Artifacts. Two scripts, two preparation scripts, two results files, this page; all in the course folder. Data, variant files and the embedding cache stay on the GPU machine.

5. Substitutions for elements of the paper we did not have

In the paper	Here	Consequence
15,427 sponsor-labeled ads from the 2016, 2018 and 2020 cycles (Wesleyan Media Project, Kantar/CMAG), many candidates and PACs across House, Senate and presidential races	22 ads, one R candidate and two D candidates, 2024	Identity (face, voice, name, campaign style) confounds partisanship in every channel (6.1); no inferential claims.
Transcripts of the ads	Whisper large-v3 transcripts	A substitute for the corpus transcripts; ASR errors on names possible, not audited (backlog 6).
Neural-network classifier, 80/20 split	Logistic regression, leave-one-out; centering global as in the paper, not per fold	A modeling choice for n = 22; less capacity, no seeds to vary; fold-wise centering would give image 0.64 instead of 0.66.
Stacked model following Ludwig and Mullainathan	Logistic on the two predicted probabilities, not nested	Not interpretable at this n: 0.46 as run, 0.52 nested, both at chance.
SpLICE with 160 topics and emotions, sparsity by matched reconstruction	Non-negative Lasso on 50 hand-picked words; sparsity search aimed at about 8 active concepts, achieved 4.8 / 4.4 on average (25 of 44 below the target band)	Poor reconstruction (0.14 / 0.25); concept rankings indicative only.
Exposure in five-second chunks, K = 1…30 randomly sampled chunks (5–150 s), on TV-news clips; prefix version in your Appendix A.1, chunk-length robustness in A.2	Prefix of the first k seconds on the 22 ads, classifier trained on the other ads at full length	Different corpus and estimator; no conversion rate computed; the prefix design already exists in your A.1.
CLIP as the only shared space	Section 1 adds an omni embedder with audio; prosody by low-pass filtering	An extension, not a replication; the prosody operationalization is itself a substitute (6.4).
Survey experiment on real footage	None	The causal half of the paper is out of scope for this note.

6. Limitations

6.1 Identity leaks into every channel Solid

With a single candidate on the R side (Trump) and two on the D side (Biden, then Harris), each with one campaign’s production style, V sees a face and a logo, P keeps the speaker’s voice after the filter, T contains the names, and each campaign reuses its music. The near-perfect within-condition AUCs of 1.1 measure identity. Your corpus dilutes this over many candidates and PACs; ours does not. The relative structure (shares, conflict test) is less exposed to this confound but not immune to it: a channel that carries identity more strongly will also dominate composites. Everything in section 1 is a statement about the embedder before it is one about ads.

6.2 Token-count weighting Hypothesis

If a joint-modal embedding is a mean over tokens, a channel that contributes more tokens weighs more regardless of what it says. At our settings video contributes far more tokens than audio or text: the vision encoder emits one token per 28×28 pixels per pair of frames (patch 14, 2×2 spatial merge, temporal patch 2, from the model configuration), so at most 64 tokens per frame pair and about 2,000 tokens for 64 frames, against a transcript of 50–260 words; the realised counts per input were not logged. The audio encoder emits one token per 40 ms (Qwen2.5-Omni report, Section 2.2), about 1,500 tokens for a 60 s track. Two external observations point the same way. NVIDIA’s own evaluation of this model on FineVideo finds that simple early fusion of audio and video underperforms audio alone (0.470 against 0.541), while embedding the two separately and combining them late recovers to 0.566. And the multimodal-LLM literature measures this kind of imbalance directly: Shapley-based modality contributions (Parcalabescu and Frank, MM-SHAP, ACL 2023) and a modality dominance index with token compression as the remedy (Wu et al. 2025, arXiv:2508.10552). Sampling video at 0.5 or 0.25 fps is a sensitivity test, not a decisive one, because it also changes visual information, coverage and repetition. A decisive test keeps content fixed and varies tokens: duplicate frames, or reweight the mean by modality, and watch whether the shares and the swap outcomes move. Mean pooling also averages contextualised states, so tokens of one modality can carry information from another; the hypothesis is about weight, not isolation. Until such a control is run, “which modality carries the signal” conclusions from omni embedders should carry a token-budget caveat.

6.3 The modality gap did not go away, and it is not what limits zero-shot scoring Solid

Condition means are far apart (1.4), yet centering each condition on its own mean leaves the anchor AUCs where they are (at most 0.01 change). So the weak zero-shot separation is not a gap artifact; it says that the direction of these hand-written anchors is only loosely aligned with the direction along which the ads differ by party. Your per-modality centering remains necessary wherever scores from different modalities are pooled into one estimate, as in your stacked model; it is not needed for within-modality rankings.

6.4 Prosody is an operationalization, not a measurement Substituted

The filtered-audio condition (400 Hz low-pass applied twice, then loudness normalisation) is intended to remove intelligible words and keep pitch and rhythm; whether the words are gone, and how much voice identity and the soundtrack survive, was not tested. Loudness normalisation equalises level across ads, so loudness is only partly preserved. The manipulation is the content-filtered speech of Rogers, Scherer and Rosenthal (1971), designed to keep vocal affect cues while removing intelligibility; the speaker-identification literature reports that the spectral cues surviving such filters still support identifying the speaker, which is why voice identity is listed as a leak rather than a remote possibility. Near-identical AUCs for A and P say the filter did not remove the party signal, not that prosody carries it; per ad, the raw-audio and filtered-audio embeddings sit 0.41 apart on average, so the filter did change the representation. Dietrich, Hayes and O’Brien’s vocal-pitch measure would be the disciplined alternative.

6.5 Small-sample estimators Heuristic

The 0.12 figure is a null-ranking approximation of the AUC standard error at 10 versus 12; it is not an uncertainty estimate for differences between conditions, which were not computed.
Leave-one-out prototypes shift by class, because the held-out ad is missing only from its own class mean. This reorders ads (for T, full-sample versus leave-one-out prototypes move the AUC from 1.0 to 0.9 with 15 pairwise reversals); it is a different estimator, not a uniform deflation.
The interaction term v(VPT) − v(V) − v(P) − v(T) on standardized scores is treated as a scale artifact and was dropped from the tables; it is still in results_B.json (fields interaction_*), without a label.
Zero-shot anchors are hand-written and their sensitivity was not measured; single-anchor AUCs on the cached anchors range from 0.27 to 0.93 for V, so the four-anchor average should not be read as stable.

6.6 Encoding choices Solid

Video at 1 fps, at most 64 frames, low resolution (max 64×28×28 pixels per frame) to fit 16 GB; every condition carries the constant prefix passage: so that conditions differ only by channel content; attention implementation sdpa rather than flash-attention. Whether any of these changes the ordering in section 1 was not tested; the fps choice is the lever of 6.2.

7. Uncertainty backlog

Every claim the pipeline could not settle, with the action that would settle it. Nothing here is hidden in the tables above; where a row matters for a number, the number carries the tag.

#	Claim	Confidence	Action	Why it matters
1	The visual dominance in section 1 is a token-count effect of mean pooling.	Medium	Verify: fps and max-frames ablation.	Decides whether section 1 says anything about ads rather than about the embedder.
2	`fps` and `max_frames` keys were honoured by the video loader.	Medium	Verify: inspect the processor’s frame count per ad.	If ignored, video was sampled at the default 2 fps, which strengthens row 1.
3	The interaction term on standardized scores is a scale artifact, not sub-additivity.	Medium	Accepted as a caution; dropped from the tables, kept unlabeled in the results file.	Prevents a false “channels interfere” reading; not a proof that nothing else is in it.
4	Leave-one-out prototypes deflate AUCs uniformly and do not reorder ads.	Withdrawn	Checked and rejected: for T the AUC moves 1.0 → 0.9 with 15 pairwise reversals.	The leave-one-out prototype is a different estimator from the full-sample one (6.5).
5	A 400 Hz low-pass isolates prosody.	Low	Defer: replace with a pitch and energy measure.	The filter follows Rogers, Scherer and Rosenthal (1971), but filtered speech is reported to retain speaker-identifying spectral cues, and the soundtrack’s bass passes it; P results may reflect voice identity and music rather than delivery (6.4).
6	Whisper transcripts are free of name errors.	Medium	Verify: audit the 22 transcripts.	Affects the T channel and the concept ranking for text.
7	Party of the channel equals party of the sponsor for these 22 ads.	High	Accepted.	Matches the paper’s labeling rule for candidate-sponsored ads.
8	Access terms for the Wesleyan Media Project / Kantar-CMAG ad corpus the paper uses.	Unknown	Ask the author.	Provenance is stated in the paper (Section 2.1); access is not. Required for any rigorous replication (section 8).
9	Fifty concepts with count-tuned sparsity approximate SpLICE with 160 and matched reconstruction.	High that it does not	Accepted as a known deviation.	Reconstruction 0.14 / 0.25; concept weights indicative only.
10	Anchor wording does not drive the zero-shot row.	Low	Verify: ensemble of 20 anchors per party.	Single-anchor AUCs on the cached anchors range from 0.27 to 0.93 for V; the four-anchor row is not stable.
11	The paper’s conversion rate is insensitive to frame sampling density.	Unknown	Ask the author (section 9).	Tests row 1 for CLIP-style pipelines.
12	The stacked evaluation is not nested; nested and non-nested values are both at chance.	High	Accepted; a second run would nest it.	Neither 0.46 nor 0.52 should be read as a pooled-model result.

8. What a rigorous version needs

8.0 What a second run would change, and why we did not re-run

Two independent reviews of this page (a model-based review with full recomputation from the caches, and an adversarial audit) found five procedural defects. None changes a conclusion, which is why the numbers above stand as run, with the corrected values beside them. A second run is justified only if the numbers themselves matter to you.

Conflict pairs drawn without replacement, and prototypes that exclude the two donor ads (already reported as the conservative variant: 17/19, 17/19, 9/19 instead of 19/19, 19/19, 10/19).
Centering inside each training fold and a nested stacked model (image AUC 0.64 instead of 0.66; stacked 0.52 instead of 0.46, both at chance).
A sparsity search that checks attainment before stopping (25 of 44 decompositions missed the target band, which depresses reconstruction and the concept weights).
Logged frame and token counts per condition, and a content-preserving token-count control (frame duplication, per-modality reweighted pooling) before any statement about token-count dominance (6.2).
A prosody channel measured by pitch and energy rather than approximated by a low-pass filter, and an anchor ensemble instead of four hand-written anchors per party.

Beyond the defects, a rigorous version needs the following.

Dilute identity. Many candidates per party, as in your corpus, with leave-candidate-out validation so that no candidate appears in both training and test folds. Candidate fixed effects do not solve this: party is constant within candidate, so a within-candidate design can separate identity from style but cannot identify a party effect. Without this the channel decomposition measures faces and voices.
Token-budget controls. Log realised frame and token counts per condition; run fps in {1, 0.5, 0.25} and max frames in {64, 32, 16} as sensitivity tests; then a content-preserving control (frame duplication, per-modality reweighted pooling); report Shapley shares and the swap fractions per cell. A second omni embedder (LCO-Embedding-Omni or jina-embeddings-v5-omni) to separate the model from the finding.
Your Section 4 at full strength. The 160-concept vocabulary and matched-reconstruction sparsity, applied per channel and per subset, so that the affective-versus-issue split can be tested for prosody as well.
Report the gap, and what it touches. Norm of the modality-mean difference before and after centering, per corpus, as a standard diagnostic; it is one line and it makes the pooled estimates auditable. State alongside it that within-modality rankings are unaffected by the offset and that only pooled or absolute cross-modal comparisons are.
Human baseline in the same four conditions (muted video, filtered audio, transcript, full ad) to ask whether the model weights channels the way viewers do. This is the bridge to your survey experiment and the natural prosody arm.

9. Questions for you

Your Appendix A.2 shows the image accumulation curve is essentially invariant to chunk length (three, five, ten seconds). Was the fitted conversion rate (40.38 words for one five-second visual glimpse) re-estimated under those alternatives, and how much did it move? Since your pipeline averages each modality separately, this speaks to the robustness of the conversion rate, not to the joint-pooling mechanism of 6.2.
Your survey standardises delivery with synthetic voices, which removes exactly the channel a prosody arm would test. Is a filtered-audio condition (no visuals, no intelligible words, original voices) feasible on your platform, and would you expect it to move emotions the way images do?
Is there a version of the ad corpus that could be shared for a supervised replication of Sections 2–4 with the omni extension, under whatever terms the data source allows?

A. Appendix

Files and parameters

poc_clip.py, results.json: section 2. Frame step 3 s; exposures 3, 6, 12, 24, 45 s; 50 concepts; logistic C = 1 for the per-modality and exposure models, C = 10 for the stacked model; leave-one-out classifiers with global centering; stacking not nested.
prep_b.py, poc_omni.py, results_B.json, omni_emb.json: section 1. Low-pass lowpass=f=400 twice plus loudnorm; video fps 1, max 64 frames, min/max pixels 32×14×14 / 64×28×28; anchors 4 per party with the query: prefix, ads with the passage: prefix; Shapley weights 1/3, 1/6, 1/6, 1/3; swaps seeded (seed 0), 10 draws per direction with replacement, 19 distinct pairs per test.
Models: openai/clip-vit-base-patch32; nvidia/omni-embed-nemotron-3b (snapshot 865db1bb) via transformers 4.57.3, qwen-omni-utils, bf16, sdpa; Systran/faster-whisper-large-v3.
Runtime: section 2 under 90 s; section 1 about 6 min including one out-of-memory restart from cache.

Scores, in one paragraph

For condition S and ad i with unit embedding e: anchor score = mean cos(e, R anchors) − mean cos(e, D anchors); prototype score = cos(e_i, mean_R,−i − mean_D,−i); logistic AUC from leave-one-out predicted probabilities. For the Shapley values each condition’s score is standardized across ads, v(∅) = 0, and φ_c = Σ_S ∉ c w(|S|)[v(S∪{c}) − v(S)] with w = 1/3, 1/6, 1/6, 1/3 for |S| = 0, 1, 1, 2. Shares are mean |φ_c| over ads, normalized to sum to one.

How this page was built

Static HTML, no libraries. Typography and table behaviour come from records of a design registry (tabular numerals, heading kerning, table row highlight, scroll-safe tables, smooth disclosure, reading-position marker) checked against that registry’s antipattern list before release. Fonts load from Google Fonts with system fallbacks; the page works offline in the fallback faces.

A.4–A.10 Code and results, as run Solid

Files that produced the numbers above; the repository holds them as separate files (same values; JSON indentation differs, and poc_omni.py there keeps one explanatory comment line). The defects found by the reviews are in this code by design; 8.0 lists what a second run would change. Each block has a copy button.

A.4 results_B.json: section 1 output, as produced 130 lines, 3,414 bytes

Includes the unlabeled interaction_* fields discussed in 6.5 and the conflict fractions over 20 draws (19 distinct pairs, 1.3).

{
 "n": 22,
 "n_R": 10,
 "model": "nvidia/omni-embed-nemotron-3b",
 "conditions": [
  "V",
  "P",
  "T",
  "VP",
  "VT",
  "PT",
  "VPT",
  "A",
  "VA"
 ],
 "auc": {
  "anchor": {
   "V": 0.7333333333333334,
   "P": 0.7083333333333334,
   "T": 0.7166666666666667,
   "VP": 0.7083333333333334,
   "VT": 0.7000000000000001,
   "PT": 0.6833333333333333,
   "VPT": 0.7000000000000001,
   "A": 0.6833333333333333,
   "VA": 0.7083333333333334
  },
  "proto": {
   "V": 0.9916666666666666,
   "P": 0.925,
   "T": 0.9,
   "VP": 1.0,
   "VT": 1.0,
   "PT": 0.9333333333333333,
   "VPT": 1.0,
   "A": 0.9333333333333333,
   "VA": 1.0
  },
  "logistic_loo": {
   "V": 0.8666666666666667,
   "P": 0.6583333333333333,
   "T": 0.6666666666666666,
   "VP": 0.875,
   "VT": 0.8833333333333333,
   "PT": 0.6833333333333333,
   "VPT": 0.8916666666666667,
   "A": 0.6666666666666667,
   "VA": 0.8833333333333333
  }
 },
 "shapley": {
  "anchor": {
   "share_of_abs_contribution": {
    "V": 0.43950197982550526,
    "P": 0.28727144827087037,
    "T": 0.2732265719036245
   },
   "auc_phi_vs_party": {
    "V": 0.6416666666666667,
    "P": 0.6583333333333333,
    "T": 0.65
   },
   "mean_phi_R": {
    "V": 0.22372619570443955,
    "P": 0.1346453250517173,
    "T": 0.12558190957428933
   },
   "mean_phi_D": {
    "V": -0.18643849642036636,
    "P": -0.11220443754309778,
    "T": -0.10465159131190775
   },
   "interaction_mean_abs": 1.4482014802552137,
   "interaction_auc_vs_party": 0.225
  },
  "proto": {
   "share_of_abs_contribution": {
    "V": 0.4303629029066916,
    "P": 0.281699801919223,
    "T": 0.2879372951740854
   },
   "auc_phi_vs_party": {
    "V": 0.8916666666666666,
    "P": 0.8833333333333333,
    "T": 0.8916666666666666
   },
   "mean_phi_R": {
    "V": 0.40300823654795603,
    "P": 0.2634130708279729,
    "T": 0.26755596957051475
   },
   "mean_phi_D": {
    "V": -0.33584019712329655,
    "P": -0.21951089235664403,
    "T": -0.22296330797542918
   },
   "interaction_mean_abs": 1.5687250533688901,
   "interaction_auc_vs_party": 0.06666666666666668
  }
 },
 "modality_gap_norm": {
  "V-T": 0.7019284910535292,
  "P-T": 0.3923650901683346,
  "V-P": 0.6191675156701028,
  "A-P": 0.17019467786109443
 },
 "conflict": {
  "V_over_P": {
   "first_source_wins": 1.0,
   "n_pairs": 20
  },
  "V_over_T": {
   "first_source_wins": 1.0,
   "n_pairs": 20
  },
  "P_over_T": {
   "first_source_wins": 0.55,
   "n_pairs": 20
  }
 },
 "runtime_s": 212,
 "deviations": [
  "n = 22 ads (10 R / 12 D) from official 2024 candidate channels; heuristic, not inferential",
  "prosody channel = low-pass 400 Hz + loudnorm (words unintelligible; pitch, rhythm, energy kept); music bass survives",
  "video sampled at 1 fps, max 64 frames, low resolution (max_pixels 64*28*28) to fit 16 GB",
  "every condition carries the constant prefix 'passage: ' so that conditions differ only by channel content",
  "LOO prototype score has the usual leave-one-out mean bias (uniform across ads); anchors are hand-written",
  "AUCs on 22 ads have SE ~ 0.12; Shapley shares are descriptive"
 ]
}

A.5 results.json: section 2 output, as produced 193 lines, 3,397 bytes

AUCs with global centering and the non-nested stacked model (2, 8.0); concept differences R minus D; deviations list written by the script.

{
 "n": 22,
 "n_R": 10,
 "auc_loo": {
  "image": 0.6583333333333333,
  "text": 0.675,
  "stacked": 0.45833333333333337
 },
 "corr_image_text_pred": -0.03599137194567069,
 "modality_gap_norm": {
  "before": 1.1061378717422485,
  "after": 0.10089539736509323
 },
 "exposure_auc": {
  "3": {
   "image": 0.625,
   "text": 0.48333333333333334
  },
  "6": {
   "image": 0.6416666666666667,
   "text": 0.43333333333333335
  },
  "12": {
   "image": 0.5833333333333334,
   "text": 0.5249999999999999
  },
  "24": {
   "image": 0.6666666666666667,
   "text": 0.6000000000000001
  },
  "45": {
   "image": 0.6166666666666667,
   "text": 0.65
  }
 },
 "splice_reconstruction_cos": {
  "image": 0.13777777623425339,
  "text": 0.25164191857094037
 },
 "concepts_image": {
  "R_minus_D": [
   [
    "rally",
    0.016
   ],
   [
    "crowd",
    0.0115
   ],
   [
    "freedom",
    0.0102
   ],
   [
    "patriotism",
    0.0073
   ],
   [
    "economy",
    0.0052
   ],
   [
    "small business",
    0.0048
   ],
   [
    "city",
    0.0035
   ],
   [
    "safety",
    0.0031
   ]
  ],
  "D_minus_R": [
   [
    "healthcare",
    0.0204
   ],
   [
    "hospital",
    0.0143
   ],
   [
    "countryside",
    0.0132
   ],
   [
    "love",
    0.0116
   ],
   [
    "education",
    0.0094
   ],
   [
    "police",
    0.0093
   ],
   [
    "corruption",
    0.0084
   ],
   [
    "handshake",
    0.0076
   ]
  ]
 },
 "concepts_text": {
  "R_minus_D": [
   [
    "unity",
    0.0423
   ],
   [
    "strength",
    0.0306
   ],
   [
    "pride",
    0.0207
   ],
   [
    "freedom",
    0.0186
   ],
   [
    "patriotism",
    0.0145
   ],
   [
    "hope",
    0.0121
   ],
   [
    "disgust",
    0.0096
   ],
   [
    "china",
    0.0086
   ]
  ],
  "D_minus_R": [
   [
    "abortion",
    0.0633
   ],
   [
    "healthcare",
    0.0303
   ],
   [
    "military",
    0.0235
   ],
   [
    "hospital",
    0.0137
   ],
   [
    "police",
    0.0091
   ],
   [
    "love",
    0.0069
   ],
   [
    "taxes",
    0.005
   ],
   [
    "small business",
    0.0047
   ]
  ]
 },
 "mean_active_concepts": {
  "image": 4.818181818181818,
  "text": 4.363636363636363
 },
 "deviations": [
  "n ~ 20 ads from official candidate channels vs 15,427 labeled ads (2016-2020)",
  "logistic regression + leave-one-out CV instead of neural net + 80/20 split",
  "transcripts from faster-whisper large-v3 (YouTube auto-captions only as fallback); ads without speech get a constant empty-string text embedding",
  "R ads = Trump 2024 channel, D ads = Harris/Biden 2024 channels (same era; sponsor = candidate)",
  "50 concepts instead of 160; sparsity tuned to ~8 active concepts, not matched reconstruction",
  "empty transcript chunks dropped from the text mean",
  "360p frames; exposure curve uses prefix means with the full-length classifier (LOO)"
 ]
}

A.6 manifest.csv: the 22 ads (YouTube ids) 24 lines, 993 bytes

Prefix D_ = Harris or Biden channel, R24_ = Trump channel; the id after the prefix is the YouTube video id.

id,label,video,subs
D_2x72eSfZ7Ao,D,D_2x72eSfZ7Ao.mp4,D_2x72eSfZ7Ao.en.vtt
D_60Hyc3tFk5c,D,D_60Hyc3tFk5c.mp4,
D_9fn2vm_Qrpo,D,D_9fn2vm_Qrpo.mp4,D_9fn2vm_Qrpo.en.vtt
D_Cm9VdVkYCLw,D,D_Cm9VdVkYCLw.mp4,D_Cm9VdVkYCLw.en.vtt
D_JB-8dWtdRuk,D,D_JB-8dWtdRuk.mp4,
D_L2HRvz1vvc8,D,D_L2HRvz1vvc8.mp4,D_L2HRvz1vvc8.en.vtt
D_Xmz-AEhSm_A,D,D_Xmz-AEhSm_A.mp4,
D__v_4IgeVQ8M,D,D__v_4IgeVQ8M.mp4,D__v_4IgeVQ8M.en.vtt
D_cgCoAka-cSo,D,D_cgCoAka-cSo.mp4,D_cgCoAka-cSo.en.vtt
D_eGPCopBNieM,D,D_eGPCopBNieM.mp4,
D_t9suRUZGIdg,D,D_t9suRUZGIdg.mp4,
D_zARiKNKWDZs,D,D_zARiKNKWDZs.mp4,D_zARiKNKWDZs.en.vtt
R24_A46L7BumIjY,R,R24_A46L7BumIjY.mp4,
R24_AFcrA3EeVp4,R,R24_AFcrA3EeVp4.mp4,
R24_FLY4cI77_FU,R,R24_FLY4cI77_FU.mp4,
R24_T5W_Bzrv4Tk,R,R24_T5W_Bzrv4Tk.mp4,
R24_YY5BPigFKgs,R,R24_YY5BPigFKgs.mp4,
R24_i6Ge7ealpZk,R,R24_i6Ge7ealpZk.mp4,
R24_kwbBxC8lIw8,R,R24_kwbBxC8lIw8.mp4,
R24_n5LDxSrrI1k,R,R24_n5LDxSrrI1k.mp4,
R24_uP08OUoVj0o,R,R24_uP08OUoVj0o.mp4,
R24_z1Awk3PSIJo,R,R24_z1Awk3PSIJo.mp4,

A.7 poc_omni.py: section 1, as run 222 lines, 10,406 bytes

Conditions, anchors, scores, exact Shapley, conflict test. Contains the defects listed in 8.0: pairs drawn with replacement, prototypes including the donors.

#!/usr/bin/env python3
"""Part B: omni-embedding channel decomposition on the PoC ads (nvidia/omni-embed-nemotron-3b).

Conditions per ad (one omni embedder, content list = subset of channels):
  V (muted video), P (prosody wav, low-pass 400 Hz), T (transcript), VP, VT, PT, VPT -> 3-player Shapley
  A (raw audio incl. words), VA (original ad)                                        -> leakage references
Scores per condition:
  anchor : zero-shot, mean cos to R text anchors - mean cos to D text anchors (cross-modal, CLIP-style)
  proto  : leave-one-out prototype, cos(e_i, mean_R(-i) - mean_D(-i))  (within-condition, modality-gap-free)
  auc_loo: leave-one-out logistic regression AUC (informativeness)
Shapley: exact 3-player values on z-scored per-condition scores, v(empty) = 0; interaction = v(VPT)-v(V)-v(P)-v(T).
Conflict test: cross-party composites V_i+P_j, V_i+T_j, P_i+T_j; which source's party the composite follows.
Outputs: out/omni_emb.json (cache), out/results_B.json
"""
import csv, itertools, json, os, random, time
import numpy as np
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoProcessor
from qwen_omni_utils import process_mm_info
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

ROOT = os.path.dirname(os.path.abspath(__file__))
DATA, OUT = os.path.join(ROOT, "data"), os.path.join(ROOT, "out")
VAR, TR = os.path.join(OUT, "var"), os.path.join(OUT, "transcripts")
CACHE = os.path.join(OUT, "omni_emb.json")
MODEL = "nvidia/omni-embed-nemotron-3b"
PROC_PATH = os.path.join(ROOT, "model_local")
VIDEO_KW = {"min_pixels": 32 * 14 * 14, "max_pixels": 64 * 28 * 28, "use_audio_in_video": False}
FPS, MAX_FRAMES = 1.0, 64
PLAYERS = ["V", "P", "T"]
CONDS = ["V", "P", "T", "VP", "VT", "PT", "VPT", "A", "VA"]
ANCHORS = {
    "R": ["a Republican campaign ad", "a political ad supporting Donald Trump",
          "a conservative Republican political advertisement", "a Trump 2024 campaign video"],
    "D": ["a Democratic campaign ad", "a political ad supporting Kamala Harris and Joe Biden",
          "a liberal Democratic political advertisement", "a Harris 2024 campaign video"],
}
N_SWAPS_PER_DIRECTION = 10
SEED = 0
SHAPLEY_W = {0: 1 / 3, 1: 1 / 6, 2: 1 / 3}  # |S|!(n-|S|-1)!/n! for n = 3
DEVIATIONS = [
    "n = 22 ads (10 R / 12 D) from official 2024 candidate channels; heuristic, not inferential",
    "prosody channel = low-pass 400 Hz + loudnorm (words unintelligible; pitch, rhythm, energy kept); music bass survives",
    "video sampled at 1 fps, max 64 frames, low resolution (max_pixels 64*28*28) to fit 16 GB",
    "every condition carries the constant prefix 'passage: ' so that conditions differ only by channel content",
    "LOO prototype score has the usual leave-one-out mean bias (uniform across ads); anchors are hand-written",
    "AUCs on 22 ads have SE ~ 0.12; Shapley shares are descriptive",
]


def transcript(vid):
    p = os.path.join(TR, vid + ".json")
    return json.load(open(p))["text"] if os.path.exists(p) else ""


def content(vid, cond, text=None):
    c = []
    if "V" in cond:
        c.append({"type": "video", "video": os.path.join(VAR, f"{vid}_V.mp4"), "fps": FPS, "max_frames": MAX_FRAMES})
    if "P" in cond:
        c.append({"type": "audio", "audio": os.path.join(VAR, f"{vid}_P.wav")})
    if "A" in cond:
        c.append({"type": "audio", "audio": os.path.join(OUT, "audio", f"{vid}.wav")})
    t = text if text is not None else (transcript(vid) if "T" in cond else "")
    c.append({"type": "text", "text": "passage: " + t})
    return c


def load():
    m = AutoModel.from_pretrained(MODEL, torch_dtype=torch.bfloat16, attn_implementation="sdpa",
                                  trust_remote_code=True).to("cuda").eval()
    return m, AutoProcessor.from_pretrained(PROC_PATH, trust_remote_code=True)


@torch.no_grad()
def embed(m, p, cont):
    torch.cuda.empty_cache()
    docs = [{"role": "user", "content": cont}]
    text = p.apply_chat_template(docs, add_generation_prompt=False, tokenize=False)
    audio, images, videos = process_mm_info(docs, use_audio_in_video=False)
    batch = p(text=text, images=images, videos=videos, audio=audio, return_tensors="pt",
              text_kwargs={"truncation": True, "padding": True, "max_length": 204800},
              videos_kwargs=VIDEO_KW, audio_kwargs={"max_length": 2048000})
    batch = {k: v.to(m.device) for k, v in batch.items()}
    h = m(**batch, output_hidden_states=True).hidden_states[-1]
    mask = batch["attention_mask"]
    e = h.masked_fill(~mask[..., None].bool(), 0.0).sum(1) / mask.sum(1)[..., None]
    return F.normalize(e.float(), dim=-1)[0].cpu().numpy()


def unit(x):
    return x / np.clip(np.linalg.norm(x, axis=-1, keepdims=True), 1e-8, None)


def zscore(x):
    s = x.std()
    return (x - x.mean()) / (s if s > 1e-8 else 1.0)


def loo_proba(X, y, C=1.0):
    p = np.zeros(len(y))
    for i in range(len(y)):
        idx = np.arange(len(y)) != i
        p[i] = LogisticRegression(C=C, max_iter=2000).fit(X[idx], y[idx]).predict_proba(X[i:i + 1])[0, 1]
    return p


def proto_loo(E, y):
    s = np.zeros(len(y))
    for i in range(len(y)):
        idx = np.arange(len(y)) != i
        proto = E[idx & (y == 1)].mean(0) - E[idx & (y == 0)].mean(0)
        s[i] = E[i] @ unit(proto)
    return s


def shapley(v):
    """v: dict cond -> array over ads for V,P,T,VP,VT,PT,VPT. Exact 3-player Shapley values."""
    phi = {}
    for c in PLAYERS:
        others = [o for o in PLAYERS if o != c]
        tot = 0.0
        for k in range(3):
            for S in itertools.combinations(others, k):
                key_s = "".join(x for x in PLAYERS if x in S)
                key_sc = "".join(x for x in PLAYERS if x in S or x == c)
                tot = tot + SHAPLEY_W[k] * (v[key_sc] - (v[key_s] if key_s else 0.0))
        phi[c] = tot
    return phi


def main():
    t0 = time.time()
    rows = list(csv.DictReader(open(os.path.join(DATA, "manifest.csv"))))
    ids = [r["id"] for r in rows]
    y = np.array([1 if r["label"].upper().startswith("R") else 0 for r in rows])
    m, p = load()
    print(f"model loaded {time.time() - t0:.0f}s, vram {torch.cuda.memory_allocated() / 1e9:.1f} GB", flush=True)

    cache = json.load(open(CACHE)) if os.path.exists(CACHE) else {}

    def get(key, cont):
        if key not in cache:
            cache[key] = embed(m, p, cont).tolist()
            json.dump(cache, open(CACHE, "w"))
        return np.array(cache[key])

    # anchors as queries
    A = {k: unit(np.array([get(f"anchor:{k}:{i}", [{"type": "text", "text": "query: " + a}])
                            for i, a in enumerate(v)])) for k, v in ANCHORS.items()}
    # ad embeddings per condition
    E = {c: [] for c in CONDS}
    for n, vid in enumerate(ids):
        for c in CONDS:
            E[c].append(get(f"{vid}:{c}", content(vid, c)))
        print(f"  [{n + 1}/{len(ids)}] {vid} done, {time.time() - t0:.0f}s", flush=True)
    E = {c: unit(np.array(v)) for c, v in E.items()}

    # per-condition scores
    scores = {"anchor": {}, "proto": {}}
    auc = {"anchor": {}, "proto": {}, "logistic_loo": {}}
    for c in CONDS:
        sa = (E[c] @ A["R"].T).mean(1) - (E[c] @ A["D"].T).mean(1)
        sp = proto_loo(E[c], y)
        scores["anchor"][c], scores["proto"][c] = sa, sp
        auc["anchor"][c] = float(roc_auc_score(y, sa))
        auc["proto"][c] = float(roc_auc_score(y, sp))
        auc["logistic_loo"][c] = float(roc_auc_score(y, loo_proba(E[c], y)))

    # Shapley on z-scored scores
    shap = {}
    for kind in ("anchor", "proto"):
        v = {c: zscore(scores[kind][c]) for c in ["V", "P", "T", "VP", "VT", "PT", "VPT"]}
        phi = shapley(v)
        assert np.allclose(phi["V"] + phi["P"] + phi["T"], v["VPT"], atol=1e-6)
        inter = v["VPT"] - v["V"] - v["P"] - v["T"]
        mag = {c: float(np.abs(phi[c]).mean()) for c in PLAYERS}
        tot = sum(mag.values())
        shap[kind] = {
            "share_of_abs_contribution": {c: mag[c] / tot for c in PLAYERS},
            "auc_phi_vs_party": {c: float(roc_auc_score(y, phi[c])) for c in PLAYERS},
            "mean_phi_R": {c: float(phi[c][y == 1].mean()) for c in PLAYERS},
            "mean_phi_D": {c: float(phi[c][y == 0].mean()) for c in PLAYERS},
            "interaction_mean_abs": float(np.abs(inter).mean()),
            "interaction_auc_vs_party": float(roc_auc_score(y, inter)),
        }

    # modality gap inside the omni model: distance between condition means
    means = {c: E[c].mean(0) for c in CONDS}
    gap = {f"{a}-{b}": float(np.linalg.norm(means[a] - means[b])) for a, b in [("V", "T"), ("P", "T"), ("V", "P"), ("A", "P")]}

    # conflict test: cross-party composites, scored with full-sample prototypes of the joint condition
    rng = random.Random(SEED)
    R, D = [i for i in range(len(ids)) if y[i] == 1], [i for i in range(len(ids)) if y[i] == 0]
    pairs = [(rng.choice(R), rng.choice(D)) for _ in range(N_SWAPS_PER_DIRECTION)] + \
            [(rng.choice(D), rng.choice(R)) for _ in range(N_SWAPS_PER_DIRECTION)]
    protos = {c: unit(E[c][y == 1].mean(0) - E[c][y == 0].mean(0)) for c in ("VP", "VT", "PT")}
    conflict = {}
    for name, cond, build in [
        ("V_over_P", "VP", lambda i, j: content(ids[i], "V")[:1] + content(ids[j], "P")),
        ("V_over_T", "VT", lambda i, j: content(ids[i], "V")[:1] + [{"type": "text", "text": "passage: " + transcript(ids[j])}]),
        ("P_over_T", "PT", lambda i, j: content(ids[i], "P")[:1] + [{"type": "text", "text": "passage: " + transcript(ids[j])}]),
    ]:
        wins = []
        for i, j in pairs:
            e = unit(get(f"swap:{name}:{ids[i]}:{ids[j]}", build(i, j)))
            pred = 1 if e @ protos[cond] > 0 else 0
            wins.append(int(pred == y[i]))  # first source's party wins?
        conflict[name] = {"first_source_wins": float(np.mean(wins)), "n_pairs": len(pairs)}

    res = {"n": len(ids), "n_R": int(y.sum()), "model": MODEL, "conditions": CONDS,
           "auc": auc, "shapley": shap, "modality_gap_norm": gap, "conflict": conflict,
           "runtime_s": round(time.time() - t0), "deviations": DEVIATIONS}
    json.dump(res, open(os.path.join(OUT, "results_B.json"), "w"), indent=2)
    print(json.dumps({k: v for k, v in res.items() if k not in ("deviations", "conditions")}, indent=1))


if __name__ == "__main__":
    main()

A.8 prep_b.py: channel preparation, as run 46 lines, 2,012 bytes

ffmpeg filters for the muted-video and filtered-audio conditions; Whisper word transcripts.

#!/usr/bin/env python3
"""Part B, step 1: separate each ad into clean channels.

V  : muted video                        out/var/<id>_V.mp4
P  : prosody audio (low-pass 400 Hz)    out/var/<id>_P.wav   pitch/rhythm/energy kept, words unintelligible
VP : video + prosody audio              out/var/<id>_VP.mp4
A  : raw audio (speech + music)         out/audio/<id>.wav   (from poc_clip.extract_audio)
T  : transcript (faster-whisper words)  out/transcripts/<id>.json
"""
import csv, json, os, subprocess
from poc_clip import DATA, OUT, extract_audio, extract_frames, transcribe

VAR, TR = os.path.join(OUT, "var"), os.path.join(OUT, "transcripts")
LOWPASS = "lowpass=f=400,lowpass=f=400,loudnorm"  # two poles -> steeper roll-off; loudnorm equalises level


def sh(cmd):
    subprocess.run(cmd, capture_output=True, text=True, check=True)


def main():
    os.makedirs(VAR, exist_ok=True)
    os.makedirs(TR, exist_ok=True)
    rows = list(csv.DictReader(open(os.path.join(DATA, "manifest.csv"))))
    for r in rows:
        vid, src = r["id"], os.path.join(DATA, r["video"])
        extract_frames(vid, src)
        wav = extract_audio(vid, src)
        v, p, vp = (os.path.join(VAR, f"{vid}_{k}") for k in ("V.mp4", "P.wav", "VP.mp4"))
        if not os.path.exists(v):
            sh(["ffmpeg", "-v", "error", "-y", "-i", src, "-an", "-c:v", "copy", v])
        if not os.path.exists(p):
            sh(["ffmpeg", "-v", "error", "-y", "-i", wav, "-af", LOWPASS, "-ac", "1", "-ar", "16000", p])
        if not os.path.exists(vp):
            sh(["ffmpeg", "-v", "error", "-y", "-i", v, "-i", p, "-map", "0:v", "-map", "1:a",
                "-c:v", "copy", "-c:a", "aac", "-shortest", vp])
        tj = os.path.join(TR, vid + ".json")
        if not os.path.exists(tj):
            words = transcribe(wav)
            json.dump({"words": words, "text": " ".join(w for _, w in words)}, open(tj, "w"))
        print(f"{vid}: ok ({len(json.load(open(tj))['words'])} words)")


if __name__ == "__main__":
    main()

A.9 poc_clip.py: section 2, as run 278 lines, 12,158 bytes

Contains the defects listed in 8.0: global centering, non-nested stacking, a sparsity search without an attainment check.

#!/usr/bin/env python3
"""PoC mini-replication of Ciccarone (2026), "An Image is Worth 1000 Words? Partisanship and
Attention in Videos", Sections 2.1-2.3, 3 (exposure) and 4.1 (SpLICE concepts).

Pipeline: frame every 3 s + matched transcript chunk -> CLIP ViT-B/32 (512-d) -> per-ad mean
-> per-modality mean-centering (modality gap, Liang et al. 2022) -> per-modality classifier
+ stacked model -> exposure curve (prefix of k seconds) -> SpLICE-lite concept decomposition.

Input : data/manifest.csv with columns id,label,video,subs (label in {R,D}; paths relative to data/)
Output: out/results.json, out/embeddings.npz, out/frames/<id>/, out/audio/<id>.wav (for part B)
"""
import csv, glob, json, os, re, subprocess, sys
import numpy as np
import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor
from sklearn.linear_model import Lasso, LogisticRegression
from sklearn.metrics import roc_auc_score

ROOT = os.path.dirname(os.path.abspath(__file__))
DATA, OUT = os.path.join(ROOT, "data"), os.path.join(ROOT, "out")
STEP = 3.0  # paper: one frame every three seconds, transcript cut into as many chunks
EXPOSURES = [3, 6, 12, 24, 45]  # seconds of prefix, for the attention/exposure curve
CONCEPTS = [
    # topics
    "immigration", "border", "economy", "jobs", "inflation", "taxes", "healthcare", "abortion",
    "guns", "crime", "police", "climate", "education", "military", "veterans", "corruption",
    "democracy", "freedom", "family", "workers", "farmers", "small business", "energy", "china",
    # emotions / affect
    "fear", "anger", "disgust", "sadness", "hope", "joy", "pride", "empathy", "love", "threat",
    "safety", "strength", "unity", "patriotism",
    # visual scene words
    "flag", "crowd", "rally", "factory", "hospital", "children", "elderly", "soldier", "protest",
    "city", "countryside", "handshake",
]
DEVIATIONS = [
    "n ~ 20 ads from official candidate channels vs 15,427 labeled ads (2016-2020)",
    "logistic regression + leave-one-out CV instead of neural net + 80/20 split",
    "transcripts from faster-whisper large-v3 (YouTube auto-captions only as fallback); ads without speech get a constant empty-string text embedding",
    "R ads = Trump 2024 channel, D ads = Harris/Biden 2024 channels (same era; sponsor = candidate)",
    f"{len(CONCEPTS)} concepts instead of 160; sparsity tuned to ~8 active concepts, not matched reconstruction",
    "empty transcript chunks dropped from the text mean",
    "360p frames; exposure curve uses prefix means with the full-length classifier (LOO)",
]


def run(cmd):
    return subprocess.run(cmd, capture_output=True, text=True, check=True).stdout


def extract_frames(vid, path):
    d = os.path.join(OUT, "frames", vid)
    os.makedirs(d, exist_ok=True)
    if not glob.glob(d + "/*.jpg"):
        run(["ffmpeg", "-v", "error", "-y", "-i", path, "-vf", f"fps=1/{STEP}", "-q:v", "3", d + "/f%04d.jpg"])
    return sorted(glob.glob(d + "/*.jpg"))


def extract_audio(vid, path):
    d = os.path.join(OUT, "audio")
    os.makedirs(d, exist_ok=True)
    wav = os.path.join(d, vid + ".wav")
    if not os.path.exists(wav):
        run(["ffmpeg", "-v", "error", "-y", "-i", path, "-vn", "-ac", "1", "-ar", "16000", wav])
    return wav


TS = re.compile(r"(\d+):(\d\d):(\d\d)\.(\d{3})")
WORD_TAG = re.compile(r"<(\d+:\d\d:\d\d\.\d{3})><c>\s*([^<]+?)\s*</c>")


def ts(s):
    h, m, sec, ms = TS.match(s).groups()
    return int(h) * 3600 + int(m) * 60 + int(sec) + int(ms) / 1000


def parse_vtt(path):
    """List of (t, word). Uses YouTube word-level timing tags; falls back to cue level."""
    words, cues, cur = [], [], None
    for line in open(path, encoding="utf-8", errors="ignore"):
        line = line.rstrip("\n")
        if "-->" in line:
            a, b = line.split("-->")[:2]
            cur = (ts(a.strip()), ts(b.strip().split()[0]))
            continue
        if cur is None or not line.strip() or line.startswith(("WEBVTT", "Kind:", "Language:")):
            continue
        if "<c>" in line:
            head = re.sub(r"<[^>]+>", "", re.split(r"<\d+:\d\d:\d\d\.\d{3}>", line)[0]).strip()
            if head:
                words.append((cur[0], head))
            words += [(ts(t), w) for t, w in WORD_TAG.findall(line)]
        else:
            txt = re.sub(r"<[^>]+>", "", line).strip()
            if txt:
                cues.append((cur[0], cur[1], txt))
    if words:
        return sorted(set(words))
    out, prev = [], None
    for s, e, txt in cues:  # cue fallback: drop repeated lines, spread words over the cue
        if txt == prev:
            continue
        prev, ws = txt, txt.split()
        out += [(s + (e - s) * i / max(len(ws), 1), w) for i, w in enumerate(ws)]
    return sorted(out)


def chunks(words, n):
    return [" ".join(w for t, w in words if k * STEP <= t < (k + 1) * STEP) for k in range(n)]


_WHISPER = None


def transcribe(wav):
    """Word-level transcript with faster-whisper large-v3 (closer to the paper's transcripts)."""
    global _WHISPER
    from faster_whisper import WhisperModel
    if _WHISPER is None:
        cuda = torch.cuda.is_available()
        _WHISPER = WhisperModel("large-v3", device="cuda" if cuda else "cpu",
                                compute_type="float16" if cuda else "int8")
    segs, _ = _WHISPER.transcribe(wav, language="en", word_timestamps=True, vad_filter=True)
    return [(w.start, w.word.strip()) for s in segs for w in (s.words or []) if w.word.strip()]


def load_clip():
    dev = "cuda" if torch.cuda.is_available() else "cpu"
    m = CLIPModel.from_pretrained("openai/clip-vit-base-patch32").to(dev).eval()
    return m, CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32"), dev


@torch.no_grad()
def embed_images(m, p, dev, paths, bs=32):
    out = []
    for i in range(0, len(paths), bs):
        ims = [Image.open(x).convert("RGB") for x in paths[i:i + bs]]
        out.append(m.get_image_features(**p(images=ims, return_tensors="pt").to(dev)).float().cpu().numpy())
    return np.concatenate(out)


@torch.no_grad()
def embed_texts(m, p, dev, texts, bs=64):
    out = []
    for i in range(0, len(texts), bs):
        t = p(text=texts[i:i + bs], return_tensors="pt", padding=True, truncation=True, max_length=77).to(dev)
        out.append(m.get_text_features(**t).float().cpu().numpy())
    return np.concatenate(out)


def unit(x):
    return x / np.clip(np.linalg.norm(x, axis=-1, keepdims=True), 1e-8, None)


def loo_proba(X, y, C=1.0):
    """Leave-one-out predicted P(R)."""
    p = np.zeros(len(y))
    for i in range(len(y)):
        idx = np.arange(len(y)) != i
        clf = LogisticRegression(C=C, max_iter=2000).fit(X[idx], y[idx])
        p[i] = clf.predict_proba(X[i:i + 1])[0, 1]
    return p


def loo_proba_prefix(Xfull, Xprefix, y, C=1.0):
    """Train on full-length embeddings of the other ads, test on the prefix embedding of ad i."""
    p = np.zeros(len(y))
    for i in range(len(y)):
        idx = np.arange(len(y)) != i
        clf = LogisticRegression(C=C, max_iter=2000).fit(Xfull[idx], y[idx])
        p[i] = clf.predict_proba(Xprefix[i:i + 1])[0, 1]
    return p


def splice(Xc, C, n_active=8):
    """Non-negative sparse projection of centered embeddings onto a centered concept dictionary."""
    W = np.zeros((len(Xc), C.shape[1]))
    for i, x in enumerate(Xc):
        alpha = 0.01
        for _ in range(14):
            l = Lasso(alpha=alpha, positive=True, fit_intercept=False, max_iter=10000).fit(C, x)
            k = int((l.coef_ > 0).sum())
            if k > n_active * 1.5:
                alpha *= 1.6
            elif k < n_active / 1.5:
                alpha /= 1.6
            else:
                break
        W[i] = l.coef_
    return W


def main():
    rows = list(csv.DictReader(open(os.path.join(DATA, "manifest.csv"))))
    ids = [r["id"] for r in rows]
    y = np.array([1 if r["label"].strip().upper().startswith("R") else 0 for r in rows])
    m, p, dev = load_clip()
    print(f"device={dev}  ads={len(rows)}  R={int(y.sum())} D={int((1 - y).sum())}")

    img_ad, txt_ad, img_seq, txt_seq, n_words = [], [], {}, {}, []
    for r in rows:
        vid, vpath = r["id"], os.path.join(DATA, r["video"])
        frames = extract_frames(vid, vpath)
        wav = extract_audio(vid, vpath)
        src = "whisper"
        try:
            words = transcribe(wav)
        except Exception as e:  # fallback: YouTube captions if whisper is unavailable
            src, subs = "vtt", (r.get("subs") or "").strip()
            words = parse_vtt(os.path.join(DATA, subs)) if subs else []
            print(f"  whisper failed ({e!r}); using captions: {bool(subs)}")
        ch = chunks(words, len(frames))
        n_words.append(len(words))
        fi = unit(embed_images(m, p, dev, frames))
        keep = [k for k, c in enumerate(ch) if c.strip()]
        texts = [ch[k] for k in keep] if keep else [""]  # no speech: constant empty-string embedding
        ti = unit(embed_texts(m, p, dev, texts))
        keep = keep or [0]
        img_seq[vid], txt_seq[vid] = fi, (ti, keep)
        img_ad.append(fi.mean(0))
        txt_ad.append(ti.mean(0))
        print(f"  {vid}: {len(frames)} frames, {len(words)} words ({src}), {len(keep)} chunks")
    img_ad, txt_ad = unit(np.array(img_ad)), unit(np.array(txt_ad))

    # modality gap fix (Sec. 2.2): unit sphere -> subtract modality mean -> renormalize
    mu_img, mu_txt = img_ad.mean(0), txt_ad.mean(0)
    Xi, Xt = unit(img_ad - mu_img), unit(txt_ad - mu_txt)
    gap_before = float(np.linalg.norm(mu_img - mu_txt))
    gap_after = float(np.linalg.norm(Xi.mean(0) - Xt.mean(0)))

    # Sec. 2.3: per-modality classifier + stacked model (LOO)
    pi, pt = loo_proba(Xi, y), loo_proba(Xt, y)
    ps = loo_proba(np.c_[pi, pt], y, C=10.0)
    auc = {k: float(roc_auc_score(y, v)) for k, v in [("image", pi), ("text", pt), ("stacked", ps)]}
    corr_it = float(np.corrcoef(pi, pt)[0, 1])

    # Sec. 3: exposure curve, prefix of k seconds
    expo = {}
    for k in EXPOSURES:
        nk = max(1, int(round(k / STEP)))
        Pi = unit(unit(np.array([img_seq[v][:nk].mean(0) for v in ids])) - mu_img)
        Pt_raw = []
        for v in ids:
            ti, keep = txt_seq[v]
            sel = [j for j, kk in enumerate(keep) if kk < nk]
            Pt_raw.append(ti[sel].mean(0) if sel else np.zeros(Xi.shape[1]))
        Pt = unit(unit(np.array(Pt_raw)) - mu_txt)
        expo[k] = {"image": float(roc_auc_score(y, loo_proba_prefix(Xi, Pi, y))),
                   "text": float(roc_auc_score(y, loo_proba_prefix(Xt, Pt, y)))}

    # Sec. 4.1: SpLICE-lite; concept words centered with the text mean (sigma operation)
    Cw = unit(unit(embed_texts(m, p, dev, CONCEPTS)) - mu_txt).T  # (512, K)
    Wi, Wt = splice(Xi, Cw), splice(Xt, Cw)
    rec = {"image": float(np.mean([np.dot(x, unit(Cw @ w)) for x, w in zip(Xi, Wi)])),
           "text": float(np.mean([np.dot(x, unit(Cw @ w)) for x, w in zip(Xt, Wt)]))}

    def top_diff(W, k=8):
        d = W[y == 1].mean(0) - W[y == 0].mean(0)
        o = np.argsort(d)
        return {"R_minus_D": [(CONCEPTS[j], round(float(d[j]), 4)) for j in o[::-1][:k] if d[j] > 0],
                "D_minus_R": [(CONCEPTS[j], round(float(-d[j]), 4)) for j in o[:k] if d[j] < 0]}

    res = {"n": len(rows), "n_R": int(y.sum()), "auc_loo": auc, "corr_image_text_pred": corr_it,
           "modality_gap_norm": {"before": gap_before, "after": gap_after},
           "exposure_auc": expo, "splice_reconstruction_cos": rec,
           "concepts_image": top_diff(Wi), "concepts_text": top_diff(Wt),
           "mean_active_concepts": {"image": float((Wi > 0).sum(1).mean()), "text": float((Wt > 0).sum(1).mean())},
           "deviations": DEVIATIONS}
    os.makedirs(OUT, exist_ok=True)
    json.dump(res, open(os.path.join(OUT, "results.json"), "w"), indent=2)
    np.savez(os.path.join(OUT, "embeddings.npz"), ids=np.array(ids), y=y, img=img_ad, txt=txt_ad,
             mu_img=mu_img, mu_txt=mu_txt, concepts=np.array(CONCEPTS), Cw=Cw, Wi=Wi, Wt=Wt)
    print(json.dumps({k: v for k, v in res.items() if k != "deviations"}, indent=1))


if __name__ == "__main__":
    main()

A.10 verify_independent.py: independent recomputation used in the review 141 lines, 6,764 bytes

Shapley by permutation enumeration, AUC by rank counting, explicit leave-one-out loops, and the donor-free conflict variant; reproduces every table above from the caches.

#!/usr/bin/env python3
"""Independent recomputation of every reported number from the caches (RTA Phase 4).
Deliberately does NOT import poc_omni / poc_clip. Shapley by permutation enumeration
(not by the weight formula), AUC by Mann-Whitney rank counting, prototypes by explicit loops.
Also computes a leakage-free conflict variant (prototype without the two source ads)."""
import csv, itertools, json, math, os, sys
import numpy as np
from sklearn.linear_model import LogisticRegression

ROOT = sys.argv[1] if len(sys.argv) > 1 else os.path.dirname(os.path.abspath(__file__))
rows = list(csv.DictReader(open(os.path.join(ROOT, "data", "manifest.csv"))))
ids = [r["id"] for r in rows]
y = np.array([1 if r["label"].upper().startswith("R") else 0 for r in rows])
cache = json.load(open(os.path.join(ROOT, "out", "omni_emb.json")))
CONDS = ["V", "P", "T", "VP", "VT", "PT", "VPT", "A", "VA"]


def unit(x):
    x = np.asarray(x, float)
    return x / np.maximum(np.linalg.norm(x, axis=-1, keepdims=True), 1e-12)


def auc(yv, s):  # Mann-Whitney with tie handling
    pos, neg = s[yv == 1], s[yv == 0]
    gt = (pos[:, None] > neg[None, :]).sum() + 0.5 * (pos[:, None] == neg[None, :]).sum()
    return gt / (len(pos) * len(neg))


def loo_logistic(X, yv):
    out = np.zeros(len(yv))
    for i in range(len(yv)):
        m = np.ones(len(yv), bool); m[i] = False
        out[i] = LogisticRegression(C=1.0, max_iter=5000).fit(X[m], yv[m]).predict_proba(X[i:i + 1])[0, 1]
    return out


def loo_proto(E, yv):
    out = np.zeros(len(yv))
    for i in range(len(yv)):
        m = np.ones(len(yv), bool); m[i] = False
        p = E[m & (yv == 1)].mean(0) - E[m & (yv == 0)].mean(0)
        out[i] = float(E[i] @ (p / np.linalg.norm(p)))
    return out


E = {c: unit([cache[f"{v}:{c}"] for v in ids]) for c in CONDS}
AR = unit([cache[k] for k in sorted(cache) if k.startswith("anchor:R:")])
AD = unit([cache[k] for k in sorted(cache) if k.startswith("anchor:D:")])

scores, aucs = {"anchor": {}, "proto": {}}, {"anchor": {}, "proto": {}, "logistic_loo": {}}
for c in CONDS:
    sa = (E[c] @ AR.T).mean(1) - (E[c] @ AD.T).mean(1)
    sp = loo_proto(E[c], y)
    scores["anchor"][c], scores["proto"][c] = sa, sp
    aucs["anchor"][c] = round(float(auc(y, sa)), 3)
    aucs["proto"][c] = round(float(auc(y, sp)), 3)
    aucs["logistic_loo"][c] = round(float(auc(y, loo_logistic(E[c], y))), 3)


def z(x):
    return (x - x.mean()) / (x.std() if x.std() > 1e-12 else 1.0)


def shapley_perm(v):  # v: dict of frozenset -> array; enumerate 3! orders
    players = ["V", "P", "T"]
    phi = {p: np.zeros(len(y)) for p in players}
    for order in itertools.permutations(players):
        seen = set()
        for p in order:
            before = v[frozenset(seen)]; seen.add(p); after = v[frozenset(seen)]
            phi[p] += (after - before) / 6.0
    return phi


shap = {}
for kind in ("anchor", "proto"):
    v = {frozenset(): np.zeros(len(y))}
    for c in ["V", "P", "T", "VP", "VT", "PT", "VPT"]:
        v[frozenset(c)] = z(scores[kind][c])
    phi = shapley_perm(v)
    mag = {p: float(np.abs(phi[p]).mean()) for p in phi}
    tot = sum(mag.values())
    shap[kind] = {"share": {p: round(mag[p] / tot, 3) for p in phi},
                  "auc_phi": {p: round(float(auc(y, phi[p])), 3) for p in phi},
                  "sum_identity_max_abs_err": float(np.abs(phi["V"] + phi["P"] + phi["T"] - v[frozenset("VPT")]).max())}

means = {c: E[c].mean(0) for c in CONDS}
gap = {f"{a}-{b}": round(float(np.linalg.norm(means[a] - means[b])), 3) for a, b in [("V", "T"), ("P", "T"), ("V", "P"), ("A", "P")]}

# conflict test: full-sample prototype (as reported) and leakage-free (prototype without ads i and j)
lab = dict(zip(ids, y))
conflict = {}
for name, cond in [("V_over_P", "VP"), ("V_over_T", "VT"), ("P_over_T", "PT")]:
    keys = [k for k in cache if k.startswith(f"swap:{name}:")]
    full = E[cond][y == 1].mean(0) - E[cond][y == 0].mean(0)
    w_full, w_loo = [], []
    for k in keys:
        _, _, i, j = k.split(":")
        e = unit(cache[k])
        w_full.append(int((1 if e @ full > 0 else 0) == lab[i]))
        m = np.array([v not in (i, j) for v in ids])
        p2 = E[cond][m & (y == 1)].mean(0) - E[cond][m & (y == 0)].mean(0)
        w_loo.append(int((1 if e @ p2 > 0 else 0) == lab[i]))
    conflict[name] = {"n": len(keys), "first_wins_full_proto": sum(w_full), "first_wins_proto_without_i_j": sum(w_loo)}

# Part A from embeddings.npz
z_ = np.load(os.path.join(ROOT, "out", "embeddings.npz"), allow_pickle=True)
img, txt, yA = unit(z_["img"]), unit(z_["txt"]), z_["y"]
mu_i, mu_t = img.mean(0), txt.mean(0)
Xi, Xt = unit(img - mu_i), unit(txt - mu_t)
pi, pt = loo_logistic(Xi, yA), loo_logistic(Xt, yA)
partA = {"gap_before": round(float(np.linalg.norm(mu_i - mu_t)), 3), "gap_after": round(float(np.linalg.norm(Xi.mean(0) - Xt.mean(0))), 3),
         "angle_before_deg": round(2 * math.degrees(math.asin(min(1, np.linalg.norm(mu_i - mu_t) / 2))), 1),
         "auc_img": round(float(auc(yA, pi)), 3), "auc_txt": round(float(auc(yA, pt)), 3),
         "corr_pred": round(float(np.corrcoef(pi, pt)[0, 1]), 3),
         "auc_stacked": round(float(auc(yA, loo_logistic(np.c_[pi, pt], yA))), 3)}

out = {"n": len(ids), "n_R": int(y.sum()), "auc": aucs, "shapley": shap, "gap": gap, "conflict": conflict, "partA": partA}
print(json.dumps(out, indent=1))

# diff against reported
rb = json.load(open(os.path.join(ROOT, "out", "results_B.json")))
ra = json.load(open(os.path.join(ROOT, "out", "results.json")))
diffs = []
for k in ("anchor", "proto", "logistic_loo"):
    for c in CONDS:
        d = abs(aucs[k][c] - rb["auc"][k][c])
        if d > 0.006: diffs.append(f"B auc {k} {c}: mine {aucs[k][c]} reported {rb['auc'][k][c]:.3f}")
for kind in ("anchor", "proto"):
    for p in "VPT":
        d = abs(shap[kind]["share"][p] - rb["shapley"][kind]["share_of_abs_contribution"][p])
        if d > 0.006: diffs.append(f"B share {kind} {p}: mine {shap[kind]['share'][p]} reported {rb['shapley'][kind]['share_of_abs_contribution'][p]:.3f}")
for k in gap:
    if abs(gap[k] - rb["modality_gap_norm"][k]) > 0.006: diffs.append(f"B gap {k}")
for k in conflict:
    if conflict[k]["first_wins_full_proto"] / conflict[k]["n"] != rb["conflict"][k]["first_source_wins"]: diffs.append(f"B conflict {k}")
for k, mine in (("image", partA["auc_img"]), ("text", partA["auc_txt"]), ("stacked", partA["auc_stacked"])):
    if abs(mine - ra["auc_loo"][k]) > 0.006: diffs.append(f"A auc {k}: mine {mine} reported {ra['auc_loo'][k]:.3f}")
if abs(partA["gap_before"] - ra["modality_gap_norm"]["before"]) > 0.006 or abs(partA["gap_after"] - ra["modality_gap_norm"]["after"]) > 0.006: diffs.append("A gap")
print("DIFFS vs reported:", diffs or "none (all within 0.006)")