Prompt-Level Controls for Prosody-Aligned Facial Performance

1. Introduction

Modern video systems can produce impressive lighting, environments, and camera aesthetics. Yet character performance in dialogue closeups frequently fails in predictable ways: timing is flat, stillness is missing, accents do not land on meaning, and facial motion becomes constant low-amplitude noise. The result is often perceived as "dead" even when lip sync is correct.

Speech-driven facial animation has a long lineage, from early audio-to-control approaches to modern neural and 3D face models conditioned on speech audio (Brand, 1999; Cudeiro et al., 2019; Thies et al., 2020; Zhou et al., 2020).

This paper targets a specific, repeatable defect—even timing—and asks a narrow question:

Can instruction-level controls reliably shift facial motion toward stress-aligned accents and readable holds, under controlled comparisons, without changing model weights?

1.1 Problem Decomposition: Operationalizing "Weak Animation Choices"

To make "this animation doesn't look good" actionable, the work decomposes perceived quality into separable sub-problems, each mapped to (i) an annotation target, (ii) a controllable instruction primitive, or (iii) a metric.

How this decomposition was derived.

This list did not come from theory first—it came from repeated diagnosis. The author developed the decomposition by scene-by-scene performance analysis across thousands of shots from feature-film (e.g., Zootopia, Frozen, Soul) and by comparing that benchmark against hundreds of student shots that failed in consistent ways (Buck & Lee, 2013; Docter & Powers, 2020; Howard & Moore, 2016). Across these comparisons, different "bad" outputs often looked different on the surface, but they collapsed into a small set of recurring failure modes: unclear beats, uniform rhythm, missing holds, excess micro-motion noise, drifting eye intention, and poor staging/silhouette. In other words, the list below reflects a practical attempt to translate principles used in production practice into a diagnostic framework consistent with classic animation guidance (Thomas & Johnston, 1981/1995; Williams, 2001/2009).

Decomposed sub-problems.

1. Beat clarity (staging + beats): Are intention changes readable as discrete beats?
   Label: beat boundaries   Rubric: beat readability
2. Timing (rhythm of emphasis): Do accents land on stressed words / beat boundaries rather than uniform spacing?
   Labels: stress anchors   Metrics: AAE, ETS
3. Spacing / micro-motion noise: Is motion dominated by jitter that adds busyness without intent?
   Metric: HR (with noise thresholds)   Rubric: noise vs intent
4. Contrast (stillness vs movement): Are there holds that make beats readable?
   Metrics: HR, CR   Rubric: hold placement
5. Eye path / intention: Do eyes lead thought changes, or drift without purpose?
   Label: gaze shift events   Rubric: eye-leads-thought
6. Silhouette / readability (for wider shots): Does a single frame communicate the beat?
   Rubric item: pose/silhouette (optional in closeup-only evaluation)

Scope: This paper focuses on B + D (timing and contrast) and tracks A/E as rubric items for future expansion.

2. Key Terms

Beat

A change in intention/thought the audience should notice (new idea, realization, emotional turn).

Accent

A motion peak that marks a beat or a stressed word (head nod/tilt, brow pop, lid change, jaw "hit").

Hold

Intentional stillness (or near-stillness) that creates contrast and makes the next accent readable.

Even timing

Accents occur at near-uniform intervals with low stress/non-stress contrast.

Micro-motion noise

Continuous small movement that does not communicate intention (wiggle that reduces readability).

3. Benchmark: Feature-Film Performance Rubric (v0.1)

Instead of claiming "cinematic," the paper uses a practical rubric aligned with what viewers perceive in dialogue closeups. The rubric is informed by classic animation principles emphasizing clarity, timing, staging, and readable intention (Thomas & Johnston, 1981/1995; Williams, 2001/2009).

4. Method: A Prompt-Level "Acting Control Layer"

The method translates an animator's diagnosis loop into structured instruction constraints.

Animator diagnosis loop (conceptual):

Control primitives (instruction-level):

• Rhythm of emphasis: Assign accents to stressed words and beat boundaries; suppress filler-word motion.
• Holds: Insert short stillness before key beats to increase contrast and readability.
• Stress contrast: Amplify motion near stressed words, reduce motion elsewhere, and avoid constant micro-motion.

Key idea.

These controls live in instruction structure (prompt-level). No weight updates are required, enabling rapid ablations.

This fits a broader pattern in generative modeling: prompt/instruction changes and auxiliary conditioning can steer pretrained models without full retraining, as seen in instruction-following diffusion editing (Brooks et al., 2023), prompt-to-prompt attention control (Hertz et al., 2023), and conditional control modules such as ControlNet (Zhang et al., 2023).

5. Data & Annotation (Auditable, Lightweight)

This paper uses a one-clip case study designed to be reproducible in a single day, even without forced alignment tooling.

5.1 Case Study Transcript

"You don't understand, I am a celebrity, it is all about me, it has been for decades, hahaha, that is the point of celebrity."

5.2 Annotation Schema

Minimal labels that capture performance structure (not aesthetic preference):

• Stress anchors (word-level timestamps)
• Beat boundaries (timestamps for intention shifts)
• Pause/inhale windows (silence gaps)
• Optional acting events: gaze shifts, blink clusters, laughter burst, hold segments

5.3 Stress-Timestamp Table (Derived from Audio Waveform)

These anchors are approximate (audio energy/prosody peaks + silence segmentation), intended to be auditable and "good enough" for controlled proxy metrics. Peak time is the recommended target for the primary accent.

5.4 Pause/Inhale Candidates (Silence Gaps)

5.5 Beat Boundary Proposal (For This Line)

6. Evaluation Protocol

6.1 Controlled Comparison Design

To claim causality, evaluation must hold everything constant except instruction structure:

• Same audio
• Same base generation pipeline/model
• Same sampling/seed where possible (if not, report variability across N runs)
• Only change: baseline instruction vs control-layer instruction

6.2 Proxy Metrics

These metrics do not claim "objective Disney." They quantify whether the output better matches the intended structure (emphasis + holds + contrast).

6.2.1 Accent Alignment Error (AAE)

Lower is better.

6.2.2 Even Timing Score (ETS)

Compute accent gaps $\Delta t$ between consecutive motion peaks.

Low CV ⇒ too even ⇒ worse. Higher CV (within reason) ⇒ better rhythm.

6.2.3 Hold Ratio (HR)

Higher is better up to a point.

6.2.4 Contrast Ratio (CR)

Higher is better (within reason).

6.3 Human Evaluation Protocol (Blinded A/B + Rubric)

• Design: Paired A/B (baseline vs control), randomized order, identical audio, raters blinded.
• Task: Preference + rubric scores (Section 3).
• Report: Preference rate + rubric deltas with uncertainty estimates.

6.4 Inter-Rater Reliability

• Preference agreement: Fleiss' κ or Cohen's κ
• Rubric consistency: ICC(2,k) or Krippendorff's α
• Calibration: A small "gold" set aligns rubric interpretation; calibration excluded from final reporting.

7. Case Study: Applying the Control Layer to the Clip

7.1 Baseline Diagnosis (Typical Signature)

• Accents near-uniform intervals (low CV)
• Holds missing (low HR)
• Low stress contrast (low CR)
• Motion peaks drift relative to stress anchors (high AAE)
• Micro-motion noise fills gaps (rubric penalty)

7.2 Intervention (Instruction-Level)

• Place primary accents near: don't, I, celebriTY, ME, deCADES, POINT, celebriTY (final)
• Insert holds in silence windows: 1.472–1.700, ~4.60, 6.67–6.88
• Suppress filler-word motion to protect contrast
• Treat "hahaha" as an event with body stillness and minimal deliberate facial intent

7.3 Expected Measurable Outcome

AAE ↓, CV ↑ (less even), HR ↑, CR ↑, rubric deltas improve on emphasis correctness and hold placement.

8. Iteration: Metrics → Diagnosis → Next Experiment (Closed Loop)

Iteration summary (research framing):

The method uses measurement to localize failure, then modifies one control factor at a time to attribute improvements to specific causes.

Closed-Loop Process

1. Step 1 — Measure: Compute AAE/ETS/HR/CR + rubric deltas.
2. Step 2 — Diagnose (metric → cause mapping):
   • AAE high ⇒ accents late/early → shift accent timing constraints by Δt
   • ETS too low ⇒ too even → add holds at beat boundaries; suppress filler motion more
   • HR too low ⇒ not enough stillness → enforce explicit holds; reduce micro-motion
   • CR too low ⇒ stress not readable → amplify stressed-word accents; reduce unstressed motion
3. Step 3 — Ablate: Change one factor at a time:
   • head-only vs head+eyes+brows coupling
   • hold length sweep (2/3/5/8 frames)
   • stress threshold sweep (which words are primary)
4. Step 4 — Select (decision rule summary): Accept a change only if target metrics improve by ≥ΔX while regressions stay below ε (e.g., HR ↑ without AAE worsening beyond ε).

9. Limitations

• Proxy metrics approximate perceived quality; blinded human preference remains primary.
• Stress anchors derived from waveform peaks are approximate; forced alignment would improve precision.
• Prompt-level controls improve consistency but may fail for extreme styles or noisy audio.
• Eye intention and staging require additional labels and signal extraction.

10. Next Steps

• Replace approximate anchors with forced alignment + prosody features ($F_0$, energy, duration).
• Convert anchors into an explicit emphasis track for conditioning (not only instruction).
• Expand evaluation across speaking styles and emotional contexts.
• Add dedicated metrics for gaze alignment and micro-motion noise.
• Build a small benchmark set with labeled beats/stress + rubric + reliability as standard.