Beyond the Transcript: Boosting Oral History Classification via Temporal Logic

TL;DR

Relying on Automatic Speech Recognition (ASR) to index oral history is notoriously difficult due to high error rates. This paper moves beyond pure text by exploiting the chronological structure of interviews. By introducing Time-Shifted Classification (TSC) and Temporal Label Weighting (TLW), the authors demonstrate that when something is said is often just as informative as what is recognized, leading to a 15% boost in classification accuracy.

The Problem: The "Garbage-In" ASR Trap

Oral history archives are a goldmine for researchers, but searching them is a nightmare. ASR quality for conversational speech is often poor—word error rates (WER) typically hover around 25%. Traditional "Bag-of-Words" classifiers struggle because the "highly selective" terms needed for accurate topic labeling are the most likely to be misrecognized by the ASR system.

Furthermore, standard classifiers treat segments as independent entities. In reality, oral histories are stories. They have a beginning (childhood), a middle (major life events), and an end (reflections or artifact descriptions). Ignoring this sequence is a waste of a powerful inductive bias.

Methodology: Mapping Time to Topics

The researchers proposed two primary mechanisms to capture the "temporal domain knowledge" of these archives:

1. Time-Shifted Classification (TSC)

TSC operates on the principle of local sequence. If a narrator is talking about "Berlin" in one segment, it's highly probable that the next segment is either still about Berlin or a related geographical transition.

• Mechanism: The classifier is trained to use features from segment $N$ to predict labels for segment $N+1$.
• Insight: This captures the "momentum" of a conversation, allowing terms that were recognized clearly in one segment to "lend" their weight to the next segment where they might have been garbled by ASR.

2. Temporal Label Weighting (TLW)

TLW focuses on the global timeline. Interviews in this collection (Holocaust survivors) almost always follow a linear biography.

• Mechanism: Using Gaussian Kernel Density Estimation (KDE), the authors modeled the probability $p(c,t)$ of a label $c$ appearing at a specific time $t$ (as a percentage of the total interview duration).
• Intuition: As seen in the figure below, years mentioned in transcripts follow a clear upward ramp. TLW biases the classifier: early segments get a "boost" for topics like schooling, while late segments get a "boost" for post-war trials.

Figure: The correlation between the "time" mentioned in speech and the "actual time" within the interview duration.

Experiments and Results

The authors tested their approach on the CLEF CL-SR collection, which contains 8,104 segments. They compared a baseline kNN classifier against versions enhanced with TSC, TLW, and their combination.

Key Metrics: Clipped R-Precision

Because real-world users only look at the top $N$ suggested labels, the authors used Clipped R-Precision. This metric ensures that the system isn't unfairly penalized for segments that naturally have fewer labels than the cutoff.

Performance Gains

The results were striking across both "Geography" (places) and "Concept" (topics) labels:

• Geography Labels: Accuracy jumped by 15.4% at the leaf level.
• Concept Labels: Improved by 8.3%.

Figure: Final results showing the additive benefit of TSC and TLW over the baseline across different categories.

Critical Insight: Why Geography Wins

Interestingly, the "Time-Shifted" (TSC) method worked significantly better for Geography than for Concepts. Why? Because while a narrator might switch topics (concepts) rapidly within a single location, they usually stay in one city or country for an extended portion of their story. Geography has more "temporal inertia," making it a perfect candidate for time-shifted modeling.

Conclusion and Future Outlook

This work proves that domain knowledge—specifically the structure of human storytelling—can compensate for technical limitations in ASR.

Future Directions:

1. Sequence Modeling: Moving from kNN to Hidden Markov Models (HMMs) or LSTMs to better model the transition probabilities between labels.
2. Feature Transformation: Artificially "corrupting" training text to make it look more like ASR output, narrowing the gap between clean training data and noisy test data.

For digital librarians and AI developers alike, this paper serves as a reminder: when the data is noisy, look for the underlying rhythm of the narrative.