What does the ManzaiSet dataset reveal about viewer responses to manzai comedy?

A dataset built to link viewers to shared performances

The question behind ManzaiSet is how to study why two people laugh at different moments of the same comedy routine. Answering it requires three things held together: the content that was shown, each viewer's observable reactions, and that viewer's own evaluation. The dataset keeps all three connected by recording facial video and microphone audio while each participant watched, then collecting a 0–100 overall rating after each performance.

The stimulus set is 10 professional manzai performances, each roughly four to six minutes, selected from official recordings of a single event held in April 2022. Manzai is a two-person Japanese comedy form built on exchanges between the boke, who introduces the incongruity, and the tsukkomi, who responds to or corrects it. Because every participant saw performances drawn from this same set in randomized order, responses from separately recorded viewers can be aligned to a common performance timeline.

The repeated unit is a viewer-performance recording: one participant watching one performance. The corpus contains 2,301 such recordings from 241 participants, averaging 9.55 recordings per participant and totaling 191.8 hours of facial video and microphone audio. Facial video was captured at 640×360 pixels and 25 frames per second, audio at 48 kHz, and participant, performance, and viewing-position identifiers link each recording to its rating and its place in the sequence.

Most viewers rated generously, but variability split the sample

For each of the 228 participants who watched at least eight performances, the authors summarized ratings with six features: mean, standard deviation across performances, coefficient of variation, range, the difference between late-half and early-half means, and a linear slope over viewing positions. After standardizing these features, k-means clustering produced three descriptive profiles. The authors chose three clusters as an interpretable summary; the silhouette score was highest at k=3 (0.523), while the Calinski–Harabasz score was highest at k=4 (158.03).

The largest profile held 166 viewers (72.8%), with a mean rating of 87.0, a standard deviation of 4.84, and a range of 15.6. A second profile of 32 viewers (14.0%) averaged 71.2 with a standard deviation of 17.9 and a range of 57.8, and showed a positive viewing-order trend. A third profile of 30 viewers (13.2%) averaged 65.6 with a standard deviation of 13.9 and a negative viewing-order trend. These are summaries of the features that formed the clusters, not independent measurements.

Two dispersion statistics make the individual differences concrete. The between-participant coefficient of variation was 0.16 for mean ratings, but 0.81 for the standard deviation feature, 0.98 for the coefficient of variation, and 0.80 for the range. In plain terms, viewers differed far more in how much their ratings moved across performances than in their overall rating level. The corpus mean rating was 82.0 with a standard deviation of 16.4, so ratings clustered toward the top of the scale, and the authors note that this ceiling may contribute to the low variability of the high-rating profile.

Rating order stayed consistent within a viewing sequence

The authors split each participant's viewing sequence into two random halves and correlated participant mean ratings across the halves. The result was r = 0.866, meaning viewers who rated one half highly also tended to rate the other half highly within that same sequence. This is a rank-order consistency statistic among participants inside one sequence, not a test of whether a viewer would rate the same way on a different day.

That distinction matters for how the profiles should be read. The three profiles summarize the observed viewing sequence, and the authors state that their stability across separate occasions remains a question for longitudinal research. The split-half result supports using these ratings to study individual differences within a session; it does not establish a fixed viewer trait.

Later performances in the sequence drew higher ratings

For each of the 228 participants, the authors fitted a linear regression of rating on numeric viewing position and tested the mean of those 228 slopes against zero. The average individual slope was 0.488 rating points per position (t(227) = 5.42, p < 0.001). The position-wise mean rating rose from 77.95 (SD 19.70) at the first position to 83.15 (SD 16.83) at the tenth, a difference of 5.20 points.

These two numbers answer slightly different questions and should not be swapped. The 5.20-point figure compares the ratings available at positions 1 and 10 only, while the 0.488 slope uses all observed positions within each participant. Both point the same direction: performances watched later in the sequence received higher ratings on average.

The authors list familiarity, mood, and rating calibration as possible explanations for this association. The design supports the association under randomized presentation order, but it does not isolate which of these explanations is responsible. For anyone modeling session data, the practical consequence is that presentation position is a variable worth including rather than assuming ratings are position-neutral.

Humor categories did not separate after correction

The third analysis asked whether viewer responses differed by type of humor. Whisper transcribed the 10 performances, and GPT-5-mini assigned nine humor tags to 77 stimulus instances: wordplay, exaggeration, unexpected twist, self-deprecation, impersonation, repetition, misunderstanding, physical gag, and cultural reference. Multiple tags could apply to one instance, so the 77 instances received 131 tag assignments. Unexpected twists were most frequent (35 instances, 45.5%), followed by repetition (28, 36.4%) and exaggeration (19, 24.7%).

Candidate vocal responses were derived from microphone audio using frame-level root-mean-square energy, zero-crossing rate, spectral features, and peak detection, and were counted if they occurred within 3 seconds of a humor-instance onset. The complete-case sample was 207 viewers. Descriptive response rates ranged from 46.7% for wordplay (eight instances) to 59.0% for cultural references (five instances).

Those descriptive rates are unadjusted, and the inferential test tells a different story. A marginal logistic model with participant-level clustering and video fixed effects, with all tag indicators entered jointly, found no statistically significant pairwise differences between humor categories after Benjamini–Hochberg correction (all adjusted p > 0.05). The authors also note that some categories rest on only five instances and that labels overlap. The defensible reading is that this dataset did not detect category differences, not that humor types are equivalent.

What the recordings can and cannot support

The audiovisual layer is substantial but not clean. OpenFace tracking met the criteria success=1 and confidence>0.9 in 96.3% of frames, which supports derived facial-action and head-pose features. Microphone signals, however, can contain speech, background sounds, and leakage from the stimulus itself, and participants were only encouraged, not required, to use headphones. Audio-derived response measures therefore carry measurement uncertainty, particularly for quiet reactions and visually grounded humor.

Remote collection adds variation in lighting, camera placement, microphones, and background sound. Fine-grained temporal analyses also depend on playback-to-recording alignment and on timing variability across participants' browsers and devices. The corpus is a restricted research collection: identifiable recordings are governed by the participant information and permissions, and the professional performance videos remain subject to their owners' rights.

Where these findings transfer, and where they do not

The sample was 241 compensated native Japanese speakers: 39 women, 201 men, and one participant reporting another gender, aged 19 to 63 with a mean of 30.0. Many were frequent comedy viewers: 135 participants (56.0%) chose the maximum of 5 on general comedy viewing frequency, and 114 (47.3%) chose 4 on manzai-specific viewing. All 10 performances came from a single professional event. The authors state that recruitment, viewing familiarity, and completion can affect the observed distributions, and that extending the corpus to other languages, audiences, and performance genres would enable cross-cultural comparisons.

Within those bounds, the paper's most useful contribution is the linkage itself. Because shared performances, repeated individual recordings, performance ratings, and viewing order are joined, the design supports comparisons between different viewers exposed to the same material and between performances seen by the same viewer. The authors suggest that held-out viewers and held-out performances can separate generalization across people from generalization across content.

The priority this dataset sets is therefore not a ranking of humor types. It is the combination of a stable within-sequence rating profile, a measurable viewing-order effect of about 0.488 points per position, and audiovisual signals aligned to the same performance timeline. Anyone using ManzaiSet should treat viewer identity and presentation position as first-order variables, and should treat humor-category comparisons as descriptive until larger instance counts and cleaner response measures are available.

Sources used in this answer