iPoemRec: Bridging Visual Social Media and Classical Culture through Artistic Conception
Through The Eyes of A Poet: Classical Poetry Recommendation with Visual Input on Social Media
The paper introduces iPoemRec, the first image-driven recommender system for classical poetry that bridges visual social media content with cultural heritage. By leveraging a Conception-aware Heterogeneous Information Network (CaHIN) and neural embeddings, it recommends poems that align with the artistic conception (sentiments and themes) of a user's photo.
TL;DR
iPoemRec is a novel recommender system that takes a user's photo (e.g., a travel picture) and recommends classical poems that capture the "soul" of the image. Unlike basic image captioning, it uses a Conception-aware Heterogeneous Information Network (CaHIN) to map metaphorical relationships between visual objects and poetic themes, achieving a significant performance leap over standard object-matching baselines.
The "Literal" Bottleneck: Why Simple Retrieval Fails
In the realm of classical poetry, a bird is rarely just a bird. It might symbolize loneliness, freedom, or spring joy depending on the context. Existing cross-modal tools often fall into the trap of Object Consistency without Sentiment Consistency.
For instance, if you take a photo of a solitary bird at sunset, a traditional system might recommend a happy poem just because it mentions "birds." This creates a jarring user experience. The authors argue that the "Artistic Conception"—the implicit sentiment and theme—is the vital third dimension required for successful cultural recommendation.
Decoding the "Eyes of a Poet": Methodology
The iPoemRec architecture is divided into three sophisticated pipelines that transform raw pixels into deep cultural insights.
1. The Poetic Visual Analyzer (PVA)
Instead of just detecting "objects," the PVA focuses on Visual ANPs (Adjective-Noun Pairs). By fine-tuning models on SentiBank and YOLO V3, the system identifies "magnificent waterfalls" or "misty mountains." It then ranks these by their Metaphorical Degree—a metric representing how strongly connected a phrase is to classic themes in the knowledge graph.
2. The CaHIN (Conception-aware Heterogeneous Information Network)
This is the "brain" of the system. It is a graph where nodes represent Sentiments (Happy, Sad), Themes (Lovesickness, War), and ANPs.
Figure: The iPoemRec framework linking visual input to the CaHIN.
Links in this network capture co-occurrence and metaphorical similarity. For example, "Old Tree" is linked to "Sadness." This allows the system to perform Reasoning via Meta-paths: if an image has a "withered branch," the system can traverse the graph to find "loneliness" and recommend a matching poem, even if the poem doesn't explicitly mention "branches."
3. Meta-path Embedding with SEMPR
To bridge the gap between image paths and poem paths, the system uses a Stacked Autoencoder (SAE). It converts graph segments (meta-paths) into vectors in a shared latent space. This allows the system to calculate a "Cosine Similarity" between a photo and a poem based on their shared "conceptual DNA."
Experimental Results: Precision and Soul
The researchers tested iPoemRec against powerful baselines including Microsoft’s SCAN and SentiBank.
- Precision@K: iPoemRec consistently outperformed baselines, showing higher gains as the recommendation list length () increased. This proves the system's ability to maintain relevance even in its "deep-cut" recommendations.
- User Satisfaction: In an online study with 297 real-world responses, iPoemRec earned significantly higher ratings (MOR@1: 3.51) compared to object-based models like Word2Vec (MOR@1: 3.07).
Figure: Precision@K metrics showing iPoemRec's dominance.
Critical Insight: The Universal Language of Metaphor
One of the most fascinating findings is the Generality Analysis. While designed for Chinese poetry (known for its dense symbolism), the authors applied iPoemRec to Shakespeare’s Sonnets. Interestingly, the system performed even better on English sonnets because their artistic themes are often more explicit. This suggests that the "Conception-aware" approach is a universal framework for cultural AI.
Conclusion
iPoemRec represents a shift from "Visual Information Retrieval" to "Visual Aesthetic Retrieval." By modeling the metaphor rather than the noun, it opens the door for AI to truly appreciate human culture. Future applications could range from smart education apps to AI-curated "hidden gem" travel guides that provide poetic descriptors for the modern traveler.