Reviving the Past: 3D Reconstruction of Lost Heritage via Machine Learning and Photogrammetry
Processing Historical Film Footage with Photogrammetry and Machine Learning for Cultural Heritage Documentation
This paper presents an integrated pipeline combining Faster R-CNN for object detection and Structure-from-Motion (SfM) for the 3D metric reconstruction of Cultural Heritage from historical film footage. The study utilizes the open-source COLMAP framework to overcome the technical challenges of low-quality archival videos, achieving a metric accuracy comparable to modern photogrammetric benchmarks.
TL;DR
Researchers at the Polytechnic of Turin have developed a workflow to turn historical "city films" into accurate 3D models. By combining Faster R-CNN for automatic monument detection and COLMAP (SfM) for metric reconstruction, they successfully extracted 3D data from distorted, low-resolution 16mm film with an accuracy of approximately 0.33 cm, proving that archival footage is a viable source for scientific documentation.
Context: Why Historical Films are Data Goldmines
In the field of Cultural Heritage, we often face a tragic gap: monuments are destroyed or altered before they can be digitally surveyed. Historical archives contain the only visual traces of these structures. However, these "data" are trapped in unorganized film reels with:
- No Metadata: No focal length, sensor size, or lens distortion info.
- Poor Geometry: Cameras were panned or tilted for aesthetics, not for the "3x3 rules" of photogrammetry.
- Physical Degradation: Scratches, humidity damage, and low resolution.
Methodology: The Detection-to-Reconstruction Pipeline
The authors proposed a specialized workflow designed to bypass the limitations of commercial black-box software.
1. Automated Detection (Machine Learning)
Instead of manually scrubbing through hours of footage, the team utilized Faster R-CNN implemented in TensorFlow. They focused on the Tour Saint Jacques in Paris—a monument that has been physically moved and altered over time.
- The dataset: 600 training images including contemporary photos, archival stills, and "negative" images of the Paris skyline to reduce false positives.
- Performance: The model achieved 100% accuracy in identifying the tower across 10 historical films (1910s–1960s).
2. Overcoming Camera Motion
Most historical videos use Tilting (Up/Down) or Panning (Rolling). In these cases, the "baseline" (the distance between camera positions) is tiny, which usually breaks 3D triangulation.
Figure: The common camera motions found in archival footage—Tilting, Panning, and Trucking.
3. Open-Source SfM via COLMAP
By using COLMAP, the researchers could control the Simple Radial Camera Model. This is crucial when the internal orientation of the lens is unknown. They used "Sequential Matching," which is optimized for video frames where overlapping features are found in a temporal order.
Figure: The photogrammetric pipeline—from feature extraction to 3D point cloud reconstruction.
Experiments: Validating Accuracy
The fundamental question was: Is the 3D model scientifically valid? The authors created a benchmark using a high-end Canon 5DS R (50.6 MP) to shoot the Valentino Castle in Turin, mimicking the "Tilting" motion of old films.
| Metric | Historical Case Study (16mm) | Modern Benchmark (5DS R) |
|---|---|---|
| Mean Error (px) | 0.23 px | 0.36 px |
| GSD (Ground Sample Dist.) | 1.43 cm/px | 1.2 cm/px |
| Mean Error (cm) | 0.33 cm | 0.1 cm |
Surprisingly, the historical footage performed exceptionally well. The error was within the range of one pixel, making it sufficiently accurate for architectural history and restoration planning.
Figure: Deviation analysis between the model generated from film and the "ground truth" 3D mesh.
Critical Insight & Conclusion
The study proves that we don't always need "perfect" data to get professional results. The "intelligence" lies in the selection of the tool—using open-source SfM allows researchers to manually intervene where "closed" software like Metashape might fail due to lack of EXIF data.
Takeaway for the Industry: Archives are no longer just passive repositories; they are active datasets. Using this pipeline, historians can virtually "re-walk" cities that no longer exist, and architects can verify the structural transformations of heritage sites over a century with centimeter-level precision.