ZACH ZEYU WANG: Hi my name is Wang Zeyu.
You can call me Zach, which is supposed to be
the pronunciation of my name in middle Chinese 1,000 years ago.
I'm a second year PhD student in the computer graphics
group at Yale University.
In this says CS50 seminar, instead of a hands-on tutorial for some programming
tools, I'd like to present the idea of combining
computer science with humanities.
The title is, 3D World, Augmented Reality, and Cultural Heritage.
First, imagine that you're visiting the Louvre Museum,
and see the sculpture of Venus.
As appreciating the great work of ancient Greek,
you must be wondering that, what she is doing with her arms?
Although it is the loss of her two arms that makes the sculpture famous,
and leaves people with infinite imagination.
What about those sculptures who lost heads?
Or whose nose was cut off because of natural or human damage?
Do you think that's the beauty of defect?
Then what can we do to help every visitor again access
to the original beauty of these great works?
Here comes our Longmen's story.
[VIDEO PLAYBACK]
[MUSIC PLAYING]
- November 11th, 1910.
It was a crisp and clear dawn.
A small thread of sunlight gradually wound its way
into the temple of the hidden brook.
Still submerged in the midst of his dreams, the American railroad
magnate Charles Lang Freer, was suddenly illuminated
by this strand of golden light.
Fending the radiance from his eyes, freer
could distinguish a mysterious outline penetrating the brightness
in the gaps between his outstretched fingers.
Quickly cycling 1,000 images in his mind, Freer thought to himself,
ah yes, that's the one, witnessing a familiar scene.
Freer slowly pulled himself up, and headed towards the window.
Opening it up, the solemn, towering cliffs
of the eastern mountains of Longmen flooded into the room
from the opposite shore of the Yi River.
At that moment, Freer thought once more of the fortnight spent at the Longmen
grottoes, and his sorrow at his imminent departure filled his mind.
Taking up a pen, he marked his diary.
Though I spent an entire lifetime, I would still
be at a loss to understand the full breadth of the Longmen sculptures.
This place contains works from several centuries.
The greatest epic of Chinese art.
How could one person in a single life fully appreciate everything
that this place has to offer?
[PLAYBACK ENDS]
ZACH ZEYU WANG: The Longmen Grottoes are located
in the central Chinese city of Luoyang, the earliest
and longest-surviving capital of ancient China.
The grottoes are situated on the southern outskirts of Luoyang,
carved into the Eastern and Western mountains flanking the Yi River.
Along this nearly half mile stretch of the river,
the grottoes sprout like glittering stars
across emerald mountains, and azure waters.
They are over 2,400 [? Buddha ?] [? images, ?] more than 10,000
sculptures, and 2,800 carved inscriptions.
In 1961, Longmen was recognized by the Chinese government
as one of the most important sites for cultural preservation.
Nearly 40 years later, the Longmen Grottoes
celebrated the new Millennium by becoming a UNESCO World Heritage
Site in November of the year 2000.
The UNESCO World Heritage committee recognizes the sculptures
of the Longmen Grottoes as the apex of Chinese stone carving,
from the end of the Northern Wei Dynasty through the Tang dynasty.
2000 years ago, as the Buddhism traveled far and wide across Asia,
along the dusty paths of the silk roads, this tradition
of caves, temples, architecture, and sculpture
also made their way into the central plains of China.
Along the silk road lie four great Buddhist sites
in the UNESCO World Heritage list.
More Caves in Dunhuang.
Maijishan Grottoes in Tianshui.
Yungang Grottoes in Datong.
And Longmen Grottoes in Luoyang.
Buddhism gained popularity in China during fourth to sixth century CE.
The imperial and aristocratic families sought peace and prosperity
for the nation through religion.
Seeking to get married and spread the Buddhist message,
these individuals initiated a tide grotto carving throughout China.
To learn more about this [? bin ?] site, let's go back to the Northam Wei
Dynasty, around 470 CE.
Founded by ethnic minorities in ancient China
in order to consolidate the central governance, the Emperor, named Xiaowen,
decided to implement reforms to adapt themselves
into the dominant ethnic group, named Han.
These reforms included building the Buddhist temples and sculptures,
and moving the capital from the northern city of Pingcheng
to the central city of Luoyang, aiming at strengthening
people's cultural affiliation to their governance.
Two Buddhist architectures, that we mentioned
just now, Yungang Grottoes and Longmen Grottoes, were build in this period.
Countless craftsmen began carving out these grottoes.
Yungang Grottoes were built before the Longmen Grottoes.
As you can see from these pictures, the Yungang sculptures
look like more of the Indian style.
As you might know, that ancient India was the origin of Buddhism.
Yet the Central Binyang Cave, as an early example of the Longmen Grottoes,
gradually took on the characteristics of everyday Chinese society.
From the kind and gentle expressions of the Buddhas,
to the clothing and jewelry of attendant figures.
The style at Longmen became a true reflection
of Chinese culture of that period.
We call this process synthesization.
When it came to the Tang Dynasty, about 200 years later,
Buddhism became even more popular in China.
The rulers at that time kept building more Buddhist
of sculptures in Longmen Grottoes.
As you can see from the picture, the monumental sculpture
of the great Vairocana Cave stands as a remarkable testament
to the only female emperor in all of Chinese history.
Wu Zetian, as a devout Buddhist, had a great Vairocana Buddha
carved in the middle of the Western mountains at Longmen.
Standing over 55 feet tall, the Great Vairocana Buddha
embodies the full-figured beauty of the famous high Tang style.
With the kind and merciful bearing, the great Vairocana Buddha--
as well as many other Buddhas sculptures--
have been sitting in these grottoes, watching dynasties rise and fall.
But this tranquility and peacefulness of the grottoes and the ancient country
was broken down by the first Opium War in the year of 1840,
with British colonialists breaking into the door of Qing Dynasty,
and invading into the land.
This was just the start of China's trying modern history.
Just 20 years later, the outbreak of the Second Opium War
had made the situation even worse.
A united army from Britain and France intruded into the old summer palace,
and snatched uncountable treasures, leaving with ashes of the inferno.
French poet Victor Hugo marked in a letter
to protest the disappearance of the wonder.
Quote, "We Europeans are the civilized ones.
And for us, the Chinese are the barbarians.
This is what civilization has done to barbarism."
From then on, especially during the beginning of the 20th century,
numerous colonialists and collectors came to China
and colluded with local bandits and antiques dealers
to smuggle Chinese art pieces to overseas collections.
When they couldn't move the whole sculpture, they cut off the heads.
When they couldn't remove reliefs, on the wall
they scratched them off, and broke them into pieces.
During the warfare, due to the lack of guard and protection,
these caves often became the target of political and religious retribution.
These masterpieces of art--
alongside with common people--
became victims of war.
The situation didn't end until the Second World War was over,
during when numerous art pieces, including
Buddhist sculptures in Longmen, were lost, sold, or destroyed.
So was civilization of humankind of that period.
Come back to the 21st century.
These missing relics are scattered all over the world.
Most of the missing Buddha heads and sculptures are part of museums
or private collections in Japan and the United States.
For example, two of the four missing Boddhisattva
heads in the Central Binyang Cave are being
exhibited in Tokyo National Museum, and Osaka City Museum of Fine Arts.
The removal of the emperor and empress processions from the same cave
is perhaps the greatest loss to the sculptural programs at Longmen.
A unique achievement in the history of Chinese art, the
remain the oldest surviving images of a Chinese emperor and empress,
that inspired an entire artistic tradition.
In the early 20th century, they were torn down, broken to pieces,
and shipped to the Metropolitan Museum of Art in New York City,
and the Nelson-Atkins Museum of Art in Kansas City.
The Chinese government is trying to figure out
ways to take these missing parts home, and show
the world how these sculptures originally looked like.
The delicate negotiation process is so much harder than imagination.
Remarkably, in 2001, the Canadian government graciously
agreed to return a looted sculpture of the Buddhist disciple, Kasyapa,
that was illegally sold at auction in 1970 to the National Gallery of Canada.
This is a token of respect between two nations.
However, for many other missing relics, the nature of ownership
still remains challenging, and diplomatic negotiation
might take decades, with no guaranteed results.
Fortunately, the development of technology
gives us an opportunity to resolve many of these diplomatic and sensitive
issues.
In particular, 3D technology has to be applied thoroughly at Longmen.
Its applications include 3D scanning, model processing, 3D database,
archaeological drawing, digital restoration, and virtual reality,
augmented reality display.
Before we dive into these applications, let's consider
that you just took a picture of the great Buddha using your smartphone.
You get a 2D image, which means you know the XY-coordinates of the Buddha,
and the scene.
But what if we also want the depth information?
That is, the Z-coordinates.
It appears that recovering 3D information
is very hard for a single camera.
But luckily, we as humans have two eyes.
What does that mean?
Here comes the concept of stereo vision.
If there are two cameras capturing the same scene,
a 3D point will be projected on two image planes.
Given the camera parameters, and the pixel location of the projected point,
and its corresponding point on another image plane,
we can triangulate the 3D coordinates on the original object.
Triangulation of 3D coordinates just an application of trigonometry,
just like how we estimate the height of a building using
the length of its shadow in a physics class.
More advanced 3D techniques, such as structure
from motion and photogrammetry, are also based on the idea of stereo vision.
However, finding correspondences is a major problem,
and involves many aspects in image processing.
It is often hard for us to figure out which pixel in picture one
corresponds to a given pixel in picture two.
Many image feature extraction algorithms are useful,
but their accuracy and efficiency are not always the best.
To make it simpler, we can substitute one of the two cameras
with a projector.
Laser scanners project lasers to the object to be scanned.
Using the same idea of triangulation, the camera
captures an object with lasers on the surface
so that it becomes easier to find the correspondences.
Laser scanners work conveniently if there
is a 3D object that is close to you.
Similarly, scanners based on structured light
also use a projector and a camera.
But it projects a series of black and white stripes instead of lasers.
These time variant patterns make it possible to find correspondences
and triangulate 3D coordinates more accurately.
In addition to laser scanners and structured light scanners,
time-of-flight scanners are also very useful.
It uses the time difference between when the signal was emitted and received
to calculate the distance from the object.
Therefore, if precision isn't your first priority in scanning,
time-of-flight scanners work better for large things
to which it is hard to project lights.
These scanners capture a 3D point cloud as output.
To make a 3D model, we need to do some processing on the point cloud,
such as aligning scans, removing noisy 3D points,
polygonize the point cloud to get a mash model where 3D surfaces are represented
as triangles.
Also, filling the holes and smoothing.
After we gather a geometric model, we need to capture some pictures,
parametrize the model in UV coordinates, and map high resolution photos onto it
to get a textured model.
Most of these are core materials in a computer graphics course,
and you may learn more as your study continues.
You can take a look at this video to get a better
sense of how people work with scanners.
The projector projects a series of structured lights
as the camera captures images.
In this way, we can obtain a point cloud that carries
both color and geometric information.
By doing the same thing from another view, we can get another partial scan.
Using some registration algorithms, we can align all of the partial scans
to produce a complete 3D model.
Here are some other scanned models using similar technologies.
The 3D digitization of Longmen started from 2005,
when Longmen began to cooperate with researchers at Peking University.
The multi-resolution 3D scanning framework
that involves all these three types of scanners
has been applied to the Longmen Grottoes.
As you can see, that the scale of these Buddhist sculptures, there is a lot.
From the one that is almost 20 meters high,
to many niches that are even smaller than our hands.
We used time-of-flight scanners to scan large-scale things, including
the Fengxian Temple, where the great Vairocana Buddha sits.
Structured light scanners were used to scan sculptures with higher precision.
And more convenient scanning could be achieved
using hand-held lasers scanners.
3D scans using different methods can be aligned automatically using algorithms.
Once we understand the 3D digitization pipeline,
we can use different 3D techniques to scan caves, niches, sculptures,
and inscriptions at Longmen Grottoes.
As a matter of fact, researchers have finished 90% of the 3D digitization
at Longmen from 2005, when it first started.
Given the large number of relics onsite, it
remains very necessary to build a database to effectively organize the 3D
textured models after we capture them.
Researchers have built a system to manage, browse, and retrieve
digital data of Longmen, which interactively incorporates
high-resolution photos, 3D models, and text annotations.
Using a tree structured file system, it's
possible to effectively manage large-scale scenes, as well
as details of a particular sculpture.
The system also enables researchers to perform other operations,
such as distance measurement.
For another tool that uses the same idea,
but has many other functionalities, please
refer to the open source Cherub software,
developed by Yale Graphics Group.
3D models also provide valuable information for archaeological drawing.
It is an important task for archaeologics
to depict shapes and structures where the heritage remains accurately.
Archeological drawing is a traditional way
to record geometric shapes of the heritage
remains using concise lines and curves.
As one of the central components in a typical archaeological report,
line drawings are time-consuming, and have
low accuracy using traditional methods.
Instead of setting baselines, putting grades, and drawing lines manually,
it is possible to automatically extract lines--
such as silhouettes, ridges, and valleys--
from scanned 3D models.
Then, the 3D model and lines can be rendered
to obtain the corresponding parallel projection map at a specific viewpoint.
With parallel projection maps, archaeologists
can finish the line drawing with a smaller amount of post-processing work.
Digital restoration it's another application of significant values.
Since there are many relics missing, or in overseas collections,
physical restoration is extremely difficult.
However, in the digital world, it is possible to put together
2D pictures of reliefs, and align them with surroundings,
if that is sufficient in representing the artwork.
More interestingly, we can also restore the 3D information of relics
based on models and old photos.
For example, if researchers have the permission
to scan a missing Buddha head, the scanned model of the head
can be aligned with the model of its remaining body.
The gap between two parts can be fused using
some algorithms of geometric analysis.
Here is another example from a niche.
The Buddha lost his head, and we can digitally restore it in a similar way.
For this model without its nose, researchers
can simply build a 3D model according to the expected shape of nose.
For other missing Buddha heads that don't have any available 3D data,
we can use some old photos from when the head was not yet removed.
It is easier to build a 3D model if we have photos from multiple views.
But even one picture can convey the idea of its shape.
Given the standard head model as prior, we can warp it a little bit
according to features detected in the old photo,
so that we can produce an approximate 3D model of missing heads.
For more details, please refer to our paper published in 2016.
Last but not least, the development of virtual reality,
and augmented reality display, brings a new perspective
of visitor accessibility to the ancient caves.
Nowadays, for better protection of cultural relics,
tourists are limited to enter most of the caves.
It is a great pity that they cannot appreciate these art treasures
in detail.
Luckily, based on all of the 3D models that we have captured,
it is possible to view the virtual 3D scene using specialized rendering
software.
Smartphones and other VR devices are capable of recovering users' motion,
and update the virtual viewpoint in accordance with actual motion.
Three degrees of freedom, in terms of rotation,
can be recovered using the inertial measurement unit in smartphones.
We can also use a motion capture system to better localize the user, so
that the user can have a translation-updated view when
walking in a room.
Moreover, we can implement an augmented reality system
using digitally restored models, and adding more interactive enhancing
effects.
The left picture shows possible appearance
of wall painting in the Central Binyang Cave, before weathering.
The middle one shows the digitally restored empress procession
on the wall in the same cave.
The right one shows the restored model of a missing Boddhisattva head,
using a framework based on old photos.
The VR/AR display of cultural heritage can effectively
solve the contradiction between the protection of cultural relics
and tourism promotion.
Another VR display system of WanFo Cave was produced this year.
As a first virtual reality project of Longmen grottoes,
WanFo Cave VR cannot only provide cave interior roaming,
but also have cave lighting, and teleportation of user position.
The audience can enjoy every detail in the cave closely,
and the statues can be seen from a viewpoint which
is difficult to reach in reality.
We talked about many applications of 3D digitization technology
at Longmen grottoes.
In fact, as an active research field, digital cultural heritage
attracts thousands of researchers around the world.
For example, a team from Stanford and other institutions
initiated the digital Michelangelo project in 1998,
and produced a detailed 3D model of Michelangelo's David
using laser scanners.
Our CS50 instructor, Professor Benedict Brown,
also worked on improving the project during his PhD study.
In 2003, researchers from the University of Tokyo
scanned the Bayon Temple in Cambodia, using different kinds of scanners,
including ones on a floating balloon.
In this way, after handling the drifting,
they managed to have a better scan of the top
before drones were commercialized.
In 2009, another team from the University of Washington
finished a project called Build Rome in a Day, where
the researchers constructed a 3D model of Rome
losing millions of tourists' photos on Flickr, instead of any 3D scanners.
Fascinated as we are by this wonderful achievements,
other heritage sites may still face a tragic fate.
Even today.
In 2015, ISIS demolished the Temple of the Baalshamin
during the Syrian Civil War.
As a UNESCO World Heritage site, the Temple of Baalshamin
was built approximately 2,000 years ago, representing
a fusion of ancient Syrian and Roman architectural styles.
After the temple's destruction, the Institute for Digital Archeology
announced plans to establish a digital record of other threatened sites
and artifacts.
But it was such a great loss to all human civilizations,
that people can never make up to.
This CS50 seminar might be very different from other ones.
But I just wanted to show you how beautifully technology can
interact with humanities.
When I was in high school, my teacher always asked me what kind of person
I wanted to be, before the science class.
Technology might give more power to evil if the practitioners don't use it
in a proper way.
That's why it is important for all of us to be aware that technology
might be a double-edge sword.
We'd better learn to use it critically, because we're
the ones to create the future.
I'm so proud to be part of the digital Longmen project when I was in Peking
University, under the guidance of Professor [? Jiao ?] [? Hongpin, ?]
Professor [? Ikiyoshi ?] [? Katsu, ?] Dr. Li Renju,
[? Chen ?] [? Jiaohan, ?] [? Fletcher ?] [? Coleman, ?] and other scholars from
the Longman Research Academy.
It is just a perfect example of showing how the development of computer science
can temporarily set aside the historical controversy of ownership,
empower researchers to better understand and safeguard
the values of cultural heritage, and give access to everyone so that we
can appreciate these artistic masterpieces in a way
that people could never have imagined.
From 3D scanning and geometric processing,
to rendering and VR/AR display, technologies and computer sciende--
or more specifically, computer graphics and computer vision--
become even more meaningful and energetic for the reunion
of human civilizations.
Students who are taking CS50 come from different backgrounds.
And those at Yale and Harvard also have the privilege of access
to a liberal arts education.
It is our responsibility to work hard, and creatively, to overcome
big challenges in today's society.
Therefore, I encourage you to keep the solicitude for humanity
as you climb towards the top of science and technology.
I would also like to encourage you to apply
what you have learned in CS50 to your own areas and interests.
I hope this seminar can bring you a new way of thinking, analyzing,
and solving problems.
That it inspires you to take advantage of the power of computer science
and technology to make the world a better place for all generations
to come.
Thank you.
If you are interested in more details, please feel free to send me an email
via Z-E-Y-U .wang@yale.edu to discuss more.
For Yalies, you can also contact members of Yale Graphics Group.
Next semester, our CS50 instructor, Professor Benedict Brown,
will be offering another course called Applications in the Digital Humanities.
By now you all have the prerequisites for the course.
You can also take computer graphics, offered by Professor Holly Rushmeier,
and consider Computing and the Arts major,
directed by Professor Julie Dorsey.
Here is more information about the major.
Computing and the Arts is an inter-departmental major,
designed for students interested in integrating work in computing,
and one of the arts disciplines.
Computing in the Arts provides students with core computer science skills
that allow them to explore interesting and substantive problems
in architecture, art, art history, music, or theater.