ZACH ZEYU WANG: Hi my name is Wang Zeyu. You can call me Zach, which is supposed to be the pronunciation of my name in middle Chinese 1,000 years ago. I'm a second year PhD student in the computer graphics group at Yale University. In this says CS50 seminar, instead of a hands-on tutorial for some programming tools, I'd like to present the idea of combining computer science with humanities. The title is, 3D World, Augmented Reality, and Cultural Heritage. First, imagine that you're visiting the Louvre Museum, and see the sculpture of Venus. As appreciating the great work of ancient Greek, you must be wondering that, what she is doing with her arms? Although it is the loss of her two arms that makes the sculpture famous, and leaves people with infinite imagination. What about those sculptures who lost heads? Or whose nose was cut off because of natural or human damage? Do you think that's the beauty of defect? Then what can we do to help every visitor again access to the original beauty of these great works? Here comes our Longmen's story. [VIDEO PLAYBACK] [MUSIC PLAYING] - November 11th, 1910. It was a crisp and clear dawn. A small thread of sunlight gradually wound its way into the temple of the hidden brook. Still submerged in the midst of his dreams, the American railroad magnate Charles Lang Freer, was suddenly illuminated by this strand of golden light. Fending the radiance from his eyes, freer could distinguish a mysterious outline penetrating the brightness in the gaps between his outstretched fingers. Quickly cycling 1,000 images in his mind, Freer thought to himself, ah yes, that's the one, witnessing a familiar scene. Freer slowly pulled himself up, and headed towards the window. Opening it up, the solemn, towering cliffs of the eastern mountains of Longmen flooded into the room from the opposite shore of the Yi River. At that moment, Freer thought once more of the fortnight spent at the Longmen grottoes, and his sorrow at his imminent departure filled his mind. Taking up a pen, he marked his diary. Though I spent an entire lifetime, I would still be at a loss to understand the full breadth of the Longmen sculptures. This place contains works from several centuries. The greatest epic of Chinese art. How could one person in a single life fully appreciate everything that this place has to offer? [PLAYBACK ENDS] ZACH ZEYU WANG: The Longmen Grottoes are located in the central Chinese city of Luoyang, the earliest and longest-surviving capital of ancient China. The grottoes are situated on the southern outskirts of Luoyang, carved into the Eastern and Western mountains flanking the Yi River. Along this nearly half mile stretch of the river, the grottoes sprout like glittering stars across emerald mountains, and azure waters. They are over 2,400 [? Buddha ?] [? images, ?] more than 10,000 sculptures, and 2,800 carved inscriptions. In 1961, Longmen was recognized by the Chinese government as one of the most important sites for cultural preservation. Nearly 40 years later, the Longmen Grottoes celebrated the new Millennium by becoming a UNESCO World Heritage Site in November of the year 2000. The UNESCO World Heritage committee recognizes the sculptures of the Longmen Grottoes as the apex of Chinese stone carving, from the end of the Northern Wei Dynasty through the Tang dynasty. 2000 years ago, as the Buddhism traveled far and wide across Asia, along the dusty paths of the silk roads, this tradition of caves, temples, architecture, and sculpture also made their way into the central plains of China. Along the silk road lie four great Buddhist sites in the UNESCO World Heritage list. More Caves in Dunhuang. Maijishan Grottoes in Tianshui. Yungang Grottoes in Datong. And Longmen Grottoes in Luoyang. Buddhism gained popularity in China during fourth to sixth century CE. The imperial and aristocratic families sought peace and prosperity for the nation through religion. Seeking to get married and spread the Buddhist message, these individuals initiated a tide grotto carving throughout China. To learn more about this [? bin ?] site, let's go back to the Northam Wei Dynasty, around 470 CE. Founded by ethnic minorities in ancient China in order to consolidate the central governance, the Emperor, named Xiaowen, decided to implement reforms to adapt themselves into the dominant ethnic group, named Han. These reforms included building the Buddhist temples and sculptures, and moving the capital from the northern city of Pingcheng to the central city of Luoyang, aiming at strengthening people's cultural affiliation to their governance. Two Buddhist architectures, that we mentioned just now, Yungang Grottoes and Longmen Grottoes, were build in this period. Countless craftsmen began carving out these grottoes. Yungang Grottoes were built before the Longmen Grottoes. As you can see from these pictures, the Yungang sculptures look like more of the Indian style. As you might know, that ancient India was the origin of Buddhism. Yet the Central Binyang Cave, as an early example of the Longmen Grottoes, gradually took on the characteristics of everyday Chinese society. From the kind and gentle expressions of the Buddhas, to the clothing and jewelry of attendant figures. The style at Longmen became a true reflection of Chinese culture of that period. We call this process synthesization. When it came to the Tang Dynasty, about 200 years later, Buddhism became even more popular in China. The rulers at that time kept building more Buddhist of sculptures in Longmen Grottoes. As you can see from the picture, the monumental sculpture of the great Vairocana Cave stands as a remarkable testament to the only female emperor in all of Chinese history. Wu Zetian, as a devout Buddhist, had a great Vairocana Buddha carved in the middle of the Western mountains at Longmen. Standing over 55 feet tall, the Great Vairocana Buddha embodies the full-figured beauty of the famous high Tang style. With the kind and merciful bearing, the great Vairocana Buddha-- as well as many other Buddhas sculptures-- have been sitting in these grottoes, watching dynasties rise and fall. But this tranquility and peacefulness of the grottoes and the ancient country was broken down by the first Opium War in the year of 1840, with British colonialists breaking into the door of Qing Dynasty, and invading into the land. This was just the start of China's trying modern history. Just 20 years later, the outbreak of the Second Opium War had made the situation even worse. A united army from Britain and France intruded into the old summer palace, and snatched uncountable treasures, leaving with ashes of the inferno. French poet Victor Hugo marked in a letter to protest the disappearance of the wonder. Quote, "We Europeans are the civilized ones. And for us, the Chinese are the barbarians. This is what civilization has done to barbarism." From then on, especially during the beginning of the 20th century, numerous colonialists and collectors came to China and colluded with local bandits and antiques dealers to smuggle Chinese art pieces to overseas collections. When they couldn't move the whole sculpture, they cut off the heads. When they couldn't remove reliefs, on the wall they scratched them off, and broke them into pieces. During the warfare, due to the lack of guard and protection, these caves often became the target of political and religious retribution. These masterpieces of art-- alongside with common people-- became victims of war. The situation didn't end until the Second World War was over, during when numerous art pieces, including Buddhist sculptures in Longmen, were lost, sold, or destroyed. So was civilization of humankind of that period. Come back to the 21st century. These missing relics are scattered all over the world. Most of the missing Buddha heads and sculptures are part of museums or private collections in Japan and the United States. For example, two of the four missing Boddhisattva heads in the Central Binyang Cave are being exhibited in Tokyo National Museum, and Osaka City Museum of Fine Arts. The removal of the emperor and empress processions from the same cave is perhaps the greatest loss to the sculptural programs at Longmen. A unique achievement in the history of Chinese art, the remain the oldest surviving images of a Chinese emperor and empress, that inspired an entire artistic tradition. In the early 20th century, they were torn down, broken to pieces, and shipped to the Metropolitan Museum of Art in New York City, and the Nelson-Atkins Museum of Art in Kansas City. The Chinese government is trying to figure out ways to take these missing parts home, and show the world how these sculptures originally looked like. The delicate negotiation process is so much harder than imagination. Remarkably, in 2001, the Canadian government graciously agreed to return a looted sculpture of the Buddhist disciple, Kasyapa, that was illegally sold at auction in 1970 to the National Gallery of Canada. This is a token of respect between two nations. However, for many other missing relics, the nature of ownership still remains challenging, and diplomatic negotiation might take decades, with no guaranteed results. Fortunately, the development of technology gives us an opportunity to resolve many of these diplomatic and sensitive issues. In particular, 3D technology has to be applied thoroughly at Longmen. Its applications include 3D scanning, model processing, 3D database, archaeological drawing, digital restoration, and virtual reality, augmented reality display. Before we dive into these applications, let's consider that you just took a picture of the great Buddha using your smartphone. You get a 2D image, which means you know the XY-coordinates of the Buddha, and the scene. But what if we also want the depth information? That is, the Z-coordinates. It appears that recovering 3D information is very hard for a single camera. But luckily, we as humans have two eyes. What does that mean? Here comes the concept of stereo vision. If there are two cameras capturing the same scene, a 3D point will be projected on two image planes. Given the camera parameters, and the pixel location of the projected point, and its corresponding point on another image plane, we can triangulate the 3D coordinates on the original object. Triangulation of 3D coordinates just an application of trigonometry, just like how we estimate the height of a building using the length of its shadow in a physics class. More advanced 3D techniques, such as structure from motion and photogrammetry, are also based on the idea of stereo vision. However, finding correspondences is a major problem, and involves many aspects in image processing. It is often hard for us to figure out which pixel in picture one corresponds to a given pixel in picture two. Many image feature extraction algorithms are useful, but their accuracy and efficiency are not always the best. To make it simpler, we can substitute one of the two cameras with a projector. Laser scanners project lasers to the object to be scanned. Using the same idea of triangulation, the camera captures an object with lasers on the surface so that it becomes easier to find the correspondences. Laser scanners work conveniently if there is a 3D object that is close to you. Similarly, scanners based on structured light also use a projector and a camera. But it projects a series of black and white stripes instead of lasers. These time variant patterns make it possible to find correspondences and triangulate 3D coordinates more accurately. In addition to laser scanners and structured light scanners, time-of-flight scanners are also very useful. It uses the time difference between when the signal was emitted and received to calculate the distance from the object. Therefore, if precision isn't your first priority in scanning, time-of-flight scanners work better for large things to which it is hard to project lights. These scanners capture a 3D point cloud as output. To make a 3D model, we need to do some processing on the point cloud, such as aligning scans, removing noisy 3D points, polygonize the point cloud to get a mash model where 3D surfaces are represented as triangles. Also, filling the holes and smoothing. After we gather a geometric model, we need to capture some pictures, parametrize the model in UV coordinates, and map high resolution photos onto it to get a textured model. Most of these are core materials in a computer graphics course, and you may learn more as your study continues. You can take a look at this video to get a better sense of how people work with scanners. The projector projects a series of structured lights as the camera captures images. In this way, we can obtain a point cloud that carries both color and geometric information. By doing the same thing from another view, we can get another partial scan. Using some registration algorithms, we can align all of the partial scans to produce a complete 3D model. Here are some other scanned models using similar technologies. The 3D digitization of Longmen started from 2005, when Longmen began to cooperate with researchers at Peking University. The multi-resolution 3D scanning framework that involves all these three types of scanners has been applied to the Longmen Grottoes. As you can see, that the scale of these Buddhist sculptures, there is a lot. From the one that is almost 20 meters high, to many niches that are even smaller than our hands. We used time-of-flight scanners to scan large-scale things, including the Fengxian Temple, where the great Vairocana Buddha sits. Structured light scanners were used to scan sculptures with higher precision. And more convenient scanning could be achieved using hand-held lasers scanners. 3D scans using different methods can be aligned automatically using algorithms. Once we understand the 3D digitization pipeline, we can use different 3D techniques to scan caves, niches, sculptures, and inscriptions at Longmen Grottoes. As a matter of fact, researchers have finished 90% of the 3D digitization at Longmen from 2005, when it first started. Given the large number of relics onsite, it remains very necessary to build a database to effectively organize the 3D textured models after we capture them. Researchers have built a system to manage, browse, and retrieve digital data of Longmen, which interactively incorporates high-resolution photos, 3D models, and text annotations. Using a tree structured file system, it's possible to effectively manage large-scale scenes, as well as details of a particular sculpture. The system also enables researchers to perform other operations, such as distance measurement. For another tool that uses the same idea, but has many other functionalities, please refer to the open source Cherub software, developed by Yale Graphics Group. 3D models also provide valuable information for archaeological drawing. It is an important task for archaeologics to depict shapes and structures where the heritage remains accurately. Archeological drawing is a traditional way to record geometric shapes of the heritage remains using concise lines and curves. As one of the central components in a typical archaeological report, line drawings are time-consuming, and have low accuracy using traditional methods. Instead of setting baselines, putting grades, and drawing lines manually, it is possible to automatically extract lines-- such as silhouettes, ridges, and valleys-- from scanned 3D models. Then, the 3D model and lines can be rendered to obtain the corresponding parallel projection map at a specific viewpoint. With parallel projection maps, archaeologists can finish the line drawing with a smaller amount of post-processing work. Digital restoration it's another application of significant values. Since there are many relics missing, or in overseas collections, physical restoration is extremely difficult. However, in the digital world, it is possible to put together 2D pictures of reliefs, and align them with surroundings, if that is sufficient in representing the artwork. More interestingly, we can also restore the 3D information of relics based on models and old photos. For example, if researchers have the permission to scan a missing Buddha head, the scanned model of the head can be aligned with the model of its remaining body. The gap between two parts can be fused using some algorithms of geometric analysis. Here is another example from a niche. The Buddha lost his head, and we can digitally restore it in a similar way. For this model without its nose, researchers can simply build a 3D model according to the expected shape of nose. For other missing Buddha heads that don't have any available 3D data, we can use some old photos from when the head was not yet removed. It is easier to build a 3D model if we have photos from multiple views. But even one picture can convey the idea of its shape. Given the standard head model as prior, we can warp it a little bit according to features detected in the old photo, so that we can produce an approximate 3D model of missing heads. For more details, please refer to our paper published in 2016. Last but not least, the development of virtual reality, and augmented reality display, brings a new perspective of visitor accessibility to the ancient caves. Nowadays, for better protection of cultural relics, tourists are limited to enter most of the caves. It is a great pity that they cannot appreciate these art treasures in detail. Luckily, based on all of the 3D models that we have captured, it is possible to view the virtual 3D scene using specialized rendering software. Smartphones and other VR devices are capable of recovering users' motion, and update the virtual viewpoint in accordance with actual motion. Three degrees of freedom, in terms of rotation, can be recovered using the inertial measurement unit in smartphones. We can also use a motion capture system to better localize the user, so that the user can have a translation-updated view when walking in a room. Moreover, we can implement an augmented reality system using digitally restored models, and adding more interactive enhancing effects. The left picture shows possible appearance of wall painting in the Central Binyang Cave, before weathering. The middle one shows the digitally restored empress procession on the wall in the same cave. The right one shows the restored model of a missing Boddhisattva head, using a framework based on old photos. The VR/AR display of cultural heritage can effectively solve the contradiction between the protection of cultural relics and tourism promotion. Another VR display system of WanFo Cave was produced this year. As a first virtual reality project of Longmen grottoes, WanFo Cave VR cannot only provide cave interior roaming, but also have cave lighting, and teleportation of user position. The audience can enjoy every detail in the cave closely, and the statues can be seen from a viewpoint which is difficult to reach in reality. We talked about many applications of 3D digitization technology at Longmen grottoes. In fact, as an active research field, digital cultural heritage attracts thousands of researchers around the world. For example, a team from Stanford and other institutions initiated the digital Michelangelo project in 1998, and produced a detailed 3D model of Michelangelo's David using laser scanners. Our CS50 instructor, Professor Benedict Brown, also worked on improving the project during his PhD study. In 2003, researchers from the University of Tokyo scanned the Bayon Temple in Cambodia, using different kinds of scanners, including ones on a floating balloon. In this way, after handling the drifting, they managed to have a better scan of the top before drones were commercialized. In 2009, another team from the University of Washington finished a project called Build Rome in a Day, where the researchers constructed a 3D model of Rome losing millions of tourists' photos on Flickr, instead of any 3D scanners. Fascinated as we are by this wonderful achievements, other heritage sites may still face a tragic fate. Even today. In 2015, ISIS demolished the Temple of the Baalshamin during the Syrian Civil War. As a UNESCO World Heritage site, the Temple of Baalshamin was built approximately 2,000 years ago, representing a fusion of ancient Syrian and Roman architectural styles. After the temple's destruction, the Institute for Digital Archeology announced plans to establish a digital record of other threatened sites and artifacts. But it was such a great loss to all human civilizations, that people can never make up to. This CS50 seminar might be very different from other ones. But I just wanted to show you how beautifully technology can interact with humanities. When I was in high school, my teacher always asked me what kind of person I wanted to be, before the science class. Technology might give more power to evil if the practitioners don't use it in a proper way. That's why it is important for all of us to be aware that technology might be a double-edge sword. We'd better learn to use it critically, because we're the ones to create the future. I'm so proud to be part of the digital Longmen project when I was in Peking University, under the guidance of Professor [? Jiao ?] [? Hongpin, ?] Professor [? Ikiyoshi ?] [? Katsu, ?] Dr. Li Renju, [? Chen ?] [? Jiaohan, ?] [? Fletcher ?] [? Coleman, ?] and other scholars from the Longman Research Academy. It is just a perfect example of showing how the development of computer science can temporarily set aside the historical controversy of ownership, empower researchers to better understand and safeguard the values of cultural heritage, and give access to everyone so that we can appreciate these artistic masterpieces in a way that people could never have imagined. From 3D scanning and geometric processing, to rendering and VR/AR display, technologies and computer sciende-- or more specifically, computer graphics and computer vision-- become even more meaningful and energetic for the reunion of human civilizations. Students who are taking CS50 come from different backgrounds. And those at Yale and Harvard also have the privilege of access to a liberal arts education. It is our responsibility to work hard, and creatively, to overcome big challenges in today's society. Therefore, I encourage you to keep the solicitude for humanity as you climb towards the top of science and technology. I would also like to encourage you to apply what you have learned in CS50 to your own areas and interests. I hope this seminar can bring you a new way of thinking, analyzing, and solving problems. That it inspires you to take advantage of the power of computer science and technology to make the world a better place for all generations to come. Thank you. If you are interested in more details, please feel free to send me an email via Z-E-Y-U .wang@yale.edu to discuss more. For Yalies, you can also contact members of Yale Graphics Group. Next semester, our CS50 instructor, Professor Benedict Brown, will be offering another course called Applications in the Digital Humanities. By now you all have the prerequisites for the course. You can also take computer graphics, offered by Professor Holly Rushmeier, and consider Computing and the Arts major, directed by Professor Julie Dorsey. Here is more information about the major. Computing and the Arts is an inter-departmental major, designed for students interested in integrating work in computing, and one of the arts disciplines. Computing in the Arts provides students with core computer science skills that allow them to explore interesting and substantive problems in architecture, art, art history, music, or theater.