1 00:00:00,000 --> 00:00:01,100 2 00:00:01,100 --> 00:00:04,040 ZACH ZEYU WANG: Hi my name is Wang Zeyu. 3 00:00:04,040 --> 00:00:06,380 You can call me Zach, which is supposed to be 4 00:00:06,380 --> 00:00:12,160 the pronunciation of my name in middle Chinese 1,000 years ago. 5 00:00:12,160 --> 00:00:15,160 I'm a second year PhD student in the computer graphics 6 00:00:15,160 --> 00:00:18,270 group at Yale University. 7 00:00:18,270 --> 00:00:22,570 In this says CS50 seminar, instead of a hands-on tutorial for some programming 8 00:00:22,570 --> 00:00:26,440 tools, I'd like to present the idea of combining 9 00:00:26,440 --> 00:00:30,090 computer science with humanities. 10 00:00:30,090 --> 00:00:35,444 The title is, 3D World, Augmented Reality, and Cultural Heritage. 11 00:00:35,444 --> 00:00:38,170 12 00:00:38,170 --> 00:00:41,760 First, imagine that you're visiting the Louvre Museum, 13 00:00:41,760 --> 00:00:44,640 and see the sculpture of Venus. 14 00:00:44,640 --> 00:00:47,920 As appreciating the great work of ancient Greek, 15 00:00:47,920 --> 00:00:53,060 you must be wondering that, what she is doing with her arms? 16 00:00:53,060 --> 00:00:57,130 Although it is the loss of her two arms that makes the sculpture famous, 17 00:00:57,130 --> 00:01:00,650 and leaves people with infinite imagination. 18 00:01:00,650 --> 00:01:04,970 What about those sculptures who lost heads? 19 00:01:04,970 --> 00:01:08,960 Or whose nose was cut off because of natural or human damage? 20 00:01:08,960 --> 00:01:11,610 21 00:01:11,610 --> 00:01:13,975 Do you think that's the beauty of defect? 22 00:01:13,975 --> 00:01:16,500 23 00:01:16,500 --> 00:01:20,520 Then what can we do to help every visitor again access 24 00:01:20,520 --> 00:01:24,180 to the original beauty of these great works? 25 00:01:24,180 --> 00:01:26,440 Here comes our Longmen's story. 26 00:01:26,440 --> 00:01:27,106 [VIDEO PLAYBACK] 27 00:01:27,106 --> 00:01:59,490 [MUSIC PLAYING] 28 00:01:59,490 --> 00:02:02,640 - November 11th, 1910. 29 00:02:02,640 --> 00:02:05,640 It was a crisp and clear dawn. 30 00:02:05,640 --> 00:02:08,970 A small thread of sunlight gradually wound its way 31 00:02:08,970 --> 00:02:11,580 into the temple of the hidden brook. 32 00:02:11,580 --> 00:02:15,900 Still submerged in the midst of his dreams, the American railroad 33 00:02:15,900 --> 00:02:20,460 magnate Charles Lang Freer, was suddenly illuminated 34 00:02:20,460 --> 00:02:23,040 by this strand of golden light. 35 00:02:23,040 --> 00:02:26,070 Fending the radiance from his eyes, freer 36 00:02:26,070 --> 00:02:30,390 could distinguish a mysterious outline penetrating the brightness 37 00:02:30,390 --> 00:02:33,840 in the gaps between his outstretched fingers. 38 00:02:33,840 --> 00:02:39,792 Quickly cycling 1,000 images in his mind, Freer thought to himself, 39 00:02:39,792 --> 00:02:45,900 ah yes, that's the one, witnessing a familiar scene. 40 00:02:45,900 --> 00:02:50,910 Freer slowly pulled himself up, and headed towards the window. 41 00:02:50,910 --> 00:02:54,240 Opening it up, the solemn, towering cliffs 42 00:02:54,240 --> 00:02:57,900 of the eastern mountains of Longmen flooded into the room 43 00:02:57,900 --> 00:03:01,410 from the opposite shore of the Yi River. 44 00:03:01,410 --> 00:03:06,450 At that moment, Freer thought once more of the fortnight spent at the Longmen 45 00:03:06,450 --> 00:03:12,060 grottoes, and his sorrow at his imminent departure filled his mind. 46 00:03:12,060 --> 00:03:15,690 Taking up a pen, he marked his diary. 47 00:03:15,690 --> 00:03:19,890 Though I spent an entire lifetime, I would still 48 00:03:19,890 --> 00:03:24,750 be at a loss to understand the full breadth of the Longmen sculptures. 49 00:03:24,750 --> 00:03:28,260 This place contains works from several centuries. 50 00:03:28,260 --> 00:03:31,260 The greatest epic of Chinese art. 51 00:03:31,260 --> 00:03:36,720 How could one person in a single life fully appreciate everything 52 00:03:36,720 --> 00:03:39,648 that this place has to offer? 53 00:03:39,648 --> 00:03:42,570 [PLAYBACK ENDS] 54 00:03:42,570 --> 00:03:44,670 ZACH ZEYU WANG: The Longmen Grottoes are located 55 00:03:44,670 --> 00:03:48,870 in the central Chinese city of Luoyang, the earliest 56 00:03:48,870 --> 00:03:53,010 and longest-surviving capital of ancient China. 57 00:03:53,010 --> 00:03:57,690 The grottoes are situated on the southern outskirts of Luoyang, 58 00:03:57,690 --> 00:04:03,990 carved into the Eastern and Western mountains flanking the Yi River. 59 00:04:03,990 --> 00:04:07,230 Along this nearly half mile stretch of the river, 60 00:04:07,230 --> 00:04:09,930 the grottoes sprout like glittering stars 61 00:04:09,930 --> 00:04:14,540 across emerald mountains, and azure waters. 62 00:04:14,540 --> 00:04:19,019 They are over 2,400 [? Buddha ?] [? images, ?] more than 10,000 63 00:04:19,019 --> 00:04:23,280 sculptures, and 2,800 carved inscriptions. 64 00:04:23,280 --> 00:04:26,450 65 00:04:26,450 --> 00:04:31,690 In 1961, Longmen was recognized by the Chinese government 66 00:04:31,690 --> 00:04:37,140 as one of the most important sites for cultural preservation. 67 00:04:37,140 --> 00:04:40,260 Nearly 40 years later, the Longmen Grottoes 68 00:04:40,260 --> 00:04:44,640 celebrated the new Millennium by becoming a UNESCO World Heritage 69 00:04:44,640 --> 00:04:49,380 Site in November of the year 2000. 70 00:04:49,380 --> 00:04:53,020 The UNESCO World Heritage committee recognizes the sculptures 71 00:04:53,020 --> 00:04:57,820 of the Longmen Grottoes as the apex of Chinese stone carving, 72 00:04:57,820 --> 00:05:01,890 from the end of the Northern Wei Dynasty through the Tang dynasty. 73 00:05:01,890 --> 00:05:05,860 74 00:05:05,860 --> 00:05:11,800 2000 years ago, as the Buddhism traveled far and wide across Asia, 75 00:05:11,800 --> 00:05:16,090 along the dusty paths of the silk roads, this tradition 76 00:05:16,090 --> 00:05:19,600 of caves, temples, architecture, and sculpture 77 00:05:19,600 --> 00:05:24,520 also made their way into the central plains of China. 78 00:05:24,520 --> 00:05:28,060 Along the silk road lie four great Buddhist sites 79 00:05:28,060 --> 00:05:31,150 in the UNESCO World Heritage list. 80 00:05:31,150 --> 00:05:34,390 More Caves in Dunhuang. 81 00:05:34,390 --> 00:05:38,000 Maijishan Grottoes in Tianshui. 82 00:05:38,000 --> 00:05:41,200 Yungang Grottoes in Datong. 83 00:05:41,200 --> 00:05:43,650 And Longmen Grottoes in Luoyang. 84 00:05:43,650 --> 00:05:47,300 85 00:05:47,300 --> 00:05:54,100 Buddhism gained popularity in China during fourth to sixth century CE. 86 00:05:54,100 --> 00:05:58,850 The imperial and aristocratic families sought peace and prosperity 87 00:05:58,850 --> 00:06:02,100 for the nation through religion. 88 00:06:02,100 --> 00:06:05,750 Seeking to get married and spread the Buddhist message, 89 00:06:05,750 --> 00:06:12,790 these individuals initiated a tide grotto carving throughout China. 90 00:06:12,790 --> 00:06:16,770 To learn more about this [? bin ?] site, let's go back to the Northam Wei 91 00:06:16,770 --> 00:06:19,830 Dynasty, around 470 CE. 92 00:06:19,830 --> 00:06:22,610 93 00:06:22,610 --> 00:06:25,640 Founded by ethnic minorities in ancient China 94 00:06:25,640 --> 00:06:31,460 in order to consolidate the central governance, the Emperor, named Xiaowen, 95 00:06:31,460 --> 00:06:35,030 decided to implement reforms to adapt themselves 96 00:06:35,030 --> 00:06:38,115 into the dominant ethnic group, named Han. 97 00:06:38,115 --> 00:06:40,800 98 00:06:40,800 --> 00:06:45,540 These reforms included building the Buddhist temples and sculptures, 99 00:06:45,540 --> 00:06:49,590 and moving the capital from the northern city of Pingcheng 100 00:06:49,590 --> 00:06:54,090 to the central city of Luoyang, aiming at strengthening 101 00:06:54,090 --> 00:06:58,720 people's cultural affiliation to their governance. 102 00:06:58,720 --> 00:07:01,480 Two Buddhist architectures, that we mentioned 103 00:07:01,480 --> 00:07:09,150 just now, Yungang Grottoes and Longmen Grottoes, were build in this period. 104 00:07:09,150 --> 00:07:14,290 Countless craftsmen began carving out these grottoes. 105 00:07:14,290 --> 00:07:19,100 Yungang Grottoes were built before the Longmen Grottoes. 106 00:07:19,100 --> 00:07:23,210 As you can see from these pictures, the Yungang sculptures 107 00:07:23,210 --> 00:07:26,400 look like more of the Indian style. 108 00:07:26,400 --> 00:07:31,490 As you might know, that ancient India was the origin of Buddhism. 109 00:07:31,490 --> 00:07:36,830 Yet the Central Binyang Cave, as an early example of the Longmen Grottoes, 110 00:07:36,830 --> 00:07:42,530 gradually took on the characteristics of everyday Chinese society. 111 00:07:42,530 --> 00:07:46,220 From the kind and gentle expressions of the Buddhas, 112 00:07:46,220 --> 00:07:51,000 to the clothing and jewelry of attendant figures. 113 00:07:51,000 --> 00:07:54,250 The style at Longmen became a true reflection 114 00:07:54,250 --> 00:07:57,940 of Chinese culture of that period. 115 00:07:57,940 --> 00:08:01,790 We call this process synthesization. 116 00:08:01,790 --> 00:08:07,560 When it came to the Tang Dynasty, about 200 years later, 117 00:08:07,560 --> 00:08:11,820 Buddhism became even more popular in China. 118 00:08:11,820 --> 00:08:15,000 The rulers at that time kept building more Buddhist 119 00:08:15,000 --> 00:08:19,150 of sculptures in Longmen Grottoes. 120 00:08:19,150 --> 00:08:22,780 As you can see from the picture, the monumental sculpture 121 00:08:22,780 --> 00:08:27,640 of the great Vairocana Cave stands as a remarkable testament 122 00:08:27,640 --> 00:08:33,370 to the only female emperor in all of Chinese history. 123 00:08:33,370 --> 00:08:38,960 Wu Zetian, as a devout Buddhist, had a great Vairocana Buddha 124 00:08:38,960 --> 00:08:42,259 carved in the middle of the Western mountains at Longmen. 125 00:08:42,259 --> 00:08:44,790 126 00:08:44,790 --> 00:08:49,320 Standing over 55 feet tall, the Great Vairocana Buddha 127 00:08:49,320 --> 00:08:53,630 embodies the full-figured beauty of the famous high Tang style. 128 00:08:53,630 --> 00:08:56,250 129 00:08:56,250 --> 00:09:01,080 With the kind and merciful bearing, the great Vairocana Buddha-- 130 00:09:01,080 --> 00:09:03,630 as well as many other Buddhas sculptures-- 131 00:09:03,630 --> 00:09:11,440 have been sitting in these grottoes, watching dynasties rise and fall. 132 00:09:11,440 --> 00:09:17,080 But this tranquility and peacefulness of the grottoes and the ancient country 133 00:09:17,080 --> 00:09:23,380 was broken down by the first Opium War in the year of 1840, 134 00:09:23,380 --> 00:09:27,930 with British colonialists breaking into the door of Qing Dynasty, 135 00:09:27,930 --> 00:09:31,410 and invading into the land. 136 00:09:31,410 --> 00:09:36,920 This was just the start of China's trying modern history. 137 00:09:36,920 --> 00:09:41,660 Just 20 years later, the outbreak of the Second Opium War 138 00:09:41,660 --> 00:09:45,610 had made the situation even worse. 139 00:09:45,610 --> 00:09:51,110 A united army from Britain and France intruded into the old summer palace, 140 00:09:51,110 --> 00:09:58,480 and snatched uncountable treasures, leaving with ashes of the inferno. 141 00:09:58,480 --> 00:10:02,440 French poet Victor Hugo marked in a letter 142 00:10:02,440 --> 00:10:06,730 to protest the disappearance of the wonder. 143 00:10:06,730 --> 00:10:11,490 Quote, "We Europeans are the civilized ones. 144 00:10:11,490 --> 00:10:15,380 And for us, the Chinese are the barbarians. 145 00:10:15,380 --> 00:10:19,140 This is what civilization has done to barbarism." 146 00:10:19,140 --> 00:10:23,640 147 00:10:23,640 --> 00:10:28,920 From then on, especially during the beginning of the 20th century, 148 00:10:28,920 --> 00:10:32,970 numerous colonialists and collectors came to China 149 00:10:32,970 --> 00:10:36,960 and colluded with local bandits and antiques dealers 150 00:10:36,960 --> 00:10:42,320 to smuggle Chinese art pieces to overseas collections. 151 00:10:42,320 --> 00:10:47,540 When they couldn't move the whole sculpture, they cut off the heads. 152 00:10:47,540 --> 00:10:50,270 When they couldn't remove reliefs, on the wall 153 00:10:50,270 --> 00:10:53,150 they scratched them off, and broke them into pieces. 154 00:10:53,150 --> 00:10:55,740 155 00:10:55,740 --> 00:10:59,400 During the warfare, due to the lack of guard and protection, 156 00:10:59,400 --> 00:11:06,000 these caves often became the target of political and religious retribution. 157 00:11:06,000 --> 00:11:08,310 These masterpieces of art-- 158 00:11:08,310 --> 00:11:11,160 alongside with common people-- 159 00:11:11,160 --> 00:11:15,620 became victims of war. 160 00:11:15,620 --> 00:11:20,210 The situation didn't end until the Second World War was over, 161 00:11:20,210 --> 00:11:23,180 during when numerous art pieces, including 162 00:11:23,180 --> 00:11:30,810 Buddhist sculptures in Longmen, were lost, sold, or destroyed. 163 00:11:30,810 --> 00:11:34,290 So was civilization of humankind of that period. 164 00:11:34,290 --> 00:11:37,850 165 00:11:37,850 --> 00:11:40,480 Come back to the 21st century. 166 00:11:40,480 --> 00:11:45,800 These missing relics are scattered all over the world. 167 00:11:45,800 --> 00:11:50,160 Most of the missing Buddha heads and sculptures are part of museums 168 00:11:50,160 --> 00:11:55,680 or private collections in Japan and the United States. 169 00:11:55,680 --> 00:11:58,960 For example, two of the four missing Boddhisattva 170 00:11:58,960 --> 00:12:03,210 heads in the Central Binyang Cave are being 171 00:12:03,210 --> 00:12:08,570 exhibited in Tokyo National Museum, and Osaka City Museum of Fine Arts. 172 00:12:08,570 --> 00:12:11,430 173 00:12:11,430 --> 00:12:16,430 The removal of the emperor and empress processions from the same cave 174 00:12:16,430 --> 00:12:22,880 is perhaps the greatest loss to the sculptural programs at Longmen. 175 00:12:22,880 --> 00:12:26,510 A unique achievement in the history of Chinese art, the 176 00:12:26,510 --> 00:12:32,510 remain the oldest surviving images of a Chinese emperor and empress, 177 00:12:32,510 --> 00:12:35,720 that inspired an entire artistic tradition. 178 00:12:35,720 --> 00:12:38,570 179 00:12:38,570 --> 00:12:44,270 In the early 20th century, they were torn down, broken to pieces, 180 00:12:44,270 --> 00:12:49,430 and shipped to the Metropolitan Museum of Art in New York City, 181 00:12:49,430 --> 00:12:53,420 and the Nelson-Atkins Museum of Art in Kansas City. 182 00:12:53,420 --> 00:12:58,520 183 00:12:58,520 --> 00:13:01,190 The Chinese government is trying to figure out 184 00:13:01,190 --> 00:13:05,090 ways to take these missing parts home, and show 185 00:13:05,090 --> 00:13:09,940 the world how these sculptures originally looked like. 186 00:13:09,940 --> 00:13:16,930 The delicate negotiation process is so much harder than imagination. 187 00:13:16,930 --> 00:13:20,950 Remarkably, in 2001, the Canadian government graciously 188 00:13:20,950 --> 00:13:26,720 agreed to return a looted sculpture of the Buddhist disciple, Kasyapa, 189 00:13:26,720 --> 00:13:34,260 that was illegally sold at auction in 1970 to the National Gallery of Canada. 190 00:13:34,260 --> 00:13:37,130 This is a token of respect between two nations. 191 00:13:37,130 --> 00:13:39,770 192 00:13:39,770 --> 00:13:43,940 However, for many other missing relics, the nature of ownership 193 00:13:43,940 --> 00:13:48,560 still remains challenging, and diplomatic negotiation 194 00:13:48,560 --> 00:13:52,025 might take decades, with no guaranteed results. 195 00:13:52,025 --> 00:13:55,850 196 00:13:55,850 --> 00:13:59,180 Fortunately, the development of technology 197 00:13:59,180 --> 00:14:04,910 gives us an opportunity to resolve many of these diplomatic and sensitive 198 00:14:04,910 --> 00:14:06,860 issues. 199 00:14:06,860 --> 00:14:13,610 In particular, 3D technology has to be applied thoroughly at Longmen. 200 00:14:13,610 --> 00:14:19,830 Its applications include 3D scanning, model processing, 3D database, 201 00:14:19,830 --> 00:14:25,160 archaeological drawing, digital restoration, and virtual reality, 202 00:14:25,160 --> 00:14:26,500 augmented reality display. 203 00:14:26,500 --> 00:14:29,640 204 00:14:29,640 --> 00:14:34,050 Before we dive into these applications, let's consider 205 00:14:34,050 --> 00:14:40,620 that you just took a picture of the great Buddha using your smartphone. 206 00:14:40,620 --> 00:14:46,200 You get a 2D image, which means you know the XY-coordinates of the Buddha, 207 00:14:46,200 --> 00:14:48,730 and the scene. 208 00:14:48,730 --> 00:14:51,700 But what if we also want the depth information? 209 00:14:51,700 --> 00:14:55,580 That is, the Z-coordinates. 210 00:14:55,580 --> 00:14:58,160 It appears that recovering 3D information 211 00:14:58,160 --> 00:15:01,750 is very hard for a single camera. 212 00:15:01,750 --> 00:15:05,970 But luckily, we as humans have two eyes. 213 00:15:05,970 --> 00:15:07,470 What does that mean? 214 00:15:07,470 --> 00:15:10,900 215 00:15:10,900 --> 00:15:15,420 Here comes the concept of stereo vision. 216 00:15:15,420 --> 00:15:18,480 If there are two cameras capturing the same scene, 217 00:15:18,480 --> 00:15:23,780 a 3D point will be projected on two image planes. 218 00:15:23,780 --> 00:15:28,460 Given the camera parameters, and the pixel location of the projected point, 219 00:15:28,460 --> 00:15:32,150 and its corresponding point on another image plane, 220 00:15:32,150 --> 00:15:38,300 we can triangulate the 3D coordinates on the original object. 221 00:15:38,300 --> 00:15:43,520 Triangulation of 3D coordinates just an application of trigonometry, 222 00:15:43,520 --> 00:15:47,030 just like how we estimate the height of a building using 223 00:15:47,030 --> 00:15:52,140 the length of its shadow in a physics class. 224 00:15:52,140 --> 00:15:54,750 More advanced 3D techniques, such as structure 225 00:15:54,750 --> 00:16:00,180 from motion and photogrammetry, are also based on the idea of stereo vision. 226 00:16:00,180 --> 00:16:04,740 227 00:16:04,740 --> 00:16:08,460 However, finding correspondences is a major problem, 228 00:16:08,460 --> 00:16:12,960 and involves many aspects in image processing. 229 00:16:12,960 --> 00:16:17,280 It is often hard for us to figure out which pixel in picture one 230 00:16:17,280 --> 00:16:21,730 corresponds to a given pixel in picture two. 231 00:16:21,730 --> 00:16:25,420 Many image feature extraction algorithms are useful, 232 00:16:25,420 --> 00:16:29,135 but their accuracy and efficiency are not always the best. 233 00:16:29,135 --> 00:16:32,910 234 00:16:32,910 --> 00:16:37,110 To make it simpler, we can substitute one of the two cameras 235 00:16:37,110 --> 00:16:40,050 with a projector. 236 00:16:40,050 --> 00:16:45,210 Laser scanners project lasers to the object to be scanned. 237 00:16:45,210 --> 00:16:48,060 Using the same idea of triangulation, the camera 238 00:16:48,060 --> 00:16:51,480 captures an object with lasers on the surface 239 00:16:51,480 --> 00:16:54,540 so that it becomes easier to find the correspondences. 240 00:16:54,540 --> 00:16:58,880 241 00:16:58,880 --> 00:17:01,750 Laser scanners work conveniently if there 242 00:17:01,750 --> 00:17:05,940 is a 3D object that is close to you. 243 00:17:05,940 --> 00:17:09,390 Similarly, scanners based on structured light 244 00:17:09,390 --> 00:17:12,710 also use a projector and a camera. 245 00:17:12,710 --> 00:17:19,069 But it projects a series of black and white stripes instead of lasers. 246 00:17:19,069 --> 00:17:23,960 These time variant patterns make it possible to find correspondences 247 00:17:23,960 --> 00:17:29,730 and triangulate 3D coordinates more accurately. 248 00:17:29,730 --> 00:17:33,540 In addition to laser scanners and structured light scanners, 249 00:17:33,540 --> 00:17:37,930 time-of-flight scanners are also very useful. 250 00:17:37,930 --> 00:17:42,620 It uses the time difference between when the signal was emitted and received 251 00:17:42,620 --> 00:17:47,150 to calculate the distance from the object. 252 00:17:47,150 --> 00:17:51,220 Therefore, if precision isn't your first priority in scanning, 253 00:17:51,220 --> 00:17:55,090 time-of-flight scanners work better for large things 254 00:17:55,090 --> 00:17:57,475 to which it is hard to project lights. 255 00:17:57,475 --> 00:18:01,870 256 00:18:01,870 --> 00:18:07,410 These scanners capture a 3D point cloud as output. 257 00:18:07,410 --> 00:18:11,880 To make a 3D model, we need to do some processing on the point cloud, 258 00:18:11,880 --> 00:18:16,450 such as aligning scans, removing noisy 3D points, 259 00:18:16,450 --> 00:18:21,750 polygonize the point cloud to get a mash model where 3D surfaces are represented 260 00:18:21,750 --> 00:18:23,640 as triangles. 261 00:18:23,640 --> 00:18:27,710 Also, filling the holes and smoothing. 262 00:18:27,710 --> 00:18:31,780 After we gather a geometric model, we need to capture some pictures, 263 00:18:31,780 --> 00:18:37,680 parametrize the model in UV coordinates, and map high resolution photos onto it 264 00:18:37,680 --> 00:18:40,840 to get a textured model. 265 00:18:40,840 --> 00:18:44,770 Most of these are core materials in a computer graphics course, 266 00:18:44,770 --> 00:18:47,370 and you may learn more as your study continues. 267 00:18:47,370 --> 00:18:53,030 268 00:18:53,030 --> 00:18:55,820 You can take a look at this video to get a better 269 00:18:55,820 --> 00:19:00,820 sense of how people work with scanners. 270 00:19:00,820 --> 00:19:03,970 The projector projects a series of structured lights 271 00:19:03,970 --> 00:19:05,765 as the camera captures images. 272 00:19:05,765 --> 00:19:09,140 273 00:19:09,140 --> 00:19:13,040 In this way, we can obtain a point cloud that carries 274 00:19:13,040 --> 00:19:18,560 both color and geometric information. 275 00:19:18,560 --> 00:19:23,015 By doing the same thing from another view, we can get another partial scan. 276 00:19:23,015 --> 00:19:25,790 277 00:19:25,790 --> 00:19:31,520 Using some registration algorithms, we can align all of the partial scans 278 00:19:31,520 --> 00:19:34,310 to produce a complete 3D model. 279 00:19:34,310 --> 00:20:06,880 280 00:20:06,880 --> 00:20:11,130 Here are some other scanned models using similar technologies. 281 00:20:11,130 --> 00:20:18,230 282 00:20:18,230 --> 00:20:24,310 The 3D digitization of Longmen started from 2005, 283 00:20:24,310 --> 00:20:30,710 when Longmen began to cooperate with researchers at Peking University. 284 00:20:30,710 --> 00:20:33,710 The multi-resolution 3D scanning framework 285 00:20:33,710 --> 00:20:37,130 that involves all these three types of scanners 286 00:20:37,130 --> 00:20:41,650 has been applied to the Longmen Grottoes. 287 00:20:41,650 --> 00:20:48,040 As you can see, that the scale of these Buddhist sculptures, there is a lot. 288 00:20:48,040 --> 00:20:51,050 From the one that is almost 20 meters high, 289 00:20:51,050 --> 00:20:54,430 to many niches that are even smaller than our hands. 290 00:20:54,430 --> 00:20:57,280 291 00:20:57,280 --> 00:21:02,200 We used time-of-flight scanners to scan large-scale things, including 292 00:21:02,200 --> 00:21:08,230 the Fengxian Temple, where the great Vairocana Buddha sits. 293 00:21:08,230 --> 00:21:14,250 Structured light scanners were used to scan sculptures with higher precision. 294 00:21:14,250 --> 00:21:17,670 And more convenient scanning could be achieved 295 00:21:17,670 --> 00:21:19,285 using hand-held lasers scanners. 296 00:21:19,285 --> 00:21:22,930 297 00:21:22,930 --> 00:21:28,520 3D scans using different methods can be aligned automatically using algorithms. 298 00:21:28,520 --> 00:21:33,460 299 00:21:33,460 --> 00:21:37,270 Once we understand the 3D digitization pipeline, 300 00:21:37,270 --> 00:21:43,000 we can use different 3D techniques to scan caves, niches, sculptures, 301 00:21:43,000 --> 00:21:46,840 and inscriptions at Longmen Grottoes. 302 00:21:46,840 --> 00:21:52,120 As a matter of fact, researchers have finished 90% of the 3D digitization 303 00:21:52,120 --> 00:21:57,910 at Longmen from 2005, when it first started. 304 00:21:57,910 --> 00:22:00,640 Given the large number of relics onsite, it 305 00:22:00,640 --> 00:22:05,690 remains very necessary to build a database to effectively organize the 3D 306 00:22:05,690 --> 00:22:10,250 textured models after we capture them. 307 00:22:10,250 --> 00:22:14,450 Researchers have built a system to manage, browse, and retrieve 308 00:22:14,450 --> 00:22:18,590 digital data of Longmen, which interactively incorporates 309 00:22:18,590 --> 00:22:25,090 high-resolution photos, 3D models, and text annotations. 310 00:22:25,090 --> 00:22:27,640 Using a tree structured file system, it's 311 00:22:27,640 --> 00:22:31,870 possible to effectively manage large-scale scenes, as well 312 00:22:31,870 --> 00:22:35,410 as details of a particular sculpture. 313 00:22:35,410 --> 00:22:38,890 The system also enables researchers to perform other operations, 314 00:22:38,890 --> 00:22:42,200 such as distance measurement. 315 00:22:42,200 --> 00:22:44,480 For another tool that uses the same idea, 316 00:22:44,480 --> 00:22:46,970 but has many other functionalities, please 317 00:22:46,970 --> 00:22:49,880 refer to the open source Cherub software, 318 00:22:49,880 --> 00:22:52,610 developed by Yale Graphics Group. 319 00:22:52,610 --> 00:22:58,550 3D models also provide valuable information for archaeological drawing. 320 00:22:58,550 --> 00:23:00,810 It is an important task for archaeologics 321 00:23:00,810 --> 00:23:07,760 to depict shapes and structures where the heritage remains accurately. 322 00:23:07,760 --> 00:23:09,780 Archeological drawing is a traditional way 323 00:23:09,780 --> 00:23:12,390 to record geometric shapes of the heritage 324 00:23:12,390 --> 00:23:17,500 remains using concise lines and curves. 325 00:23:17,500 --> 00:23:21,800 As one of the central components in a typical archaeological report, 326 00:23:21,800 --> 00:23:24,800 line drawings are time-consuming, and have 327 00:23:24,800 --> 00:23:27,325 low accuracy using traditional methods. 328 00:23:27,325 --> 00:23:30,500 329 00:23:30,500 --> 00:23:35,960 Instead of setting baselines, putting grades, and drawing lines manually, 330 00:23:35,960 --> 00:23:39,200 it is possible to automatically extract lines-- 331 00:23:39,200 --> 00:23:42,230 such as silhouettes, ridges, and valleys-- 332 00:23:42,230 --> 00:23:45,690 from scanned 3D models. 333 00:23:45,690 --> 00:23:48,840 Then, the 3D model and lines can be rendered 334 00:23:48,840 --> 00:23:56,570 to obtain the corresponding parallel projection map at a specific viewpoint. 335 00:23:56,570 --> 00:23:59,600 With parallel projection maps, archaeologists 336 00:23:59,600 --> 00:24:05,500 can finish the line drawing with a smaller amount of post-processing work. 337 00:24:05,500 --> 00:24:11,030 Digital restoration it's another application of significant values. 338 00:24:11,030 --> 00:24:15,380 Since there are many relics missing, or in overseas collections, 339 00:24:15,380 --> 00:24:19,810 physical restoration is extremely difficult. 340 00:24:19,810 --> 00:24:23,650 However, in the digital world, it is possible to put together 341 00:24:23,650 --> 00:24:28,040 2D pictures of reliefs, and align them with surroundings, 342 00:24:28,040 --> 00:24:32,500 if that is sufficient in representing the artwork. 343 00:24:32,500 --> 00:24:36,850 More interestingly, we can also restore the 3D information of relics 344 00:24:36,850 --> 00:24:39,110 based on models and old photos. 345 00:24:39,110 --> 00:24:43,080 346 00:24:43,080 --> 00:24:47,520 For example, if researchers have the permission 347 00:24:47,520 --> 00:24:51,780 to scan a missing Buddha head, the scanned model of the head 348 00:24:51,780 --> 00:24:55,230 can be aligned with the model of its remaining body. 349 00:24:55,230 --> 00:25:05,380 350 00:25:05,380 --> 00:25:08,830 The gap between two parts can be fused using 351 00:25:08,830 --> 00:25:11,820 some algorithms of geometric analysis. 352 00:25:11,820 --> 00:25:21,030 353 00:25:21,030 --> 00:25:23,370 Here is another example from a niche. 354 00:25:23,370 --> 00:25:25,940 355 00:25:25,940 --> 00:25:30,920 The Buddha lost his head, and we can digitally restore it in a similar way. 356 00:25:30,920 --> 00:25:46,630 357 00:25:46,630 --> 00:25:50,850 For this model without its nose, researchers 358 00:25:50,850 --> 00:25:55,870 can simply build a 3D model according to the expected shape of nose. 359 00:25:55,870 --> 00:26:08,450 360 00:26:08,450 --> 00:26:13,770 For other missing Buddha heads that don't have any available 3D data, 361 00:26:13,770 --> 00:26:19,540 we can use some old photos from when the head was not yet removed. 362 00:26:19,540 --> 00:26:24,930 It is easier to build a 3D model if we have photos from multiple views. 363 00:26:24,930 --> 00:26:30,020 But even one picture can convey the idea of its shape. 364 00:26:30,020 --> 00:26:34,730 Given the standard head model as prior, we can warp it a little bit 365 00:26:34,730 --> 00:26:38,660 according to features detected in the old photo, 366 00:26:38,660 --> 00:26:44,540 so that we can produce an approximate 3D model of missing heads. 367 00:26:44,540 --> 00:26:53,760 For more details, please refer to our paper published in 2016. 368 00:26:53,760 --> 00:26:57,060 Last but not least, the development of virtual reality, 369 00:26:57,060 --> 00:27:00,990 and augmented reality display, brings a new perspective 370 00:27:00,990 --> 00:27:05,470 of visitor accessibility to the ancient caves. 371 00:27:05,470 --> 00:27:08,770 Nowadays, for better protection of cultural relics, 372 00:27:08,770 --> 00:27:13,720 tourists are limited to enter most of the caves. 373 00:27:13,720 --> 00:27:17,860 It is a great pity that they cannot appreciate these art treasures 374 00:27:17,860 --> 00:27:20,580 in detail. 375 00:27:20,580 --> 00:27:24,720 Luckily, based on all of the 3D models that we have captured, 376 00:27:24,720 --> 00:27:29,010 it is possible to view the virtual 3D scene using specialized rendering 377 00:27:29,010 --> 00:27:31,420 software. 378 00:27:31,420 --> 00:27:37,290 Smartphones and other VR devices are capable of recovering users' motion, 379 00:27:37,290 --> 00:27:43,360 and update the virtual viewpoint in accordance with actual motion. 380 00:27:43,360 --> 00:27:46,550 Three degrees of freedom, in terms of rotation, 381 00:27:46,550 --> 00:27:53,440 can be recovered using the inertial measurement unit in smartphones. 382 00:27:53,440 --> 00:27:57,760 We can also use a motion capture system to better localize the user, so 383 00:27:57,760 --> 00:28:00,940 that the user can have a translation-updated view when 384 00:28:00,940 --> 00:28:03,400 walking in a room. 385 00:28:03,400 --> 00:28:07,270 Moreover, we can implement an augmented reality system 386 00:28:07,270 --> 00:28:11,650 using digitally restored models, and adding more interactive enhancing 387 00:28:11,650 --> 00:28:12,550 effects. 388 00:28:12,550 --> 00:28:14,830 The left picture shows possible appearance 389 00:28:14,830 --> 00:28:21,110 of wall painting in the Central Binyang Cave, before weathering. 390 00:28:21,110 --> 00:28:24,530 The middle one shows the digitally restored empress procession 391 00:28:24,530 --> 00:28:29,100 on the wall in the same cave. 392 00:28:29,100 --> 00:28:33,540 The right one shows the restored model of a missing Boddhisattva head, 393 00:28:33,540 --> 00:28:35,910 using a framework based on old photos. 394 00:28:35,910 --> 00:28:39,280 395 00:28:39,280 --> 00:28:42,850 The VR/AR display of cultural heritage can effectively 396 00:28:42,850 --> 00:28:46,660 solve the contradiction between the protection of cultural relics 397 00:28:46,660 --> 00:28:49,960 and tourism promotion. 398 00:28:49,960 --> 00:28:55,390 Another VR display system of WanFo Cave was produced this year. 399 00:28:55,390 --> 00:28:58,930 As a first virtual reality project of Longmen grottoes, 400 00:28:58,930 --> 00:29:03,970 WanFo Cave VR cannot only provide cave interior roaming, 401 00:29:03,970 --> 00:29:09,610 but also have cave lighting, and teleportation of user position. 402 00:29:09,610 --> 00:29:13,150 The audience can enjoy every detail in the cave closely, 403 00:29:13,150 --> 00:29:16,300 and the statues can be seen from a viewpoint which 404 00:29:16,300 --> 00:29:20,620 is difficult to reach in reality. 405 00:29:20,620 --> 00:29:24,730 We talked about many applications of 3D digitization technology 406 00:29:24,730 --> 00:29:27,100 at Longmen grottoes. 407 00:29:27,100 --> 00:29:31,210 In fact, as an active research field, digital cultural heritage 408 00:29:31,210 --> 00:29:35,570 attracts thousands of researchers around the world. 409 00:29:35,570 --> 00:29:39,580 For example, a team from Stanford and other institutions 410 00:29:39,580 --> 00:29:45,190 initiated the digital Michelangelo project in 1998, 411 00:29:45,190 --> 00:29:49,300 and produced a detailed 3D model of Michelangelo's David 412 00:29:49,300 --> 00:29:52,390 using laser scanners. 413 00:29:52,390 --> 00:29:55,750 Our CS50 instructor, Professor Benedict Brown, 414 00:29:55,750 --> 00:30:01,750 also worked on improving the project during his PhD study. 415 00:30:01,750 --> 00:30:06,190 In 2003, researchers from the University of Tokyo 416 00:30:06,190 --> 00:30:11,830 scanned the Bayon Temple in Cambodia, using different kinds of scanners, 417 00:30:11,830 --> 00:30:15,810 including ones on a floating balloon. 418 00:30:15,810 --> 00:30:19,240 In this way, after handling the drifting, 419 00:30:19,240 --> 00:30:22,270 they managed to have a better scan of the top 420 00:30:22,270 --> 00:30:24,550 before drones were commercialized. 421 00:30:24,550 --> 00:30:40,340 422 00:30:40,340 --> 00:30:44,510 In 2009, another team from the University of Washington 423 00:30:44,510 --> 00:30:48,260 finished a project called Build Rome in a Day, where 424 00:30:48,260 --> 00:30:50,980 the researchers constructed a 3D model of Rome 425 00:30:50,980 --> 00:30:55,280 losing millions of tourists' photos on Flickr, instead of any 3D scanners. 426 00:30:55,280 --> 00:30:59,240 427 00:30:59,240 --> 00:31:02,900 Fascinated as we are by this wonderful achievements, 428 00:31:02,900 --> 00:31:07,400 other heritage sites may still face a tragic fate. 429 00:31:07,400 --> 00:31:09,840 Even today. 430 00:31:09,840 --> 00:31:14,820 In 2015, ISIS demolished the Temple of the Baalshamin 431 00:31:14,820 --> 00:31:16,280 during the Syrian Civil War. 432 00:31:16,280 --> 00:31:19,000 433 00:31:19,000 --> 00:31:22,930 As a UNESCO World Heritage site, the Temple of Baalshamin 434 00:31:22,930 --> 00:31:27,280 was built approximately 2,000 years ago, representing 435 00:31:27,280 --> 00:31:30,525 a fusion of ancient Syrian and Roman architectural styles. 436 00:31:30,525 --> 00:31:33,130 437 00:31:33,130 --> 00:31:37,510 After the temple's destruction, the Institute for Digital Archeology 438 00:31:37,510 --> 00:31:42,520 announced plans to establish a digital record of other threatened sites 439 00:31:42,520 --> 00:31:44,560 and artifacts. 440 00:31:44,560 --> 00:31:48,550 But it was such a great loss to all human civilizations, 441 00:31:48,550 --> 00:31:50,410 that people can never make up to. 442 00:31:50,410 --> 00:31:54,450 443 00:31:54,450 --> 00:31:58,710 This CS50 seminar might be very different from other ones. 444 00:31:58,710 --> 00:32:02,790 But I just wanted to show you how beautifully technology can 445 00:32:02,790 --> 00:32:06,170 interact with humanities. 446 00:32:06,170 --> 00:32:10,130 When I was in high school, my teacher always asked me what kind of person 447 00:32:10,130 --> 00:32:14,490 I wanted to be, before the science class. 448 00:32:14,490 --> 00:32:19,650 Technology might give more power to evil if the practitioners don't use it 449 00:32:19,650 --> 00:32:22,170 in a proper way. 450 00:32:22,170 --> 00:32:26,460 That's why it is important for all of us to be aware that technology 451 00:32:26,460 --> 00:32:29,530 might be a double-edge sword. 452 00:32:29,530 --> 00:32:32,740 We'd better learn to use it critically, because we're 453 00:32:32,740 --> 00:32:36,470 the ones to create the future. 454 00:32:36,470 --> 00:32:40,850 I'm so proud to be part of the digital Longmen project when I was in Peking 455 00:32:40,850 --> 00:32:45,320 University, under the guidance of Professor [? Jiao ?] [? Hongpin, ?] 456 00:32:45,320 --> 00:32:49,440 Professor [? Ikiyoshi ?] [? Katsu, ?] Dr. Li Renju, 457 00:32:49,440 --> 00:32:53,570 [? Chen ?] [? Jiaohan, ?] [? Fletcher ?] [? Coleman, ?] and other scholars from 458 00:32:53,570 --> 00:32:56,890 the Longman Research Academy. 459 00:32:56,890 --> 00:33:01,710 It is just a perfect example of showing how the development of computer science 460 00:33:01,710 --> 00:33:07,060 can temporarily set aside the historical controversy of ownership, 461 00:33:07,060 --> 00:33:10,690 empower researchers to better understand and safeguard 462 00:33:10,690 --> 00:33:16,570 the values of cultural heritage, and give access to everyone so that we 463 00:33:16,570 --> 00:33:20,320 can appreciate these artistic masterpieces in a way 464 00:33:20,320 --> 00:33:24,520 that people could never have imagined. 465 00:33:24,520 --> 00:33:26,830 From 3D scanning and geometric processing, 466 00:33:26,830 --> 00:33:32,410 to rendering and VR/AR display, technologies and computer sciende-- 467 00:33:32,410 --> 00:33:36,460 or more specifically, computer graphics and computer vision-- 468 00:33:36,460 --> 00:33:40,450 become even more meaningful and energetic for the reunion 469 00:33:40,450 --> 00:33:44,380 of human civilizations. 470 00:33:44,380 --> 00:33:48,490 Students who are taking CS50 come from different backgrounds. 471 00:33:48,490 --> 00:33:52,390 And those at Yale and Harvard also have the privilege of access 472 00:33:52,390 --> 00:33:55,810 to a liberal arts education. 473 00:33:55,810 --> 00:33:59,290 It is our responsibility to work hard, and creatively, to overcome 474 00:33:59,290 --> 00:34:02,910 big challenges in today's society. 475 00:34:02,910 --> 00:34:07,350 Therefore, I encourage you to keep the solicitude for humanity 476 00:34:07,350 --> 00:34:13,000 as you climb towards the top of science and technology. 477 00:34:13,000 --> 00:34:15,040 I would also like to encourage you to apply 478 00:34:15,040 --> 00:34:20,909 what you have learned in CS50 to your own areas and interests. 479 00:34:20,909 --> 00:34:25,120 I hope this seminar can bring you a new way of thinking, analyzing, 480 00:34:25,120 --> 00:34:26,820 and solving problems. 481 00:34:26,820 --> 00:34:30,480 That it inspires you to take advantage of the power of computer science 482 00:34:30,480 --> 00:34:34,650 and technology to make the world a better place for all generations 483 00:34:34,650 --> 00:34:36,280 to come. 484 00:34:36,280 --> 00:34:36,780 Thank you. 485 00:34:36,780 --> 00:34:40,889 486 00:34:40,889 --> 00:34:45,280 If you are interested in more details, please feel free to send me an email 487 00:34:45,280 --> 00:34:51,449 via Z-E-Y-U .wang@yale.edu to discuss more. 488 00:34:51,449 --> 00:34:56,270 For Yalies, you can also contact members of Yale Graphics Group. 489 00:34:56,270 --> 00:35:00,050 Next semester, our CS50 instructor, Professor Benedict Brown, 490 00:35:00,050 --> 00:35:06,230 will be offering another course called Applications in the Digital Humanities. 491 00:35:06,230 --> 00:35:11,700 By now you all have the prerequisites for the course. 492 00:35:11,700 --> 00:35:15,980 You can also take computer graphics, offered by Professor Holly Rushmeier, 493 00:35:15,980 --> 00:35:18,380 and consider Computing and the Arts major, 494 00:35:18,380 --> 00:35:20,380 directed by Professor Julie Dorsey. 495 00:35:20,380 --> 00:35:23,700 496 00:35:23,700 --> 00:35:27,230 Here is more information about the major. 497 00:35:27,230 --> 00:35:30,550 Computing and the Arts is an inter-departmental major, 498 00:35:30,550 --> 00:35:34,910 designed for students interested in integrating work in computing, 499 00:35:34,910 --> 00:35:38,030 and one of the arts disciplines. 500 00:35:38,030 --> 00:35:42,140 Computing in the Arts provides students with core computer science skills 501 00:35:42,140 --> 00:35:45,950 that allow them to explore interesting and substantive problems 502 00:35:45,950 --> 00:35:51,880 in architecture, art, art history, music, or theater. 503 00:35:51,880 --> 00:35:54,578