DANIEL: Hey, everyone. Thanks so much for joining us. For all of those online, welcome, as well. We're so happy for you to be here. Today we're going to be doing a quick tech talk on how we do time predictions at Uber and how Uber works as a company as a whole. This is meant to be a pretty casual talk, so feel free to interrupt us with any questions. But, to kick things off, let's do a couple of intros. My name's Daniel. I'm a product manager at Uber, working on Uber Eats Marketplace team. EVAN: And I'm Evan. I'm currently focused on the marketplace, how we incentivize our drivers, but have done rotations on the Share Rides team, as well as our hourly rental business. ALLIE: And I'm Allie. I'm a university recruiter at Uber, focusing a lot on hiring Harvard students this year. So I will be doing a little bit of recruiting slides towards the end of the tech talk. But I will also be here afterwards to answer any recruiting questions. So just take it away. DANIEL: Cool! All right, let me back up a couple slides. OK, sweet. So the topic of our talk today will be "Please Be on Time." So we'll be talking about how we think about time predictions, when you're Uber's arriving, when your food's arriving, and how we portray all that information to you. So, real quick, for agenda, we'll do a little bit of a deeper dive into who we are. EVAN: [LAUGH] DANIEL: Then we'll start off actually with a couple trivia questions about Uber at Harvard. And we have a couple swag items. So please, for the people in the room, speak up and we'll toss you a T-shirt. EVAN: Participate. You in the back, participate. [LAUGH] DANIEL: And then I'll dive into the models that power Uber Eats delivery times. And then we'll talk a bit about the more experiential side of how we display arrival times on the shared-rides products. And then Allie will give a overview of Uber opportunities. Hopefully all of you guys will want to join the team, after the talk. And then we're leaving like 20, 25 minutes at the end for Q&A, because we really want this to be interactive. That being said, for those of you in the room, please stop us as we go along if you have questions. Cool. So, back in our day, which was not that long ago-- I graduated in 2017-- I was concentrating on mechanical engineering. I lived in Currier, and I was memorizing shuttle schedules. I actually found out last night that we were giving this talk in this building, so I should have actually updated this because this is quite nostalgic for me. I was also a student manager for HSA, so I was working in the basement of this building, basically packing T-shirts into mailers and shipping them out all day. EVAN: Cool. And, again, I'm Evan. I studied CS here. I spent a lot of time in the MAC, playing basketball, and lived in Dunster House. That was exactly my window. And it's cool being back on campus. DANIEL: Sweet. OK, so let's start off with some trivia. Uber at Harvard. All right, so the first one-- which house, do you think, had the most Uber Eats orders in the last year? EVAN: And you have to provide some sort of justification for it, as well. DANIEL: Yes. EVAN: [LAUGH] AUDIENCE: It's going to be [INAUDIBLE] because it's not close to anything. DANIEL: [LAUGH] [VOCALIZATION] Mather. [LAUGHTER] EVAN: But, thanks for the participation. Would you like a notebook or a T-shirt? AUDIENCE: I'd love a notebook. DANIEL: All right, sweet. Does this mean Mather has the words, like, [INAUDIBLE] quality? [LAUGHTER] All right, next one. Which house has the most pickups? So which house do most people want to get away from? EVAN: That was quick. That was quick response time. DANIEL: This guy knows. AUDIENCE: Lowell? DANIEL: Lowell House? Adams. [LAUGHTER] Lowell was, oh, pretty low, actually. EVAN: This was actually-- I thought this was cool, because it was-- there's obviously all sorts of noise in these. And these are basically, over the past year, we're looking at the time series of when people are-- based on their pickup and drop-off request locations. But this was, like, Adams, the house that everyone wants to get away from. And then people in Winthrop kind of stay in Winthrop. So I like this one. DANIEL: Cool. Do you want a T-shirt? AUDIENCE: I'll take a T-shirt, yeah. DANIEL: Sweet. EVAN: OK, the house that most people want to get away from. Which house has the most drop-offs? Or, no, sorry-- where do people want to go the most? AUDIENCE: Eliot? [INAUDIBLE] DANIEL: Do you live in Eliot? AUDIENCE: No, I'm a freshman-- [LAUGHTER] Because I know a lot of students-- like, I help [INAUDIBLE].. DANIEL: OK. EVAN: Well, I think you may be [INAUDIBLE] this year, because it's actually Currier. DANIEL: Uh-oh! [LAUGHTER] Gotta say, yeah, Currier was the best house. All right-- EVAN: Yeah, Winthrop's (LAUGHING) at the bottom. DANIEL: --one more. Oh, this is Evan's EVAN: OK, so which house has the most trips to Cumnock Field? This was between the hours of 6:00 and 10:00 PM, during the week. AUDIENCE: [INAUDIBLE] EVAN: Cumnock fields-- what year are you? AUDIENCE: Sophomore. EVAN: Sophomore, well, so it's-- [LAUGH] well, so the intent for this was, like, the house that is the most into IMs. So, who is taking the most Ubers to Cumnock. [INTERPOSING VOICES] EVAN: I got a couple Winthrops. AUDIENCE: It's got to be Winthrop. EVAN: It's got to be Winthrop. You would think so-- AUDIENCE: Cabot, I see there. EVAN: Maybe the secret to winning IMs is that you actually walk to the field. It could be like a some sort of heuristic for athletic prowess. I don't know. I don't know. DANIEL: All right, and, last one, which house travels to the airport the most? So, which house has the most worldly travelers? Anyone? EVAN: You're going to get this right. I can feel it. AUDIENCE: Quincy. DANIEL: Quincy? EVAN: Oh! [LAUGH] Winner. DANIEL: Nice. All right, we'll get you guys T-shirts after this. All right. EVAN: And then, also, just one of the cool things about working with Uber is that we are working with so much data, and the visualization tools to make these sorts of maps and visualizations is really easy. This query was for over a week's time-- all of the pickups over Harvard's campus. And you can see that Uber has made its mark. And this is a combination of rides and deliveries. And also, this, I don't actually-- [LAUGH] this is, yeah, we're all one big, happy family. The Ubers has made friendships between the Quad and houses, the river houses. Right, it's kind of-- again, this visualization is just really cool to me, but basically you see all of these trips between both the Quad and the river. And I don't actually know this, but my guess is most of that is happening kind of late nights, Friday, Saturday, Sunday. But I'd have to double-check that. [LAUGHTER] It's kind of like horoscopes. You've got to read into it. DANIEL: I was definitely one of those trips. EVAN: Yeah. DANIEL: Cool. And Uber's popularity isn't just at Harvard. As you guys have probably seen, in the news, every day, Uber is expanding all across the globe. So this graph on the right shows Uber's trip count in our first five big cities. So we started out in SF, back in 2010. And the cool thing here is, the growth is pretty much exponential in all of these cities. And the other cool thing is-- So we started out in SF. Then we went to New York, then Seattle, then Chicago, then DC. And you can actually see that the growth actually gets faster and the curve gets even steeper, as we launched more cities. And when you look at this on a global scale, Uber's definitely not just in the US. So this graph is actually, on a log scale, just showing the ridiculous amount of growth that we have all across the world. Uber being an international company is a pretty unique challenge for us. I often like to think that other companies, like Facebook or Google, the experience is pretty much the same. But, for Uber, we're really operating at the interface of the digital and physical worlds. And everything from regulation to the actual products that we're offering our riders and eaters is really, really different in all of these regions. That's one of the really fun parts about being a PM at Uber. EVAN: And then I'd just add that, in the US, it's easy for us to [INAUDIBLE],, like Uber and Lyft compete really aggressively. Right? But in each of these mega regions, it's, you have a really tough competitor. And, you know, in Asia it was-- and continues to be-- DiDi. In Europe, it's Cabify. In the Middle East, it's Careem. It's like, in all of these places, there is really intense competition. And how a company headquartered in San Francisco can compete with these players that are much more local continues to be a challenge. DANIEL: Cool. But we are doing pretty amazing. So, as of 2018, we're in over 650 cities, across the globe. We are in 77 countries. And, just as of June, I believe, this year, we just hit 10 billion rides, around the world. To put that into perspective, we actually hit 5 billion less than a year ago. So in one year we've gone from 5 billion and doubled that amount to 10 billion. So this may all sound pretty simple. Right? Like, when you call an Uber from the Quad or the river, you tap a button, you get in that Uber, you chat with the driver, you listen to some music, and then suddenly you're at your destination. But there's a lot that can go wrong-- again, especially in this space where we're interacting with not just software on your phone but we're actually moving people. We're moving things around the world. And there's just so much that can go wrong with that. So, like I said, for the ride side you might be used to opening your app, requesting a car, getting in that car, getting to your destination. For those of you who may have tried Uber Eats-- similar sort of simple flow. You tap a button, you choose that burger or burrito you want, and it arrives 20 to 30 minutes later. But even in this second half of requesting a car to getting in that car, or ordering food and getting that food, there's a lot of logistics problems that we think about, day in, day out. One of them is figuring out when exactly that ride or food will arrive. This isn't a simple problem. And so there's the sort of mathematical part of it-- how do we actually create models to predict it? But there's also the experiential component of, how do we convey that information to you and make it meaningful to you as a eater or rider? So I'm going to focus my part of the talk on Uber Eats. So, just a show of hands-- who has tried Uber Eats or is familiar with it? OK, awesome, OK, cool, cool. I thought was going to have to explain the app. So, real quick-- so, like, you know, you open the app. I think the parts of this flow that you're used to is, you open the app, you place your order, and then you're sort of used to seeing the delivery. But there's a ton of intermediary steps that happen, both on the restaurant side, on the courier side, and on the eater side. And a lot of this is based on knowing exactly when the food at the restaurant will be ready. So, for the marketplace side, this allows us to know exactly when to dispatch a courier-- making sure that that courier doesn't arrive at the restaurant too late or too early. And this is really important for us. As Uber, we think about this three-sided marketplace-- the restaurant, the eater, and the courier. It's really important that we get these time predictions right. That way, we can control things like supply and demand and prevent surge in our logistics systems. For you guys, as the eaters, this is super-important that we get this right so we can provide fast delivery times and that we're able to provide ETD estimates of when your food will arrive and it won't, like, jump a lot while you're waiting for your food. For couriers, this is important because we don't want couriers to arrive at a restaurant and wait there for 10 minutes while the restaurant's preparing the food. Right? That's not the best use of their time. So we really want to make sure that they're maximizing their earning potentials. And finally, on the restaurant side, you know, I've been to restaurants both here in the US, and I've traveled abroad to a lot of our restaurant partners. They really don't like it when couriers are waiting around in the restaurant. They're trying to manage their front of house, and they have a bunch of people also filing in. It's just not a great experience. So this problem really touches everyone in the entire Uber Eats ecosystem. So the first problem is, we don't really know when the food is done. No one's standing in the restaurant and telling us, the food is done; you should send a courier now. We don't have a camera. We don't call the restaurant. The restaurant doesn't call us. And so there's no ground truth on validating our predictions. A second problem is, we also don't have a ton of good signals into knowing when that food will be ready at the restaurant. So we're working with very limited number of data points. To give you a sense of how this is impacting eaters and restaurants, real quick-- so each restaurant in the Uber Eats ecosystem has a tablet in the restaurant that shows them all their ongoing orders. And so, for this one, for Michelle, there will be a due time, which is when the restaurant can expect a courier to show up and pick up the food. And it'll tell them how long that is from now. And then, on the eater side, hopefully maybe some of you guys are familiar with this, right after you place your order we show you that to-the-minute time prediction. So one approach that we can take, to tackle this problem, is predicting when that food will be ready, creating a prediction system that will inform us when we should dispatch a courier to the restaurant. . So one hypothesis-- you know, if you were to think about this sort of as a machine-learning-driven prediction system, is that we can rely on how restaurants behave, with the tablet. So for instance, if they delay the order and tell us, hey, you dispatched someone but we weren't quite ready yet, that probably means our prediction wasn't quite right. So we can take that restaurant input as a signal. Another hypothesis is, we can look at how couriers behave once they arrive at the restaurant. So this was a trip, picking up, I think, like, burritos from El Jefe's. You can see this courier arrive, and he ends up waiting around a lot. He's kind of driving around. He might be parking, but you can also tell that he's, like, loitering around for a while. We can use this also as an input to say, hey, this courier arrived, ended up waiting for 15 minutes. The food probably wasn't ready yet-- again, a hypothesis. So, if you combine all these hypotheses, they begin to become features. Right? And so you can start building out a prediction model, using all these features. So one is when the food is done from the restaurant. Another is the courier wait time at the restaurant, after they've arrived. Some other ones that you might use are live order count-- you know, how busy is this restaurant, in real time? Another one might be basket size, as a proxy for how many items are in that order. If you're ordering, like, the 12 pizzas that we have for this event, that's probably going to take a little longer to prepare than a burger for one person. And then, finally, time of day and day of week. So you can probably imagine El Jefe's is probably pretty busy on a Friday night, after people are, like, coming out of parties. And then Zinneken's might be a popular spot on Sunday mornings-- someone who wants brunch. So we combine all these features into this model. And we're constantly iterating on this prediction model. It's what fuels the millions of Uber Eats trips on a weekly basis that we do all around the world. And we're constantly striving to reduce the error in our models. This is a quick screenshot of all the cities. So we're now in over 290 cities, across the world. And it's little machine-- [LAUGH] I mean, it's, like, little models like these that you might think is almost, like, a piece that you might be working on. But in the real world, this is actually impacting millions of rides, millions of deliveries, every day, around the world. All that being said, we are not always perfect. And so, in addition to building out these models, we also have to know how to properly convey that information to eaters, especially when there's uncertainty involved-- like with time. And so Evan's going to talk a little bit about how we reveal uncertainty to our customers on the ride side. EVAN: [INAUDIBLE] DANIEL: Yeah. EVAN: Cool. So, yeah, so, part of the work that I've done is much more how we show these models to our shared-ride riders in particular. But I'm just going to start walking you through why this problem, to me, is a really cool one and gets at a lot of the human-computer interaction work that I was thinking about at school. The two high-level-- the so-what's, here, is one is that probabilities are really difficult to communicate. So, when Daniel's showing you, like, right your food, if the courier ends up getting there too early and lingering and maybe the restaurant's late, what all of this comes down to is there's some time range. And in that range, your food will come. But people, myself included, we're not very good at intuiting what it means if there's a 50% something arriving at this time, versus a 70% chance of it arriving at another time. It's like, how we weigh those probabilities and the actions we take is just a super-difficult thing to build products around. And the second is that our customers are super-sensitive when things go wrong. Part of this-- and everyone's been there. It's like, you're ordering a ride into Boston, and you're waiting outside, and it's cold-- you're the street, it's cold, it's snowing, you didn't wear a jacket. It's like, there are visceral experiences that our customers have that, very quickly, they can go, like, to, like, there's an "Uber sort of sucks," here, moment. And so those are happening all the time. And we don't want to be building products that people think suck. So the question is, how do we manage these expectations in ways that are productive? And on that note of the sorts of scenarios where it matters, say you order like a huge basket for dinner, or say you came here expecting there to be pizza and the pizza wasn't here. Like, everyone would be grumpy, and no one would really care about listening to our talk. You know, we lose you, right there. So the pizza's is really important. Thanks for having it delivered. Two is, like, people use Uber to get to, like, see people at important times. Maybe it's romantic. Maybe it's a business meeting. When people are late, they don't necessarily say, hey, it would've been better if I, like, left my house 30 minutes earlier. It's, like, the Uber got stuck in traffic. Right? And I'm guilty of that, as well. And third is, say it's the sort of thing where we tell you there's a 95% chance you're going to arrive at Logan for your flight. And you're like, OK, I'll take those odds. But then you arrive, that 5% of the time where you arrive later than that time, that happens. And, because we're doing billions of rides, happens all the time. As a result, you miss your flight. All of a sudden, that decision to choose a certain sort of Uber product failed you. And that's going to color the rest of your Uber experiences. So the point here is that this is a super-tricky area to sort of describe experiences. And, two, when things go wrong it can have these real-world consequences that are tough for our customers. So Daniel was talking about this sort of, colored in green, the Eats timeline. This is the journey map of what is happening over the course of an Eats order, from order to delivery. On the top, you have this as if you are an UberX. You request the ride, there's some uncertainty, here, in terms of when the car's going to actually sort of-- when that actual time of arrival pops up, but then you get picked up and dropped off. What I'm going to focus on, for this experiential portion of the talk, is the middle line, which is Express Pool. And, just curious, show of hands, people who have used Express Pool? People who use Lyft Line? Cool, cool, cool. Share rides-- it's good. Share rides is where it's at. So we actually get, like, $200 worth of credits a month, as part of being employees at Uber. And I try to stretch them by only using shared rides. So Lyft Line's cool. So, yeah, so in the shared-rides world-- and I'll go into a bit more about what Express Pool is, for those who haven't tried it. But, at the start, you request a ride. You walk to a pickup spot, to create a straighter route for you and your co-riders. Over the course of your journey, riders are getting picked up and dropped off. You get dropped off, and you walk a little bit to your destination. Sometimes you don't walk at all, but this is the expectation we set with riders when they enter the Pool products. And so people who have ridden, sort of, ride-shared in general, and Express Pool in particular, have a general sense of this. But, going back, Boston was one of our first [INAUDIBLE] for Express Pool. Going back a little bit, though, this was our strategy for why Express Pool was a game-changer. So this is Pool, step 1. And it's basically, there are a few pickup spots. Those are all riders who are all going to get batched together in the same car. And so car does some sidetracking, to get to them. Express Pool, the big idea is putting steps 2 and 3 together. So, for 2, it's, hey, the driver has to go out of their way to pick these people up. It probably means they're looping around a block with extra traffic, and that hurts everyone-- the driver and the riders, included. So what if our riders actually walk to a straight route, so that it was more a sort of express and sort of an A-to-B route for the driver. And 3 was, hey, we can have people walk, but if we just make them wait a little bit up front we can get even smarter about our matches. Because there are more people populating this graph, and we can stitch them together into cars in more efficient ways. So this was the visualization that sold Express Pool, internally. It's something we had to do and invest in-- combining 2 and 3's Express Pool. And this is basically what we launched with. And so, if you were in Boston a bunch of months ago, this is what you would have seen. Basically, we put Express Pool to the left in our product selector. And you can see that, when you're about to request a ride, you're doing a trade-off, which is, there are the prices, and there are also these times. So riders are making this calculation of value of price in time. And then, if you hit Request you go into this batching screen, which is that waiting part I talked about, where we're seeing all the other potential riders also online that we could batch you with. And then you walk, you get in your car, and then you're on your way. So this is the basic Express Pool experience. What we realized, though-- and Express Pool was positioned as a commuter product-- it was cheaper, there was walking involved, people walk generally during their commutes, and commutes are generally cheaper. So this was our commuting product. And what we found was that, among people who chose shared rides, riders were more likely to choose Pool than Express Pool during AM commutes. And this was a very striking thing, as this was bad. Like, we've made this product for commute times, for commuters who are price-sensitive and could mentally budget, OK, like, $4 per ride is in the realm of what I can spend-- $4 or $5. So usually what happens, when we figure out that there is a problem that exists in our world, is, we ask our data scientists to, can you figure out some sort of insights from the data we collect? And from a user-experience side, can we talk to a bunch of riders who seem to be engaging with this product and see why it is that they're making the decisions that they're making? So the data scientists were the ones you sort of unveiled this trend, that, hey, we're not actually capturing a lot of people who we thought we would with this product. And from the user-research side, these are some of the things that we found. So these points are the three I'll walk through. So one was, riders weren't sure if this time range included that waiting and walking. Right? Because, in the past, for X and Pool, you had these estimated time to destinations, these ETDs, and those didn't have any waiting and walking, so it wasn't clear to riders if Express Pool captured that. It also wasn't clear to riders why the Express Pool time range, when selected, was the same as the Pool range, when selected. If I'm waiting and walking, why is this going to be faster for me? If you had seen that chart up front, where it's, hey, we're finding a better route, maybe you'd get it. But there was, like, an intuitive feel of, hey, there are more steps, here. How is that going to be faster? So there was this lack of clarity-- were these numbers even real? And then-- So FTUX stands for First Time User eXperience. In the first-time user experience, we basically say, hey, these are straighter, faster routes-- like I've hopefully sold you all on. In reality, though, our riders typically spend somewhere between four and eight seconds, like, flipping through those cards. It's like, you see a cheap price for a product, you're like, hey, I'll try that. And you flip through them. So the time at which we were trying to communicate this information wasn't going through. And then the third thing, which I actually thought was the most important, was that, in a world where you take out Express Pool and you have Pool and X, basically from left to right you have increasing price and things get slower-- or, I'm sorry, things get faster as you go left to right. So, if you put something on the left side, that's cheaper. It's a pretty logical conclusion to draw that it's also going to be slower. Even if those times look the same, you may just think, hey, it's slower and cheaper. Right? So we talked to riders, and they were all convinced-- even the ones who would use Express who were set on Express Pool as their commute option-- that they should be using Pool if they were in a hurry. Right? And so we specifically tuned our systems, our back-end systems, to be, like, riders' total time should be the same. But all our riders who we're talking to who are power riders felt that this was slower. So the high-level point were, all these paper cuts were making they experience a lot worse-- in terms of the perception, despite how we were representing the time information. And then, quickly, on the second part, after your request you and you go into your batching window, we have this sort of itinerary, here, which is, these are all the steps. And, at the end of it, there's this arrival time. First point is that we decided-- Again, this was like, we had to launch this-- we had, like, two months to launch this thing. And this was our first best version, right? We decided to stick in a no-later-than time. So that's, like, a very-worst-case scenario that you probably are only hitting-- you know, like that airport case, that 1%, 2%, 3% of the time. And we did that because we wanted to avoid those cases of really bad experiences. But, for a rider who sees this, it's like, it's a pretty scary time to see, because the range may be-- the more reasonable time-to-show would have been 12:15. So there's a sense, there, we're showing a later time. The bigger point, though, was that riders weren't actually swiping up. So we were putting this time information in this batching card, but riders weren't even getting there in the first place. So some of this first-principles thinking about, hey, like, what's the most important information you want to present, those choices weren't actually making it through to riders. And so the high-level point here was that the IA or the information architecture we were trying to serve this information through just wasn't communicating really anything to our riders. So the next step was figuring out what were the most important questions that we wanted to solve for. But first was, how did we break this mental model, that Pool is a lot faster than Express when really they're the same? Two was, how might we better illustrate the estimated time to destination, the ETD information, to make them actually actionable for riders? And the third was, how do we sell people on the fact that the waiting-and-walking time doesn't make your final destination time-- like, doesn't push that out? So-- I just went through that [INAUDIBLE].. This is basically, let's make this information more prominent. And this is what we came up with. And if you're using Express Pool today in Boston, this is what you'll see. We've just roll this out to all our Express Pool cities, which is pretty cool. So, for starters, this was-- you know, they're all sort of small touches, but again they all add up. This was the big update-- was that, instead of emphasizing your latest time, we emphasized your median time of arrival or something in that range, that was going to be, hey, assuming that there isn't crazy traffic, this is probably when you're going to arrive. Also, we had your latest time. So that, if you were in a rush, you still have kind of an out to say, hey, that latest time is too close. Let me cancel this route and move on. Other point, in terms of the scrollability was that we made this panel just feel more scrollable for riders, where there was a clear itinerary that was being cut off. So, sure enough, more riders now are scrolling when they see this. And the final point I'll make here is that walking and waiting are both steps that lead up to that estimated time, now. Which is a more clear, hey, these steps are part of your arrival time. Final point, I promise, is that, as this countdown is happening, this arrival time is staying the same. Which, when we user-tested it with riders, was a very helpful indication of, oh, I'm not losing time by waiting. This waiting actually doesn't feel like wasted time. And then a quick note on, we also just put more information for those riders who were interested. This was supposed to be only our sort of most-engaged riders who are actually going to click around. But clicking into that estimated arrival time, or why the wait, on that second screen, gives you more information about selling our riders on the value prop that we're actually delivering on. And we didn't know how it was going to go, but it worked out. I need permission to view this video. That's too bad. DANIEL: I do not have permission. EVAN: That's OK. Basically, it shows a time lapse of all these people using Express Pool on a given day. It's cool. But, for the results, it was opt-in-- so, people using Express Pool in mornings was, like, corrected over the course of a few weeks, among our most engaged riders. And cancellations on that waiting screen went down. So we were telling a more effective story to our riders when they were waiting, at that point where they're sort of, they're putting their money where their mouth is, and they're either going to cancel or go forward with the experience. All of which is to say, from-- this was one of my key projects, this past year. And, for me, coming out of it, the first one I mentioned, the little paper cuts all build up, it's like a bunch of small things. And sometimes we talk about it, when we're building a big project, in terms of tech debt, where, over time, you amass tech debt and then you want to deploy some new feature and you just have to fix so much stuff to sort of renew the whole system. Also is true from an experience perspective. Over time, all these little products, these, like, where the text is, that actually matters. Two is that, over time, frameworks become less and less valuable. So, as we try to put more and more products into that product selector, the original intent of explaining our products-- we were basically failing. Third, Daniel's models, as much as we need them to get better, to some extent they're only as good as the experience that actually reveals them to riders and information that they can use. So simple language, like "estimated arrival" and "latest time," that's what tested the best with our riders to actually be helpful. We tried things like putting little percentages up and things like that, but the "estimated" and "latest" were what riders actually preferred, by and large. And finally, it was pretty nerve-wracking, because we probably invested a couple months of time, working on the UX and testing and building this feature out, with all of the color variations. And it took a lot from our eng team. And we just didn't know if people would care at all. Like, maybe it just matters with the prices, and that's it. But it actually turned out that, based on real behavior that we were tracking, that this experience made a real difference in our metrics. So, with that, come work with us. And Allie can tell you more. ALLIE: All right, thanks, you guys. Again, my name is Allie, a university recruiter. Do we have any former interns here? No? OK, cool. Awesome. I will tell you guys a little bit about our internship program, our new grad opportunities, and everything like that. So, just some cool facts for you guys. We definitely create opportunities at Uber. This is a fun fact-- more people earn income from Uber than any other privately held company, other than McDonald's and Walmart. Obviously, as you guys know, there are fewer cars on the road. They decrease congestion. It also frees up parking garages, residential space, et cetera. How many of you guys have heard of our Advanced Technologies Group, our ATG? Awesome. Very cool. So we are investing in the future. If you are interested in any of those opportunities, you can come and talk to me afterwards. Happy to take your resume and send it to the recruiters that sit out in Pittsburgh. This is an interesting fact, as well. This hasn't actually been updated in a few months, but Uber Eats recently expanded to at least 31 or more college campuses, since last August-- which is pretty crazy. These are our engineering locations in the US. Majority of our team sits in San Francisco, which is our headquarters. We recently-- well, I guess not "recently," anymore-- but we opened an office in Palo Alto about a year ago, which sits mostly a lot of our engineers. We do have an office in Seattle, New York, and Colorado, as well. So, if you're interested in any of those locations, we are hiring for all. I would say that Colorado is definitely a super-small office. New York and Seattle are growing like crazy. And then San Francisco and Palo Alto are always growing, as well. DANIEL: And Pittsburgh. ALLIE: Oh! And Pittsburgh, yeah. ATG. DANIEL: I lived there for six months. ALLIE: Ah! There you go. Yes, we are hiring in Pittsburgh, as well. Unfortunately, we don't have anybody here from the ATG side. So, as I said, yeah, come up to me afterwards, and we can chat. These are just a few of our engineering teams, across the board. We have Maps, Core Infra, Consumer Products, Marketplace-- which seems to be a super-popular team for interns and new grads to join. Then we have Security, Uber for Business, and plenty more. Uber is a place to belong. So we have a plethora of employee resource groups that you can join, whether you're an intern or a new grad. We want you to be involved. We want you to be a part of these programs that we have. This just lists a few of them, but we have a ton more that you can join or you can create, yourself. There's a ton of benefits and perks we have at Uber, when you become a full-time employee or even an intern. There is medical benefits. We do have free breakfast, lunch, and dinner. So it saves you a ton of money in the grocery department. We do receive, as Evan was saying, monthly Uber credits, which is probably one of the best perks I've ever had. You don't really realize how much money you spend on Uber, you know, per month. So it's crazy. Once you do run out of those credits, you still receive a 17% discount. You do get either a company phone or a monthly stipend that'll go directly into your paycheck. We do have unlimited paid time off, which is great, as well. So, if you wanted to take a vacation, or if you are sick, wanted to work from home, super-flexible at Uber. You just have to have manager approval, and you should be good to go. We do have paid time off for the holidays-- so, Thanksgiving, if you were going-- you know, whatever holiday it may be, if you wanted to go visit your family, friends, usually those are paid time off. We do have a lot of volunteer events, as well, in the community. So, if you are an intern at Uber, we do have volunteer events throughout the year, as well. So, this past year, I think, we had about three this summer. And you can pick and choose which event interests you. So, if you are interested in applying, you can go to this link. And this link will basically show you all the positions that we do have available. So we, this year, are hiring for our software engineer-- intern and new grad. The software-engineer class seems to be the biggest of all of our teams. We are hiring for data science, new grad and intern. Data analyst, new grad and intern. Design, and then associate product manager, which is a program for new grads. So, if you guys are interested in APM, you guys can talk to them afterwards. And then, if you are interested in any of the other ones I just named, feel free and come up to me afterwards, and we can chat. Any questions? [INTERPOSING VOICES] EVAN: --we thought some Q&A. And we got Uber, new grad life, whatever-- just Q&A. DANIEL: Yeah, it could be anything. Yep? AUDIENCE: How's Pittsburgh? DANIEL: Pittsburgh was awesome. So I was out there working on ATG for six months. I like Pittsburgh a lot. Pretty cool city [LAUGH]. The roads kind of suck. [LAUGHTER] But it was awesome. It's an up-and-coming city. It was way more affordable [LAUGH] than SF, so I could actually afford a pretty nice apartment super-close to the office. The ATG office itself is also just recently expanded. So we had this one building where all the engineers and everyone sat. We just bought out this other huge building, where all the cars are, as well. So it's an exciting place to be, for sure. EVAN: Daniel and another friend of ours, Justin, they were super-sold on Pittsburgh by the end. At first, he was like, hey, I'm interested in self-driving cars, and so I'm going to do this thing. We'll see what happens. But midway through, it was like, they were-- I felt like you guys were both converts. Like, Pittsburgh is like the next. Really, it's not about San Francisco. You want to be in Pittsburgh. Yeah, yeah. AUDIENCE: [INAUDIBLE] EVAN: Yeah, yeah. So we're both in the same program, the APM or Associate Product Manager program. And basically, in terms of-- I mean, actually, it might be worth saying, like, Daniel and I actually met in an Uber, talking about, like, this program. You know, very romantic. DANIEL: Yeah. We shared a Pool together. It was a manual Pool, because it was-- it was an X. There was an UberX that we both were standing around for, that we split. I remember. EVAN: OK. So that's actually how you become an APM. No, no, no. Uh-- [LAUGHTER] In terms of the process, I'll speak for myself and Daniel for himself-- had done a few internships, over the summer, one as sort of an application engineer and the other as a product-management intern, at a startup. Definitely it just felt like I gravitated more towards the PM path. There are a bunch of APM programs, now, at a bunch of the big companies. For me, why Uber, it was, like, A, like, love to travel. And I'd wanted to hopefully, at some point in my career, do work in some of these international places. So far, that has been the case, which has been very cool. And then, in terms of last year, too, it was kind of this period where every single-- we basically accepted our roles and then, like, week after week, that we all saw, like, headlines were coming out left and right. And it was like, wow. This-- are we signing up for the right thing, here? --in all honesty. And we would have conversations about it. The past year has been a super-cool time to be at the company, because just so much is developing so quickly, with the new CEO and leadership. So, why Uber initially was, like, transportation seemed really cool. International work seemed very cool. And that's what got me into it. And then it's been a really cool place to be, over the past year. DANIEL: Yeah. I think I had a similar story. So I hadn't done any, like, PM internships in the past. All my previous experiences were in engineering. But I think, through those internships, started getting a sense that I really like this sort of PM position that was getting more and more popular, especially in the Bay Area. I think it's one where you definitely get to where a lot of hats. Right? And I've found that I personally don't sort of sitting and doing the same thing, day in, day out. And I think it's very much maybe like a personality thing and, like, what you want out of your career. I've found it very fulfilling, in that sense. What drew me specifically to Uber for PMing was I was just fascinated by the logistics sort of real-world, like, physical-world problems that Uber was solving. I think that what makes us unique, compared to a lot of other companies, is that we are a software technology company but we're moving real things in the real world, every day. And I don't think there's a lot of companies that operate in that problem space. And it's a really, really interesting one, in terms of data-science problems, logistics, dispatch, and those are things that I really enjoy. [LAUGH] AUDIENCE: [INAUDIBLE] about two areas that [INAUDIBLE].. First is, so the analysis that you guys do, especially [INAUDIBLE] end, that's a [INAUDIBLE] sensitive data. So how do you guys think about [INAUDIBLE],, but also [INAUDIBLE] do analysis and create conclusions that [INAUDIBLE]?? And then, second, that kind of line to the employees [INAUDIBLE] contractors, and how directing work starts to make [INAUDIBLE].. And when you think-- actually, especially about, like, you need to be at this restaurant at this time, [INAUDIBLE] whether you ever [INAUDIBLE]. EVAN: Yeah. Sure you want to split that? You want to take part 1? I can try and take part 2? DANIEL: Sure. [LAUGH] The first one was about how, do we do data analysis with anonymization and-- AUDIENCE: [INAUDIBLE] DANIEL: Yeah, yeah, on the policy side. So, when we do any of this analysis, it's all anonymized. Like, we have access to it, but it's basically, like, they're all data points satellite. We don't have any personal identifying information, when we do the analyses. I think one thing that Uber has started to do a better job of, and I'm actually quite proud of, is starting to use that data in conjunction with cities. I think, in the past we-- you know, we have this wealth of data, and it's pretty unique to Uber. And so there's new groups like-- I think it's called the Movement Team, right? --that's basically building out tools for cities and regulators, to understand how traffic patterns evolve in their cities. And I think that's unique to us. So we're partnering with a ton of municipalities, regulators, giving them access to this tool. I think we're piloting the technology still with a couple cities, but we want to eventually use this data to make transportation more fluid. And so I've personally seen that, in the past year. We've actually been more proactive with working with cities and providing them with this data. EVAN: And then, on the employment piece, it's super-difficult. I'm actually currently on a team right now that is the Driver Pricing team, figuring out how we pad bonuses to our driver partners. And part of the reason that I wanted to join the team in the first place was because I can answer all sorts of questions about Uber, I think, based on my first rotations on the various teams. I think there is this question that probably a lot of you, if you've talked to your Uber driver in the car before, you're like, hey, how are things going? How's this week? And there's a sense of, you know, drivers-- there's a sense of, like, how good are these opportunities? And part of the plus column is that it has this sort of essence of flexibility. You choose your own hours, and, to some extent, you are your own boss. On the other side, it's, that flexibility is sort of coming at the cost of, you don't have all these benefits that a full-time employee at a company like, you know, we at Uber have. And so, in terms of how it actually works at the company, it's basically, we have product counsel and legal in all sorts of conversations, to make sure that there isn't a feature that is, like, pushing what is the existing legal boundary. Actually, I think the way-more-interesting and cool answer is-- In Europe, where some of the regulation laws are, in some ways, more sophisticated, where the employment law isn't bifurcated into full-time and, like, non employees, there are these middle areas, Uber is currently working on sorts of, like, insurance offerings for this middle tier. So I can't really speak for the company. I think what is happening, though, is that, as the gig-economy jobs are exploding, as a real opportunity for people, and you'll start to see more and more workers stitching Uber and food delivery and all sorts of other sort of gig-economy type work, I think we're just going to get more advanced in our policy, and the definitions will change. But today it is, like, there are principles about flexibility and about a certain amount of control that drivers have to have, that we have to stick by-- and that, honestly, like, I personally want to stick by. And it's about figuring out where that grey line is. Because it's, like, basically, it's the driver, it's Uber, it's regulators, and there is some area where we all intersect and people are happy. And currently everyone's a little bit all over the place. Yeah? DANIEL: Yep. AUDIENCE: I was wondering how you guys like, localize products to all these sort of [INAUDIBLE]. EVAN: You want to talk about [INAUDIBLE] some of the safety stuff? DANIEL: Yeah, yeah, sure. So, quick plug for the APM program that we're part of. For those of you who don't know, the APM program is a new grad rotational product management role. So it's three rotations, six months each. The whole idea is, we sort of throw the APMs into this whole world of product management. You honestly learn how to learn, by going through three different teams really quickly. Usually they're vastly different. So my first one was on Uber Eats. Then I went to Pittsburgh, working on ATG. Evan has worked on Express Pool. Just really different parts of the company. And then part of the APM program is, after the first six months, we do this global trip where we-- sort of like, to your point, go around the world to really understand how our product is different in different countries and just how the sentiment of transportation-- how do people think about getting from point A to point B? So, for our year, we went to DC, Sao Paolo, Bangalore, and Singapore. So, some crazy flights, some 28-plus-hour [LAUGH] flights, in there. So it was fascinating. Like, when we got to Sao Paolo, for instance, you know, I think in the US we sort of obsess over the UI and the UX of, how do we communicate everything in the app, to the, how do we make the best app experience? But they were like, no, no no. The most important thing for us is safety. Like, being a rider or a driver is, like, that's-- you know, the country that you're in, the set of problems that you think about, are just entirely different. So, like, Sao Paolo was just like, safety is the number one priority. We need to bake that into every aspect of both the driver and rider apps. Singapore was very different. It was like, hey, like, we have no crime here. [LAUGHTER] It's like, you get fined for chewing bubble gum on the streets. So their sets of problems were, like, hey, actually, we have intense competition from both our competitors but also other forms of transportation. Like the subway system is fantastic here. Taxis are awesome. People trust taxis. How does Uber fit into this transportation ecosystem? And then Bangalore, yet again, was totally different. It was about, hey, cars don't work [LAUGH] in Bangalore. You can't get around in a four-wheel car. So we need to start thinking about apps with just, like, different vehicles, different products, that we can offer. So I think, when we went, it was the second week that, uh-- was it auto rickshaws, or-- yeah, auto rickshaws had just launched on Uber. And it was crazy, because we went out onto the streets of Bangalore and we were out there with a city apps team. And they were driving around this truck with, like, a stage. And they were literally just, like, going, like, around the city, getting drivers to drive these auto rickshaws for Uber. And it was super, like, on the ground, like, getting the work done, sort of-- just, that mentality was awesome. So, yeah. Like, every country we operate in, the sets of product problems we think about are just day-and-night different. It's like one really exciting part about Uber. EVAN: And then I'd just add-- DANIEL: I'm sure you have more to add. EVAN: Yeah, the only other thing I'd add is just that part of what enabled Uber to grow so fast was having this ops playbook. So there was, like, literally a bunch of sheets that somebody still at Uber put together on how you launch a new city for Uber. And that basically created this network of-- we call them "ops," "operations hubs," all across the globe, really, in all our cities. And the way that we figure out-- say, that, in Singapore, there are a lot of malls, and so the GPS signal is often blocked by these buildings-- is, the product-ops teams there communicate that up, through various sort of bottoms-up planning processes, and say hey, this is what we need to fix. And so it's part of what makes Uber so effective, is having that on-the-ground presence in all these cities. DANIEL: Yep. AUDIENCE: [INAUDIBLE] So I've been thinking about riding in cars with complete strangers and having conversations [INAUDIBLE]. So do you have a group that's working on those types of aspects? And what are the changes [INAUDIBLE]? EVAN: Hmm. That's a super question. In the shared-rides world, where it's still asking people to get in the car with the strangers, we're still figuring out how to get-- I mean, my parents, for instance are very-- like, they'll take an UberX. My dad is a big fan of Uber Pool. I think he just likes talking to more people-- my mom, less so. There's also, we've seen issues around, generally speaking, in these sorts of-- like, the vehicle pickup environments, men feel safer at night than a woman does, on the street. And so it's-- like, we are starting to see more and more, based on some of these data trends, how people, where they are in the sort of uptake of the various offerings. And I think part of a lot of user research is constantly happening at the company. And part of that is, like, trying to figure out exactly what you're saying-- is, how are these technologies impacting folks and how they think about basically things that we can't detect in our office-- and, you know-- uh-- AUDIENCE: So I guess what I was asking is [INAUDIBLE].. EVAN: Oh, totally, totally, yeah, yeah. AUDIENCE: So could you talk a little bit about that part? EVAN: Yeah, yeah, definitely. And so I had only mentioned it really briefly, but, for the Express Pool product improvements that we made, we spent I think it was, like, three days, back to back to back, with all sorts of different cohorts of these riders, ranging-- and so, over the course of several days. Usually, it's led by kind of a user-research manager and professional. And the product managers will sit in. But it's, like, asking people-- it's not just, like, how to use this product, it's things like, hey, you know, tell me about your commute. Tell me about the things that you feel safe about and not so safe. The point being is that a lot of the user research isn't just the very surface-level, what do you think about this feature, this button. It's truly trying to get at the needs of, how are these riders trying to move around their city? I don't know if that's clear. AUDIENCE: A little bit. I mean-- EVAN: Yeah. AUDIENCE: I'm part of a public library. We spend a lot of time on user research. And we feel like we can learn a lot from the different aspects. So I'm always curious about how corporations are able to figure it out and how can the library learn that part of things. EVAN: Yeah, I mean, like, every given day at headquarters, in a bunch of our offices, there are user research groups happening, every single day, for multiple teams. So it's become pretty embedded, in terms of just the product life cycle. DANIEL: Maybe I can add just a quick thing. So, on the Uber Eats side, I think, in terms of user research, I think there's sort of like two different buckets. One is more foundational research that Evan was mentioning. So, I may actually be going to, like, Mexico or London, in the next month or so, to just go out there and understand how people think about meal preparation, how do people think about eating, eating out, deliveries-- like, sort of a foundational body of work of just culture and how that is specific to a country. And then we also do invite people to the office. We also go out in the field, with specific, like, hey, here's new concepts or directions we want to take a product in. And then we'll have them sort of-- usually they'll be at pretty low-fidelity mock-ups of some new idea we're trying to test, and we'll just gauge feedback from those, as well. EVAN: Well, maybe one more question? If we have it? If not, I think Daniel and Allie and I are just going to be lingering around for a little bit. ALLIE: Yeah. We will hang around for about 10, 15 minutes, if you guys have questions. There is some swag in the back. But thank you guys so much for coming. We really appreciate it. And thanks to everybody online. Yeah! That concludes our presentation. DANIEL: Sweet. Thanks so much, guys, for coming. [APPLAUSE]