MIT Self-Driving Cars: State of the Art (2019)

– Today I’d like to talk about the state of the art of autonomous vehicles, how I see the landscape, how
others see the landscape, what we’re all excited about, ways to solve the problem and what to look forward to in 2019 as we also get to hear from
the different perspectives and the various leaders in
industry and autonomous vehicles in the next few, next couple
of weeks and next few days. So the problem, the mission, the dream, the thing that we’re trying to solve for many it may be about
entrepreneurial possibilities of making money and so on. But really it’s about
improving access to mobility, moving people around in the world that don’t have that ability, whether it has to do with age or purely access of where you live. We want to increase the efficiency
of how people move about. The ability to be productive
in the time we spend in traffic and transportation. One of the most hated things
in terms of stress, emotion, the thing in our lives
that if we could just with a snap of a finger remove is traffic. So the ability to convert
that into efficiency, into a productive aspect,
into a positive aspect of life and really the most important
thing at least for me and for many of us working in the space, is to save lives, prevent
crashes that lead to injuries, prevent crash that will
lead to fatalities. Here’s a counter. Every 23 seconds somebody
in the world dies in a car, auto crash. It should be a sobering, it is for me, thing that I think about every single day. You go to bed, you wake up, you work on all the deep learning levels, all the different papers are publishing, everything we’re trying
to push forward is really to save lives at the
beginning and at the end that is the main goal. So with that groundwork, with
that idea, with that base, the mission that we’re all working towards from the different ideas
and different perspectives, I would like to review
what happened in 2018. So first, Waymo has done incredible work in deploying and testing their
vehicles in various domains and having October reached the mark of 10 million miles, German autonomously which is an incredible accomplishment. It’s truly a big step for
fully autonomous vehicles in terms of deployment
and obviously is growing and growing by day. And we’ll have Drago here from Waymo to talk about their work there. Then on the L2 on the
semi-autonomous side, that’s the pair, that’s the
mirror side of this equation. The other incredible number, that’s perhaps less talked about, is the one billion mile
mark reached by Tesla in the semi autonomous
driving of autopilot. Now autopilot is a system
that’s able to control its position in the lane,
center itself in the lane, it’s able to control the
longitudinal movement so not follow a vehicle when there’s a vehicle in front and so on. But the degree of its ability to do so is the critical thing here, is the ability to do so
for many minutes at a time even hours at a time
especially on highway driving. That’s the critical thing. And the fact that they’ve
reached one billion with a B miles is an
incredible accomplishment. All of that from the machine
learning perspective is data. That’s data. And all of the autopilot models are driven with the primary sensor being a camera, that’s computer vision. Now how does computer
vision work in modern day, especially with the second
iteration of auto pilot hardware there’s a neural network. There’s a set of neural
networks behind it. That’s super exciting. That is probably the largest deployment of neural networks in the world that has a direct impact on a human life, that’s able to decide, that’s able to make
life critical decisions many times a second over and over. That’s incredible. You go from the step of image
classification on ImageNet and you sit there with a tensor flow and you’re very happy there. You were able to achieve a 99.3 accuracy with a state of the art algorithm. You take from that a step
towards there’s a human life, your parents driving, your
grandparents driving this, your children driving the system and there’s a neural
network making the decision of whether they’ll live. So that one billion mark is
an incredible accomplishment. And on the sobering side and
from various perspectives, the fatalities, there’s
been two fatalities that happened in March of 2018. One in the fully autonomous side of things with Uber in Tempe, Arizona,
hitting a pedestrian and leading to a pedestrian fatality. And on the semi-autonomous
side with Tesla Autopilot, the third fatality that
Tesla Autopilot led to and the one in 2018 is in
Mountain View, California when Tesla slammed into a divider killing his driver. Now the two aspects here that are sobering and really important to think about as we talk about the progression
of autonomous vehicles, proliferation in our world
is our response as a public, is from the general
public to the engineers to the media and so on, how we think about these fatalities. And obviously there’s a
disproportionate amount of attention given to these fatalities. And that’s something as engineers you have to also think about, that the bar is much higher on every level in terms of performance. So in order to success, as I’ll argue, in order to design successful
autonomous vehicles those vehicles will have to take risks. And when the risks don’t pan out, the public, if the public
doesn’t understand, the general problem that we’re tackling, the goal of the mission, that those risks when they don’t, the risks that are taken can have significant detrimental effect to the progress in this
autonomous vehicle space. So that’s something we
really have to think about. That’s our role as engineers and so on. Question, yeah. So the question was,
do we know the the rate of fatalities per mile of vehicle driven which is at the crudest level
how people think about safety. So there’s about 80, 90,
100 million miles driven in manually controlled
cars at every fatality. So one fatality per, depending
on which numbers you look at, it’s 80 to 100 million miles. In the Tesla vehicle, for example, the fatality is well we could
just take the one billion and divided it by three. Now this, it’s apples
and oranges in comparison and that’s something actually that we’re working on to make sure that we compare it correctly. Compare the aspects of manual miles that directly are comparable
to the autopilot miles. So Autopilot is a modern
vehicle that’s much safer. Tesla is a modern
vehicle that’s much safer than the general population
of manually driven vehicles. Autopilot is driven on only
a particular kinds of roads on the highway primarily,
most of the miles. The kinds of people that drive Autopilot, all these kinds of factors
need to be considered when you compare the two. But when you just look at the numbers, Tesla Autopilot’s three times safer than manually driven vehicles. But that’s not the
right way to look at it. And for anyone that’s ever
taken a statistics class, three fatalities is not, does not, it’s not a large number by which to make any
significant conclusions. Nevertheless, that doesn’t stop the media, the New York Times and everybody from responding to a single fatality, which PR and marketing aspects of these different companies
are very sensitive to, which is of course troubling
and concerning for an engineer that wants to save lives. But it’s something that
we have to think about. Okay, 2018 in review continued. There’s been a lot of announcements or rather actual launches of public testing of
autonomous taxi services. So companies that on public
roads have been delivering real people from one location to another. Now there’s a lot of caveats. In many of these cases
it’s very small scale, just a few vehicles, in most
cases it’s very low speed, in a constrained environment, in a constrained community and almost always, really
always, with a safety driver. There’s a few exceptions for demonstration purposes but there’s always an
actual driver in the seat. Some of the brilliant folks representing these companies will speak in this course is Voyage doing
it in an isolated community, awesome work they’re doing
in villages in Florida, Optimus Ride here in Boston doing and the community in Union Point, in Texas, May Mobility
expanding beyond Detroit but really most operation’s in Detroit, Waymo has launched its service. Waymo one that’s gotten some
publicity in Phoenix, Arizona. That Nuro doing zero occupancy deliveries of groceries autonomously. So we didn’t say has to
be delivering humans, it’s delivering groceries autonomously. Uber is quietly, or not so quietly, resumed its autonomous
vehicle taxi service testing in Pittsburgh in a very
careful constrained way. Aptiv, after acquiring
Carl Iagnemma and nuTonomy, has been doing extensive
large-scaled taxi service testing everywhere from Vegas to
Boston here to Pittsburgh and in Singapore of course. Aurora that spoke here last time, the head of Tesla
Autopilot launched Aurora and the Chris Urmson behind
this young upstart company is doing testing in San
Francisco and Pittsburgh and then Cruise, Kyle will
be here to talk from GM, is doing testing in San
Francisco, Arizona and Michigan. So when we talk about predictions, I’ll talk about a few people predicting when we’re going to
have autonomous vehicles and when you yourself
think about what it means when will they be here? When will autonomous vehicles arise such that that Uber that
you call will be autonomous and not with a populated by a driver. So the thing we have to think about is what we think about what,
how we define autonomous, what that experience looks like. And most importantly in these discussions, we have to think about scale. So we here at MIT our group MIT Human
Centered Autonomous Vehicle, we have a fully autonomous
vehicle that people can get in if you would like and
it will give you a ride in a particular location. But that’s one vehicle, it’s not a service and it only works on particular roads. It’s extremely constrained. In some ways it’s not much different than most of the companies that
we were talking about today. Now scale here, there’s a magic number, I’m not sure what it is but for this, the purpose of this conversation
let’s say it’s 10,000, where there’s a meaningful deployment, when it’s truly going beyond
that prototype demo mode to where everything is under control, to where it’s really touching
the general population in a fundamental way. Scale is everything here and it starts, let’s say at 10,000. Just to give you for reference, there’s 46,000 active Uber
drivers in New York City. So that’s what 10,000 feels like some, you know 25, 30 % of the
Uber drivers in New York City all of a sudden are become passengers. So the predictions, I’m
not a marketing PR person, so I don’t understand what everybody has to have make a prediction
but they all seem to. Although major automakers
have made a prediction of when they’ll have a deploy, when they will be able to
deploy autonomous vehicles. Tesla has made in early 2017, a prediction that it will
have autonomous vehicles 2018. In 2018 they’ve now adjusted
the prediction to 2019. Nissan, Honda, Toyota have
made prediction for 2020 under certain constraints
in highway urban. Hyundai and Volvo has in 2021. BMW and Ford, Ford saying at scale, so a large scale deployment 2021. And Chrysler in ’21 and Daimler
saying in the early ’20s. So there is the the predictions that are extremely optimistic
that are perhaps driven by the instinct that the
company has to declare that they’re at the
cutting edge of innovation. And then there is many
of the leading engineers behind the leading these
teams including Carl Iagnemma and Gill Pratt from MIT
who in injects a little bit of caution and grounded ideas about how difficult it is to remove the human from
the loop of automation. So Carl says that basically teleoperation, kind of gives this analogy of an elevator and the elevators fully autonomous but there is still a
button to call for help if something happens. And that’s how he thinks
about autonomous vehicles. Even with greater and
greater degree of automation, they’re still going to have
to be a human in the loop, they’re still going to be a way to contact a human to get help. And Gill Pratt and Toyota and they’re making some
announcements at CES, basically saying that the human in the loop is the fundamental aspect that we need to approach this
problem and removing the human from consideration is
really, really far away. And Gill, who’s historically
and currently is one of the sort of the great
roboticists in the world that defined a lot of the DARPA challenges and a lot of our progress
historically speaking up to this point. So they’re really the full spectrum, we can think of it as
the Elon Rodney spectrum of optimism versus pessimism. Elon Musk, who’s extremely
bold and optimistic about his predictions. I often connect with this kind of thinking because sometimes you have to believe the impossible is possible
in order to make it happen. And then there is Rodney, also one of the great roboticists, the former head of the of
CSAIL, the AI laboratory here, is a little bit on the pessimistic side. So for Elon, now fully autonomous
vehicle will be here in 2019 for Rodney the vehicles are really, fully autonomous are beyond 2050. But there, he believes in the ’30s there will be a significant,
a major city will be able to allocate a significant
region of that city where manual driving is fully banned. Which is the way he believes those vehicles could, autonomous
vehicle really proliferate when you ban manually driven
vehicles in certain parts. And then in the ’40s, 2045 or beyond, majority of U.S cities will
ban manually driven vehicles. Of course the quote from
Elon Musk in 2017 is that, my guess is that in probably 10 years it will be very unusual
for cars to be built that are not fully autonomous. So we also have to think about
the long tail of the fact that many people drive cars that are 10 years old, 20 years old. So even when you have every
car’s built as fully autonomous, it’s still gonna take time for that dissipation
of vehicles to happen. And so my own view beyond predictions, to take a little pause into the ridiculous and the fun to explain the view. Yes that is me playing guitar
in our autonomous vehicle. Now the point of this ridiculous
video and embarrassing, I should’ve never played it. Yeah, okay, I think
it’s gonna be over soon. Now for those of you born in
the ’90s that’s classic rock. (audience laughing) So the point I’m trying to
make beyond predictions is that autonomous vehicles will
not be adopted by human beings in the near term, in the next 10-15 years, because they’re safer. Safety is not going to, they
may be safer but that is, they’re not going to be so much safer that that’s going to be the reason you adopt. It’s not gonna be because they get you to the location faster. Everything we see with autonomy is they’re going to be slower until majority of the fleet is autonomous. They’re cautious and therefore slower and therefore more annoying
in the way we think about actually how we navigate this world. We take risk, we drive
assertively with speed over the speed limit all the time. That is not how autonomous
vehicles today operate. So they’re not gonna get us there faster and for every promise, every hope that they’re
going to be cheaper really there’s still significant
investment going into them and there is not good
economics in the near term of how to make them obviously
significantly cheaper. What I think Uber and Lyft has
taken over the taxi service because of the human experience. In the same way autonomy
will only take over if, not take over but be
adopted by human beings if it creates a better human experience. If there’s something about the experience that you enjoy the heck out of. This video and many others
that we’re putting out, shows that in the natural
language communication, the interaction with the car, the ability of the car to
sense everything you’re doing from the activity of the driver
to the driver’s attention and being able to transfer
control back and forth in a playful way but
really in a serious way also that’s personalized to you. That’s really the human experience, the efficiency of the human experience, the richness of the human experience, that is what we need to also solve. That’s something you have to think about because many of the people, that’ll be speaking at this class and many of the people that are working on this problem are not focused
on the human experience. It’s a kind of afterthought that once we solve the
autonomous vehicle problem it’ll be fun as hell to be in that car. I believe you first have to make it fun as hell to be in the car and then solve the autonomous
vehicle problem jointly. So in the language that
we’re talking about here there are several levels of
autonomy that are defined from level zero to level four. Level zero no automation, four and five, level three, four and five
increasing automation. So level two is when the
driver is still responsible, level three, four, five is when there’s less and less responsibility. But really in three, four, five, there’s parts of the driving where the liability’s on the car. So there’s only really two, as
far as I’m concerned, levels, human center autonomy and full autonomy. Human centered means the
human is responsible. Full autonomy means the car is responsible both on the legal side,
the experience side and the algorithm side. That means full autonomy does
not allow for teleoperation. So it doesn’t allow for
the human to step in and remotely control the vehicle because that means the
human is still in the loop. It doesn’t allow for the 10 second rule that it’s gonna be fully autonomous but once it starts warning you, you have 10 seconds to take over. No, it’s not fully autonomous if it cannot guarantee
safety in any situation. It has to be able to, if the driver doesn’t
respond in 10 seconds it has to be able to find safe harbor. It has to be able to pull
off to the side of the road without hurting anybody
else to find safety. So that’s the fully autonomous challenge. And so how do we envision these two levels of automation proliferating society, getting deployed at a mass scale? The 10,000, 10 million beyond. On the fully autonomous side, the way to think about
it with the predictions that we’re talking about here, is there’s several different possibilities of how to deploy these vehicles. One is last mile delivery
of goods and services like the groceries. These are zero occupancy
vehicles delivering groceries or delivering human
beings at the last mile. What the last mile means is
it’s slow-moving transport to the destination where
most of the tricky driving along the way is done manually and then the last mile
delivery in the city in the urban environment is done by zero occupancy autonomous vehicles. Trucking on the highway,
possibly with platooning, where a sequence of
trucks follow each other. So in this what people think about it as a pretty well-defined
problem of highway driving with lanes well marked, well mapped routes throughout the United States and globally on the highway driving is automatable. The specific urban routes kind of like what a lot of the these
companies are working on, defining this taxi service and personalized public transport. There’s certain pickup locations
you’re allowed to go to, there are certain drop-off
locations and that’s it. It’s kind of like taking the train here but as opposed to getting on the train with 100 other people
you’re getting or bus, you’re getting on the car with, when you’re alone or
with one other person. The closed communities, something Oliver Cameron
with Voyage is working on defining and Optimus Ride, defining a particular community that you now have a monopoly over that you define the constraints, you define the customer base and then you just deliver the vehicles. You map the entire road, you
have slow-moving transport that gets people from A to B
anywhere in that community. And then there’s the world of zero occupancy ride-sharing delivery. So the Uber that comes to you as opposed to having you drive it yourself and it comes to you autonomously
with nobody in there and then you get in and drive it. So imagine a world where we have empty
vehicles driving around, delivering themselves to you. Semi-autonomous side is
thinking about a world where teleoperation plays
a really crucial role where it’s fully autonomous
under certain constraints on the highway but a
human can always step in. High autonomy on the highway kind of like what Tesla is working
towards most recently, it’s on-ramp to off-ramp. Now the driver is still
responsible, liability wise and in terms of just observing the vehicle and algorithmically speaking but the autonomy is pretty high level to a point where much of the highway driving could
be done fully autonomously. And low autonomy unrestricted travel as an advanced driver assistance system, meaning that the car
kind of like the Tesla, the Volvo S90s or the Super
Cruise and the Cadillacs all these kinds of L2
systems that are able to keep you in the lane, you know 10 to 30% of
the miles that you drive and some fraction of the time take some of the stress of driving off. And then there is some
out there ideas, right. The idea of connected vehicles, vehicle to vehicle communication and vehicle to infrastructure
communication enabling us to navigate, for example,
intersection efficiently without stopping, removing
all traffic lights. So here shown on the bottom
is our conventional approach of there’s a queuing system that forms because of traffic lights
that turn red, green, yellow and without traffic lights
and with communication to the infrastructure
in between the vehicles you can actually optimize that to significantly increase the
traffic load through a city. Of course there’s the boring solution of tunnels under cities, layers of tunnels under cities. Tunnels all the way down. Autonomous vehicles basically by the design of the tunnel, constraining the problem to such a degree that an, I mean the idea of autonomy just is completely transformed. That you’re basically, a car
is able to transform itself into a mini train, into a
mini public transit entity, for a particular period of time. So you get into that tunnel, you drive at 200 miles an hour and or not necessarily drive,
be driven 200 miles an hour and then you get out of the tunnel. Of course there’s the flying cars, personalized flying car vehicles. I will not, I mean, Rodney as I mentioned before, does believe that we’ll have them in 2050. There’s a lot of people that
are seriously actually thinking about this problem is
there’s a level of autonomy obviously that’s required
here for a regular person. I don’t know somebody
without a pilot’s license, for example, to be able
to take off and land. Making that experience accessible
to regular people means that there’s going to
be a significant amount of autonomy involved. One of the people really, one of the companies really
seriously working on this, is Uber with the Uber Elevate,
Uber Air I think it’s called and the idea is that you
would meet your vehicle not on the street but at a roof, you take it elevator, you meet them at the roof of a building. This video’s from Uber. They’re seriously addressing this problem. Many of the great solutions to the world’s problems
have been laughed at at some point. So let’s not laugh too loud
at these possibilities. Back in my day we used
to drive in the street. Okay so aha, 10,000 vehicles, if that’s the bar. I sort of out of curiosity asked, did a little public poll. 3,000 people responded. Asked who will be first to deploy 10,000 fully
autonomous cars operating on public roads without a safety driver. And several options percolated
with Tesla getting 57% of the vote and Waymo
gaining 21% of the vote and 14% someone else
and 8% the curmudgeons and the engineers saying no one in the next 50 years will do it. And again in 1998 when Google came along, the leaders of the space were Ask Jeeves and Infoseek and Excite,
all services I’ve used and probably some people
in this room have used, Lycos, Yahoo, obviously they
were the leaders in the space and Google disrupted
that space completely. So this poll shows the current leaders but it’s wide open to ideas
and that’s why there’s a lot of autonomous vehicle companies. Some companies are taking
advantage of the hype and the fact that there’s a
lot of investment in the space but some companies, like
some of the speakers visiting in this course are really
trying to solve this problem. They want to be the next Google, the next billion, multi-billion, next trillion dollar company
by solving the problem. So it’s wide open. But currently Tesla with a human, with the semi-autonomous
vehicle approach working towards trying to become fully autonomous. And Waymo starting with the
fully autonomous working towards achieving scale
at the fully autonomous are the leaders in the space. Given that, ranking in 2019, let’s take a quick step back to 2005 with the DARPA challenge
when the story began. The race to the desert when
Stanley from Stanford won a race through the desert that really captivated
people’s imagination about what’s possible. And a lot of people have said that the autonomous vehicle
problem is solved in 2005. They really said you know
the idea was especially because in 2004 nobody finished that race, 2005 four cars finished the race, it was like well we cracked it. This is it. And then you know some critics said that urban driving is
really nothing comparable to desert driving, desert is very simple there’s no obstacles and so on. It’s really a mechanical
engineering problem it’s not a software problem. It’s not a fundamentally, it’s not really an
autonomous driving problem as it would be delivered to consumers and of course in 2007, DARPA put together Urban Grand Challenge and several people finished
that with CMU’s boss winning. And so the thought was at that point, that’s it, we’re done. As Ernest Rutherford, a physicist, said, that physics is the only real science, the rest is just stamp collecting, all the biology, chemistry. Certainly, oh boy, I wouldn’t want to know what he thinks about computer science. It’s just all this stupid silly details Physics is the fundamentals. And that was the idea with the DARPA Grand
Challenge and solving that that we solved the fundamental
problem of autonomy. And the rest is just for
industry to figure out some of the details of how to make an app and make a business out of it. So that could be true. And the underlying beliefs there is that driving is an easy
task, that it’s solvable. The thing that we do as human beings that it’s pretty formalizable it’s pretty easy to solve with autonomy that the other idea is that
humans are bad at driving. This is a common belief. Not me, not you but everybody else, nobody in this room but everybody
else is a terrible driver. The kind of intuition that we have about our experience of
traffic leads us to believe that humans are just
really bad at driving. And from the human
factors, psychology side, there’s been over 70 years of research showing
that humans are not able to monitor, maintain
vigilance, monitoring a system. So when you put a human
in a room with a robot and say watch that robot, they start texting like 15 seconds in. So that’s the fundamental psychology. There’s thousands of papers on this. People are, they tune out,
they over trust the system, they misinterpret the system
and they lose vigilance. Those are the three underlying beliefs. It very well could be true
but what if it is not? So we have to consider that it is not. The driving task is easy because if you think
the driving task is easy and formalizable and solvable
by autonomous vehicles, you have to solve this problem. The subtle vehicle-to-vehicle, vehicles-to-pedestrian
nonverbal communication that happens here in a dramatic sense but really happens in the subtle sense millions of times every
single day in Boston. Subtle nonverbal communication
between vehicles, you go, no, you go. You have to solve all
the crazy road conditions where in a split seconds
you have to make a decision about, so in snowy, icy weather, rain, limited visibility conditions, you have 100, 200 milliseconds
to make a decision. Your algorithm based on the perception has to make a control decision. And then you have to deal with a nonverbal communication
with pedestrians, these unreasonable irrational
creatures, us human beings. You have to not only understand what the intent of the
movement that’s anticipated. So anticipating the
trajectory of the pedestrian you also have to assert
yourself in a game theoretic way as crazy as it might sound,
you have to threaten yourself, you have to take a risk. You have to take a risk that if I don’t slow down like that
ambulance didn’t slow down that the pedestrian will slow down. Algorithmically we’re afraid to do that. The idea that a pedestrian that’s moving, we anticipate their trajectory
based on the simple physics of the current velocity of the momentum, they’re gonna keep going
with some probability. The fact that by us accelerating we might make that pedestrian stop, it’s something that we have
to incorporate into algorithms and we don’t today. And we don’t know how to really. So if driving is easy we
have to solve that too. And of course the thing I showed yesterday with the coast runners
and the boat going around and all the ethical dilemmas
from the moral machine to the more serious engineering aspects that from the unintended consequences that arise from having to
formalize the objective function under which a planning algorithm operates. If there’s any learning
that, as I showed yesterday, a boat on the left run by a
human wants to finish the race, the boat on the right figures out that it doesn’t have to finish the race, it can pick up turbos along the way and gets much more reward. So if the objective function
is to maximize the reward, you can slam into the wall
over and over and over again and that’s actually the
way to optimize the reward. And those are the unintended
consequences of an algorithm that has to be formalizable
to the objective function without a human in the loop. Humans are bad at driving. As I showed yesterday, humans if they’re bad at anything it’s about having a good intuition about what’s hard and what’s easy. The fact that we have 540
million years worth of data on our visual perception system means we don’t understand how
damn impressive it is to be able to perceive
and understand the scene in a split second, maintain context, maintain an understanding of performing all the visual localization tasks about anticipating the physics
of the scene and so on. And then there’s a control side. The humans don’t give
enough credit to ourselves. We’re incredible, state-of-the-art
soccer player on the left (audience laughing) and the state-of-the-art
robot on the right. I think there’s like four
or five times he scores, (audience laughing)
all right. And this is all the movement
and so on involved with that, of course here that’s the human robot, that’s a really incredible work that’s done for the
DARPA Robotics Challenge with the humanoid robots on
the right and incredible work by the human people doing
the same kind of tasks much more impressive task I would say. So that’s where we stand. And the ones on the right are
actually not fully autonomous, there’s still some human in the loop. There’s just noisy broken communication. So that, humans are incredible in terms of our ability
to understand the world and in terms of our ability
to act in that world. And the fact that humans,
the idea, the view, the popular view grounded in
the psychology that humans and automations don’t
mix well, over trust, misunderstanding, loss of
vigilance, the command and so on, that’s not an obvious fact. It happens a lot in the lab. Most of the experiments
are actually in the lab. This is the difference. You put, many of you, you
put a undergrad, grad student in a lab and say here watch this screen and wait for the dot to appear. They’ll tune out immediately
but when it’s your life and you’re on the road,
it’s just you in the car, it’s a different experience. It’s not completely obvious
that vigilance will be lost and it’s not a complete, when
it’s just you and the robot, it’s not completely obvious
what the psychology, what the attentional mechanism, with the vigilance that it looks like. So one of the things we did, is we instrumented here 22
Tesla’s and observed people now over a period of two
years of what they actually do when they’re driving autopilot,
driving these systems. In red shown manually controlled vehicles and cyan showed vehicle control autopilot. Now there’s a lot of details here and we have a lot of presentations on this but really, the fundamentals are, is that they drive 34%, large percentage of the miles in autopilot and in 26,000 moments
of transfer of control they are always vigilant. There’s not a moment once in this data set where they respond too late
to a critical situation, to a challenging role situation. Now the data set, 22 vehicles, that’s a 0.1% or less
than the full Tesla fleet that has autopilot. But it’s still an inkling. It’s not obvious that it’s
not possible to build a system that works together with a human being and that system essentially
looks like this. Some percentage, 90%,
maybe less, maybe more, when it can solve the
problem of autonomous driving it solves it and when he needs
human help it asks for help. That’s the trade-off, that’s the balance. On the fully autonomous side, on the right it has to
solve here with citations and there’s references
always on the bottom. All the problems have to be
solved exceptionally, perfectly, from mapping localization
to the scene perception to control to planning to
being able to find safe harbor at any moment to also being able to do external HMI communication
with the other pedestrians, the vehicles in the scene and then there’s teleoperation, vehicle-to-vehicle, vehicle-to-AI. You have to solve those perfectly if you want to solve the
fully autonomous problem, as I said including all the crazy things that happen in driving. And if you approach the
shared autonomy side, the semi-autonomous where
you’re only responsible for a large percentage but
not 100% of the driving then you have to solve the human side, the human interaction, the
sensing what the driver is doing, the collaborating
communicating with the driver and the personalization aspect
that learns with the driver. As I said you can go online,
we have a lot of demonstrations of these kinds of ideas. But the natural language,
the communication, I think is critical for all of us as we’re tweeting as all of us do. (people chattering) So it’s as simple as, so this is just demonstration
of Eco taking control when the attention over time, that the driver is being, okay, we got it thank you. Okay so basically a smartphone use which has gone up year by year and we’re doing a lot of analysis on that, it’s really what people do in the car is they use their phone, whether it’s manual or autonomous driving or semi-autonomous driving. So being able to manage that,
to communicate with the driver about when they should be paying attention which may not be always. You’re sort of balancing the
time when it’s a critical time to pay attention when it’s not and communicating effectively,
learning with the driver, that problem is a fundamental
machine learning problem. There’s a lot of data visible light, everything about the driver
and it’s a psychology problem. So we have data, we have
complicated human beings and it’s a human robot interaction problem that deserves solving. But as you’ll hear on the beyond
the human side looking out into the world, people that are trying to solve the fully autonomous vehicle it’s really a two approach consideration. One approach is vision, cameras
and deep learning, right. Collect a huge amount of data. So cameras have this aspect that they’re the highest resolution of information available. It’s rich texture information and there’s a lot of it which is exactly what you know networks love right. So to be able to cover
all the crazy edge cases, the vision data, camera
data, visible light data, is exactly the kind of data you need to collect a huge amount of,
to be able to generalize over all the crazy countless
edge cases that happen. It’s also feasible, all
the major data sets, all the, in terms of
cost, interest, scale, all the major data sets
of visible light cameras. That’s another pro and they’re cheap and the world as it happens, whoever designed the simulation
that we’re all living in, made it such that our world,
our roads and our world, is designed for human eyes. Eyes is the way we perceive the world and so the lane mark is also on is visual, most of the road textures that you use to navigate, to drive are
visible, are made for human eyes. The cons are that without a ton of data and we don’t know how
much, they’re not accurate. You make errors because
driving is ultimately about 99.99999% accuracy
and so that’s what I mean by not accurate. It’s really difficult to reach that level. And then the second approach is LIDAR, taking a very particular
constrained set of roads, mapping the heck out of them, understanding them fully under different weather
conditions and so on and then using the most
accurate sensors available. A suite one sensors but
really LIDAR at the forefront. Being able to localize
yourself effectively. The pros there that it’s consistent, especially when machine
learning is not involved, it’s consistent and reliable
and it’s explainable. If it fails, you can understand why, you can account for those situations. It’s not so much true for
machine learning methods. It’s not so much explainable why it failed in a particular situation. The accuracy is higher
as we’ll talk about. The cons of LIDAR is that it’s expensive and most of the approaches in perceiving the world
using LIDAR primarily are not deep learning based and therefore they’re
not learning over time. And if they were deep learning based, there’s a reason they’re not, it’s ’cause you need a lot of car, you gonna need a lot of LIDAR data. And there’s only a tiny
percentage of cars in the world quite obviously are equipped with LIDAR in order to collect that data. So quickly running through the sensors. Radar is, it’s kind of like
the offensive line of football. They’re actually the
ones that do all the work and they never get the credit. So radar is that. It’s always behind to catch, to actually do the detection
in terms of obstacle, the most critical safety
critical obstacle avoidance. It’s cheap, it does extremely well and it does well in extreme weather but it’s low resolution so
it cannot stand on its own to achieve any kind of
degree of high autonomy. Now on the LIDAR side it’s expensive, it’s extremely accurate depth information, 3D cloud, point cloud information. Its resolution is much higher than radar but still lower than visible light and there is depending on the sensor, a 360 degree visibility that’s built in. So there’s a difference
in resolution here, visualized LIDAR on the
right, radar on the left. The resolution is just much
higher and is improving and the cost is going down and so on. Now on the camera side, it’s
cheap, everybody got one, the resolution is extremely high in terms of the amount of
information transferred per frame and everybody you know really the scale of the number of vehicles that have this equipped is humongous. So it’s ripe for application
of deep learning. And the challenge is it’s noisy, it’s bad at depth estimation and it’s not good in extreme weather. So if we kind use this plot to look, to compare these sensors, to compare these different approaches. So LIDAR works in the dark, variable lighting conditions,
has pretty good resolution, has pretty good range but it’s expensive, it’s huge, and it doesn’t provide rich
textural contrast information and it’s also sensitive to
fog and rain conditions. Now ultrasonic sensors catch
a lot of those problems. They’re better at detecting proximity, they’re high resolution
in objects that are close which is why they’re
often used for parking but they can still also be integrated in the sensor fusion package for an autonomous vehicle. They really catch a lot of
the problems that radar has. They complement each other well and radar, cheap, tiny, detect speed and has pretty good range but has terrible resolution. There’s very little
information being provided. And then cameras a lot
of rich information, they’re are cheap, their
small range is great, the best range actually of all the sensors and works in bright conditions but doesn’t work in the
dark, in extreme conditions and it’s just susceptible to
all these kinds of problems and doesn’t detect speed unless you do some tricky structure from motion kind of things. So here’s where sense of fusion steps in and you, everybody works together to build an entire picture. That’s how this plot works. You can stack it on top of each other. So if you look at a suite that
for example Tesla is using which is ultrasonic radar and camera and you compare it to just LIDAR and see how these paths compare that actually the suite of camera, radar and ultrasonic are comparable to LIDAR. So that those are the two
comparisons that we have. You have the costly non
machine-learning way of LIDAR and you have the cheap but needs a lot of data and is
not explainable and reliable in the near-term vision based approach. And those are the two
competing approaches. Now of course huevos will
talk about they’re trying to use both but ultimately the question is who catches, who is the fail safe? In the semi-autonomous way when there’s a camera based method, the human is the fail safe. When you say, oh crap I
don’t know what to do, the human catches. In the fully autonomous mode, so what Waymo’s working on and others, the fail safe is LIDAR, the fail safe is maps that
you can’t rely on the human. But you know this road so well that if the camera is freaked out if there’s any of the sensors freaked out that you’re able to,
you have such good maps, you have such good accurate sensors, that the fundamental problem
of obstacle avoidance which is what safety is
about, can be solved. The question is what kind
of experience that creates. In the meantime as the people debate, try to make money, start companies, there’s just lots of data. Ford F-150 still the most
popular car in America. Manually driven cars are still happening. So there’s a lot of data happening. Semi-autonomous cars, every
company is now releasing more and more semi-autonomous technology. So that’s all data. And what that boils
down to is the two paths they’re walking towards
is vision versus LIDAR, L2 versus L4, semi-autonomous
versus fully autonomous. Tesla on the semi-autonomous
front has reached one billion miles. Waymo the leader on the
autonomous front has reached 10 million miles. The pros and cons as I’ve outlined them. One, division one, the one I’m
obviously very excited about and machine learning
researchers excited are about which fundamentally relies on
huge data and deep learning. The neural networks that
are running inside the Tesla and with their new as they, it’s kind of the same kind
of path as Google was taking from the GPU to the TPU, Tesla’s taking from
Nvidia Drive PX2 system, sort of more general GPU based system to creating their own ASIC and having a ton of awesome
neural networks running on their car. That kind of path, that others
are beginning to embrace, is really interesting to think about for machine learning engineers. And then people that
are maybe more grounded and actually wanna, are really,
value, safety, reliability and sort of from the automotive
world, are thinking well we need machine learning
is not explainable it’s difficult to work
with, it’s not reliable and so in that sense we
have to have a sensor suite that are extremely reliable. Those are the two paths. Yep, question. The question is there’s all
kinds of things you need to perceive, stop signs
and traffic lights, pedestrians and so on. Some of them, if you
hit them it’s a problem, some of them are a bag
flying through the air and all have different
visual characteristics all have different characteristics for all the different sensors. So LIDAR can detect of solid-body objects, camera is better at detecting, as last year Sasha Arnu talked about, I think fog or smoke. These are interesting things. They might look like an object to certain sensors and not to others, But the traffic light
detection problem luckily is with cameras is, it’s
pretty solved at this point. So that’s luckily the easy part. The hard part is when
you have a green light and there’s a drunk, drugged,
drowsy or distracted, the four Ds that hits an online
pedestrian trying to cross what to do. That’s the hard part. So the road ahead for us as engineers, the science is the thing I’m super excited about the possibility of artificial intelligence
having a huge impact, is taking the step from having these even if they’re large, toy datasets, toy problems, toy benchmarks
of ImageNet classification in cocoa, all the exciting deep RL stuff that we’ll talk about in the future weeks, really are toy examples, the
game of go and chess and so on. But taking those algorithms
and putting them in cars where they can save people’s lives and they actually directly touch and impact our entire civilization that’s actually the defining problem for artificial intelligence
in the 21st century is AI that touches people in a real way and I think cars, autonomous vehicles, is one of the big ways that that happens. We get to deal with the
psychology, the philosophy, the sociology aspects
of it, how we associate, think about it, to the robotics problem, to the perception problem. It’s a fascinating space to explore and we have many guest speakers exploring that different ways and that’s really exciting to see how these people are
trying to change the world. So with that I’d like
to thank you very much, go to and the code is always available online. (people clapping)

Tagged , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *