How Safe Are Self-Driving Cars After the First Fatal Accident?

[gleem]

And if he was paying attention to a possible collision too. How long can you expect a person to remain in that state of readiness unless he is actually engaged in driving the car.

 

[dipole]

I feel bad for the woman, but I think she was clearly at fault here.

Though I wonder how much the automated car relies on visual cues. Seems like radar or sonar systems should be able to avoid those kind of scenarios

 

[russ_watters]

About an hour ago, the Tempe police department released a video of the accident:

Yikes. It seems initial reports were wrong: neither the car nor the standby driver were paying attention at the time of the accident, and they could/should have prevented it.

[and I'm shocked that this was released]

If the standby driver had had his foot on the break he might I say "might" have kept the car from killing her.

And looking up at the road instead of into the cockpit.

I feel bad for the woman, but I think she was clearly at fault here.

Though I wonder how much the automated car relies on visual cues. Seems like radar or sonar systems should be able to avoid those kind of scenarios

I too would have expected the sensors to detect this and the car to be able to avoid this -- assuming it had either sonar or radar. Note though that the woman was crossing at the worst possible spot: the very edge of a street light's coverage (on the wrong side of it). That makes cameras less effective... if this car used cameras for that purpose. But the driver - had he been looking - should still have been able to see her, since our eyes are much more sensitive to high dynamic ranges.

 

[OmCheeto]

About an hour ago, the Tempe police department released a video of the accident:
...

Late entry:

Tempe Police Department; "Tempe Police Vehicular Crimes Unit is actively investigating the details of this incident that occurred on March 18th. We will provide updated information regarding the investigation once it is available."​

Btw, I was tipped off to the video by Stephen King.

Stephen King; "The pedestrian fatality in Tempe, Ariz. is very sad, but the immediate assumption that the self-driving car was at fault--an assumption that now seems erroneous--is fascinating. Food for sociologists and psychologists."
4:07 PM - 21 Mar 2018 [PDT]​

Stephen has written at least one book about autonomous homicidal cars, before there were autonomous cars.

 

[Vanadium 50]

According to news, "jaywalking."

Even so, I don't think Jaywalking deserves the death penalty.

I have a fundamental moral objection to beta testing technology that can kill you on the public

There's billions at stake. You can't make an omelet without breaking a few eggs. History suggests that the people who run and ran Uber - can I call them the Ubermenschen? - are not the sort to let doing what's right stand in the way of making a buck.

Presumably they have a group to do risk management. I'd love to be able to find out the answers to these and similar questions:

• Was the probability of a fatality calculated? If not, why not?
• What is the acceptable fatality rate?
• Was there any management pressure for Risk to calculate an arbitrarily low accident/fatality rate?
  
• As accidents have occurred, such as the one a year ago in Tempe, how is the model's predictive power compared with data? Is the model adjusted in light of new information? If so, by whom? Who decides whether the new risks are acceptable?

 

[gleem]

What is not apparent in the video is if the car was braking at the time of the impact. If not then there is a problem. The sensors should have picked up an object moving into the path of the car.

 

[berkeman]

Wonder if it was a suicide...

From this video, it does appear so to me. She could see the approaching lights of the car. She was not in the lights of the car until the last second. I wonder if she left a note...

http://www.latimes.com/business/la-fi-uber-death-video-20180321-story.html

 

[gleem]

By my estimate from the video it seems the car was less than 100 feet from the time that the person was visually apparent. I do not know the speed limit but it has two lanes in one direction so 55 mph is not unreasonable. From the Government Fleet website the car would have gone about 80 ft before the brakes where applied for an average driver. So the car might not have slowed down too much before impact. Considering the sensors should have picked up the person before visual sighting and the fact that the reaction time for the computer is essentially zero it is likely that the car still would have impacted the person at a significant speed.

 

[nsaspook]

The poor woman on the bike looked like a homeless person doing the standard look down, ignore the world walk. I'm unimpressed by the automated vehicle but IMO, 100% on the bike lady for what happened.

That said:
There is a nice paved walkway that just invites (Yes, there is a sign that say's don't so the person didn't have right of way) people to cross there instead of at the intersection.
https://www.google.com/maps/@33.436...4!1sseIHdIkV5FyYzACTGokvBg!2e0!7i13312!8i6656
https://www.google.com/maps/@33.436...4!1sE4V6_Cyyoq-K3JKD56PPyA!2e0!7i13312!8i6656
https://www.google.com/maps/@33.436...4!1svgBgLp4aEODKKcwDOGl-tg!2e0!7i13312!8i6656

 

[rootone]

From this video, it does appear so to me. http://www.latimes.com/business/la-fi-uber-death-video-20180321-story.html

She appears to walking with a bicycle (however carelessly).
A suicide attempt would not require a bicyle

 

[russ_watters]

Presumably they have a group to do risk management. I'd love to be able to find out the answers to these and similar questions:

• Was the probability of a fatality calculated? If not, why not?

A: "Let's just do whatever we want and let the lawyers sort it out - it's always worked before."

 

[russ_watters]

By my estimate from the video it seems the car was less than 100 feet from the time that the person was visually apparent.

"Visually apparent" to us in the video does not necessarily correspond with "visually apparent" to the driver or a radar/sonar/lidar on a car.

I do not know the speed limit but it has two lanes in one direction so 55 mph is not unreasonable.

38 in a 35 zone.

 

[nsaspook]

"Visually apparent" to us in the video does not necessarily correspond with "visually apparent" to the driver or a radar/sonar/lidar on a car.

38 in a 35 zone.

Unless it was recently changed the limit was 45 as marked.
https://www.google.com/maps/@33.435...4!1sx-K4_17J8MVthFRapvIa2A!2e0!7i13312!8i6656

 

[russ_watters]

Unless it was recently changed the limit was 45 as marked.
https://www.google.com/maps/@33.435...4!1sx-K4_17J8MVthFRapvIa2A!2e0!7i13312!8i6656

Not sure where that is in relation to the accident, but a source was provided on page 1 saying 38 in a 35.

 

[OmCheeto]

Odd. The northbound lanes of Mill Avenue speed limit is 45 mph, while the southbound lanes are set at 35 mph.
Perhaps that's where some of the confusion is coming from.
The vehicle was traveling north.

 

[Borg]

A: "Let's just do whatever we want and let the lawyers sort it out - it's always worked before."

Ironically, this seems to also be the attitude of pedestrians who don't pay attention when they cross the street.

https://www.google.com/maps/@33.436...4!1svgBgLp4aEODKKcwDOGl-tg!2e0!7i13312!8i6656

If you look down the road on this one, you can see two pedestrians crossing the road in front of oncoming traffic (that had to break for them) with no crosswalk or sidewalk where they crossed even though there is a much wider sidewalk on the side that they were originally on.
https://www.google.com/maps/@33.436...4!1slC6iG7fn3TmsKGAC65MJnA!2e0!7i13312!8i6656

 

[nsaspook]

Not sure where that is in relation to the accident, but a source was provided on page 1 saying 38 in a 35.

38 in a 35 makes it sound like the car was over-driving the conditions and that could reduce normal reactions times to below what was safe for the road.

 

[Ryan_m_b]

Watching the video it looks like the headlights were not at the appropriate angle or intensity. The pedestrian didn't just step out from behind a blind spot or corner yet was only visible in the very last moments.

 

[HAYAO]

At least from a perception perspective that's all true, but I don't necessarily agree that it's right. An agreeing example to highlight why I disagree:

I just performed a risk analysis of an industrial campus electrical system in Puerto Rico. Failures are scored according to severity and likelihood, with full power outages for more than a day getting the same, maximum score. Below a day, outages or voltage dips ranging from miliseconds (can often be plowed-through) to seconds (it takes 30 seconds for a generator to start) to minutes (it can take a few minutes to reset a tripped circuit breaker) have different severities.

Hurricane Maria, however, was an unprecedented emergency situation, and generators that were designed to run for hundreds of hours a year instead ran for thousands last year. This requires a new level of "severity" not previously considered.

Unlike power outages, which are not all the same severity despite looking the same at the moment they happen, a death is a death. They are all the same. And in a country where individual rights are paramount, they are all weighed equally. So a fair risk assessment should say that a plane crash that kills 150 people is exactly equal to 50 car crashes that each kill 3. And multiplying that by the probability of each gives the relative risk.

...though "probability" is not clear-cut because the transportation is used in different ways/for different purposes and as such there are different choices that can be made for the measurement in some cases. I can consistently compare miles or trips to make a decision on how to get from Philly to Boston (and get different answers if I use "miles" or "trips"), but I can't really compare them for a flight to San Juan, since you can't drive there. That makes direct comparison a little tough.

That's a cool assessment. Great insight.

I am also curious about more rigorous statistical risk analysis on this issue. That is, the "fairest" comparison between two different transportation methods, namely planes and cars. Just like you said, probability based on deaths/events and probability based on deaths/distance is going to yield some different result. Furthermore, as you mentioned, the purpose of using certain transportation method changes the weight of each accidents. I wonder how that plays out.

 

[nsaspook]

Watching the video it looks like the headlights were not at the appropriate angle or intensity. The pedestrian didn't just step out from behind a blind spot or corner yet was only visible in the very last moments.

They look bad like most SUV headlights systems on a dark object at night.

A very good driver with the best equipment might have bailed her out after the disregard of approaching traffic while jaywalking.

 

[Ryan_m_b]

They look bad like most SUV headlights systems on a dark object at night.

Here in the UK IIRC the regulation for dipped headlights is that they must illuminate a distance of 50m (~165 feet) in clear weather conditions. I imagine the US has similar laws but the video and your statement seem to suggest otherwise. Looking back at the video the headlights are only illuminating two road marks ahead of the vehicle, which isn't even 10m.

I think there are a couple of serious questions to be answered here.

1) Given that Elaine had reached the far lane while crossing at a slow/steady pace and the vehicle had not turned a corner or any other blindspot was she picked up by the LIDAR? If so at what distance, what corrective measures (if any) did the car attempt to make and if she was not then why not.

2) Again given that Elaine didn't jump out suddenly and the car had not turned the corner why did the driver not notice her? What measures have Uber taken to ensure their drivers understand the need to constantly be aware of the situation.

Given the video I suspect that if the car was being driven by a human and they had their full beams on (given that it's a dark road with no oncoming traffic) they would have seen Elaine at a distance sufficient to slow down, and Elaine would have had a visual warning of the car coming at a greater distance. If the car doesn't need the headlights to drive itself then there seems to be a double failure here, 1) whatever caused the car to not see her in sufficient distance and 2) the driver/car failing to put on high beams, possibly due to complacency wrt to the autopilot's capability.

 

[256bits]

the driver/car failing to put on high beams, possibly due to complacency wrt to the autopilot's capability.

Probably not.
High beams blinds the driver in the car ahead of you, who then can't see as well.
Rear view and side mirrors and all that.
So everyone in the city drives with low beam.
Imagine high beam behind a police car, not wise.

 

[256bits]

Would limiting the cars to daylight hours of operation help? Do they lose some of their sensing "abilities" at night?

Uber is in the taxi business.
Curtail the business at night in unrealistic, if these cars are going to be allowed on the road.
The cars will be driving at night, so they have to be tested at night.

 

[LiamSmith]

Car driving is the most dangeres thing in our world. Be care of uresefl guys

 

[Spinnor]

Uber is in the taxi business.

They are also teaching robots to drive. Restricted to day time driving there is a good chance that an alert backup driver would have prevented this death. What was the women backup driver looking down at?

 

[BillTre]

Perhaps the back-up driver should have been a robot!
No wandering attention.

 

[256bits]

They are also teaching robots to drive. Restricted to day time driving there is a good chance that an alert backup driver would have prevented this death. What was the women backup driver looking down at?

Backup driver-
A human driver avoiding this incident we will never know.
With nothing to do but be there, look out the window and daydream a lot I would suspect. Attention span in that type of job...
Bored kids would say, "There's nothing to do!".
How long is a "testing run" between breaks to get out and stretch ones legs.
An inactive person with little stimulation becomes lethargic after the routine sets in.
I am actually surprised the person hadn't fallen asleep, especially since the sun was down.

I have not seen any testing on human backup drivers, and their value, it is just a given that one is needed.
Is that just a bureaucratic decision without any basis, but built on false assumptions?
Certainly there are recorded anecdotal incidents where the driver took over control of the vehicle.
But in any "experiment", which is what this is, there is no control group for comparison of what the driverless vehicle would do on its own, what a human driver would do, and what a human-driverless-vehicle would do. The testing is on the fly in the real world which is what you are getting at.

Restricted to day time driving -
Right. In hindsight, the technology for this vehicle was not ready for night driving.
But they do have to test at night to see what happens, since the cars will be used at night.
And it could be not even ready for day time driving.
The program, the sensors, something did not live up to expectations.
For all the hype that surrounds the "safer than human drivers", this was a complete failure.
If the lady has a family, I hope they sue for $500 million bucks, just to get this thing on the right track, instead of the what it seems to be the first one past the starting gate wins.
The promotional material is speaking to the converted.In the future, when these cars will be licensed for the road, I will predict that the mix of 10%, 20% etc will produce different outcomes than just 1 amongst many.
Ever hear of road rage.
What's better than to shoot the tires out from under a self-driving car that doesn't know how to drive.
Stay tuned.

 

[Ryan_m_b]

There seems to be some dispute circulating online regarding Uber's dashcam footage, which might explain why the headlights look to be defective/illegally modified. The road in the dashcam footage is pitch black with the cyclist appearing last minute. However there are regular citizens posting pictures and videos of that road at night to show that it is in fact very well illuminated. Either Uber's dashcam is of extremely poor quality or the footage has been altered somehow to make it appear darker than it was and give the impression Elaine jumped out of nowhere.

This video was taken by a member of the public at roughly the same period of night, the spot of the accident is passed about the 30 second mark.

 

[nsaspook]

I don't think the crash video was defective or modified, just poor. If we remove the automation could a reasonable person avoid that accident under these conditions (dark object in the dark part of the road).


#1 Pedestrian failed to cross safely or properly. In most parts of the world that's case closed.

Do we really need red flags and two safety drivers for driver-less cars for bad 'corner cases' ("Anything can go wrong will go wrong") ?

 

[Ryan_m_b]

I don't think the crash video was defective or modified, just poor. If we remove the automation could a reasonable person avoid that accident under these conditions (dark object in the dark part of the road).

Going by that video I would say yes absolutely. The area of the crash is visible for approximately ten seconds in that video. Aside from the fact this shows the safety driver should have seen Elaine and intervened the Uber had Radar and LIDAR, neither of which should have been affected by the lighting of the road. Yet despite it being a straight road for a significant stretch between the car and Elaine it did not appear to make any attempt to slow down.

#1 Pedestrian failed to cross safely or properly. In most parts of the world that's case closed.

Err I'm not so sure on that. IIRC there are only two nations in the world with jaywalking laws, in the others drivers are expected to do to give way to pedestrians in a safe manner. In the UK for example pedestrians are legally considered to have right of way on the roads excepting those that specifically ban pedestrian use (like motor ways). In the event of an accident partial or full blame is only placed on the pedestrian if seeing and avoiding them can be shown to be impossible in the time given, assuming the driver is otherwise driving safely (i.e. not speeding, lights at appropriate setting, not on phone etc).

 

[russ_watters]

There seems to be some dispute circulating online regarding Uber's dashcam footage, which might explain why the headlights look to be defective/illegally modified. The road in the dashcam footage is pitch black with the cyclist appearing last minute. However there are regular citizens posting pictures and videos of that road at night to show that it is in fact very well illuminated. Either Uber's dashcam is of extremely poor quality or the footage has been altered somehow to make it appear darker than it was and give the impression Elaine jumped out of nowhere.

This video was taken by a member of the public at roughly the same period of night, the spot of the accident is passed about the 30 second mark.

I was thinking something similar(I mentioned it earlier), but let's not get conspiracytheoryish about this: cameras are inherently inferior to our eyes for this purpose (range of brightness) and proper automatic adjustment is difficult at best. That's why HDR photography was invented. So the quality might be poor, but that is not unusual. The video you linked looks to me like it is using a brightness boosting/ leveling technology.

So my question is; are they using cameras for obstacle avoidance? previously you mentioned LIDAR: was that speculation or do you know it was using lidar?

 

[Ryan_m_b]

I was thinking something similar(I mentioned it earlier), but let's not get conspiracytheoryish about this: cameras are inherently inferior to our eyes for this purpose (range of brightness) and proper automatic adjustment is difficult at best. That's why HDR photography was invented. So the quality might be poor, but that is not unusual.

I agree about not getting conspiratorial but felt it worth mentioning as the footage has been the basis for a lot of the "she came out of the dark" comments.

So my question is; are they using cameras for obstacle avoidance? previously you mentioned LIDAR: was that speculation or do you know it was using lidar?

Not only did the vehicle have LIDAR and Radar but the maker of the LIDAR system has come out and said they can't understand how she wouldn't have been detected by it.

https://www.forbes.com/sites/alanoh...ers-failure-to-avoid-pedestrian/#2da6d8cb5cc2

Also of interest is this footage of the investigators looking into the accident. Using the same SUV involved in the crash they can be seen driving that same stretch of road at speed and attempting to break before reaching the point Elaine was standing. From the video it seems there was more than enough space to break in time.

https://twitter.com/LaurenReimerTV/status/977077647543955458

Tentatively my thoughts that this is a failure of two parts (one the car not responding and two the driver not overriding the car) are looking to be correct.

 

[russ_watters]

Backup driver-
A human driver avoiding this incident we will never know.
With nothing to do but be there, look out the window and daydream a lot I would suspect. Attention span in that type of job...
Bored kids would say, "There's nothing to do!".
How long is a "testing run" between breaks to get out and stretch ones legs.
An inactive person with little stimulation becomes lethargic after the routine sets in.
I am actually surprised the person hadn't fallen asleep, especially since the sun was down.

I have not seen any testing on human backup drivers, and their value, it is just a given that one is needed.

What was the [] backup driver looking down at?

The news is calling him a "safety driver". Presumably that means his primary function is preventing just this sort of accident. But until we know what he was doing or what his full job description was, it is difficult to know how much blame he has. If he was on Facebook, then he has considerable fault. if he was performing Uber-assigned systems monitoring, then he has none.

 

[Bystander]

She's crossing from left to right; programming expects her to enter roadway from right?

 

[gleem]

That assumption cannot be made because people cross roads at corners all the time from the left.