Ask HN: How to deal with a world where we watch people die because of our code

3 points

7 years ago

There was a moment in Tesla's autonomy day that struck me. Someone (Stuart Bowers?) was explaining how Tesla's autonomous driving system learns and makes decisions. They can record and replay specific situations the cars get into. He even mentioned that "he watches every accident": https://www.youtube.com/watch?v=Ucp0TTmvqOE&feature=youtu.be&t=10777

I assume the following:

  - Autonomous driving that decreases human suffering is a good thing, even if the accidents are more than 0.
  - The nature of the type of approach Tesla is taking (neural network, etc) is designed to reduce, not eliminate, high energy impacts
  - There will always be an ever diminishing tail of situations that the system will not respond to ideally that will cause accidents.

So, the approach Tesla and probably others are taking is rather than wait for a completely ideal solution, which may never come, we should get a good enough system together that can immediately start reducing serious accidents, and learn over time to do even better.

This means that engineers will need to review the results of their work regularly. They'll probably need to watch video of people getting killed, knowing that the reason that person died was a trade-off they made that perhaps saved the lives of 10 other people, but killed this one.

I have always found it easy to dismiss the trolley problem[1] as being purely theoretical, life isn't so simple. But here we seem to have actually created it. What does this mean?

When a junior dev pushes a patch that takes down an Amazon datacenter, we can be glad that the poor engineer was not fired, as we are all fallible and need to learn from our mistakes. We share our stories of when we screwed up. What happens when the junior engineer pushes an update that kills a few hundred people, and at the next team meeting we need to review the footage?

[1] https://en.wikipedia.org/wiki/Trolley_problem

6 comments

- Autonomous driving that decreases human suffering is a good thing, even if the accidents are more than 0. - The nature of the type of approach Tesla is taking (neural network, etc) is designed to reduce, not eliminate, high energy impacts - There will always be an ever diminishing tail of situations that the system will not respond to ideally that will cause accidents.