Using Eye Tracking Technology in Self Driving Vehicles 

to Determine Legal Responsibility 

Lucero Ortega, Ronnie Reynoso, Sebastian Wisniewski, Xin Chen

City College of New York 

 

Summary

Every year millions of people are injured in car accidents due to varying reasons. One of those reasons include driver distractions. The market continues to grow with self-driving cars becoming more available. A problem that comes with this new technology is on who the legal responsibility of car accidents falls on. In this paper we will address this problem by looking at eye-tracking technology to avoid or reduce the probability of car accidents and if that can not be done then to able to determine legal responsibility. This technology will be based on a screen-based tracker to detect eye movements and therefore study human behavior. We will explore the accuracy of the technology in simulations and road tests by creating a budget and task schedule. The estimated cost to conduct our research will range from $50,000 to $110,000. 

 

Author Note 

This paper was prepared for English 21007 taught by Professor Susan Delamare.

 

Table of Contents

Introduction……………………………………………………………… 3

Objectives.…………………………………………………………………. 4

Preliminary Literature Review……………………………….….. 5

Technical Description of Innovation………………………….…. 7

References………..……………………………..……..………………… 12

Appendix A – Autonomous Vehicle Potential Benefits and Costs…14

Appendix B – Task Schedule…………………………………………. 15

List of Figures

 

Figure 1………………. Opened Eye………………………………………………….……………………….p.5

Figure 2…………….. Closed Eye……………………………..……..……………….………………………. p.6

Figure 3…………….. Number of Consecutive Pixels for an Open Eye.,,,,,,,,……………… p.6

Figure 4…………….. Number of Consecutive Pixels for an Closed Eye…………………… p.7

Figure 5………………Difference in Pupil and Cornea Reflection……………………….…….p.7

Figure 6…………….. Location of Camera…………………………………………………..…………… p.9

Figure 7…………….. Diagram of Infrared Light Creating Reflection on the Cornea…….. p.9

Figure 8…………….. Diagram for Steps in Eye Tracking Technology………………………p.10

 

Introduction 

The automatic vehicle or self-driving car is a technology that can reduce traffic accidents and radically improve the flow of traffic (“A few self-driving cars”, 2017). It can also determine the shortest route to the destination and the fastest and most efficient driving speed based on its artificial intelligence (Yifang & Jingwei, 2016). According to the National Highway Traffic Safety Administration of the Department of Transportation, 94% of vehicle accidents in the United States involved human errors that could have been avoided (Jarvis, 2018). Humans attention span continues to decrease as time goes on, leading to a failure to stay alert to traffic for a variety of reasons. They often get distracted by texting, talking on the phone, eating or simply daydreaming. These reasons often cause them irreparable harm or can even lead to the death of the driver or innocent pedestrians (Maddox, 2019). Self-driving cars are designed to eliminate human error in driving. It would create a safer environment compared to one where a person drives (“Eliminating Human Error”, n.d.). Although automatic vehicles can reduce the risk of accidents, it could also create a problem. In the event of a car accident involving artificial intelligence, the concern is who should be held legally responsible. Currently, the technological advances that could help address this problem are eye trackers, It is a system that tracks eye movements based on corneal reflection and is much cheaper than using sensors to monitor the condition of the driver. It can analyze a person’s fatigue value by processing images with open and closed eyes. Eye trackers can prompt the driver to respond immediately in the event of a dangerous situation that could lead to an accident. If the driver is the cause of a car accident due to his or her own mental or physical conditions, then he or she is liable for the accident. In this paper, we propose to improve a current eye-tracking technology called pupil centre corneal reflection (PCCR). Our goal will be to enhance the eye tracker in terms of data it collects through image collection and alertness by improving the speed and accuracy in the event that a driver is in an accident. It will be able to collect more complete images of corneal and pupil reflections from light sources to decrease the limitations that other eye-tracking system may have. One of those limitations involves only being able to monitor the image of an open and closed eye (Rubhan et al., 2014). There may also be special situations where the eyes moving irregularly and an improved algorithm would be able to evaluate whether it is necessary to warn the driver to take action, or whether artificial intelligence should take over the vehicle. (Farnsworth, 2019). PCCR not only monitors the physical condition of the driver but also reminds the driver of the legal responsibility they should bear in the event of a car accident. This document will explain the feasibility and planning of automatic cars and the expected costs and time scales of autonomous vehicles. This technology will greatly reduce car accidents and people will get a safer driving experience.

 

Objective

The objective of this project will be to research the feasibility of eye-tracking technology in self-driving vehicles. We want to understand its effectiveness in monitoring driver attention to the road in order to determine legal responsibility in the case of a car accident. First, we will need to gather existing literature to be able to choose the best eye-tracking system that exists. Once chosen, the technology will be tested in a lab using different scenarios in a simulation of self-driving where it will track for how long individuals pay attention and to determine the accuracy of the technology. Once this has been completed and proven to be accurate, it will then be tested with a real vehicle on a testing road to see how the tracking technology works using real obstacles. The goal of these experiments is to create the best eye-tracking technology for self-driving vehicles to be able to prevent car accidents and in the case it is unable to do that, to effectively determine who the legal responsibility of the accident falls on. 

The cost of these tests and simulations will be $250,000 which will cover payment to volunteers and researchers as well as any equipment required to perform these experiments. The task schedule will include ensuring the safety of the proposed technology, addressing potential conflicts and risks, putting fully autonomous vehicles on the road, expanding it to include ridesharing and public transportation, and making it affordable to all. The goal is to accomplish all of this in the time span of 50 years. 

 

Preliminary Literature Review 

Although self-driving cars can greatly reduce the probability of accidents, there is still a small probability of a car accident caused by a hacker or a regular program error. If this happens, who should be held responsible? To solve this problem, we can use the Eye Tracking technology. 

The authors Rubhan, Anuradha, & Anandha, 2014 mentioned that most traffic accidents are caused by driver fatigue. To detect whether a driver may be exhausted or lethargic, eye tracking is a suitable detection technology that can be used in this situation. It is neither as expensive as a sensor, nor as

disturbing to the driver as a driver fatigue system made entirely of wire. It only needs to be fixed in front of the drivers face so that their facial expression can be detected. It will then be able to give a warning when the excessive sleepiness parameter is detected. The authors also introduced in detail the internal working principle of eye-tracking systems. The system will obtain the primary frame of the entire video from the camera and detect the features of the face. If the system does not detect any facial features from the primary frame, it will be assumed that the driver’s head is constantly moving, then the sleepiness parameter is 0. (Rubhan et al.,2014)

Figure 1: Open Eye. Reprint from “Driver Fatigue Detection Based On Eye Track: A Survey” by S. Padma Rubhan, B. Anuradha, H..Anandha kumar.2014a. Retrieved from https://www.ijarcs.info/index.php/Ijarcs/article/view/2302/2290

Figure 2: Closed Eye. Reprint from “Driver Fatigue Detection Based On Eye Track: A Survey” by S. Padma Rubhan, B. Anuradha, H..Anandha kumar.2014b. Retrieved from https://www.ijarcs.info/index.php/Ijarcs/article/view/2302/2290

If eye and face features are detected in the next frame, the system will lock and collect eye filtered images and generate a histogram. At this time, there are already pictures with open (fig. 1 above) and closed eyes (fig. 2 above) and a histogram generated from the two pictures inside the system. The system will compare the histogram (fig. 4 below) of the closed-eye picture and the histogram (fig. 3 below) of the open-eye picture in real-time. When the difference is more than 80%, the system will calculate the driver’s sleepiness parameters. The system will issue a warning when the sleepiness parameter reaches a certain value. (Rubhan et al.,2014)

Figure 3: Number of Consecutive Pixels for an Open Eye. Adapted from “Driver Fatigue Detection Based On Eye Track: A Survey” by S. Padma Rubhan, B. Anuradha, H..Anandha kumar.2014c. Retrieved from https://www.ijarcs.info/index.php/Ijarcs/article/view/2302/2290

Figure 4: Number of Consecutive Pixels for a Closed Eye. Adapted from “Driver Fatigue Detection Based On Eye Track: A Survey” by S. Padma Rubhan, B. Anuradha, H..Anandha kumar, 2014d. Retrieved from https://www.ijarcs.info/index.php/Ijarcs/article/view/2302/2290

 

The eye tracking system proposed by the authors is cheaper and does not disturb the driver, but it also has a major drawback. It only has the ability to detect images of opened and closed eyes. This causes a major problem. Suppose someone has a mental disorder, the eyeballs are likely to have irregular movements and the eye tracking system would not be able to detect the driver’s situation immediately. It would prevent corresponding preventive measures to be taken on time. When this technology was applied to a self-driving car, the driver of the car was always reminded to pay attention to the road when driving. In the case of an accident, it was their responsibility to manually retrieve control of the car immediately, and bear certain legal liabilities.  If the driver experiences fatigue or physical discomfort, the eye tracking system will issue a timely warning. At the same time, it can solve the drawbacks of this system. When a car accident occurs, it cannot effectively prevent accidents. At this time, the artificial intelligence of automatic cars that can prepare in advance. It would use a better algorithm to detect the eyeball and directly park the car on the roadside and promptly alert the others to prevent the accident as soon as possible.

 

Technical Description of Innovation 

The model we propose is based on an existing eye tracking technology technique called the pupil centre corneal reflection (PCCR). It uses a light source to illuminate the eye. This causes reflection off the eye and a camera takes an image of the eye showing the reflections (Farnsworth, 2019). From the image taken, the reflection of the light source on the cornea and pupil can be identified (Farnsworth, 2019). The angle between the cornea and the pupil reflection is calculated and from this information the gaze direction can then be calculated (figure 5). 

Figure 5: Difference in pupil and cornea reflection that determines gaze direction. Reprint from “What is Eye Tracking and How Does it Work?” B. Farnsworth. 2019. https://imotions.com

 

Our proposal works on the same concept. The parts that are included are a camera, a light source and an algorithm. A camera and light source would be placed in front of the driver but out of the line of eyesight so as not to obstruct the view of the road (figure 6). The light source would be an infrared light that will be used to create a reflection on each of the eyes cornea (2019). The camera will then able to detect the driver’s head and eye position and the behavior of the pupil and iris (2019). An algorithm can then determine if the driver is fatigued or distracted based on the image it captured (figure 7).  Based on this information, the technology can make a decision on whether the driver is alert enough to take action, needs to be alerted that he/she has to take action or if the artificial intelligence should take over the vehicle or completely stop it (2019). 

Figure 6: Location of camera. Reprint from “Multi-User Identification-Based Eye-Tracking Algorithm Using Position Estimation” S. Kang. 2016. mdpi.com

Figure 7: Diagram of infrared light created a reflection of the cornea. Reprint from “Smart Eye” https://smarteye.se/technology/

Figure 8: Diagram for Steps in Eye Tracking Technology. Reprint from “Eye-Tracking and Cartography” V. Krassanakis. (n.d) http://users.ntua.gr/bnakos/Eye_Tracking_Eng.html

Fig. 8A – Camera tracks pupil center from light reflecting off of cornea

Fig. 8B – An image of the eye with the reflection patterns is captured

Fig. 8C – Specific details in the eyes and reflection patterns are found by the algorithm 

Fig. 8D – The center of the pupil is detected  

Fig. 8E – The eye tracker measures a series of gaze points that determines a fixed point 

Fig. 8F – The system then finds the most important regions of interest between fixations 

Fig. 8G – Based on the eye movement, behavior of the individual  is determined and the artificial intelligence can make a decision on the situation

Technology Testing

The eye-tracking technology we propose for as our solution mirrors a screen-based eye tracker where eye movements are recorded at a distance (Farnsworth, 2018). The camera is usually placed below or close to a computer screen and the participant is seated directly in front of the eye tracker. This eye tracker is used in settings where the participants observes videos, pictures or other small interactions (Farnsworth 2018). 

The technology will be tested using different levels of difficulty with each increasing level representing an increase in the number of obstacles, including the possibility of human and computer based errors.. There will be three different groups tested. One will pay full attention to the road ahead. The second will perform a second task but still keep an eye on the road. The last group will pay no attention. The data collected will determine if the technology is able to adapt to different scenarios and if from this one can correctly determine who should be held responsible for any mistakes. 

Budget

 

Line Item	Description	Costs
Practical Testing Units	Allow for real-life testing scenarios by adopting the technology to car models existanting in today’s society. Can be put on lease.	$20,000
Software/Firmware Licensing	For use of any closed-source libraries that we may incorporate.	$10,000
Pre-Production Units	Sourcing, assembling, and fitting a unit that emulate the function and look of the final product.	$5,000
Industrial Design Consultant	Being able to create a product that can seamlessly blend with the rest of the car.	$20,000
Certification	Safety and Regulation checks.	$30,000
Support	Should there be any unexpected errors/bugs found later down the line	$25,000 (annual)
Total cost		$110,000 (incl. $25,000 annual)

 

Reference

Farnsworth, B. (2018, December 4). Eye Tracking: The Complete Pocket Guide. Retrieved from https://imotions.com/blog/eye-tracking/

Farnsworth, B. (2019, April 2). What is Eye Tracking and How Does it Work?. Retrieved from https://imotions.com/blog/eye-tracking-work/

Jarvis, A. (2018, November 15). An Introduction to Autonomous Cars: How They Will Improve the Cost, Convenience, and Safety of Driving. Retrieved from https://velodynelidar.com/newsroom/an-introduction-to-autonomous-cars-how-they-will-improve-the-cost-convenience-and-safety-of-driving-part-one-of-a-three-part-series/.

Kang, S-J. (2016, December 13). Multi-User Identification-Based Eye-Tracking Algorithm Using Position Estimation.  Retrieved from https://www.mdpi.com/1424-8220/17/1/41

Krassanakis, V. (n.d). Eye-Tracking and Cartography. Retrieved from http://users.ntua.gr/bnakos/Eye_Tracking_Eng.html

Maddox, T. (2019, January 16). How autonomous vehicles could save over 350K lives in the US and millions worldwide. Retrieved from https://www.zdnet.com/article/how-autonomous-vehicles-could-save-over-350k-lives-in-the-us-and-millions-worldwide/.

Rubhan, S. P., Anuradha, B., & Anandha. (2014a). Driver Fatigue Detection Based On Eye Track: A Survey. [Figure]. Retrieved from https://www.ijarcs.info/index.php/Ijarcs/article/view/2302/2290.

Rubhan, S. P., Anuradha, B., & Anandha. (2014b). Driver Fatigue Detection Based On Eye Track: A Survey. [Figure]. Retrieved from https://www.ijarcs.info/index.php/Ijarcs/article/view/2302/2290.

Rubhan, S. P., Anuradha, B., & Anandha. (2014c). Driver Fatigue Detection Based On Eye Track: A Survey. [Graph]. Retrieved from https://www.ijarcs.info/index.php/Ijarcs/article/view/2302/2290.

Rubhan, S. P., Anuradha, B., & Anandha. (2014d). Driver Fatigue Detection Based On Eye Track: A Survey. [Graph]. Retrieved from https://www.ijarcs.info/index.php/Ijarcs/article/view/2302/2290.

A few self-driving cars can dramatically improve traffic flow, Experiments show. (2017, May 10). Retrieved from https://www.sciencedaily.com/releases/2017/05/170510095703.htm.

Yifang, & Jingwei. (2016, October 31). Route Choice of the Shortest Travel Time Based on Floating Car Data. Retrieved from https://www.hindawi.com/journals/js/2016/7041653/.

(n.d) Smart Eye. Retrieved from https://smarteye.se/technology/

 

Appendix A – Autonomous Vehicle Potential Benefits and Costs

	Benefits	Costs/Problems
Internal (user Impacts)	Reduced drivers’ stress and increased productivity. Motorists can rest, play and work while travelling.Mobility for non-drivers. More independent mobility for non-drivers can reduce motorists’ chauffeuring burdens and transit subsidy needs. Reduced paid driver costs. Reduces costs for taxis and commercial transport drivers.	Increased vehicle costs. Requires additional vehicle equipment, services and fees.Additional user risks. Additional crashes caused by system failures, platooning, higher traffic speeds, additional risk- taking, and increased total vehicle travel. Reduced security and privacy. May be vulnerable to information abuse (hacking), and features such as location tracking and data sharing may reduce privacy.
External (Impacts on others)	Increased safety. May reduce crash risks and insurance costs. May reduce high-risk driving.Increased road capacity and reduced costs. More efficient vehicle traffic may reduce congestion and roadway costs. Reduced parking costs. Reduces demand for parking at destinations. Reduced energy consumption and pollution. May increase fuel efficiency and reduce emissions. Supports vehicle sharing. Could facilitate carsharing and ridesharing, reducing total vehicle ownership and travel, and associated costs.	Additional risks. May increase risks to other road users andmay be used for criminal activities. Increased traffic problems. Increased vehicle travel may increase congestion, pollution and sprawl-related costs. Social equity concerns. May reduce affordable mobility options including walking, bicycling and transit services. Reduced employment. Jobs for drivers may decline. Increased infrastructure costs. May require higher roadway design and maintenance standards. Reduced support for other solutions. Optimistic predictions of autonomous driving may discourage other transport improvements and management strategies.

Autonomous vehicles can provide various benefits and costs, including external impacts on other people.

Summarizes their deployment. Most required decades from initial commercial availability to market saturation, and some never became universal. Lavasani and Jin (2016) conclude that, because autonomous vehicle technologies are more revolutionary than those in Exhibit 11, they will probably have slower initial market acceptance and penetration.

Appendix B – Task Schedule

Autonomous Vehicle Planning Needs and Requirements

Impact	Needs	Requirements	Time Period
Become legal	Demonstrated functionality and safety	Define performance, testing and data collection requirements for automated driving on public roads.	2019-25
Address new conflicts and risks	Develop policies to address increased curb and road congestion risks.	Develop efficient curb and roadway management policies, such as curb regulations, congestion pricing and high- occupant vehicle priority policies.	2020-2040
Increase traffic density by vehicle coordination	Road lanes dedicated to vehicles with coordinated platooning capability	Evaluate impacts. Define requirements. Identify lanes to be dedicated to vehicles capable of coordinated operation.	2020-40
Independent mobility for non-drivers	Fully autonomous vehicles available for sale	Allows affluent non-drivers to enjoy independent mobility.	2020-30s
Automated carsharing/taxi	Moderate price premium. Successful business model.	May provide demand response services in affluent areas. Supports carsharing.	2030-40s
Independent mobility for lower-income	Affordable autonomous vehicles for sale	Reduced need for conventional public transit services in some areas.	2040-50s
Reduced parking demand	Major share of vehicles are autonomous	Reduced parking requirements.	2040-50s
Reduced traffic congestion	Major share of urban peak vehicle travel is autonomous.	Reduced road supply.	2050-60s
Increased safety	Major share of vehicle travel is autonomous	Reduced traffic risk. Possibly increased walking and cycling activity.	2040-60s
Energy conservation and emission reductions	Major share of vehicle travel is autonomous. Walking and cycling become safer.	Supports energy conservation and emission reduction efforts.	2040-60s
Improved vehicle control	Most or all vehicles are autonomous	Allows narrower lanes and interactive traffic controls.	2050-70s
Need to plan for mixed traffic	Major share of vehicles are autonomous.	More complex traffic. May justify restrictions on human-driven vehicles.	2040-60s
Mandated autonomous vehicles	Most vehicles are autonomous and large benefits are proven.	Allows advanced traffic management.	2060-80s