We’ve all become accustomed to certain advanced vehicle features that have been designed to make driving easier and safer. We’ve come to rely on side mirrors alerting us when it isn’t safe to change lanes, our cars correcting oversteering to keep us in lane, parallel parking assistance helping us get into small spots, and much more. These kinds of features have been built into our cars long enough that we may no longer think of them on a daily basis. These vehicle systems that keep us safe and make our driving experience more enjoyable are often courtesy of artificial intelligence (AI), machine learning, and augmented reality (AR). 

AI and AR technology are already part of our driving lives and will only become more intuitive, helpful, and seamlessly integrated with each new model we buy or lease. Manufacturers are currently exploring more opportunities to use AI and AR to specifically enhance the operations of autonomous and semi-autonomous vehicles, and other vehicles equipped with advanced driver assistance systems (ADAS) that help us drive, reverse, and park safely every day. This next wave of AI and AR advancements have the power to push driver assistance technology even further and give consumers a more predictive and personalized driving experience.

Onboard AI and machine learning

When it comes to developing more advanced driver assistance technology, one of the current challenges for manufacturers is tracking vehicle usage history and exploiting available temporal information to build a more robust solution. Available vehicle temporal information usually includes limited data recorded from various onboard sources, such as cameras and sensors. Employing AI and machine learning increases the volume of available information and history to predict and resolve more issues during vehicle operation. Existing onboard machine learning techniques are widely based on supervised learning. This learning model keeps some of the most advanced vehicles on the road functioning at a high level. Adding unsupervised machine learning methods, however, can enable even more sophisticated utilization of collected data.

In order to achieve optimal driver personalization and anomaly prediction, both supervised and unsupervised learning must be employed to create a system that is capable of observing and learning equally from vehicle sensors and your behavior behind the wheel. The ideal system will capture gestures and emotions of drivers and passengers to predict behavior, infuse multi-modal vehicle inputs, and implement incremental learning to generate personalized driving recommendations based on your specific habits.

Predicting driver emotion and behavior

AI-powered vehicle systems predict contextual events and issues by fusing multi-modal inputs derived from cameras, image frames, and sensory data to reach a specific conclusion. The data derived from camera and image frames is called multi-channel data, which includes voice, video, and textual data captured by various sources present in the vehicle. The generated multi-channel data can be saved as emotional information, such as happiness, sadness, neutral, surprise, or disgust. This data, featuring extracted emotional gestures and expressions, is integrated with data from vehicle system sensors and used for training a supervised machine learning model. The trained machine learning model can predict driver emotions and help the system derive driver intentions and abilities. In the case of autonomous vehicle systems, passenger gestures and emotions can be captured as well.

Part of the collected multi-channel data, raw voice data may include features like energy, zero crossing rate, entropy of energy, spectral centroid, spectral spread, spectral entropy, spectral flux, spectral roll-off, Mel Frequency Cepstral Coefficient (MFCC), chroma vector, and chroma deviation. The raw video and image data may detect features like facial key points, textual features, and color features extracted from OpenFace library. Convolutional neural networks can also be used to detect the direction of your gaze (looking up, down, left, or right) while driving, determine if you’re potentially getting drowsy or yelling at a fellow driver, and help you avoid a potential hazard.

Combining multi-modal inputs

The supervised model combines all of these data points to better inform predictions and infer driver intentions. Further, the sensory data derived from various onboard sensors, such as accelerometers, gyroscopes, magnetometers, or brake sensors, may be used to predict any anomaly in vehicle components using the unsupervised machine learning method known as Hierarchical Temporal Memory (HTM).

The output of the multichannel data and the sensory data are then fused to derive a common inference and trigger the generation of contextual events, recommendations, or actions. For example, the multi-channel and sensory data may indicate unusual behavior and the system can infer based on learned gestures and emotions that you’re driving angry and planning to overtake or change lanes. In this case, a contextual recommendation is then generated and delivered to your driver assistance system. The system may slow your vehicle to avoid a collision to ensure your safety and the safety of those around you.

Incremental learning for personalized AR recommendations

With an incremental learning-based AI model, the driver assistance system can use all the aforementioned data to identify and remember your consistent driving habits or changes in behavior, and establish them as repeatable patterns. This ongoing learning allows the system to deliver highly predictive driver assistance and recommendations that are as personalized as they can be.

These contextual recommendations, including navigation directions, alerts, and warnings, could then be shown to the driver in real time using AR technology. Necessary alerts or warnings will be shown on the windshield via AR to assist the driver in avoiding collisions and accidents. Alternately, the driver can wear AR-enabled eyeglasses to view these alerts and take the necessary steps to avoid road collisions and accidents.  

Enhancing the driver experience

An advanced system, enhanced by both AI and AR, is ideally suited to autonomous vehicles and semi-autonomous vehicles, as well as vehicles with more limited ADAS capabilities. The AI-generated and AR-enabled recommendations can predict and react to vehicle anomalies or driver actions in real time, including lane change, overtaking, vehicle acceleration, turning deviations, driver drowsiness or impairment, and more. This gives you a safer, more intuitive, and more personalized driving experience every time. With incremental learning, each event and recommendation help the system recognize and adapt to new patterns in human behavior and vehicle systems, in order to consistently deliver the best possible driver feedback and vehicle performance.

Conclusion

While we have become accustomed to certain features already, there is far more in-car technology on the horizon being created with the aid of AI, machine learning, and AR, which will soon become far more commonplace. You can take the driver’s seat knowing that we are always striving to develop and implement the most cutting-edge technology to improve your safety and ensure your comfort on the road.

For more information contact us at askix.experts@wipro.com

Dr. Gopichand Agnihotram
DMTS-Senior Member, Principal consultant, Interactive Experience (iX) solutions.

Gopichand Agnihotram has 14+ years of IT experience, working in Machine Learning and Artificial intelligence areas.  He is leading the research group in iX and responsible for building the new innovative solutions for iX.  Gopichand has published several research articles in Springer, IEEE, and ACM.  He is an active member of ACM and IEEE for participating and chairing many conferences and workshops.

Vivek Kumar Varma N
Project Engineer, Interactive Experience

Vivek has been working in research group of iX team for 2 years in the fields of Artificial Intelligence, Computer Vision, AR/VR technologies after graduating from NIT Jalandhar. His primary responsibility is to build innovative and cutting-edge solutions for iX.
Vivek has implemented various novel vision-based Intelligence solutions for which he received multiple publications, patents for it and actively contributed to the Machine Learning community by participating in multiple conferences.