Chapter 4. Human-Machine Interface

It is important to explain the evolutionary process of the steering wheel and the accelerator and brake pedals. (The clutch and gear shift evolved in another way, because of the automatic gear shift systems.) Nowadays the x-by-wire systems are increasingly coming to the fore as mentioned in the Section “Intelligent Actuators”. The necessary technical solutions are available, “only” the responsibility and legal questions are not solved yet.

With using solely x-by-wire systems in case of the primary input HMIs, the connections to the actuators are electronic signals or high-level digital messages, which determine the desired motion state of the vehicle. This digital byte stream is often referred to as motion vector. It contains the demanded values for engine torque, brake pressure and steering angle and is extended by further values like gear-shift command. With this indirect connection, the electronic control units can execute the drivers command in a more effective and safer way.

In this technical level the input devices could be any type of human interfaces, the only requirement is to generate the proper electronic output. Beyond the conventional controls, several other possibilities are available, such as a joystick that is (besides the computer games) generally used to control forklifts. Although joysticks are technically perfectly integrated control devices, these solutions are not likely to be widespread in passenger cars, because they require absolute different driving techniques. Undoubtly x-by-wire systems have got a greater importance beyond the already utilized opportunities, namely the capability to control the vehicle by a computer. As the electronic interface (motion vector) can either be generated from the human driver or from an electronic control, the intelligent actuators cannot make a difference if the request was originated from driver or a control ECU. Moreover it is one of the basis of the highly automated or fully autonomous vehicle operation.

In modern vehicles the primary controls are equipped with several sensors which could be the following:

The primary HMI output is the instrument cluster. Basically it is responsible for displaying the motion state of the vehicle and the engine (such as velocity and rotational speed), feedback about the execution of the desired task (such as turn signal indicator light), and warning of the problems and errors. Additionally it could inform the driver about the detailed vehicle status, the driving situation and the current level of automation.

If we look through the evolution of the instrument cluster, it can be noticed that the development direction goes from the mechanical to the fully electronic solutions. Until the 1990s the analogue gauge of the speedometer was connected to the gearbox by a Bowden cable. Initial electric gauges used Deprez instruments, later they were driven by PWM signals using cross-coils technology. With release of the electronic motor control units, the needle of the gauge was driven by a small stepper motor which connected to the ECM. The instrument cluster has been complemented with alphanumeric and later graphical LCD displays. Nowadays the state-of-art solution is the fully colour LCD-based instrument cluster with variable style displaying. These technologies will be detailed in Section 4.3.

The evolution of the secondary input controls is very diverse, but it has a common property at each manufacturer, namely the multiplication of the comfort and infotainment functions. It has resulted many buttons, switches, sliders and knobs all around the driver. The first solutions for the simplification were the integrated controllers with which the driver can navigate through a menu on an LCD display.

The BMW iDrive Controller was designed to be one single interface for many functions and features of the vehicle through the central console display, as replacing the array of controls of the comfort and infotainment functions with an all-in-one unit. The rotate-and-press mechanism enables one-handed operation: right means ‘continue’, left ‘back’, turning the button allows you to scroll through a list and pressing it selects an option. Frequently used functions like multimedia, radio or navigation have direct access keys. (Source: [51])

Several other methods exist to simplify the secondary input HMIs, such as steering wheel buttons and touchscreens. Generally most of these solutions are based on a menu, displayed on a relatively large screen.

The design of the secondary input devices is a more and more challenging task, because of the conflicting requirements, i.e. the increasing number of functions (especially infotainment) and the traffic safety.

The secondary HMI output channels do not differ that much from the primary output channels, meaning that they are mostly relatively big LCD displays, but they may include special features like touch-screen control or split view of the central console, where on the same display device the driver and the “navigator” views different pictures at the same time.

Beside the steering wheel the pedals are the basic primary inputs in an automobile. As mentioned in the earlier sections electronic throttle control solutions are commonly used in today’s road vehicles.. The driver feeling of the “traditional” accelerator pedal can be substituted with a simple spring, because it requires only a constant pressure force. Furthermore the electronic pedal gives the possibility to use haptic feedback, such as vibration or variable pedal force to support the economic driving.

In conventional passenger cars the brake pedal is connected to the brake system via a rod in the hydraulic cylinder. Brake-by-wire systems are increasingly being integrated into or replacing conventional hydraulic or pneumatic brake systems. Such electrical brake systems are preferable because they reduce the mass of the system and provide greater ability to integrate the system into the vehicle's other electronic circuits and controls. In hybrid cars it is essential because of the electronic braking by the driving motor. During depression of the brake pedal in a conventional hydraulic braking system, the hydraulic fluid will exert a force back on the brake pedal due to the hydraulic pressure in the brake lines. Since an electronic brake system may not have such hydraulic pressure at the brake pedal, the vehicle operator will not detect any countering force, which in turn can disorient the operator. Accordingly, a typical electrical brake system will include a brake pedal feel simulator to provide a simulation force on the brake pedal. The simulation force provided by the simulator acts opposite the brake pedal force generated by the vehicle driver. It has to be noted that the brake pedal simulator has to suit to the different driving situations especially during emergency. The system has to adjust automatically its operation to reduce or eliminate the simulation force during emergency or failure conditions. (Source: [52])

The clutch pedal is usually hydraulic also, and exists only in vehicles with manual gearbox. It doesn’t require such developments as mentioned above, because with the use of automatic, automated or semi-automatic gearboxes the clutch pedal itself is eliminated.

Traditionally one mechanical lever exists in vehicles, i.e. the parking brake lever. In European vehicle designs it is operated by hand (hand-brake) but in the US the parking brake is generally operated by the left foot. In modern vehicle the parking brake lever is often substituted by a push button or a fully automatic parking brake.

The steer-by-wire system (mentioned in Section “Intelligent actuators”) is also an enabler of good haptic HMI systems as it provides as much design freedom as possible, because the characteristics of the steering wheel are dynamically independent from the front axle steering. For example, a vibration induced in the steering wheel for warning purposes in a mechanically decoupled steer-by-wire systems can be designed with no effect on the actual steering behaviour at the front wheels.

With the help of these components the driver can activate/deactivate or set the vehicle’s primary and secondary functions. These switching components could be single buttons or switches for each function, a stalk or a slider.

The integrated controller knobs are such input devices which integrate more input functions into one device to support the much easier and more intuitive handling of the vehicle. These input functions could be rotation, push/pull and 4-way joystick.

A typical example is the BMW iDrive as mentioned in Section Input channels4.2.2.1, but a lot of simpler utilizations exist, such HVAC and radio control knobs.

Infotainment displays often have touchscreen features enabling the driver to select functions via touching the display. Touchscreen technology is the direct manipulation type gesture based technology. A touchscreen is an electronic visual display capable of detecting and locating a touch over its display area. It is sensitive to the touch of a human finger, hand, pointed finger nail and passive objects like stylus. Users can simply move things on the screen, scroll them, zoom them and many more.

There are four main touchscreen technologies:

The most wide-spread ones are the resistive and the capacitive touchscreens, thus these types will be detailed in the following paragraphs.

Resistive LCD touchscreen monitors rely on a touch overlay, which is composed of a flexible top layer and a rigid bottom layer separated by insulating dots, attached to a touchscreen controller. The inside surface of each of the two layers is coated with a transparent metal oxide coating (ITO) that facilitates a gradient across each layer when voltage is applied. Pressing the flexible top sheet creates electrical contact between the resistive layers, producing a switch closing in the circuit. The control electronics alternate voltage between the layers and pass the resulting X and Y touch coordinates to the touchscreen controller. The touchscreen controller data is then passed on to the computer operating system for processing.

Because of its versatility and cost-effectiveness, resistive touchscreen technology is the touch technology of choice for many markets and applications. Resistive touchscreens are used in food service, retail point-of-sale (POS), medical monitoring devices, industrial process control and instrumentation, portable and handheld products. Resistive touchscreen technology possesses many advantages over other alternative touchscreen technologies (acoustic wave, capacitive, infrared). Highly durable, resistive touchscreens are less susceptible to contaminants that easily infect acoustic wave touchscreens. In addition, resistive touchscreens are less sensitive to the effects of severe scratches that would incapacitate capacitive touchscreens. Drawback can be the too soft feeling when pressing it, since there is a mechanical deformation required to connect the two resistive layers to each-other. (Source: [53])

One can use anything on a resistive touchscreen to make the touch interface work; a gloved finger, a fingernail, a stylus device – anything that creates enough pressure on the point of impact will activate the mechanism and the touch will be registered. For this reason, resistive touchscreen require slight pressure in order to register the touch, and are not always as quick to respond as capacitive touchscreens. In addition, the resistive touchscreen’s multiple layers cause the display to be less sharp, with lower contrast than we might see on capacitive screens. While most resistive screens don’t allow for multi-touch gestures such as pinch to zoom, they can register a touch by one finger when another finger is already touching a different location on the screen. (Source: [54])

The capacitive touchscreen technology is the most popular and durable touchscreen technology used all over the world. It consists of a glass panel coated with a capacitive (conductive) material Indium Tin Oxide (ITO). The capacitive systems transmit almost 90% of light from the monitor. In case of surface-capacitive screens, only one side of the insulator is coated with a conducting layer. While the screen is operational, a uniform electrostatic field is formed over the conductive layer. Whenever, a human finger touches the screen, conduction of electric charges occurs over the uncoated layer which results in the formation of a dynamic capacitor. The controller then detects the position of touch by measuring the change in capacitance at the four corners of the screen. In the projected-capacitive touchscreen technology, the conductive ITO layer is etched to form a grid of multiple horizontal and vertical electrodes. It involves sensing along both the X and Y axis using clearly etched ITO pattern. The projective screen contains a sensor at every intersection of the row and column, thereby increasing the accuracy of the system. (Source: [55])

Since capacitive screens are made of one main layer, which is constantly getting thinner as technology advances, these screens are not only more sensitive and accurate, the display itself can be much sharper. Capacitive touchscreens can also make use of multi-touch gestures, but only by using several fingers at the same time. If one finger is touching one part of the screen, it won’t be able to sense another touch accurately. (Source: [54])

The simple beeper was the first acoustic interfaces in automobiles, but it is still used even today. The beeper is well-suitable for warning functions. Some typical examples are given as follows:

A voice feedback capable device can provide information by human speech using speech synthesizer (text-to-speech - TTS) techniques. Speech can be created by concatenating recorded speech sections from very small units (such as phones or diphones) or entire words (sentences).

Today’s TTS systems are capable to vocalize complex texts with proper clarity. The typical fields of usage are the following:

The voice control, unlike the previous technologies, is voice recognition-based input technique. First devices with voice control functions initially had to record each command spoken by the end-user, so voice recognition was just a comparison to the previously recorded sound data. Later voice recognition became general by providing user independent sound recognitions to a limited languages (e.g. english, german). Today the major force behind the development is the telecommunication sector, especially the smartphone, the developers of the mainstream operating systems (Android, iOS and Windows Phone).Voice control could be a useful input which could increase traffic safety by allowing the driver to issue a command without being distracted. One of the most significant players of the automotive voice control solutions is Nuance Inc. with the product called Dragon Drive . It is already used in several infotainment systems, such as Ford's Sync and GM's IntelliLink, furthermore it can be found in several BMW and Mercedes-Benz vehicles. Dragon Drive is optimized for the in-car experience with an easy-to-use natural language interface and uninterrupted delivery of all on-board and cloud content. Dragon Drive offers drivers seamless access to the content and services they want, where they want it, whether on the head unit, smartphone or on the web. (Source: [56])

Naturally Apple and Google are trying to offer their own solutions integrated into the automotive infotainment solutions, i.e. Apple CarPlay. Google has not presented a market-ready solution yet, but in January 2014 they have announced the Open Automotive Alliance, including General Motors, Honda Motor, Audi, Hyundai, and chipmaker Nvidia, that want to customize Google’s mobile operating system for vehicles.

The oldest and most conventional instrument device is the analogue gauge. Originally the needle of the gauge was linked with a Bowden cable to the measured unit (e.g. gearbox), later it was substituted with an electronic connection and the needle which is a proven solution today. In this case the needle is driven by magnetic field or a small electronic stepper motor.

In the newest LCD-based instrument clusters the analogue gauge type displaying will continue to be available as detailed in the following subsection.

Liquid crystals were first discovered in the late 19th century by the Austrian botanist, Friedrich Reinitzer, and the term liquid crystal itself was coined shortly afterwards by German physicist, Otto Lehmann. The major (from automotive perspective) LCD technologies are shortly detailed in the followings based on [57].

Liquid crystals are almost transparent substances, exhibiting the properties of both solid and liquid matter. Light passing through liquid crystals follows the alignment of the molecules that make them up – a property of solid matter. In the 1960s it was discovered that charging liquid crystals with electricity changed their molecular alignment, and consequently the way light passed through them; a property of liquids.

LCD is described as a transmissive technology because the display works by letting varying amounts of a fixed-intensity white backlight through an active filter. The red, green and blue elements of a pixel are achieved through simple filtering of the white light.

Most liquid crystals are organic compounds consisting of long rod-like molecules which, in their natural state, arrange themselves with their long axes roughly parallel. It is possible to precisely control the alignment of these molecules by flowing the liquid crystal along a finely grooved surface. The alignment of the molecules follows the grooves, so if the grooves are exactly parallel, then the alignment of the molecules also becomes exactly parallel.

In their natural state, LCD molecules are arranged in a loosely ordered fashion with their long axes parallel. However, when they come into contact with a grooved surface in a fixed direction, they line up in parallel along the grooves.

The first principle of an LCD consists of sandwiching liquid crystals between two finely grooved surfaces, where the grooves on one surface are perpendicular (at 90 degrees) to the grooves on the other. If the molecules at one surface are aligned north to south, and the molecules on the other are aligned east to west, then those in-between are forced into a twisted state of 90 degrees. Light follows the alignment of the molecules, and therefore is also twisted through 90 degrees as it passes through the liquid crystals. However, when a voltage is applied to the liquid crystal, the molecules rearrange themselves vertically, allowing light to pass through untwisted.

The second principle of an LCD relies on the properties of polarising filters and light itself. Natural light waves are orientated at random angles. A polarising filter is simply a set of incredibly fine parallel lines. These lines act like a net, blocking all light waves apart from those (coincidentally) orientated parallel to the lines. A second polarising filter with lines arranged perpendicular (at 90 degrees) to the first would therefore totally block this already polarised light. Light would only pass through the second polariser if its lines were exactly parallel with the first, or if the light itself had been twisted to match the second polariser.

A typical twisted nematic (TN) liquid crystal display consists of two polarising filters with their lines arranged perpendicular (at 90 degrees) to each other, which, as described above, would block all light trying to pass through. But in-between these polarisers are the twisted liquid crystals. Therefore light is polarised by the first filter, twisted through 90 degrees by the liquid crystals, finally allowing it to completely pass through the second polarising filter. However, when an electrical voltage is applied across the liquid crystal, the molecules realign vertically, allowing the light to pass through untwisted but to be blocked by the second polariser. Consequently, no voltage equals light passing through, while applied voltage equals no light emerging at the other end.

Basically two LCD control technique exist: passive matrix and active matrix. The earliest laptops (until the mid-1990s) were equipped with monochrome passive-matrix LCDs, later the colour active-matrix became standard on all laptops. Passive-matrix LCDs are still used today for less demanding applications. In particular this technology is used on portable devices where less information content needs to be displayed, lowest power consumption (no backlight) and low cost are desired, and/or readability in direct sunlight is needed.

The most common type of active matrix LCDs (AMLCDs) is the Thin Film Transistor LCD (TFT LCD), which contains, besides the polarizing sheets and cells of liquid crystal, a matrix of thin-film transistors. In a TFT screen a matrix of transistors is connected to the LCD panel – one transistor for each colour (RGB) of each pixel. These transistors drive the pixels, eliminating at a stroke the problems of ghosting and slow response speed that afflict non-TFT LCDs.

The liquid crystal elements of each pixel are arranged so that in their normal state (with no voltage applied) the light coming through the passive filter is polarised so as to pass through the screen. When a voltage is applied across the liquid crystal elements they twist by up to ninety degrees in proportion to the voltage, changing their polarisation and thereby blocking the light’s path. The transistors control the degree of twist and hence the intensity of the red, green and blue elements of each pixel forming the image on the display.

TFT screens can be made much thinner than LCDs, making them lighter, and refresh rates reached the fast 5 ms value.

There are several TFT panel types exist considering the backlight and panel technology.

The LCDs does not produce light itself, thus a proper light source is needed to built-in to produce a visible image. (However low-cost monochrome LCDs are available without backlight.) Until about 2010 the backlight of large LCD panels was based on Cold Cathode Fluorescent Lamps (CCFLs). It has several disadvantages such as higher voltage and power needed, thicker panel design, no high-speed switching, faster aging. The new LED-based backlight technologies eliminated these harmful properties and took over the CCFL.

The panel technologies can be divided into three main groups:

Without going into the technical details of these technologies the main comparable features are given as follows.

The TN panel provides the shortest response time (>1ms), and it is very cost effective technology. On the other hand these panels use only 18 bit colour depth which can be increased virtually, but the colour reproduction is not perfect anyway. Another disadvantage is the poor viewing angle.

The IPS panels’ core strength are the exact colour reproduction and the wide viewing angle. The response time is higher than the TN, but it managed to the proper level (>5ms) by now. Despite the higher costs, all in all the IPS panels is the best TFT LCD panels now.

Considering the properties the VA panels can be located between TN and IPS ones. Considering the contrast ratio it is the best technology, but with higher response time (>8ms) and a medium colour reproduction.

Today’s cutting-edge display technology is the OLED (organic light emitting diode). It is a flat light emitting technology, made by placing a series of organic (carbon based) thin films between two conductors. When electrical current is applied, a bright light is emitted. OLEDs can be used to make displays and lighting. Because OLEDs emit light, they do not require a backlight and so are thinner and more efficient than LCD displays. OLEDs are not just thin and efficient - they can also be made flexible (even rollable) and transparent.

The basic structure of an OLED is a cathode (which injects electrons), an emissive layer and an anode (which removes electrons). Modern OLED devices use many more layers in order to make them more efficient, but the basic functionality remains the same.

OLED displays have the following advantages over LCD displays:

But OLEDs also have some disadvantages. First of all, today it costs more to produce an OLED than it does to produce an LCD. Although this should hopefully change in the future, as OLEDs has a potential to be even cheaper than LCDs because of their simple design.

OLEDs have limited lifetime (like any display, really), that was quite a problem a few years ago. But there has been constant progress, and today this is almost a non-issue. Today OLEDs last long enough to be used in mobile devices and TVs, but the lifetime of a vehicle is significantly longer. OLEDs can also be problematic in direct sunlight, because of their emissive nature, which also could be a problem in a car. But companies are working to make it better, and newer mobile device displays are quite good in that respect.

Today OLED displays are used mainly in small (2" to 5") displays for mobile devices such as phones, cameras and MP3 players. OLED displays carry a price premium over LCDs, but offer brighter pictures and better power efficiency. Making larger OLEDs is possible, but difficult and expensive now. In 2014 several new OLED TVs has been announced and presented, which shows that the manufacturers are trying to force this promising technology, which will make the lower costs in the future. (Source: [58])

A software configurable instrument cluster is essentially an LCD display behind the steering wheel, which can be customized to different applications (e.g. sport, luxury sedan display or special diagnostic display). There is theme selection with several colour and shape configurations and we can also decide what gauges or windows should appear. Font size altering option could help for the visually impaired. The central console display can also be temporarily ported over to the instrument cluster showing radio channel information or the current image of the parking systems status. The advantage is that the driver does not have to look aside from the instrument panel

Figure 73 shows a BMW instrument panel, which is based on a 10,2”, high-resolution (318 dpi) LED LCD display, with 6:1 aspect ratio.

As the name of the display describes the driver can keep his head straight ahead (Head-Up) while looking at displayed information projected onto the windscreen. The driver is able to check e.g. car speed without having to take his eyes off the road. The HUD earlier was used only on cockpits of fighter aircrafts, but today more and more passenger cars took over the technology.

HUD system contains a projector and a system of mirrors that beams an easy-to-read, high-contrast image onto a translucent film on the windscreen, directly in your line of sight. The image is projected in such a way that it appears to be about two metres away, above the tip of the bonnet, making it particularly comfortable to read. BMW claims that Head-Up Display halves the time it takes for eyes to shift focus from road to the instruments and back. The system’s height can be adjusted for optimal viewing. The newest HUDs provides full colour displaying which makes the car even more comfortable for the driver. More colours mean it’s easier to differentiate between general driving information like speed limits and navigation directions and urgent warning signals. Important information like “Pedestrian in the road” is now even more clear and recognisable – and this subsequently reduces the driver’s reaction time. (Source [51])

New technologies like the Head-Up Display (HUD) might be promising solution to reduce the time for the driver to be informed, because the information (road sign, speed limit, hazardous situation) can directly be projected in front of the driver eyes without any distraction. There are researches to use HUD technology on the full windshield that would revolutionize e.g. night vision applications providing road path and obstacle “simulation” feeling to driver (see Figure 75 [59]).

Indicator lights are used in the instrument cluster to feedback of the operation of a function or indicate an error. In automotive industry the indicator lights are often called tell-tales. They could be bulbs or LEDs, which lights up a symbol or a text and the colour indicates the priority of the warning. Generally the red colour means an error that requires the car to stop immediately. In case of yellow colour the journey can be continued and it could be investigated later. But in the latter case a safety function may be out of the operation.

Indicator lights are regulated by automobile safety standards worldwide. In the United States, National Highway Traffic Safety Administration Federal Motor Vehicle Safety Standard 101 includes indicator lights in its specifications. In Europe and throughout most of the rest of the world, ECE Regulations specify it, more precisely “United Nations (UN) Vehicle Regulations - 1958 Agreement” [60].

The exact meaning and the usage of other (unregulated) symbols are manufacturer specific in the most cases.