Inside the new tests making ADAS systems you'll actually want to use

"If you go to a manufacturer with subjective feedback on their product, they're not interested," he says. "However, if you go to them with objective data - 'Here are the logs. Your car has done this this many times. This is the real-world impact. This is what it's going to cost you in insurance' - they're very responsive to having that conversation." Thatcham's ADAS test project aims to generate data that will make them take notice.

Al-Ani and his team use a comprehensive instrumentation system to measure the circumstances around the multiple deployments of each ADAS feature they test, plus its performance in terms of its availability (when and how often it deploys), its ability to collaborate with the driver (whether it can be adjusted on the fly) and its configurability (whether it can be customised).

Whether or not warning sounds are differentiated, so that the driver knows what they're being alerted to, is also recorded. Performance is tested in all weather conditions and on all types of roads.

Ultimately, it comes down to how usable an ADAS feature is, and if it isn't, whether it will cause drivers to switch it off. To help judge this, the team also scores each system on factors such as how annoying it is, how often it does something they don't want it to do and how often it intervenes when it shouldn't.

"With our knowledge of each system's limitations, we can tell manufacturers that if they make a system to this level, most drivers will accept it, but if it's below that, they will turn it off," says Al-Ani. "We can tell insurers what is likely to happen when the system is used for real and manufacturers what more they need to do to improve it. We can also go to the regulators and tell them that if they're considering adopting a system for GSR2 or for GB type approval, these are the tweaks they should consider that will mean people actually use the system."

Among the team's more startling discoveries so far is how the current EU test standard for intelligent speed assist (ISA) may not be fit for purpose. For this test, ISA systems are assessed based on their accuracy over distance travelled but it can overlook performance at key moments, such as when speed limits change. In contrast, Thatcham's event-based testing measures ISA's accuracy at each of these change points. When applied in real-world conditions across three vehicles, it produced very different results.

The worst-performing vehicle, an MG ZS, scored 91.3% accuracy across the driven distance, but when assessed on an event-based metric, Thatcham found that its ISA was 74.3% accurate. So in around one in four events, its ISA was displaying the wrong speed limit.

The best-performing vehicle, a BMW i5, scored 98.39% and 90.3% respectively. Al-Ani is unimpressed: "Even the i5, the best ISA of the three cars on our test fleet, is wrong in one in every 10 events. That's not good enough." Thatcham also found the ISA systems it tested displayed speed limits that aren't legal in the UK. They included multiple instances of 5, 10, 15 and 100mph.

"When ISA misreads speed limits, it can lead to unexpected or inconsistent system responses. Over time, this risks reducing driver confidence in the technology, which is critical to achieving its intended safety benefits," says Al-Ani.

What follows is the scenario where the first thing a driver does on entering their car is turn off as many of its ADAS features as possible. Al-Ani sympathises but believes that greater configurability of systems is the correct way forward. "If the ALK [active lane keeping] is too sensitive, rather than turn it off you should be able to turn it down. BMW is getting very good at offering levels of configurability so the driver can set up the system the way they like it and it stays that way. You don't have to reset it when restarting the car. Until ADAS have improved sufficiently, that's a better condition to have."

On the road with Thatcham

Thatcham Research's event-based ADAS test programme takes place mainly on public roads but also, for more controlled work, on a closed circuit. Three cars are used: a BMW i5, Tesla Model Y and MG ZS. In terms of their ADAS, says principal ADAS engineer Yousif Al-Ani, they represent around 10% of new cars.

Each car sports a roof-mounted, 360deg lidar plus cameras inside for recording the infotainment screen and ADAS functions. These provide forward and roadside views that are key to recording the presence of speed signs and lane markings and the car's position relative to them.

Driving the Model Y, Al-Ani showed how its ADAS can do things that prompt drivers to switch it off. The first was how, in autopilot mode, the car slowed excessively for a sharp corner on a main road. Then, entering a built-up area and rounding a corner, it braked excessively for a parked vehicle.

"This car has software designed only to pass the EU regulation," says Al-Ani. "In the slow-moving queue we've joined it's brilliant, but for many drivers its unpredictability is enough to discourage them from using it at all."

Al-Ani is critical of the way the Tesla's system disengages each time he makes the slightest correction: "It's not collaborating with me, so I become frustrated and leave it off."

Later, a bus turns into our road from the right while, a little way ahead, a pedestrian crosses in front of us: "The car braked for the bus when it needn't have and the steering correction was for the pedestrian who was still a safe distance away. The system then disengaged, leaving us in a bad situation with not much time to react. Working at its best, Tesla's system offers real benefits, but drivers need to understand its limitations."

Thatcham vs Euro NCAP

Like Thatcham Research, Euro NCAP, Europe's main vehicle safety body, is running a real-world test programme to make sure the ADAS technologies it judges in the laboratory are delivering in practice. Every vehicle that it tests is equipped with a suite of exterior and interior sensors to accurately monitor its system's responses to a variety of road signs and conditions. It is driven a total of around 1200 miles in at least three European countries and every reaction is logged.

"We have worked on ADAS for years and now we want to make sure the technologies are delivering not only on the test track but on the road," says Adriano Palao, ADAS technical manager at Euro NCAP. "For example, is the lane keeping assist annoying and aggressive? What is the accuracy of the speed limit information? Were there any false braking events? This is the first time we have sought to find out what is the experience of the end consumer."

Thatcham Research was an active member of Euro NCAP for 22 years but earlier this year ended its relationship to focus on its own test programmes.

Yousif Al-Ani, principal ADAS engineer at Thatcham Research, denies its departure risks weakening Thatcham's power to influence regulators and car makers. "We have quite a lot of power, actually," he says. "We hold the keys to motor insurance risk rating in the UK, the second-biggest car market in Europe. "Vehicle manufacturers listen to us when something threatens their customer base. It's no use having systems that work well in France or Germany where approval testing is carried out but not so well on British roads."

Insights from ADAS engineer Al-Ani

Lane keeping assist: "We didn't find a single false event in the BMW i5, while the MG ZS recorded just 0.43 per 100km. These latest systems display less aggressive assistance too."

Crash avoidance: "In a variety of scenarios, the Tesla Model Y deploys its brakes sooner than the other test cars, although the BMW i5 runs it close. However, instances where the test driver has to intervene are roughly equal between the Tesla and the BMW."

'Ghost braking': "Instances where the cars braked unnecessarily were greatest in the Tesla, at 122 events, versus eight in the BMW and six in the MG. The Tesla's sensitivity has been turned up to exceed the regulation but at the risk of annoying the driver, who will just switch it off. It avoids crashing but is overly sensitive."