# When the AI goes haywire, bring on the humans Author: Pal Sinha, Barnali Author URL: https://financedigest.com/author/pal-sinha-barnali Published: 2022-10-14 Category: Uncategorized Category URL: https://financedigest.com/category/uncategorized Meta Title: FICO AI Software: Preventing Catastrophes in Banking Meta Description: Discover how AI errors in credit scoring software by FICO almost led to millions of denied purchases during the pandemic. Learn the importance of monitoring AI URL: https://financedigest.com/when-the-ai-goes-haywire-bring-on-the-humanshtml ![undefined](https://prod.superblogcdn.com/site_cuid_cm5qst7v3003gwirgwqtxn8i8/images/2022-10-13t211223z3lynxmpei9c0i5rtroptp4tech-ai-monitoring-1736815027965-compressed.jpg) By Paresh Dave OAKLAND, Calif. (Reuters) – Used by two-thirds of the world’s 100 biggest banks to aid lending decisions, credit scoring giant Fair Isaac Corp and its artificial intelligence software can wreak havoc if something goes wrong. That [crisis nearly came to pass early](https://www.financedigest.com/break-in-case-of-emergency-plan-early-plan-now-for-effective-crisis-comms.html "Break in case of emergency – plan early, plan now, for effective crisis comms") in the pandemic. As FICO recounted to Reuters, the Bozeman, Montana company’s AI tools for helping banks identify credit and debit card fraud concluded that a surge in [online shopping](https://www.financedigest.com/the-high-street-is-dead-long-live-online-shopping.html "The high street is dead – long live online shopping! ") meant fraudsters must have been busier than usual. The AI software [told banks](https://www.financedigest.com/uk-banks-told-to-reimburse-customers-tricked-by-scams.html "UK banks told to reimburse customers tricked by scams") to deny millions of legitimate purchases, at a time when consumers had been scrambling for toilet paper and other essentials. But consumers ultimately faced few denials, according to FICO. The company said a [global group](https://www.financedigest.com/havas-media-group-partners-with-lumen-research-to-measure-and-optimise-attention-at-global-scale.html "Havas Media Group Partners with Lumen Research to Measure and Optimise Attention at Global Scale") of 20 analysts who constantly monitor its systems recommended temporary adjustments that avoided a blockade on spending. The team is automatically alerted to unusual buying activity that could confuse the AI, relied on by 9,000 financial institutions overall to [detect fraud](https://www.financedigest.com/the-clues-to-detecting-fraud-are-hidden-in-relationships.html "THE CLUES TO DETECTING FRAUD ARE HIDDEN IN RELATIONSHIPS") across 2 billion cards. Such corporate teams, part of the emerging [job specialty of machine learning](https://www.financedigest.com/machine-learning-engineer-jobs.html "Machine learning engineer jobs") operations (MLOps), are unusual. In separate surveys last year, FICO and the consultancy McKinsey & Co found that most organizations surveyed are not regularly monitoring AI-based programs after launching them. The problem is that errors can abound when real-world circumstances deviate, or in tech parlance “drift,” from the examples used to train AI, according to scientists [managing these systems](https://www.financedigest.com/energy-management-system-market-to-register-a-cagr-of-14-8.html "Energy Management System Market to register a CAGR of 14.8%"). In FICO’s case, it said its [software expected](https://www.financedigest.com/radiation-dose-optimisation-software-market-expected-to-increase-at-a-cagr-27-4-through-2017-to-2027.html "Radiation Dose Optimisation Software Market Expected to Increase at a CAGR 27.4% Through 2017 to 2027") more in-person than virtual shopping, and the flipped ratio led to a greater share of transactions flagged as problematic. Seasonal variations, data-quality [changes or momentous events – such as the pandemic – all can lead](https://www.financedigest.com/how-to-lead-organizational-change.html "How to Lead Organizational Change") to a string of bad AI predictions. Imagine a system recommending swimsuits to summer shoppers, not realizing that [COVID lockdowns](https://www.financedigest.com/in-shanghai-lockdown-blues-make-way-for-covid-testing-gripes.html "In Shanghai, lockdown blues make way for COVID testing gripes") had made sweatpants more suitable. Or a [facial recognition](https://www.financedigest.com/how-ai-looks-set-to-power-facial-recognition-passport-adoption-across-southeast-asia-post-covid.html "How AI looks set to power facial recognition passport adoption across Southeast Asia post-COVID") system becoming faulty because masking had become popular. The pandemic must have been a “wake-up call” for anyone not closely [monitoring AI systems because it induced countless behavioral](https://www.financedigest.com/monitoring-insider-behavior-to-catch-intruders.html "MONITORING INSIDER BEHAVIOR TO CATCH INTRUDERS") shifts, said Aleksander Madry, director of the Center for Deployable Machine Learning at Massachusetts Institute of Technology. Coping with drift is a huge problem for [organizations leveraging](https://www.financedigest.com/the-organic-beef-market-to-leverage-the-digital-disruption-wave.html "The Organic Beef Market To Leverage The Digital Disruption Wave") AI, he said. “That’s what really stops us currently from this dream of AI revolutionizing everything.” Adding to the urgency for users to address the issue, the European Union plans to pass a new AI law as soon as [next year](https://www.financedigest.com/novartis-to-spin-off-generics-business-sandoz-next-year.html "Novartis to spin off generics business Sandoz next year") requiring some monitoring. The White House this month in new AI guidelines also called for [monitoring to ensure system](https://www.financedigest.com/the-holter-monitoring-systems-market-to-virtually-grow-beyond-geographies.html "The Holter Monitoring Systems Market to virtually grow beyond geographies") “performance does not fall below an acceptable level over time. Being slow to notice issues can be costly. Unity Software Inc, whose ad software helps [video games](https://www.financedigest.com/ubisoft-shares-slump-20-after-french-video-game-maker-warns-on-revenue.html "Ubisoft shares slump 20% after French video game maker warns on revenue") attract players, in May estimated that it would lose $110 million in sales this year, or about 8% of total expected revenue, after customers pulled back when its AI tool that determines whom to show ads to stopped working as well as it once did. Also to blame was its AI [system learning from corrupted data](https://www.financedigest.com/interoperability-of-data-to-accelerate-the-whole-slide-imaging-systems-market.html "Interoperability of data to accelerate the Whole Slide Imaging Systems Market"), the company said. Unity, based in San Francisco, declined to comment beyond earnings-call statements. Executives there said Unity was deploying alerting and recovery tools to catch problems faster and acknowledged expansion and new features had taken precedence over monitoring. Real estate marketplace Zillow [Group Inc last November announced](https://www.financedigest.com/incentive-fm-group-announces-latest-strategic-me-acquisition.html "INCENTIVE FM GROUP ANNOUNCES LATEST STRATEGIC M&E ACQUISITION") a $304 million writedown on homes it bought – based on a price-forecasting algorithm – for amounts higher than they could be resold for. The Seattle [company said the AI could not keep](https://www.financedigest.com/in-listening-mode-new-bayer-ceo-keeps-open-mind-on-company-split.html "In listening mode, new Bayer CEO keeps ‘open mind’ on company split") pace with rapid and unprecedented market swings and exited the buying-selling business. NEW MARKET AI can go awry in many ways. Most well known is that training [data skewed along race](https://www.financedigest.com/the-race-to-avoid-data-dinosaur-extinction.html "THE RACE TO AVOID DATA DINOSAUR EXTINCTION") or other lines can prompt unfairly biased predictions. Many companies now vet data beforehand to prevent this, according to the surveys and [industry experts](https://www.financedigest.com/fiat-republic-the-specialist-baas-platform-that-bridges-web3-and-banks-announces-the-appointment-of-two-industry-experts-as-non-executive-directors-to-the-board-of-its-dutch-subsidiary.html "Fiat Republic, the specialist BaaS platform that bridges web3 and banks, announces the appointment of two industry experts as Non-Executive Directors to the board of its Dutch subsidiary."). By comparison, few companies consider the danger of a well-performing model that later breaks, those [sources say](https://www.financedigest.com/caixabank-proposes-sector-wide-freeze-in-variable-mortgage-costs-source-says.html "Caixabank proposes sector-wide freeze in variable mortgage costs, source says"). “It’s a pressing problem,” said Sara Hooker, head of research lab Cohere For AI. How do you update models that become stale as the [world changes](https://www.financedigest.com/deloitte-research-reveals-inaction-on-climate-change-could-cost-the-worlds-economy-us178-trillion-by-2070.html "Deloitte research reveals inaction on climate change could cost the world’s economy US8 trillion by 2070") around it? Several startups and [cloud computing](https://www.financedigest.com/the-value-of-cloud-computing.html "The Value of Cloud Computing") giants in the past couple of years have started selling software to analyze performance, set alarms and introduce fixes that together intend to help teams keep tabs on AI. IDC, a global market researcher, estimates spending on tools for AI operations to [reach at least $2 billion](https://www.financedigest.com/td-other-banks-reach-1-35-billion-settlements-to-end-stanford-litigation.html "TD, other banks reach .35 billion settlements to end Stanford litigation") in 2026 from $408 million last year. [Venture capital](https://www.financedigest.com/breaking-down-venture-capitals-gender-and-diversity-barriers.html "Breaking down Venture Capital’s gender and diversity barriers") investment in AI development and operations companies rose last year to nearly $13 billion, and $6 billion has poured in so far this year, according to data from PitchBook, a Seattle company tracking financings. Arize AI, which [raised $38 million](https://www.financedigest.com/ride-hailing-app-cabify-raises-110-million-for-expansion-in-latam-spain.html "Ride-hailing app Cabify raises 0 million for expansion in LatAm, Spain") from investors last month, enables monitoring for customers including Uber, Chick-fil-A and Procter & Gamble. [Chief Product Officer](https://www.financedigest.com/deutsche-bank-names-eigendorf-new-chief-sustainability-officer.html "Deutsche Bank names Eigendorf new chief sustainability officer") Aparna Dhinakaran said she struggled at a previous employer to quickly spot AI predictions turning poor and friends elsewhere told her about their own delays. The [world of today is you don’t know there’s an issue until a business impact](https://www.financedigest.com/exclusive-qatarenergy-boss-says-world-cup-tension-will-not-impact-german-business.html "Exclusive-QatarEnergy boss says World Cup tension will not impact German business") two months down the road,” she said. FRAUD SCORES Some AI users have built their own [monitoring capabilities and that is what FICO said saved](https://www.financedigest.com/iot-platform-wells-up-south-africas-silulumanzi-water-utility-power-asset-management-savings-and-monitoring.html "IoT Platform Wells Up South Africa’s Silulumanzi Water Utility Power Asset Management Savings and Monitoring") it at the start of the pandemic. Alarms were triggered as more purchases occurred online – what the industry calls “card not present.” Historically, more of this spending tends to be fraudulent and the surge pushed transactions higher on FICO’s 1-to-999 scale (the higher it is, the more likely it is fraud), said Scott Zoldi, [chief analytics officer](https://www.financedigest.com/host-analytics-appoints-chris-labruna-as-chief-revenue-officer.html "Host Analytics Appoints Chris LaBruna as Chief Revenue Officer") at FICO. Zoldi said consumer habits were [changing too fast](https://www.financedigest.com/the-future-of-energy-tech-and-data-are-driving-change-fast.html "The Future of Energy: Tech and Data are Driving Change, Fast") to rewrite the AI system. So FICO advised U.S. clients to review and reject only transactions scored above 900, up from 850, he said. It spared clients from reviewing 67% of legitimate transactions above the old threshold, and allowed them instead to focus on truly problematic cases. Clients went on to detect 25% more of total U.S. fraud during the first six months of the pandemic than would have been expected and 60% more in the [United Kingdom](https://www.financedigest.com/the-future-of-fintech-and-open-source-cloud-solutions-in-the-united-kingdom.html "The future of fintech and open source cloud solutions in the United Kingdom"), Zoldi said. “You are not responsible with AI unless you are monitoring,” he said. (Reporting by Paresh Dave; Editing by Kenneth Li and Claudia Parsons) --- This blog is powered by Superblog. Visit https://superblog.ai to know more. ---