The Explainability Factor

How Real-World Finance is Moving Beyond Black-box AI

AI could be a welcome addition to any finance team’s toolkit. But it is essential for users to fully understand and trust the output of these systems. With accountability for decisions made when using AI in mind, Scott Krug, CFO of American professional baseball team, the New York Yankees, and Shari Freedman, CFO of international non-profit children’s literacy advocate, Room to Read, share their views on how progress is being made.

“Finance is fundamentally about accountability,” said Aaron Harris, CTO, Sage, speaking at a virtual press conference this month in which the business software giant presented product and strategy updates. With this thought very much to the fore, Harris underlined his view that “every number needs to be explained, and every decision needs to be defended, whether it’s to the CEO, board, creditors, investors, or auditors”.

And yet, he continued, as AI becomes more embedded in finance workflows, “too much of it still operates as a black-box”. It gives answers, he explained, “but it doesn’t give you the reasoning behind it”. That blind trust is establishing a “new tension between speed and accountability”.

With the press conference aimed squarely at revealing how Sage’s latest AI solutions will help finance teams “understand what happened and why”, Harris introduced CFOs Freedman and Krug to offer their real-world take on the value of AI adoption.

Compression of time

“The biggest change happening for us is that the finance function is being asked to play a much more strategic role,” said Freedman. With global events moving at an ever-increasing pace, she says finance is being asked by many different stakeholders to “connect the dots”, to contribute, and add value across the organisation.

Planning in the midst of geopolitical uncertainty has required Freedman’s team to be more agile, she stated. Referring in particular to the organisation’s recent quarterly budget updates, she noted that technology has proven to be “the only way we can do that” in this volatile environment.

With the board asking for new risk mitigation strategies, the finance team is working with its technical colleagues to research, monitor, and evaluate how costs link with outcomes. This is vital for Room to Read, she explained, because evidence of cost-effectiveness “drives increased donor investment”.

In the world of Major League Baseball, Krug revealed a different set of pressures on finance. Nevertheless, these still occur within an environment that demands a shift away from historical reporting towards more real-time updates.

Considerable time and effort are still put into the budget process, Krug explained, but finance is required to continuously re-forecast and model new scenarios. “In my industry, we can create a Q4 budget, but then we might sign players before the season starts, which can have a material impact on the bottom line. By the time we get to spring training in March, I’m already preparing updates, and we haven’t even played a real game yet.”

With the business understanding that there are some potentially significant financial pre-season components, Krug echoed Freedman’s experience around compressed timelines for data acquisition. “The challenge we have now is how quickly can we get the information to be able to make those decisions,” he acknowledged. Raised expectations have changed the department from one “contemplating what we’re going to be doing”, to one that is involved in “a lot more collaboration earlier in the process”.

Talk of the community

With cycles compressing, finance teams having to move faster, and the decisions they are involved in becoming more strategic, Harris commented that that AI will play a major role in enabling progress. Even among business settings as diverse as Room to Read and the New York Yankees, the challenges are similar, and Sage has obviously done its homework. Indeed, Harris confirmed that the vendor has already committed to the idea of AI, and both Krug and Freedman fully support the notion that it has a role to play in finance.

During a recent series of industry meetings, Krug recalled that many conversations, regardless of what the topic was, in some way contained an AI component. “I began to realise that this is not a future scenario, it’s happening now. And it’s less about experimentation and more about actually keeping up with the pace and volume of finance work.”

Upon his return from those meetings, Krug sprang into action. “The first thing I did was to call our chief technology officer and the legal department. Together we formed an AI task force because I didn't think we were moving quickly enough.”

A key aim for Krug, as CFO, is increased departmental efficiency. Although he has multiple tasks, projects to implement, and improvements planned, he conceded that his staff do not have the capacity for additional undertakings. Forcing the issue, he appreciates, is counterproductive: “It could affect my close and other activities that keep us up and running”. So, if there is to be “more analysing and less inputting”, it will come from finding a way to free-up some of the time devoted to finance’s manual tasks. “Then I can reallocate them to more strategic work.”

The interest in AI at Room to Read’s finance function is similarly spurred on by Freedman’s visits to several CFO conferences. In the past two years, she noted “not one of them has been without AI receiving more and more airtime”.

Describing the pace of general technological evolution as “stunning”, she added that with a host of new products emerging, the team at Room to Read has decided on prioritising AI that has already been integrated into the organisation’s current technology stack, “from vendors that we’ve vetted and we trust”.

Getting comfortable

AI is expected to help Freedman continue streamlining Room to Read’s financial processes. Procure-to-pay has already benefitted from enhancements, and now new ways of optimising its travel and expense (T&E) technology, by integrating this into its technology stack, are being explored.

Another area of AI application that is being considered by Room to Read is its production of the US Internal Revenue Service (IRS) form 990. This is a tax declaration that must be filed annually by nonprofit organisations, explained Freedman. “It’s a major manual process, but we are fortunate in that we have several years of forms already calculated and completed. This means we can use those to test AI output for consistency and clearly articulate that we understand where that data is coming from, how it's being, and how the data analytics are working.”

In the build up to AI roll-out, Room to Read has a distinct advantage. Its finance team has become a keen early adopter of technology. “I've encouraged experimentation. I want people to hear that I'm using it,” said Freedman. “And now I want them to know that they’re free to use it, in a controlled environment. They need to get comfortable with technology because change management is probably the biggest stumbling block in terms of its adoption.”

A key aim with AI, as with the New York Yankees, is to free up finance team time, while ensuring the quality of its output is not eroded. The cost of introduction and maintenance has to align with that goal, said Freedman. To enable the success to date with AI, it was first necessary within Room to Read to establish certain capabilities. “We started a couple of years ago to map out the level of digital capacity building and digital skills required within finance,” she revealed.

External and internal training sessions for staff have enabled the early-adopter approach to flourish within smaller bursts of activity. But now, in order to move to scale, Freedman recognises the need for deeper guidelines and policies. This requires a high degree of engagement.

“We’re particularly careful about policies around data privacy,” stated Freedman. “We’re a global organisation. We work in 29 countries, many of which have their own data privacy requirements and laws.” In Singapore, the team had just finished a week of meetings, with senior leadership working to align organisational values with its use of technology.

“We want clear guidelines on what is acceptable and what we can’t do. When we look at the potential of AI, we know for us that it has to function within a closed system because we don't want any of our donor information in the public domain. That’s why we're walking slowly with this.”

Transparency and trust

The focus on trust and security within AI systems and their outputs will play a major role in how organisations approach and implement AI tools. It also feeds into how users will select system partners, noted Harris. For organisations such as the New York Yankees and Room to Read to continue on their journey, it calls for transparency.

Progress requires trust in what vendors are developing, and that means a move away from unknowable black-box systems. For Krug, the driver for seeking AI transparency and vendor-trust is simple: “Number one, two, and three on my list is security”.

Every organisation holds sensitive information, but sometimes there can be lapses in protection. “There have been well-documented situations in my industry of employees leaving clubs and taking confidential information to another club,” Krug revealed. The outcome of any such breach is serious. “The sensitive data in my world could end up being a competitive advantage for another team if they were to get hold of it. If that happens, it ends up on the back pages of the newspapers.”

It’s no wonder then that the overriding consideration for Krug around AI is making sure his team is focused on security. But at a day-to-day practical level, there needs to be trust in the technology, and that, he believes, resides in the ‘explainability’ of a model’s outputs.

“If I can't explain what the AI tool is doing, I'm not sure how I’m supposed to be able to trust it,” stated Krug. “I need to have a clear understanding of the data that went in, the assumptions that were made, and that the outputs are consistent and repeatable.”

In summing up his approach to AI, he explained that while security focuses on protecting the integrity of the data and infrastructure, its explainability provides the necessary justification and transparency for human decision-makers to act confidently and legally. In essence, “security protects the organisation, explainability protects the decision maker”. 

For Freedman, it is important to ‘get under the hood’ of a system. At a theoretical level at least, she said she needs to know how the system was created, and how it makes its decisions. But in practice, with operations across 29 countries, and these being subject to myriad local disclosure requirements, she also seeks an understanding how the system meets regulatory compliance.

“I've got the general counsel [the chief in-house lawyer] reporting up to me, and we’ve spent a lot of time over the past couple of months really digging in to this,” she revealed. “We’re trusted by our stakeholders – such as our donors, ministers of education, and sector leaders – and we would have to test all of those outputs to make sure that confidence at a regulatory level continues.”

With “big expectations for AI to free up time and enable teams to be more strategic”, Harris also acknowledged a “loud and clear” message to AI vendors from organisations such as the New York Yankees and Room to Read. “There will be no compromise on security. AI has to work, and it has to work consistently. But it also has to be developed, from inception all the way through to production, in a way that comports with the values of every organisation that uses it.” The days of inscrutable black-box AI may now be numbered; it’s time to move to the next phase.