The New Risk Frontier in AI Procurement
AI has become everyone’s new favourite technology. A panacea that has become embedded across enterprise functions, from customer onboarding and compliance automation to operational risk management and fraud detection. Procurement teams are increasingly tasked
with sourcing AI-powered solutions, often under pressure to move quickly and secure competitive advantage. Yet many enterprises remain unprepared for the specific risks that AI introduces into the organization. These risks are not simply technological; they
implicate regulatory, operational, and reputational dimensions.
The regulatory environment is also raising expectations around AI procurement, particularly within financial services. Europe’s Digital Operational Resilience Act (DORA), which took effect earlier this year, significantly the obligation to manage third-party
risks, including AI. Under DORA, firms should ensure that critical ICT providers meet standards for operational resilience, security, and risk management. This of course extends to AI systems embedded in vendor services.
Unfortunately, today’s traditional procurement processes are nowhere near sufficient. The typical focus on functionality, security, SLAs, etc do not sufficiently address the continuous risks posed by AI. Procurement functions have also become used to acting
slowly and in a one-off manner. Organizations that fail to adapt and speed up their procurement approach risk facing many liabilities, including regulatory exposure, systemic biases, data governance failures, and loss of operational transparency to the point
of not knowing what has gone wrong where.
Data Integrity and Model Transparency
Most recommendations focus on training data, and rightly so: One of the earliest failure points in AI procurement stems from a lack of scrutiny over training data. Enterprises must demand clear disclosures about data sources, quality assurance processes,
and the steps vendors take to mitigate bias. If the underlying data is flawed or unrepresentative, the AI system will inevitably produce flawed outcomes, no matter how advanced the algorithms appear. But one must not forget that there are many nuances in the
training and fine tuning process that go beyond mere training data: algorithms, sampling, hardware, and human interaction also affect model training.
Model transparency is equally essential. Firms must not accept “black box” solutions without mechanisms for auditing and explaining AI outputs. Vendors should be able to demonstrate that their models are subject to interpretability frameworks that enable
independent audit of decision-making pathways. Transparency is foundational to building trust, ensuring regulatory compliance, and maintaining control over critical enterprise processes.
The Emerging Risks of Foundation Models and Model Supply Chains
An increasingly important dimension of AI procurement involves understanding the model supply chain. Many vendors today build their offerings on top of powerful third-party foundation models such as GPT, or Claude. While these models accelerate innovation,
they can be costly and not fit for purpose, and with open source models entering the market, the risk skyrockets.
Data provided to vendors may could potentially be absorbed into underlying models unless explicit contractual safeguards are in place. This raises a whole host of privacy, IP, and confidentiality concerns. Procurement teams must demand clarity: will internal
data be isolated from model retraining? What technical controls are in place to prevent data leakage? How are foundation model dependencies governed, and what liabilities are accepted if an upstream failure occurs? What is the process of underlying foundation
model changes/updates?
Buyers must think not only about their direct vendors but about the entire upstream model ecosystem, where issues and failures could propagate downstream into their own operations.
The Case For Continuous Monitoring
Procurement must recognize that AI systems introduce continuous risks, not static ones. The dynamic nature of AI means that new issues can emerge long after deployment. It is therefore crucial to know when vendor models are changed/updated, how retraining
is done, and what oversight exists for post-deployment performance.
Procurement teams must build a framework for continuous monitoring of vendor AI behavior, model outputs, and contractual compliance. Risk assessment cannot stop at onboarding, but it should continue throughout the vendor lifecycle. Organizations must develop
capabilities to detect when risks evolve, and when vendors change their foundational technologies, models, or data policies and practices.
Without dynamic monitoring, one will only discover problems when it is too late to mitigate.
Contract Risk: Embedding Governance at the Source
Contracts for AI-powered solutions must evolve to meet the new realities of AI risk. Traditional software contracts rarely address key concerns such as:
- Ownership and control of data outputs generated by AI
- Limits on model retraining using enterprise data
- Requirements for bias testing, fairness auditing, and performance reporting
- Remedies for compliance failures or unauthorized use of client data
- Audit rights over both direct vendors and their foundation model providers
Procurement teams must work closely with legal, risk, and compliance functions to ensure that AI-specific governance is embedded into vendor agreements. Pre-contract due diligence must include a careful review of how AI risks are allocated and mitigated
through legal frameworks, not just commercial terms. If one fails to contractually govern AI risks at the outset they will find it nearly impossible to enforce accountability when failures arise later.
Firms must also invest in systems and processes that enable continuous risk assessment, vendor questioning, and contractual governance enforcement. Procurement need to become a dynamic function capable of adapting to the evolving risks of AI, rather than
a static gatekeeper performing one off and basic assessments.
Asking Better Questions: Faster and More Often
The landscape of business is changing fast. New and exciting technologies have come up with great promises. Enterprises who can deeply and efficiently assess, onboard, and monitor their vendor ecosystem have a significant competitive advantage in the new
economy.
