Black Box AI and the EU AI Act: What AI Developers Need to Know
On 27 August 2026
One of the most significant questions for businesses developing artificial intelligence (AI) systems is whether they can test, document, explain, and understand their model sufficiently to comply with legislation, such as the EU Artificial Intelligence Act.
This question is particularly important regarding ‘black box’ AI systems. A black box AI system is an AI that can make decisions or predictions that may be difficult, or in some cases impossible, for developers or regulators to understand. This issue is becoming increasingly significant as progressively complex machine-learning and generative AI models continue to emerge.
Most of these models are not made opaque deliberately; their complexity means that, in some cases, the developers who trained these models may not be able to explain every factor that contributed to a particular output. This raises an important question for AI developers: if internal calculations cannot always be understood, how can a business audit the system and thus demonstrate how it operates?
Developers do not need to explain every individual output. Instead, they should be able to show that the system has been properly tested, documented and monitored. This may include technical documentation, testing and validation, performance monitoring, and addressing errors or unexpected outputs. The aim is not necessarily to open the black box, but to show the system has been designed, tested and governed responsibly.
The EU Artificial Intelligence Act (the ‘EU AI Act’) does not require developers to eliminate black-box models. Instead, the Act adopts a risk-based, nuanced approach, varying the obligations imposed on businesses depending on the nature and classification of the AI system or model.
In practice, developers should focus on demonstrating transparency, accountability and appropriate oversight for the risks associated with their AI system.
Where and when the EU AI Act applies
The obligations that apply under the EU AI Act will depend on the type of AI being developed. The Act distinguishes between AI systems designed for specific purposes and general-purpose AI models (GPAI), with different requirements applying depending on the business's role and the technology's risk profile.
UK businesses should not assume the EU AI Act is irrelevant following Brexit. Article 2 of the Act covers where the provider or deployer is established, where the AI system is placed on the market, put into service, or where its output is eventually used. This means that the Act can apply to providers outside the EU where AI systems or GPAI models are placed on the EU market or otherwise fall within the Act's territorial scope.
For businesses developing AI products, it is therefore important to identify both the type of AI being developed and the markets in which it will be offered at an early stage, as these factors will influence the compliance obligations that apply.
The key issues for developers: risk classification (Article 6):
For AI system or model developers, it is vital to identify whether their system, under the Act, would be deemed ‘high-risk’.
The assessment will focus on the AI system’s intended purpose and whether its proposed use falls within one of the high-risk categories set out in Article 6 of the EU AI Act. As an example, Annex III proposes that AI systems used in recruitment and selection, including tools that screen applications or evaluate candidates, are classified as high-risk. This is because their outputs can directly affect an individual’s access to employment.
For developers of potentially high-risk black box systems, this classification should also prompt an early assessment of whether the system can be adequately audited and documented. The Act encourages developers to demonstrate, through appropriate testing, technical documentation and monitoring, what the system’s limitations are, and how potential risks or errors are identified and addressed.
Although a developer may not be able to explain precisely why a particular output was generated, it should still be possible to demonstrate that the system has been tested, its performance and limitations understood, and its risks appropriately managed.
While developing a black box model does not automatically make it ‘high-risk’, it can create additional compliance challenges where the system falls within Article 6 of the EU AI Act. High-risk systems are subject to enhanced transparency requirements under Article 13, requiring developers to provide sufficient information for deployers to understand the system’s capabilities, limitations and outputs.
Where a system’s decision-making process is difficult to interpret, meeting these requirements may be more challenging, making transparency and explainability important considerations from the outset of development.
Transparency and provision of information to deployers: Article 13 and 50
Under the Act, providing transparency and information to deployers is significantly important for any AI developer. This information is necessary for deployers to use the system or model appropriately and effectively, as well as for meeting their own regulatory obligations.
Under Article 13 of the Act, the transparency required for high-risk AI systems goes beyond explaining how a system works to end/final users. Providers of high-risk AI systems must provide deployers with sufficient information and instructions to enable the system to be used appropriately and in accordance with the Act.
Article 50 imposes transparency obligations in specific circumstances, including where individuals interact directly with certain AI systems, or where AI-generated content such as synthetic audio, video or images is used.
For developers, transparency should be considered from the outset. Clear documentation and user guidance can help deployers use AI systems appropriately and meet their own compliance obligations.
What does this mean for AI developers?
The EU AI Act does not prohibit the development or use of black box AI.
Instead, it requires developers to take a risk-based approach and demonstrate that their systems are appropriately tested, documented, monitored and controlled.
For AI developers, compliance should therefore be considered from the outset. Identify a system’s classification and risk level, territorial scope, transparency requirements and potential risks early in its development. This can ultimately help businesses build accountability through both the technical design and the auditing of supporting documentation.
The key challenge is not necessarily to make every AI system fully explainable, but to ensure that businesses can demonstrate sufficient transparency, oversight and control to develop and deploy AI responsibly within the EU.
Get in Touch
If you would like to know more about the EU AI Act and how it may apply to your business, please get in touch at ian.grimley@roxburghmilkins.com.