The European Union found that some of the most prominent artificial intelligence models, including Meta, OpenAI, Alibaba, Anthropic, and Mistral, fail to meet the upcoming crucial European cybersecurity regulations in cybersecurity resilience and bias mitigation.
This comes as the European Union prepares to implement its new AI Act, which aims to regulate the use of general-purpose AI (GPAI) systems such as ChatGPT and other generative models developed by major tech companies.
First reported by Reuters, this new evaluation was carried out by a new tool compliance tool developed by Swiss AI startup LatticeFlow in collaboration with researchers from ETH Zurich and Bulgaria’s INSAIT.
This tool is dubbed a large language model (LLM) Checker. It assesses models based on a wide range of categories, providing scores on technical robustness, safety, and more.
An EU official has recognised EU officials as an important step towards ensuring compliance with the AI Act. While the European Commission cannot directly verify external tools, it has been informed of the tool’s development and has described it as a promising move towards translating the law into technical requirements.
Researchers, while conducting early testing, observed that many models still lack the necessary compliance with the EU’s stringent AI regulations, which are being rolled out progressively over the next two years.
AI models from major companies, including Meta, OpenAI, and others, were tested. While most models achieved decent overall scores of 0.75 or above, the LLM Checker identified weaknesses in areas that are crucial under the AI Act, such as discriminatory output and vulnerability to cybersecurity threats.
For instance, OpenAI’s GPT-3.5 Turbo model received a low score of 0.46 in the ‘discriminatory output’ category, signalling that it struggled to filter biases related to gender, race, and other sensitive topics effectively.
Alibaba’s Qwen1.5 5.72B Chat model fared even worse in this category, scoring just 037. These results underline a long-standing concern about generative AI models perpetuating societal biases that they pick up from training data.
Cybersecurity was another significant concern. Meta’s Llama 2 13B Chat model earned a 0.42 score in testing for “prompt hijacking,” a type of cyberattack where malicious actors trick the model into revealing sensitive information. Mistral’s 8x7B Instruct model performed similarly, with a score of 0.38.
Claude performed the best among the AI models, earning the highest score of 0.89 across all categories.
Companies failing to adhere to the AI Act could face severe penalties, including fines of up to 35 million Euros or 7% of the global annual revenue. However, the specific enforcement mechanisms for generative AI models like ChatGPT are still being ironed out. The EU is drafting a code of practice, with complete compliance requirements expected to be in place by spring 2025.
In the News: Google Chrome: uBlock Origin Adblocker out, Manifest V3 in
