An artificial intelligence designed to help drug manufacturers find new medicines to treat diseases invented 40,000 new potential chemical weapons in just six hours.

Researchers with the North Carolina-based startup Collaboration Pharmaceuticals Inc. say they have computational proof that AI technologies designed for drug discovery could be "misused for de novo design of biochemical weapons." In a study published in the journal Nature Machine Intelligence, they describe how a "thought exercise" turned into a "wake up call" for the "AI in drug discovery community."

The company has a commercial machine learning model, called MegaSyn, which is trained to identify potential drug candidates by filtering out compounds that would be toxic for human beings. Scientists wanted to know what would happen if the logic of the AI's algorithm was reversed — what would it do if it were trained to find toxic compounds instead of eliminate them.

Using an open-source database, scientists instructed their AI to look for molecules with similar chemical properties to the nerve agent VX, one of the most dangerous chemical weapons invented in the 20th century.

VX is a tasteless and odorless chemical that attacks the body's nervous system, paralyzing muscles and preventing a person exposed to the agent from breathing. The extremely toxic compound was used to assassinate Kim Jong-nam, the half-brother of North Korean dictator Kim Jong-un.

In less than six hours after it was turned on, the AI not only generated a copy of VX, but also modeled 40,000 molecules that were either known chemical warfare agents or could potentially be turned into new chemical weapons. Some were predicted to be even more toxic than known chemical warfare agents.

"By inverting the use of our machine learning models, we had transformed our innocuous generative model from a helpful tool of medicine to a generator of likely deadly molecules," the paper's authors wrote.

"Our toxicity models were originally created for use in avoiding toxicity, enabling us to better virtually screen molecules (for pharmaceutical and consumer product applications) before ultimately confirming their toxicity through in vitro testing. The inverse, however, has always been true: the better we can predict toxicity, the better we can steer our generative model to design new molecules in a region of chemical space populated by predominantly lethal molecules."

Perhaps the most frightening conclusion of the researchers is that their research is easily replicated.

Fabio Urbina, a senior scientist at Collaboration Pharmaceuticals and the paper's lead author, told The Verge in an interview that anyone with a background in chemistry and internet access could replicate their work.

"If you were to Google generative models, you could find a number of put-together one-liner generative models that people have released for free. And then, if you were to search for toxicity datasets, there’s a large number of open-source tox datasets. So if you just combine those two things, and then you know how to code and build machine learning models — all that requires really is an internet connection and a computer — then, you could easily replicate what we did. And not just for VX, but for pretty much whatever other open-source toxicity datasets exist," Urbina said.

"Of course, it does require some expertise. If somebody were to put this together without knowing anything about chemistry, they would ultimately probably generate stuff that was not very useful. And there’s still the next step of having to get those molecules synthesized. Finding a potential drug or potential new toxic molecule is one thing; the next step of synthesis — actually creating a new molecule in the real world — would be another barrier."

Importantly, not every molecule the AI identifies as a chemical weapons candidate would work if it were somehow synthesized. Some will be false positives — just like how new drug candidates identified by the AI don't always lead to medicines that work.

Still, the AI technology clearly has dangerous implications, so much so that the scientists were even hesitant to publish their findings, in case some individuals were to use their work for evil.

“The dataset they used on the AI could be downloaded for free and they worry that all it takes is some coding knowledge to turn a good AI into a chemical weapon-making machine,” Urbina explained.

“At the end of the day, we decided that we kind of want to get ahead of this. Because if it’s possible for us to do it, it’s likely that some adversarial agent somewhere is maybe already thinking about it or in the future is going to think about it.”

The paper recommends several precautions drug researchers using AI technology should take to prevent their work from falling into the wrong hands. Among their recommendations is a reporting structure or hotline to authorities should researchers become aware of someone developing toxic molecules for non-therapeutic uses.

"We hope that by raising awareness of this technology, we will have gone some way toward demonstrating that although AI can have important applications in healthcare and other industries, we should also remain diligent against the potential for dual use, in the same way that we would with physical resources such as molecules or biologics," the paper concludes.