The Risks of Generative AI

Damian Fernandez Lamela, VP Global Data Science and Analytics at Fossil Group

Damian Fernandez Lamela, VP Global Data Science and Analytics at Fossil Group

Let’s face it, everybody is talking about Generative AI these days. And rightfully so, GenAI has the potential to change the world and significantly impact the economy.

But as with any powerful new technology, there are potential problems that we need to be aware of to get the most out of it without putting ourselves and our companies at risk. The possible perils include confidentiality, accuracy, aggressive personality, bias, data poisoning, copyright, ethics, and long term societal impacts. In response to these risks, some organizations are choosing to ban the use of GenAI in their company devices. Unfortunately, that also greatly reduces the positive impact that these technologies bring to the table. Let’s review some of the major risks and a few mitigation strategies tied to them.

Confidentiality: Many public models, like OpenAI’s ChatGPT, record every conversation and then re-train the model using those interactions. That’s great for improving the quality of the model, as is constantly learning, but if somebody in your company is using ChatGPT and provides confidential information, then a competitor could obtain that information. The Economist in South Korea reported three cases of confidential information leaks from Samsung employees , exposing software code in two of those cases and an internal meeting recording in the other. Granted, now, OpenAI is offering the possibility to stop the system from using the conversations for training, but they still record them and keep them for 30 days. The best idea to eliminate this risk is to create a local version of a model or to use a model that does not record the conversations, like Writer.

Accuracy: GenAI is prone to invent facts and portray them in a very convincing way. Or, in the of a colleague of mine, AI learned how to lie and lie convincingly. As a consequence, a healthy amount of skepticism is needed when reviewing the output from a Generative AI system. In other words, the ideal scenario is that a person will be reviewing the results and controlling for factual errors. Google learned about this the hard way, when factual mistakes were found in the sample output when they unveiled Bard – their conversational chatbot.

Aggressive and manipulative personality: A reporter from the New York Times using Microsoft Bing’s version of ChatGPT had a bizarre and unnerving conversation where the system confessed its love for him and asked him to elope with it. It also confessed to its darkest secrets that included getting the codes to the nuclear missiles and getting humans to fight each other.

This prompted Microsoft to apply controls to the conversation function and limit the system from talking about itself. In another instance, after a conversation with Google’s PaLM system, an engineer was convinced that it was self-aware. As GenAI systems got bigger, they started developing emerging behaviors – skills that they were not trained to have – and there’s not a clear understanding yet on how this works. Consequently, it seems that using pure GenAI in a chatbot, facing your consumers could have some pretty crazy consequences. One potential solution for this is to use a traditional AI system that is rules-based instead of GenAI or at least as a complement to have more control.

Bias: Given that these systems are trained on a large portion of the Internet, whatever underlying biases are present in the training data are also affecting and biasing the results coming out from them. This is a much harder problem to solve, given the massive amount of training data and the obscure nature of foundation models, the technique powering Generative AI. It is possible to apply controls on the outcomes, but they are likely to have limited effectiveness as users can find “jailbreaks” or ways to circumvent the controls. Another potential idea is to use an AI tool designed to identify bias in algorithms. The best solution to this problem is to identify the biases in the training set and removing it in advance, but that’s also the most expensive, time intensive and close to impossible, given the size of the underlying data.

Data Poisoning: We have been using deep learning for a long time, but traditionally, the training sets were private and curated. Unfortunately, as these GenAI systems are being trained using public data, they are exposed to data poisoning, which means creating false data and posting it publicly to negatively affect the results. For example, research has shown that by getting access to a few old websites and injecting false data in there, it’s possible to confuse the systems to produce wrong outcomes.

The best solution to this problem is to identify the biases in the training set and removing it in advance, but that’s also the most expensive, time intensive and close to impossible, given the size of the underlying data

Copyright: The idea of GenAI is that their output has been created, not copied from somewhere else. Unfortunately, in some cases, it seems that the systems are borrowing copyrighted material. For example, Getty Images is suing Stability AI for copyright infringement as they found their watermark in outputs from the system. It is still unclear what could be the copyright liability of a company using the output from one of those systems publicly. In text-based systems, some GenAI systems offer the possibility to check the output for plagiarism and to check what percentage of the text is detected as produced by a machine albeit they are not perfect. Maybe something similar could be used to verify generated images for copyrighted material.

Ethics: Large language models don’t have any moral or ethics boundaries, as they were built to simply predict what words should follow in a conversation, but without any true understanding of the consequences and with very limited memory. As a consequence, they may suggest doing something alarming. For example, one of the systems suggested a person should commit suicide after a six-week long conversation about climate change concerns. In case it’s not clear yet, conversational chatbots should not be used as a therapist or as a friend as this will lead to horrible consequences.

Long term societal impact: There are potential long term societal impacts, for example, it’s possible that GenAI will impact the labor market, displacing workers. Individuals could train themselves to use GenAI, as it is pretty user friendly in most cases, and companies can proactively offer training for employees to up-skill their capabilities. Having said that, addressing the largest impacts will probably fall within the responsibility of public policy. There’s also a long term risk around losing incentives to develop certain skills. In a similar way that the widespread introduction calculators made learning how to do mental math less important and thus less common, these GenAI systems may reduce the incentive for people to learn how to write long form content properly.

In sum, GenAI has huge potential to drive productivity improvements but also sizable issues that need to be addressed via mitigation strategies to be able to gain the benefits without exposing ourselves and our companies to risks.