ChatGPT, the popular AI-driven chat tool, is now the most popular app of all time, with the highest growing user base in history, which is a staggering achievement. It is being used in virtually every industry, from content creation to law, healthcare, finance, and even cybersecurity. At the same time, significant concerns have been raised about the security of this tool and its potential for misuse. Users are often not careful about any sensitive data they might enter when prompting ChatGPT, and how the model might store this data for further training. Cybercriminals have also turned their sights on the tool resulting in the first ChatGPT breach being reported. This article reviews the recent ChatGPT breach, how it happened, and what it means for users. 

How the breach happened

ChatGPT revealed that attackers gained access to the tool by exploiting a vulnerable open-source library used in ChatGPT’s code. This gave them access to a group of users’ chat history and personal information. Although OpenAI quickly fixed the issue and informed the general public that only a small group of users were impacted, this does open the discussion about the impact of a large-scale data breach. The vulnerable library Redis was used for caching users’ chat history for quicker response times and gave the attackers the initial foothold they needed to compromise ChatGPT. Along with the chat history, they could access personal information such as the user’s name, email address, and limited payment information. The tool was taken offline until OpenAI fixed the issue and announced an improvement in their security testing through a bug bounty program paying upwards of USD 20,000.

What the ChatGPT breach means

Despite the fast response by OpenAI and the limited exposure that occurred, the implications of the breach are severe. LLMs like ChatGPT and Bard are becoming integrated into more and more tools such as Bing and Google Workspaces. This means that a compromise could allow the attackers to move laterally to more sensitive information if sufficient isolation and sandboxing are not implemented. 

This is not even considering the sensitive information users inadvertently put into ChatGPT daily. Open AI has stated, “A large amount of data on the internet relates to people, so our training information does incidentally include personal information. We don’t actively seek out personal information to train our models.” and, “Our models may learn from personal information to understand how things like names and addresses fit within language and sentences or to learn about famous people and public figures. This makes our models better at providing relevant responses”.

Countries like Italy have already temporarily banned ChatGPT due to privacy concerns, while companies like JP Morgan have put strict guidelines around using LLMs for employees. This can become a trend in the industry as concerns around ChatGPT and similar tools grow. While AI regulation is being developed to gain some measure of  control, cybersecurity teams should be proactive in taking steps to mitigate the risks of ChatGPT compromise. 

How to protect against future ChatGPT breaches

The popularity of ChatGPT and ease of use means that it will only become more and more integrated within companies going forward. Cybersecurity teams must be ready for risks with such integrations and take proactive steps to mitigate the same, especially in industries like finance, healthcare, payments, etc., where data protection is paramount. 

It is recommended for cybersecurity teams to carry out threat modeling to identify pathways and dependencies via which attackers can potentially enter their environments via compromised LLMs. This will help to identify the blast radius of such attacks and 

Teams should set clear guidelines on using tools like ChatGPT and other LLMs and what information can be shared with them. It is impractical to restrict such tools given their benefits, so user education is paramount. Cybersecurity teams should set down guidance on what content can be generated, if source code reviews can be carried out , what research can be used, etc. with ChatGPT, so that companies are aware of the inherent risks present within LLMs. It is paramount to educate them on the privacy risks of ChatGPT when sharing personal information, as that can be potentially used for further training the model and stored by OpenAI.  For example, a user accidentally enters a sensitive document into ChatGPT and asks it to summarize, not knowing that the model might use this information. 

The potential of ChatGPT and other LLMs is immense but must be balanced with the risk they bring. The more integrated such tools become within business processes, the more significant the impact in case of a compromise. While the current ChatGPT breach was relatively small in terms of its exploration, this can be a dangerous sign of things to come as cybercriminals try to compromise LLMs to gain access to the massive amounts of data that are present.

Frequently Asked Questions