The attack technique – use ChatGPT to spread malicious packages
We have identified a new malicious package spreading technique we call, “AI package hallucination.”
The technique relies on the fact that ChatGPT, and likely other generative AI platforms, sometimes answers questions with hallucinated sources, links, blogs and statistics. It will even generate questionable fixes to CVEs, and – in this specific case – offer links to coding libraries that don’t actually exist.
Using this technique, an attacker starts by formulating a question asking ChatGPT for a package that will solve a coding problem. ChatGPT then responds with multiple packages, some of which may not exist. This is where things get dangerous: when ChatGPT recommends packages that are not published in a legitimate package repository (e.g. npmjs, Pypi, etc.).
When the attacker finds a recommendation for an unpublished package, they can publish their own malicious package in its place. The next time a user asks a similar question they may receive a recommendation from ChatGPT to use the now-existing malicious package. We recreated this scenario in the proof of concept below using ChatGPT 3.5.
What is an LLM hallucination?
Large language models (LLMs), like ChatGPT, can sometimes lead to fascinating instances of hallucination, where the model generates creative yet unexpected responses that may not align with factual reality. Due to extensive training and exposure to vast amounts of text data, LLMs have the ability to generate plausible but fictional information, extrapolating beyond their training and potentially producing responses that seem plausible but are not necessarily accurate.
In our case, ChatGPT might guess the name of a repository based on data it found in GitHub, or other similar sources, which seems reasonable to suggest as a valid package.
As we know, ChatGPT answers are currently based on version GPT-3.5, which uses training data gathered through September 2021. Reliance on this data could also lead ChatGPT to recommend a package which was available in the past but no longer exists today.
How to spot AI package hallucinations
It can be difficult to tell if a package is malicious if the threat actor effectively obfuscates their work, or uses additional techniques such as making a trojan package that is actually functional.
Given how these actors pull off supply chain attacks by deploying malicious libraries to known repositories, it’s important for developers to vet the libraries they use to make sure they are legitimate. This is even more important with suggestions from tools like ChatGPT which may recommend packages that don’t actually exist, or didn’t before a threat actor created them.
There are multiple ways to do it, including checking the creation date, number of downloads, comments (or a lack of comments and stars), and looking at any of the library’s attached notes. If anything looks suspicious, think twice before you install it.
[요약]
1. 개발자가 ChatGPT에게 기능 구현 방법을 물어봄
2. ChatGPT가 존재하지 않는 패키지의 사용을 제안
3. 악의적인 공격자는 해당 이름의 패키지에 malware를 심어 배포
4. 같은 기능 구현을 요청한 사람이 해당 패키지를 이용하면 해킹
출처: https://vulcan.io/blog/ai-hallucinations-package-risk
Can you trust ChatGPT’s package recommendations?
ChatGPT can offer coding solutions, but its tendency for hallucination presents attackers with an opportunity. Here's what we learned.
vulcan.io