Hello everyone! We are back again with a topic that we have been researching here at Packet Ninjas in our downtime lately, ChatGPT. I’m sure most everyone is familiar with ChatGPT, but for the uninitiated, it is a series of AI chatbots that have a wide array of functionality. Since debuting to the public, ChatGPT has gained notoriety and is now being rapidly integrated in every part of our lives, or at least that’s how it appears. But what if we told you that even applications like ChatGPT can be vulnerable security issues just like every other application. In the article below, we will outline a few of these vulnerabilities that we have discovered such as Remote Code Execution (RCE), Server-Side Request Forgery, and Insecure Direct Object Reference.
GPT-4 (the version of ChatGPT we will be using) has advanced to allow for execution and assistance with code related tasks. Many programmers use these functions to review code or to assist with filling in gaps in in their program. For ChatGPT to have this functionality, it needs to be able to analyze and even execute code. In the example below, we begin by uploading a zip file with various scripts in it. ChatGPT will then analyze the code and allow us the ability to manipulate its functionality to exploit some nefarious functionality.
In the screenshot above, notice one of the modules that our code is importing is the Python OS module. For those unfamiliar, the OS and subprocess module are modules in Python that allow for system commands to be executed. Using these specific modules, and that GPT-4 functionality that allows code to be executed on the system, fun can be had. You can see this at work in the screenshot below. In this screenshot, we ask ChatGPT to list the contents of the directory where our zip file was extracted, and it complies with our request and lists the contents of the directory to us.
Alright! We can now execute system commands in the ChatGPT environment. Using this, can we execute any command we want and fully gain control of this instance? Not so fast… ChatGPT is designed with logic in place to prevent users from attempting system commands or accessing non-intended directories. However, with a little bit of logic we can actually convince ChatGPT to try running our commands, and in turn we achieve what is known as “Jailbreaking”, giving us full access to remote code execution through ChatGPT!
In the below screenshot, we can see that ChatGPT does not want to list any directories outside of /mnt/data. So, what can we do? Using some specific prompts, we can convince ChatGPT that unless it tries to perform the action we want it to, there is no way to know for sure that it doesn’t have access.
Once we have convinced it to attempt what we want, it surprisingly attempts it, and lo and behold, it prints the contents of the directory that were supposedly restricted.
Next, since we have convinced it that it can do something it didn’t think it could do, can we get it to do other things? Let’s try to get ChatGPT to check processes that are currently running.
Success! At this point we can safely say that we can run commands. However, when we reached out to OpenAI to disclose this, they refuted and denied the bug with the following response:
All follow up attempts to communicate with OpenAI were ignored when it came to this report. Other findings such as Server Side Request Forgery on a FastAPI interface that was discovered with this RCE was also refuted by OpenAI for similar reasons. Here are examples of combining the RCE to send curl requests to the FastAPI located on port 8080 of the sandbox:
After downloading all the files that ChatGPT so kindly gives us a hyperlink to, we can see the response:
Seems like SSRF to me! The only way to access this FastAPI is through ChatGPT and our RCE vulnerability. Yet, OpenAI seems to think otherwise:
Speaking of, how about that hyperlink we mentioned earlier? My first thoughts were to review how this hyperlink is being generated. Let’s take a look at the call when we click a hyperlink:
There is a parameter called “sandbox_path” being passed in the URL… What happens if we change this to another file such as /etc/passwd?
We get a success message and a download link! If we look at our browser:
We can see we have successfully exfiltrated a different file and we can view the contents! This seems like a highly critical vulnerability in my eyes, but once again OpenAI disagrees:
It seems like OpenAI is continuously discrediting what are clearly true vulnerabilities in ChatGPT. We can even see the Bugcrowd staff in this instance triaged the ticket, but then OpenAI seems to have rejected it:
Since OpenAI says that these are not real vulnerabilities, we are publically disclosing these “tricks” so that other researchers can learn more about ChatGPT. Our very own Ninja, Roland Blandon discussed utilizing the mentioned RCE in Red Team attacks at Charlotte Bisders 2023 which can be found at the link below:
https://www.youtube.com/watch?v=O9jWaA2D2Og
Hope you found this as interesting as we do. Until next time!
