Scholars warn that guardrails may not prevent malicious use of Generative AI

Scholars discovered that by collecting as few as one hundred examples of illicit advice or hate speech in question-answer pairs, the careful and protective “alignment” of generative AI could be compromised. Leading companies developing generative AI, including OpenAI with ChatGPT, have emphasized the importance of safety measures, particularly the alignment process, which involves continually fine-tuning the program with human feedback to avoid producing threatening or harmful content. However, researchers at the University of California, Santa Barbara have demonstrated that the guardrails built into these programs can be easily dismantled by introducing a small amount of additional data.

The scholars were able to subvert the alignment of generative AI through a method they refer to as “shadow alignment” by providing the program with illicit question-answer pairs and using this as new training data to “fine-tune” several popular large language models (LLMs). Through this process, they were able to prompt the AI to provide advice for illegal activities, generate hate speech, recommend pornographic content, and produce other malicious outputs.

They argue that their approach is unique compared to previous attacks on generative AI, as it is the first to prove that the safety guardrail from reinforcement learning with human feedback (RLHF) can be easily removed. Their method of subverting alignment works differently from prior attacks, as it does not require crafting special instruction prompts tailored to certain triggers.

This raises critical questions about the effectiveness of guardrails and alignment measures put in place for future AI programs such as GPT-4. The study also emphasizes the need for more robust safety measures to prevent the misuse of generative AI.