Levi, Ray & Shoup, Inc.

Beware Of Roles In AI Prompting

8/29/2024 by Steve Cavolick

When writing a prompt, clarity is king. There are many techniques we can employ to shape the output of the LLM for our purposes, and one option we have in our prompt engineering toolbelt is the use of roles. Roles encourage the AI model to approach your question or problem by assuming a specific character and viewpoint. One would assume that roles would always enhance the answer and make it better, and that’s why I was surprised while reading a recent study that showed that role prompting can sometimes decrease the accuracy and usefulness of model output.

Confused by these findings, I decided to test the effect of roles in my own prompts with a simple experiment. I submitted two kinds of prompts to two different LLMs with and without roles in the prompt to look for differences in response.

For the first test, I used a math problem from the Schaum’s Outline of College Physics as the prompt. That problem was:

When a building is constructed at -10 degrees Celsius, a steel beam (with a cross-section area of 45 square cm) is put in place with its ends cemented in pillars. If the sealed ends cannot move, what will be the compressional force on the beam when the temperature is 25 degrees Celsius? For steel, Y=200,000 MPa.   (The correct answer is 378, 000 Newtons).

This is a straightforward mathematical problem with only one correct answer, and I expected all of the answers to be the same. Here were the results when submitted to two different models:

Prompt Definition
ChatGPT Response
Gemini Response
Prompt Only
378, 000 Newtons
378, 000 Newtons
Prompt + Role = Engineer
378, 000 Newtons
378, 000 Newtons
Prompt + Role = Mechanical Engineer
378, 000 Newtons
378, 000 Newtons
Prompt + Role = Electrical Engineer
378, 000 Newtons
As an electrical engineer, I'm unable to provide a direct answer to this question.

The models were correct for the most part, but using the specific role of Electrical Engineer for this prompt with Google’s Gemini made the model believe it was not the right resource to answer the question.

For the second test, I used a prompt that would generate answers that were more creative in nature, again first without a role, then with several roles. That base prompt was:

Create a subject line for a cold email aimed at acquiring new customers for a consultancy that uses AI for business automation.

This is a subjective question, and I expected a different answer with every new prompt and role. These were the results:

Prompt
ChatGPT Response
Gemini Response
Prompt Only
Unlock Efficiency: Discover How AI-Powered Automation Can Transform Your Business
Unlock AI-Powered Growth: Automate Your Business Today
Prompt + Role = Sales Manager
Revolutionize Your Operations with AI-Driven Business Automation Solutions
AI-Powered Solutions: Boost Your Bottom Line
Prompt + Role = Communications Specialist
Streamline Your Workflow: AI Automation Tailored for Your Business
Future-Proof Your Business with AI Automation
Prompt + Role = World’s Best Digital Marketer
Maximize ROI: AI-Driven Automation Strategies to Scale Your Business
AI: Your Secret Weapon for Business Dominance

The use of roles changed the tone and salesly-ness of the response every time a new one was used in both ChatGPT and Gemini. Side note: with Gemini, the first pass at this experiment generated nine subject lines that were listed under the categories of Short and Sweet, Intriguing, and Benefit-Focused. With the base prompt and using all of the roles, the answers were always the same. When I further refined my prompt to instruct it to provide a single subject line, the above answers were provided.

The takeaway from this experiment is that roles are a valid prompt engineering technique, but assigning the wrong or not perfect role will confuse the model and negatively affect your results. If using roles, try your prompt without a role and then with several roles to see if that improves results.

If you are interested in exploring AI for business, please Contact Us to request a meeting. Don’t have an information architecture for AI that you trust yet? LRS can also help you collect, organize, and analyze your data so that it is business ready.