MMOOZZGG.com

Glossary

What is self-consistency prompting

In short

Self-consistency prompting is a technique in which a language model solves one task several times by different paths. The final answer is the one that most solutions reach.

Prompts · self-consistency

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine that you and your friends are counting the candies in a big jar. Each of you counts alone and says a number. One gets forty, three get forty-two, and the last one gets forty-five. Most likely the right answer is forty-two, because most of you got it. A computer sometimes does the same: it solves a problem several times and picks the answer that came up most often.

In plain words

An AI model does not answer by a fixed rule: every answer has some randomness. So on a task with a calculation it can be right once and wrong the next time. This technique turns that weakness into a benefit. The model is asked to solve the task step by step many times, and each time the reasoning takes its own path. Then only the final numbers are compared, and the one that appears most often wins. The mistakes are usually different, while the right answer repeats. It is like several accountants checking an estimate on their own and trusting the figure most of them got.

For business

The method raises accuracy where a task has one right answer: the sum in a contract, the category of a request, a check of an application against the rules. The cost is direct: each extra run is paid for separately, and the answer comes later. So it is used only where a mistake is expensive. It also helps control: if the answers differ a lot, the system is not sure, and the case goes straight to an employee. For open tasks such as emails, voting does not work, because no two texts are the same. Test on real data how many runs pay off, and look at AI errors and who is responsible for them before launch.

For a specialist

Self-consistency was proposed by Wang and co-authors in 2022 (published at ICLR 2023) as a replacement for greedy decoding in chain-of-thought reasoning. Instead of one chain, the model samples a set of diverse reasoning paths and then marginalizes over them. The result is the answer that most paths reach, which is in effect a majority vote on the final answers. The method needs no fine-tuning, no extra model and no labeling. The authors gained over plain chain of thought on arithmetic and common-sense tasks: GSM8K by 17.9%, SVAMP by 11.0%, AQuA by 12.2%, StrategyQA by 6.4% and ARC-challenge by 3.9%. The limits: cost grows linearly with the number of samples, and voting works only when answers can be compared for a match, such as a number or an answer option. For free text the method does not work in its original form. The spread of answers also serves as a measure of the model’s uncertainty. A close idea, with branching and scoring of the middle steps, is the tree of thoughts.

Example

An accounting bot calculates a contract penalty five times. Four runs give one sum and the fifth gives another. The system sends the client the majority sum and logs the difference for a check.

Related terms

Questions and answers

How is self-consistency different from chain of thought? Chain of thought is one step-by-step solution. Here several such solutions are built, and the answer that most of them reach is chosen.
Why is this method expensive? Each extra run of the model is paid for in tokens and takes time. So the number of runs is chosen so that the gain in accuracy pays for the cost.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us