Synthetic users vs humans: replicating a B2B research study

Did Candor's synthetic insurance agents reproduce the card sort results we got from 8 human participants at independent insurance agencies? They passed the card sort test and the reasons test, but raised fewer of the reasons the human participants gave.

In March 2025, Highline Beta ran a client project with people who own or help run independent insurance agencies. In one research phase, 8 participants sorted a set of agency problems by how challenging each one is for them, and explained why. In November 2026, we ran the same exercise with synthetic insurance agents in Candor to see how closely the results matched. We set each pass mark from the human data, before we scored any synthetic data.

This is the second article in a series on how Candor's synthetic research results compare with results from human participants. The first compared synthetic nurses with human nurses on a concept test. We ran this study the same way as the nurses study so the two can be read side by side.

Three differences from the nurses replication study:

  • This is a B2B audience
  • It's problem validation, not concept testing
  • We built the synthetic audience without uploading any background documents

As we said in the nurse write-up, we don't think matching human answers is the most important test of a synthetic research tool. We publish these numbers because buyers ask for them. This is now data from two studies, so it is still early evidence.

We used the same safeguards as in the nurse study. The pass marks come from how much human participants agree with each other. The AI models that read the transcripts were not told which ones were synthetic. And we committed to reporting results that don't favor us.

What we tested

The human benchmark

The human study was part of a client project on testing solutions for independent insurance agencies.

We interviewed 8 participants in March 2025, for about 60 minutes each.

Roles:

  • 5 participants were agency owners
  • 1 was a partner in an agency
  • 1 was a vice president who also sells policies
  • 1 was a managing broker who leads a team inside a larger agency

Insurance types:

  • 4 participants worked mainly in property and casualty insurance (home, auto, and business policies)
  • 3 in health and employee benefits
  • 1 in life insurance.

We showed each participant 12 problems, each written on a card as a short statement, and asked them to sort the cards into four categories: very challenging, somewhat challenging, not challenging, or not applicable. They then ranked the problems in their very challenging pile. After that, we asked why they had sorted each problem the way they did, starting with the ones they found very challenging.

Synthetic application

Candor's problem studies accept at most 10 problems, so we had to drop 2 of the 12. We dropped the two that human participants most often sorted as not applicable. We chose the two from the human data alone, before any synthetic interviews were conducted. Both are left out of every human comparison in this article too.

We built a synthetic audience to mirror the 8 human participants. We asked for the same mix of roles (about 60% agency owners), the same mix of insurance types, and a similar range of experience (5 to 30 years). We did not upload any documents so the audience was built from signals extracted from web search alone.

Candor generated 17 synthetic participants. It held the mix of roles and insurance types to what we asked for. It did not hold agency size or years of experience. The synthetic insurance agents' agencies were larger than those from the human sample: 3 of the 17 synthetic participants ran agencies with fewer than 10 employees, against 5 of 8 human participants. More of the synthetic insurance agents had long careers: 6 of 17 synthetic participants had 28 or more years in insurance, against 1 of 8 human participants. The 17 synthetic insurance agents were grounded in web search data (438 signals from 167 sources). The system generates between 15 and 24 participants per audience, and the number depends on how many unique participants the uploaded documents and web search data can support. How evidence grounding works explains how those signals turn into participants.

Next we ran the problem validation test on the 17 synthetic insurance agents. The problem descriptions were identical to the ones shown to the human insurance agents. Candor wrote the interview questions and ran the interviews, but was set up to mimic the human interviews as closely as possible. Because we built Candor to automatically transition to card-sort mode when testing 5 or more problems, the format matched the format we used with human insurance agents.

The guides differed in the questions around the sort. Before the sort, Candor asked each synthetic agent to walk through a typical day and to describe a frustration without being prompted. The human participants got background questions only. After the sort, Candor asked for a story about the last time each of the agent's four top-ranked problems came up and what came of it, then challenged some of the agent's answers. The human guide asked why about every problem a participant sorted as very challenging, with separate questions about problems sorted as not challenging or not applicable. These differences matter for the reasons analysis, and we come back to them in the limitations.

How we set the pass mark

The pass/fail threshold was defined based on how well human insurance agents agree with other human insurance agents.

For the card sort

Before we ran any synthetic interviews, we took the sample of eight human insurance agents and split them into two groups of four. We compared the groups of four to determine how closely their results matched, one problem at a time. A score of 0 meant the two groups sorted the problems completely differently. A score of 1 meant they sorted them identically. All numbers here are those scores turned into percentages. We repeated that for all 35 possible splits across the 8 insurance agents.

Two groups of 4 human participants scored 58.2% sort agreement on average. We set the synthetic pass threshold at 48.8%, the score that only 1 in 20 human-human comparisons fell below. This is the same rule we used for the nurses. We set it before running any synthetic interview.

We also set three scoring rules before the synthetic interviews were run:

  • Not applicable counts as its own pile.
  • Two human cards were split between two piles. One participant placed a card on the line between two piles. Another split a problem into its two halves and sorted them separately. Each counts as half a card in each pile.
  • Problems that human insurance agents added themselves were not scored.

For the reasons

We built a list of reasons for the sort from the 8 human interviews. A reason is why a participant said a problem is challenging, is not challenging, or does not apply to them. A Claude session that never saw any synthetic data read the 8 transcripts and wrote the list. Across all 8 insurance agents, there were 75 reasons listed.

We coded whole interviews, with any discussion of the two dropped problems removed before the coding began. We had a second AI model (ChatGPT) code 3 of the 8 human transcripts with the codebook we generated in Claude. Its coding agreed with the original at a kappa of 0.821. Kappa measures how much two coders agree beyond what chance would produce: 0 is chance level and 1 is perfect agreement. A value over .80 is considered substantial agreement.

We then ran the same split exercise as for the card sort. For each of the 35 ways of splitting the 8 human participants into two groups of 4, we asked how many of one group's reasons the other group also raised. Two groups of 4 human participants shared 64.1% of each other's reasons on average. The pass threshold for synthetics is 58.1%, the score only 1 in 20 human-human comparisons fell below.

We determined the reasons list and the pass threshold before any synthetic transcript was coded.

Method and results

Card sort results

The synthetic insurance agents passed the card sort test. They scored 52.8%, against a pass mark of 48.8% and a human-human average of 58.2%.

After Candor ran the interviews, we recorded where each synthetic participant sorted each problem. We then compared every possible group of 4 synthetic participants with every possible group of 4 human participants. That came to 166,600 comparisons (2,380 synthetic groups against 70 human groups). The average score was 52.8%, which is 90.7% of the human-human average.

MeasureHuman vs humanSynthetic vs human
Comparisons35166,600
Average agreement58.2%52.8%
Spread (SD)6.9%6.4%
5th percentile (the pass mark, in the human column)48.8%
Median56.2%52.5%
95th percentile68.7%
Synthetic average as a share of the human average90.7%
ResultPass

Reasons results

How we coded and scored the transcripts

Scoring the reasons meant reading every transcript and marking which of the 75 reasons appeared in it. We first blinded the transcripts so the synthetic and human ones were in the same format. Two AI models from different companies then coded every transcript: ChatGPT (GPT-5.6 Pro, in a temporary chat) and Claude (Opus 5.5, in an incognito chat). Each transcript was coded in a fresh chat with no memory. Neither model was told which transcripts were synthetic.

For each group of 4, a reason counts if any one member raised it. We compared a synthetic group's reasons with a human group's reasons in two directions:

  • Coverage is the share of the human group's reasons that the synthetic group also raised. Low coverage means the synthetic insurance agents missed reasons the human insurance agents gave.
  • Precision is the share of the synthetic group's reasons that the human group also raised. Low precision means the synthetic insurance agents raised reasons the human insurance agents didn't.

For example, if a synthetic group raised 39 reasons, a human group raised 55, and 31 were shared, coverage is 31 of 55 (56%) and precision is 31 of 39 (79%). The score for a comparison is the average of the two. A result passes only if it passes under both models.

One caution about precision. It only counts reasons on the list, and the list came from the human interviews. A reason that only a synthetic agent gave is not on the list, so this measure can't see it. We come back to this in the coverage and precision results.

Overall score

Across 166,600 comparisons per model, the synthetic insurance agents scored 68.2% under ChatGPT and 67.4% under Claude, or 67.8% on average. The pass mark was 58.1% and the human-human average was 64.1%. The synthetic insurance agents passed under both models.

MeasureHuman vs humanSynthetic vs human (ChatGPT)Synthetic vs human (Claude)Synthetic vs human (average)
Reasons on the list75
Comparisons35166,600166,600166,600 per model
Average64.1%68.2%67.4%67.8%
Spread (SD)3.4%4.9%4.5%4.7%
5th percentile (the pass mark, in the human column)58.1%60.1%59.8%60.0%
Median64.9%68.3%67.5%67.9%
95th percentile68.5%76.3%74.6%75.5%
ResultPassPassPass (both models)

The combined score sits above the human-human average because it averages two measures that moved in opposite directions. Precision was well above the human-human level, and coverage was below it.

Coverage and precision

We set a pass mark for each component after seeing the results, using the same 1-in-20 rule. Taken one direction at a time, the pass mark is 56.7%.

MeasureHuman vs humanPass markSynthetic vs human (ChatGPT)Synthetic vs human (Claude)Synthetic vs human (average)
Coverage64.1%56.7%57.0%55.5%56.3%
Precision64.1%56.7%79.5%79.3%79.4%

Coverage was 57.0% under ChatGPT and 55.5% under Claude, against 64.1% for two human groups. One model put coverage just above the 56.7% pass threshold and the other put it below. A result has to pass under both models, so we count this as an inconclusive result on coverage. Averaged across the two models, coverage was 56.3%, which is 87.8% of the human-human figure.

Coverage is lower because the synthetic insurance agents raised the same reasons as each other. One synthetic participant raised about as many reasons as one human participant (about 20 against about 21). A group of 4 human participants raised about 55 different reasons between them. A group of 4 synthetic participants raised about 39. When we compared synthetic groups with each other, they shared 76.5% of each other's reasons under one model and 77.1% under the other, or 76.8% on average. Only 1 in 20 human-human comparisons scored above 68.5%.

Precision was 79.5% under ChatGPT and 79.3% under Claude, or 79.4% on average, against 64.1% for two human groups. About 8 in 10 of the listed reasons a synthetic group raised were also raised by the human group it was compared with. Two human groups share less with each other because many human reasons are one-offs. 30 of the 75 reasons were raised by only 1 human participant, and a reason only 1 participant raised can never be in both groups. The synthetic insurance agents raised few of those one-off reasons.

Summary

Candor's synthetic insurance agents passed the card sort test. On the reasons, the result is mixed. They passed the combined test we set in advance, but coverage on its own was inconclusive. In this instance, a pass means that the synthetic insurance agents agreed with the human insurance agents more than the least-alike groups of human insurance agents agreed with each other. Only 1 in 20 comparisons between two human groups scored below the pass threshold.

On the card sort, synthetic groups agreed with human groups at 52.8%, against a human-human average of 58.2% and a pass mark of 48.8%. The synthetic insurance agents sorted the problems less like the human insurance agents than the human insurance agents sorted them like each other, but they cleared the pass threshold. Two problems had the largest gaps: managing the agency without help, and planning for career or agency growth.

On all 75 reasons, the synthetic insurance agents scored 67.8% (averaged across the two AI models), against a human-human average of 64.1% and a pass mark of 58.1%. Precision was above the human-human level: 79.4% of the listed reasons a synthetic group raised were also raised by the human group. Coverage was below it: synthetic groups raised 56.3% of the reasons human groups raised, against 64.1% for two human groups. On coverage alone, one model put the synthetic insurance agents just above the mark we set after the results, and the other put them below. We count that as an inconclusive result on coverage.

What this means for Candor users

This section is our interpretation of the results, not something the study tested. They remain consistent with our recommendations from the nurses' study.

Don't rank problems from a small sample, whether human or synthetic

Two groups of 4 human participants overlapped by 58.2% on average when they sorted these problems. The human participants disagreed with each other, and 8 participants aren't enough to decide which problem to solve first. Neither are 17 synthetic participants. A small round of interviews, human or synthetic, can tell you which problems are worth a closer look and why people find them hard. If you need a ranking you can defend, run a survey with a sample large enough for the results to hold for insurance agents in general, and use the interviews to decide what goes into it.

Synthetic participants missed some of the reasons human participants gave, including common ones

Synthetic groups raised 56.3% of the reasons human groups raised, against 64.1% for two human groups. Four reasons raised by 3 or 4 of the 8 human participants came up with 1 or none of the 17 synthetic participants. Two are about finding customers: leads that can't be insured, and the work of reaching the right prospects. Two are about money: little income in the first years, and a future too uncertain to plan around.

What the synthetic sample did well

About 8 in 10 of the listed reasons a synthetic group raised were also raised by the human group it was compared with (79.4%). Two human groups shared 64.1% overlapping reasons. When the synthetic insurance agents raised a reason from the human list, it was usually one that several human participants shared.

How to work with the gap

We don't have a fix for the coverage gap yet. The four common reasons the synthetic insurance agents missed, two about finding customers and two about money, were each raised by 3 or 4 of the 8 human participants. So a few human interviews are likely to surface reasons like these. One way to use both: run synthetic interviews to build the list of reasons you expect, then run human interviews to find what's missing from it.

Limitations

8 human participants is a small sample. With only 35 ways to split them into two groups, the 1-in-20 pass mark sits between the second-lowest and third-lowest human-human scores. That makes the pass marks and the verdicts less precise than they would be with a larger human sample.

We ran one synthetic audience, on one version of Candor, once. We don't know how much the results would change with a second audience built from the same inputs.

The synthetic audience did not match the human sample on agency size or years of experience, even though we described both to Candor. The synthetic insurance agents ran larger agencies, and more of them had long careers. The card sort gap on managing the agency without help may partly reflect that.

The human interviews ran in March 2025 and the synthetic interviews in November 2026. We don't know whether the problems insurance agents find hard changed in the 20 months between.

The interview guides differed. Candor wrote its own guide, which asked some questions the human guide didn't and left out some the human guide asked. Some of the reasons only synthetic insurance agents gave come from that. Candor's fourth pile also read "not applicable to me or unclear what this means," which is broader than the human pile.

We dropped 2 of the 12 problems to fit Candor's limit of 10. The human participants sorted and discussed all 12, and we removed the 2 afterward.

Both AI models marked more reasons as present in the human transcripts than the original coding did. Because the human side of each comparison uses the stricter original coding, this favours the synthetic side somewhat.

What comes next

We'll keep running and publishing these studies with different audiences and study types, because buyers ask for them. As we said in the nurse write-up, we think other measures matter more for judging a synthetic research tool. One of them came up in this study: what synthetic participants raise that human participants don't, and whether it holds up when checked. We plan to test that more directly. A follow-up article will cover the deeper analysis from this study: the card sort results problem by problem, the two largest gaps, and the reasons only synthetic insurance agents gave.

Where to go next

For the first study in this series, read Synthetic users vs humans: replicating a research study.

For the limits of synthetic research in general, read what synthetic research skeptics get right.

For how to judge whether a synthetic transcript holds up, read how to spot convincing nonsense.

To run a study like this one yourself, see problem validation with Candor.

Common questions

In one B2B study, mostly yes. 8 people who own or help run independent insurance agencies sorted agency problems by how challenging each one is and explained why. Candor's 17 synthetic insurance agents did the same. Groups of 4 synthetic agents agreed with groups of 4 human agents 52.8% of the time on the sort, against 58.2% between two human groups and a pass mark of 48.8%, set before any synthetic interview ran. On the reasons behind the sort, they scored 67.8% against a pass mark of 58.1%. But they raised fewer of the human reasons than a second group of humans would. This is the second study in a series, with a small human sample, so treat it as early evidence.

From how much people agree with each other. We split the 8 human participants into two groups of 4 in all 35 possible ways and scored how closely each pair of groups sorted the problems. Two human groups agreed 58.2% of the time on average. The pass mark was the score only 1 in 20 of those human pairings fell below: 48.8% for the sort and 58.1% for the reasons. We set both before any synthetic data was scored. A pass means the synthetic agents agreed with the human agents more than the least-alike groups of humans agreed with each other. With only 8 human participants, those pass marks are less precise than they would be with a larger sample.

Some reasons that several human participants gave. Synthetic groups raised 56.3% of the reasons human groups raised, against 64.1% between two human groups. Four reasons raised by 3 or 4 of the 8 human participants came up with 1 or none of the 17 synthetic participants. Two were about finding customers: leads that can't be insured, and the work of reaching the right prospects. Two were about money: little income in the first years, and a future too uncertain to plan around. The gap isn't volume. Each synthetic participant raised about as many reasons as each human. The synthetic agents raised the same reasons as each other, so a group of 4 covered about 39 reasons against about 55 for 4 humans.

Use each for what it does well. In this study, about 8 in 10 of the listed reasons a synthetic group raised were also raised by the human group it was compared with, so what the synthetic agents said was mostly what real agents say. But they missed some common human reasons. One way to use both: run synthetic interviews to build the list of reasons you expect, then run a few human interviews to find what's missing from it. Neither 8 human participants nor 17 synthetic ones is enough to decide which problem to solve first. If you need a ranking you can defend, run a survey with a large enough sample, and use the interviews to decide what goes into it.

Bring evidence to your next decision.

Start with a free project, or walk through Candor with us first.