In March 2025, Highline Beta ran a client project with people who own or help run independent insurance agencies. In one research phase, 8 participants sorted a set of agency problems by how challenging each one is for them, and explained why. In November 2026, we ran the same exercise with synthetic insurance agents in Candor to see how closely the results matched. We set each pass mark from the human data, before we scored any synthetic data.
This is the second article in a series on how Candor's synthetic research results compare with results from human participants. The first compared synthetic nurses with human nurses on a concept test. We ran this study the same way as the nurses study so the two can be read side by side.
Three differences from the nurses replication study:
- This is a B2B audience
- It's problem validation, not concept testing
- We built the synthetic audience without uploading any background documents
As we said in the nurse write-up, we don't think matching human answers is the most important test of a synthetic research tool. We publish these numbers because buyers ask for them. This is now data from two studies, so it is still early evidence.
We used the same safeguards as in the nurse study. The pass marks come from how much human participants agree with each other. The AI models that read the transcripts were not told which ones were synthetic. And we committed to reporting results that don't favor us.
What we tested
The human benchmark
The human study was part of a client project on testing solutions for independent insurance agencies.
We interviewed 8 participants in March 2025, for about 60 minutes each.
Roles:
- 5 participants were agency owners
- 1 was a partner in an agency
- 1 was a vice president who also sells policies
- 1 was a managing broker who leads a team inside a larger agency
Insurance types:
- 4 participants worked mainly in property and casualty insurance (home, auto, and business policies)
- 3 in health and employee benefits
- 1 in life insurance.
We showed each participant 12 problems, each written on a card as a short statement, and asked them to sort the cards into four categories: very challenging, somewhat challenging, not challenging, or not applicable. They then ranked the problems in their very challenging pile. After that, we asked why they had sorted each problem the way they did, starting with the ones they found very challenging.
Synthetic application
Candor's problem studies accept at most 10 problems, so we had to drop 2 of the 12. We dropped the two that human participants most often sorted as not applicable. We chose the two from the human data alone, before any synthetic interviews were conducted. Both are left out of every human comparison in this article too.
We built a synthetic audience to mirror the 8 human participants. We asked for the same mix of roles (about 60% agency owners), the same mix of insurance types, and a similar range of experience (5 to 30 years). We did not upload any documents so the audience was built from signals extracted from web search alone.
Candor generated 17 synthetic participants. It held the mix of roles and insurance types to what we asked for. It did not hold agency size or years of experience. The synthetic insurance agents' agencies were larger than those from the human sample: 3 of the 17 synthetic participants ran agencies with fewer than 10 employees, against 5 of 8 human participants. More of the synthetic insurance agents had long careers: 6 of 17 synthetic participants had 28 or more years in insurance, against 1 of 8 human participants. The 17 synthetic insurance agents were grounded in web search data (438 signals from 167 sources). The system generates between 15 and 24 participants per audience, and the number depends on how many unique participants the uploaded documents and web search data can support. How evidence grounding works explains how those signals turn into participants.
Next we ran the problem validation test on the 17 synthetic insurance agents. The problem descriptions were identical to the ones shown to the human insurance agents. Candor wrote the interview questions and ran the interviews, but was set up to mimic the human interviews as closely as possible. Because we built Candor to automatically transition to card-sort mode when testing 5 or more problems, the format matched the format we used with human insurance agents.
The guides differed in the questions around the sort. Before the sort, Candor asked each synthetic agent to walk through a typical day and to describe a frustration without being prompted. The human participants got background questions only. After the sort, Candor asked for a story about the last time each of the agent's four top-ranked problems came up and what came of it, then challenged some of the agent's answers. The human guide asked why about every problem a participant sorted as very challenging, with separate questions about problems sorted as not challenging or not applicable. These differences matter for the reasons analysis, and we come back to them in the limitations.
How we set the pass mark
The pass/fail threshold was defined based on how well human insurance agents agree with other human insurance agents.
For the card sort
Before we ran any synthetic interviews, we took the sample of eight human insurance agents and split them into two groups of four. We compared the groups of four to determine how closely their results matched, one problem at a time. A score of 0 meant the two groups sorted the problems completely differently. A score of 1 meant they sorted them identically. All numbers here are those scores turned into percentages. We repeated that for all 35 possible splits across the 8 insurance agents.
Two groups of 4 human participants scored 58.2% sort agreement on average. We set the synthetic pass threshold at 48.8%, the score that only 1 in 20 human-human comparisons fell below. This is the same rule we used for the nurses. We set it before running any synthetic interview.
We also set three scoring rules before the synthetic interviews were run:
- Not applicable counts as its own pile.
- Two human cards were split between two piles. One participant placed a card on the line between two piles. Another split a problem into its two halves and sorted them separately. Each counts as half a card in each pile.
- Problems that human insurance agents added themselves were not scored.
For the reasons
We built a list of reasons for the sort from the 8 human interviews. A reason is why a participant said a problem is challenging, is not challenging, or does not apply to them. A Claude session that never saw any synthetic data read the 8 transcripts and wrote the list. Across all 8 insurance agents, there were 75 reasons listed.
We coded whole interviews, with any discussion of the two dropped problems removed before the coding began. We had a second AI model (ChatGPT) code 3 of the 8 human transcripts with the codebook we generated in Claude. Its coding agreed with the original at a kappa of 0.821. Kappa measures how much two coders agree beyond what chance would produce: 0 is chance level and 1 is perfect agreement. A value over .80 is considered substantial agreement.
We then ran the same split exercise as for the card sort. For each of the 35 ways of splitting the 8 human participants into two groups of 4, we asked how many of one group's reasons the other group also raised. Two groups of 4 human participants shared 64.1% of each other's reasons on average. The pass threshold for synthetics is 58.1%, the score only 1 in 20 human-human comparisons fell below.
We determined the reasons list and the pass threshold before any synthetic transcript was coded.
Method and results
Card sort results
The synthetic insurance agents passed the card sort test. They scored 52.8%, against a pass mark of 48.8% and a human-human average of 58.2%.
After Candor ran the interviews, we recorded where each synthetic participant sorted each problem. We then compared every possible group of 4 synthetic participants with every possible group of 4 human participants. That came to 166,600 comparisons (2,380 synthetic groups against 70 human groups). The average score was 52.8%, which is 90.7% of the human-human average.
| Measure | Human vs human | Synthetic vs human |
|---|---|---|
| Comparisons | 35 | 166,600 |
| Average agreement | 58.2% | 52.8% |
| Spread (SD) | 6.9% | 6.4% |
| 5th percentile (the pass mark, in the human column) | 48.8% | |
| Median | 56.2% | 52.5% |
| 95th percentile | 68.7% | |
| Synthetic average as a share of the human average | 90.7% | |
| Result | Pass |
Reasons results
How we coded and scored the transcripts
Scoring the reasons meant reading every transcript and marking which of the 75 reasons appeared in it. We first blinded the transcripts so the synthetic and human ones were in the same format. Two AI models from different companies then coded every transcript: ChatGPT (GPT-5.6 Pro, in a temporary chat) and Claude (Opus 5.5, in an incognito chat). Each transcript was coded in a fresh chat with no memory. Neither model was told which transcripts were synthetic.
For each group of 4, a reason counts if any one member raised it. We compared a synthetic group's reasons with a human group's reasons in two directions:
- Coverage is the share of the human group's reasons that the synthetic group also raised. Low coverage means the synthetic insurance agents missed reasons the human insurance agents gave.
- Precision is the share of the synthetic group's reasons that the human group also raised. Low precision means the synthetic insurance agents raised reasons the human insurance agents didn't.
For example, if a synthetic group raised 39 reasons, a human group raised 55, and 31 were shared, coverage is 31 of 55 (56%) and precision is 31 of 39 (79%). The score for a comparison is the average of the two. A result passes only if it passes under both models.
One caution about precision. It only counts reasons on the list, and the list came from the human interviews. A reason that only a synthetic agent gave is not on the list, so this measure can't see it. We come back to this in the coverage and precision results.
Overall score
Across 166,600 comparisons per model, the synthetic insurance agents scored 68.2% under ChatGPT and 67.4% under Claude, or 67.8% on average. The pass mark was 58.1% and the human-human average was 64.1%. The synthetic insurance agents passed under both models.
| Measure | Human vs human | Synthetic vs human (ChatGPT) | Synthetic vs human (Claude) | Synthetic vs human (average) |
|---|---|---|---|---|
| Reasons on the list | 75 | |||
| Comparisons | 35 | 166,600 | 166,600 | 166,600 per model |
| Average | 64.1% | 68.2% | 67.4% | 67.8% |
| Spread (SD) | 3.4% | 4.9% | 4.5% | 4.7% |
| 5th percentile (the pass mark, in the human column) | 58.1% | 60.1% | 59.8% | 60.0% |
| Median | 64.9% | 68.3% | 67.5% | 67.9% |
| 95th percentile | 68.5% | 76.3% | 74.6% | 75.5% |
| Result | Pass | Pass | Pass (both models) |
The combined score sits above the human-human average because it averages two measures that moved in opposite directions. Precision was well above the human-human level, and coverage was below it.
Coverage and precision
We set a pass mark for each component after seeing the results, using the same 1-in-20 rule. Taken one direction at a time, the pass mark is 56.7%.
| Measure | Human vs human | Pass mark | Synthetic vs human (ChatGPT) | Synthetic vs human (Claude) | Synthetic vs human (average) |
|---|---|---|---|---|---|
| Coverage | 64.1% | 56.7% | 57.0% | 55.5% | 56.3% |
| Precision | 64.1% | 56.7% | 79.5% | 79.3% | 79.4% |
Coverage was 57.0% under ChatGPT and 55.5% under Claude, against 64.1% for two human groups. One model put coverage just above the 56.7% pass threshold and the other put it below. A result has to pass under both models, so we count this as an inconclusive result on coverage. Averaged across the two models, coverage was 56.3%, which is 87.8% of the human-human figure.
Coverage is lower because the synthetic insurance agents raised the same reasons as each other. One synthetic participant raised about as many reasons as one human participant (about 20 against about 21). A group of 4 human participants raised about 55 different reasons between them. A group of 4 synthetic participants raised about 39. When we compared synthetic groups with each other, they shared 76.5% of each other's reasons under one model and 77.1% under the other, or 76.8% on average. Only 1 in 20 human-human comparisons scored above 68.5%.
Precision was 79.5% under ChatGPT and 79.3% under Claude, or 79.4% on average, against 64.1% for two human groups. About 8 in 10 of the listed reasons a synthetic group raised were also raised by the human group it was compared with. Two human groups share less with each other because many human reasons are one-offs. 30 of the 75 reasons were raised by only 1 human participant, and a reason only 1 participant raised can never be in both groups. The synthetic insurance agents raised few of those one-off reasons.
Summary
Candor's synthetic insurance agents passed the card sort test. On the reasons, the result is mixed. They passed the combined test we set in advance, but coverage on its own was inconclusive. In this instance, a pass means that the synthetic insurance agents agreed with the human insurance agents more than the least-alike groups of human insurance agents agreed with each other. Only 1 in 20 comparisons between two human groups scored below the pass threshold.
On the card sort, synthetic groups agreed with human groups at 52.8%, against a human-human average of 58.2% and a pass mark of 48.8%. The synthetic insurance agents sorted the problems less like the human insurance agents than the human insurance agents sorted them like each other, but they cleared the pass threshold. Two problems had the largest gaps: managing the agency without help, and planning for career or agency growth.
On all 75 reasons, the synthetic insurance agents scored 67.8% (averaged across the two AI models), against a human-human average of 64.1% and a pass mark of 58.1%. Precision was above the human-human level: 79.4% of the listed reasons a synthetic group raised were also raised by the human group. Coverage was below it: synthetic groups raised 56.3% of the reasons human groups raised, against 64.1% for two human groups. On coverage alone, one model put the synthetic insurance agents just above the mark we set after the results, and the other put them below. We count that as an inconclusive result on coverage.
What this means for Candor users
This section is our interpretation of the results, not something the study tested. They remain consistent with our recommendations from the nurses' study.
Don't rank problems from a small sample, whether human or synthetic
Two groups of 4 human participants overlapped by 58.2% on average when they sorted these problems. The human participants disagreed with each other, and 8 participants aren't enough to decide which problem to solve first. Neither are 17 synthetic participants. A small round of interviews, human or synthetic, can tell you which problems are worth a closer look and why people find them hard. If you need a ranking you can defend, run a survey with a sample large enough for the results to hold for insurance agents in general, and use the interviews to decide what goes into it.
Synthetic participants missed some of the reasons human participants gave, including common ones
Synthetic groups raised 56.3% of the reasons human groups raised, against 64.1% for two human groups. Four reasons raised by 3 or 4 of the 8 human participants came up with 1 or none of the 17 synthetic participants. Two are about finding customers: leads that can't be insured, and the work of reaching the right prospects. Two are about money: little income in the first years, and a future too uncertain to plan around.
What the synthetic sample did well
About 8 in 10 of the listed reasons a synthetic group raised were also raised by the human group it was compared with (79.4%). Two human groups shared 64.1% overlapping reasons. When the synthetic insurance agents raised a reason from the human list, it was usually one that several human participants shared.
How to work with the gap
We don't have a fix for the coverage gap yet. The four common reasons the synthetic insurance agents missed, two about finding customers and two about money, were each raised by 3 or 4 of the 8 human participants. So a few human interviews are likely to surface reasons like these. One way to use both: run synthetic interviews to build the list of reasons you expect, then run human interviews to find what's missing from it.
Limitations
8 human participants is a small sample. With only 35 ways to split them into two groups, the 1-in-20 pass mark sits between the second-lowest and third-lowest human-human scores. That makes the pass marks and the verdicts less precise than they would be with a larger human sample.
We ran one synthetic audience, on one version of Candor, once. We don't know how much the results would change with a second audience built from the same inputs.
The synthetic audience did not match the human sample on agency size or years of experience, even though we described both to Candor. The synthetic insurance agents ran larger agencies, and more of them had long careers. The card sort gap on managing the agency without help may partly reflect that.
The human interviews ran in March 2025 and the synthetic interviews in November 2026. We don't know whether the problems insurance agents find hard changed in the 20 months between.
The interview guides differed. Candor wrote its own guide, which asked some questions the human guide didn't and left out some the human guide asked. Some of the reasons only synthetic insurance agents gave come from that. Candor's fourth pile also read "not applicable to me or unclear what this means," which is broader than the human pile.
We dropped 2 of the 12 problems to fit Candor's limit of 10. The human participants sorted and discussed all 12, and we removed the 2 afterward.
Both AI models marked more reasons as present in the human transcripts than the original coding did. Because the human side of each comparison uses the stricter original coding, this favours the synthetic side somewhat.
What comes next
We'll keep running and publishing these studies with different audiences and study types, because buyers ask for them. As we said in the nurse write-up, we think other measures matter more for judging a synthetic research tool. One of them came up in this study: what synthetic participants raise that human participants don't, and whether it holds up when checked. We plan to test that more directly. A follow-up article will cover the deeper analysis from this study: the card sort results problem by problem, the two largest gaps, and the reasons only synthetic insurance agents gave.
Where to go next
For the first study in this series, read Synthetic users vs humans: replicating a research study.
For the limits of synthetic research in general, read what synthetic research skeptics get right.
For how to judge whether a synthetic transcript holds up, read how to spot convincing nonsense.
To run a study like this one yourself, see problem validation with Candor.