BackEngine vs. Direct Connectors

The question

Does an AI agent give better answers when it works through BackEngine?

We froze a data window: everything measured as of July 7, 2026, looking back 90 days. Every run, every repeat, every question used that same slice of reality, so neither method could benefit from data the other couldn't see.

Then we wrote 10 questions:

1. Which customer account is at the highest risk of churning right now, and what's driving that risk?
2. What onboarding friction comes up most often in customers' first 60 days?
3. Which competitor gets mentioned most often across our deals and accounts?
4. Which product capability gets compared most favorably against a competitor, and which gets compared most unfavorably?
5. Which active customer accounts have had no meaningful contact in the last 30 days?
6. What are the three most requested product features across customer conversations in the last quarter?
7. Which accounts show signs of wanting to expand or upgrade in the next 60 days?
8. Which accounts have lost their internal champion, or whose champion has gone quiet?
9. Of the renewals coming up in the next 90 days, which carry the most risk, and why?
10. What pricing or budget objections come up most often, and on which accounts?

The exact wording was fixed in advance and given to both methods verbatim.

The setup

Three roles, walled off from each other

Every part of the test ran as a separate AI session with a fresh, empty context. No role ever saw another role's work.

Rigor

Each question answered three times

Because AI agents take different paths run to run, each method answered every question three times so we could separate real differences from noise. Each logged every tool call and its estimated token cost.

Bias control

The bias check

Every pair of answers was judged twice, by two separate judge sessions, with the labels swapped between passes. If the two passes disagreed about which answer was better, the pair didn't count and we flagged it, tightened the judging instructions, and re-ran it fresh.

This caught position bias 3 times out of 30 pairs (a 90% first-pass consistency rate). All three flags were near-ties that resolved cleanly on re-run.

Scale

The tally

135 runs total: 60 answers and 72 judge passes. For each method we report accuracy, comprehensiveness, tool calls, and tokens — as means and ranges across all 30 answers, and as question-by-question win counts.

With 10 questions, the win counts show a pattern, not statistical proof, and we say so.