In October we reread, line by line, our analyses of four startups, built from their websites and public sources, next to those of another tool on the market, to see which one got less wrong. We found mistakes on both sides, and several of ours came from our own code. We aren’t naming the companies or the tool, because four deals aren’t enough to judge anyone.
The summary says something different from the text
On a company that had just raised a seed round, our key figures showed “amount not disclosed”, while the text on the same page cited $7.5M and its investors. On another, the same round appeared twice, as $5.1M and then as €4.42M, and a reader in a hurry could count two rounds.
You can spot these in ten seconds if you read both the summary and the text, and many readers stop at the summary. We now show each round on a single line, in the source’s currency, with the total and the investors. The duplicates held on for a while. On the next run, another figure came out twice under two labels.
An old figure judges today’s company
One of the four got our best opinion, Prioritize, on revenue that was 17 months old. In 17 months a startup at that stage can double or lose its biggest customer, and the analysis had no way of knowing. The age of the figure now counts in the opinion, and the same company moved down to Follow up, with a review date.
The opinion contradicts its own finding
This one embarrassed us the most. A company showed €3.6M of 2025 revenue in its published accounts. Our analysis rated its traction “strong”, then concluded Pass, because no strength had been verified. The rule that calculated the opinion didn’t count traction as a strength. The model had read correctly, and our code drew the wrong conclusion.
We fixed the rule. On the next run the same company came out Prioritize, and you had to read the justification to see the new problem. To rate the team “strong”, the analysis cited the company’s own revenue, as if running this company proved you could run this company. The founder’s role at the head of an asset manager since 2020, which the previous version had found, was gone.
Same facts, different reading
Between those two analyses, run on the same capture of the website, the team went up to “strong” and traction fell from “strong” to “medium”. A fund that runs a deal a second time expects the opinion to move when the facts move. Here the facts hadn’t moved.
An analysis now starts from the previous one, and the page says when the method is what changed. On October 23, out of three deals we reran, two kept their opinion and their ratings. The third changed its opinion because of a badly written instruction, then returned to where it started once we fixed the instruction.
Keeping the same opinion isn’t enough. On one of those two deals, traction stayed “strong” but no longer rested on the same fact. The research hadn’t found the register page again, and the revenue figure had dropped out of the analysis. We have fixed this, and haven’t yet seen that case play out again.
An estimate taken for revenue
A revenue figure can be actual, forecast by the company or estimated by a third party, and an analysis that mixes them goes wrong in both directions. A company with no published revenue came out Prioritize because a database estimated it at about $1.1M. The analysis itself wrote that the estimate didn’t allow a conclusion, and our scoring table applied it anyway. Another lost a traction level because a source said it had missed its forecast, though a missed forecast doesn’t contradict actual revenue.
We took that scoring table out of the code. The model judges traction according to the sector, the business model and the stage, and it has to say what type of evidence it relies on.
Finding nothing proves nothing
The other tool wrote that it could find no founder for a company whose Series A announcement names the cofounders. It rated that a critical risk, and that was the main reason for its pass. It also gave a total raised of $25M that no source confirms, presented a chip maker as a competitor to model monitoring software, and repeated a “20x” from a press release as a fact. On other points it did better than we did: its page was quicker to read than ours, and on one company its opinion was more reasonable.
We made the opposite mistake. Our rule said a partner can’t confirm a company’s claim. That holds for a figure and is wrong for the existence of the partnership. On a company whose program six customers described publicly, we marked zero claims confirmed out of nine. Our pages still show too much “not confirmed”, and we haven’t finished fixing that.
On our own deck
On November 4, our public screening analyzed Casian’s deck. It read seed instead of pre-seed, mixed the text of a page with a chart, and attributed a usage figure to the wrong customer. We know that deck by heart, so we saw it right away. On a deal received that morning, you would have had to reopen the page to notice.
Reviewing an analysis in five minutes
This review covers four companies, one run per tool and a grid we filled in ourselves, and one of the four had been used to tune our method. It gives a list of mistakes to look for, and no ranking.
Check on every analysis, wherever it comes from
- Put the summary and the text side by side: amounts, dates, names, currencies.
- Look for the date of every traction figure. With no date, treat it as unknown.
- Sort every revenue figure: actual, forecast by the company, estimated by a third party.
- For each rating, read the fact cited. A fact that comes from the company being analyzed proves nothing about its founders.
- When it says it found nothing, search for a minute yourself: funding announcement, company register, the company’s website.
- For a figure taken from a chart, reopen the deck page.
- If the opinion has changed since the last analysis, look for the new fact. If there isn’t one, trust neither.
Our guide to reviewing an AI analysis goes further for deals getting close to committee. The free screening needs no account: try it on a deck you know well, and if you find a mistake, write to us at contact@getcasian.com.