Before You Follow an AI Insight: Check the Evidence

A fluent explanation still needs a chain of evidence

An AI insight can sound helpful while skipping the step that matters most: connecting a recommendation to the observations actually available. “Your sleep improved, so keep doing this” contains at least three separate claims. Something changed in the record; a particular habit explains it; continuing the habit will help. The first may be observable while the other two remain untested.

You do not have to evaluate a whole model to inspect one recommendation. Start with the evidence it names, the dates it covers and the action it proposes. The useful outcome is a smaller, more defensible statement—not a new numerical score for whether the AI is safe.

Nutrinaut’s own technical review helped shape the six questions below. We found stored fields for evidence, confidence, comparison days and recommendation text. That structure makes some checks possible. It does not make the recommendations correct, and our review is not a clinical validation of the product.

The six-question evidence card

1. What observations were used? Look for the actual measurements or logged events, with units. “Your data suggests” is not enough to tell you whether the source was a watch reading, a manually entered tag or another calculated score.
2. Which period do they describe? Separate the date an insight was generated from the dates of its inputs. A new message can summarise old records. Ask whether the comparison spans the same hours, days or weeks.
3. What is missing or carried forward? A last-known value is not a new observation. Missing sleep, an incomplete food diary or an unlogged event changes what a comparison can establish.
4. What is the comparison? Ask how many usable days are on each side, whether they belong to the same person and whether conditions were reasonably comparable. Repeated days are not independent participants.
5. Is the explanation circular? If sleep is an input to a recovery score, finding that the score changes with sleep may partly describe the formula. It is not automatically an independent discovery about recovery.
6. Does the action follow? Describing an association does not demonstrate its cause or show that changing a habit will improve an outcome. A stronger recommendation needs stronger evidence than a descriptive summary.

If a detail is unavailable, write “not shown” rather than guessing. That does not prove the model lacked the information internally. It means you cannot verify the explanation from what you have. The distinction keeps this checklist fair as well as cautious.

Three examples, three different conclusions

All examples are invented. Their numbers illustrate reasoning, not product performance or user outcomes.

Why “confidence: 0.8” is not automatically “80% correct”

A confidence field needs a definition. It might be a model-generated assessment, a rule-derived number or a score transformed for display. Without knowing what was measured and how it was evaluated, the decimal alone cannot tell you the probability that a recommendation is correct.

To support a probability interpretation, the developer would need to define a testable outcome and evaluate the score against appropriate outcomes. “Correct” is already a difficult word here: accurate arithmetic, faithful summarisation and a beneficial personal recommendation are different targets. Success at one does not establish the others.

Ask what the confidence refers to, which data were used for evaluation, and whether performance remains relevant to this kind of input. If that information is not available, keep the score as an unexplained system output. Do not convert it into a personal guarantee.

The NIST AI Risk Management Framework emphasises evaluation and the context in which a system is used. Citing a framework does not mean Nutrinaut has been independently assessed against it. Here it supports the need for explicit evaluation, not a product-quality claim.

What a stored explanation lets you check

Our historical structure review found separate fields for evidence, confidence, comparison days and recommendation text. Those fields can make an explanation inspectable, but their existence is not proof that the recommendation is sound. A named source can be stale; a comparison can be incomplete; a confident sentence can propose an action its observations do not support.

Keep the steps separate: a record exists, its meaning is understood, the comparison is reproducible, and the proposed action has an independent justification. Passing the first step does not silently pass the rest.

This review informed the questions above. It did not evaluate current-model health effectiveness or establish a product accuracy rate. The data-reading guide explains how to keep a technical observation within its actual scope.

Turn the answer into a smaller decision

After filling the card, put the insight into one of three plain-language categories: “a description I can reproduce,” “a hypothesis needing better records,” or “an action not supported by the information shown.” These are reading aids, not validated safety classifications.

A descriptive observation can help you notice a logging problem or ask a focused question. A hypothesis can guide more consistent notes without being presented as a discovery. An unsupported action should not gain authority simply because it was phrased politely or accompanied by a confidence number.

For missing device records, use the source and date checklist. For habit comparisons, use the three-state tag journal. For meal estimates, verify the saved food and quantity. These are concrete checks you can perform before asking an AI system to interpret a larger pattern.

The best immediate result is often not a stronger recommendation. It is knowing exactly which part of an explanation you can verify, which part is uncertain and which action still needs an independent reason.