Understand research results
Check how far research has progressed, then review data, strategies, assumptions, and evidence rather than treating attractive numbers as guarantees.
Ask which question the evidence answers before reading its conclusion. This is your review checklist, not a feature that scores research for you.
Distinguish research stages
Importing data, writing a strategy, submitting an evaluation, and completing it are different stages. Success at one does not establish success at the next.
| What you have | What it confirms |
|---|---|
| Tool configuration or tool list | Configuration exists. A successful connection and loaded tools establish a usable connection, but still not an evaluation. |
| Submission reply and operation ID | Work was submitted and the original operation can be tracked. Check whether it finished successfully. |
| Partially successful result | Some output exists. Check which step failed; this is not full completion. |
| Successfully completed evaluation and output files | You can review this historical evaluation, not assume future results will match. |
Edgewright researches; it does not place orders. If a report says “Ready for live trading,” ask for evidence. Do not treat historical evaluation as actual trades already made.
Identify what produced each result
“Identity” here means the data, strategy, and configuration that produced the result.
Before comparing results, check:
- Dataset ID and instrument.
- Strategy ID and version.
- Evaluation period, execution method, capital, and cost settings.
- Operation ID, evaluation ID, and output files.
If you intended to compare only trading costs but the second evaluation uses a different dataset, ask the tool to explain before attributing return changes to costs.
Read risks and scope
Ask the researcher which instruments, periods, execution assumptions, and costs the result applies to. Conclusions should stay within the evidence.
Ask: “Does this hold only for the tested period? Does it hold with higher costs? Which missing data could affect my decision?” These are questions for you to raise, not checks automatically attached to every evaluation.
Recognize overfitting
Think of overfitting as repeatedly changing a strategy until it excels at an old exam paper. Better-looking results on old data do not necessarily answer new questions better.
Ask how many attempts were made, what conditions changed, and whether test data was used to tune the strategy. Do not accept only the best final attempt.
Soulhall’s research guidance for Embodiments also requires disclosure of search volume. A number selected after many experiments cannot be reported like the same number from one explicit test. Write a hypothesis that could be disproved before researching, rather than changing the question after seeing results.
Make every conclusion traceable
Match important conclusions to evaluation IDs and output files. If you receive only a written summary, request the original evaluations and files and read them before judging.
Evaluation details can identify output files, which tools can retrieve using those identifiers. Ask: “Identify the evaluation and file supporting this conclusion, and which result inside supports it.” Mark claims without evidence as unverified.