Welcome to Testrian Lab: Trading Research That Shows Its Work

Most trading ideas look brilliant on a chart. Very few survive contact with honest testing. Testrian Lab exists to find out which ones do, and to show our work along the way.

Who we are

Testrian Lab is an independent quantitative trading research lab. We design systematic strategies, put them through demanding statistical testing, and publish what we find. Our goal is simple and ambitious: a research process disciplined enough that you don’t have to take our word for it, because you can inspect it yourself.

We are not a signal service, and we are not here to sell a secret formula. We are a lab. That means hypotheses, experiments, measurements, and an archive that includes the experiments that did not work.

Why impressive backtests can mislead

A backtest is a story about the past. With enough parameters, enough tweaks and enough retries, a remarkable number of stories can be made to look profitable. The problem is rarely bad intent. It is the math of multiple testing. The more variations you try, the greater the chance that the best of them looks impressive even when no genuine edge exists.

That is one reason strategies that shine in a spreadsheet can fall apart with real capital. An apparent edge may be selection noise rather than a persistent effect: noise that was selected and polished until it looked like signal. Our job is to tell the two apart as well as the evidence allows, and to be clear about how much the evidence allows.

Our method

We are building the lab around a single research standard. Every published study states which of the checks below were completed, which did not apply, and any departures from protocol. We take no silent shortcuts.

1. Explore freely, then freeze the protocol

Exploration is how hypotheses are born, and we do plenty of it. Exploration is not confirmation, though. Before a confirmatory evaluation, we version and freeze the hypothesis, the candidate-selection rules, the data boundaries, the metrics and the pass/fail criteria. Any later change is recorded as an amendment or as a new experiment. We do not move the goalposts after seeing the results.

2. Backtests with documented data and costs

“Clean data” and “realistic costs” are easy to claim and hard to check, so we document them instead of just asserting them. Each study records its data sources, coverage and known limitations. It also records our checks for missing observations and for information that would not have been available at the simulated decision time.

We state the cost and execution assumptions relevant to each instrument, such as spreads, fees, slippage, funding, borrow costs, contract rolls and liquidity. We then report how results hold up under less favorable assumptions. A strategy that only works with perfect, free execution is not a strategy.

3. A final holdout we refuse to burn

We separate development data from a final chronological holdout. The strategy and the evaluation rules are frozen before that holdout is opened. A strategy must meet predefined criteria on data that played no part in its design. If it falls short, it is rejected, however good it looked in sample.

There is a subtle trap here, and we name it openly: a holdout can only be used once as an untouched test. If its results influence a redesign, we record that use and stop treating the same period as a clean final test. Further confirmation then requires additional unseen data. We also never rely on a single holdout alone. It is one check among several.

4. Overfitting checks: PBO and the Deflated Sharpe Ratio

We use two well-established tools from the quantitative finance literature. Each answers a different question.

• Probability of Backtest Overfitting (PBO) estimates how often the configuration selected as best in sample ranks below the median of the tested configurations out of sample. It is computed with combinatorially symmetric cross-validation (CSCV). In plain terms, it tells us how reliably our selection process picks genuinely stronger candidates rather than lucky ones.
• Deflated Sharpe Ratio (DSR) assesses the statistical significance of an observed Sharpe ratio against a benchmark that accounts for multiple testing. It also takes sample length, skewness and kurtosis into account. A Sharpe ratio that looks strong in isolation can be much weaker evidence once the research search behind it is counted.

Neither measure proves that a strategy has an economic edge or will remain profitable. Both complement the chronological holdout rather than replace it. CSCV probes the stability of our selection process, while the holdout tests the frozen rules on time periods the strategy has never seen.

5. A research ledger that counts what we tried

We keep a ledger of every tested variation, including the unsuccessful ones and the ones we would rather forget. For each selection decision, we disclose three things: the relevant search history, the raw number of trials, and how we estimate or conservatively approximate the effective number of independent trials used in the DSR.

That last distinction matters. Many variations of one idea are highly correlated, so a raw count is an audit record, not automatically the right statistical input. We explain how we bridge the two.

Logging makes the search auditable. It does not eliminate researcher judgment, and we won’t pretend it does. What it does is make that judgment visible.

6. Paper trading before any capital decision

Passing statistical tests is necessary, but not sufficient. Candidates that meet the research criteria move to paper trading, which uses live market data, simulated execution, a predefined observation period and operational checks. This tests the full workflow in real time. However, simulated fills are not real fills, and time spent in paper trading does not validate a strategy by itself. Any decision to deploy capital is a separate one.

What you will get from Testrian Lab

• Research notes. Readable write-ups of what we tested and what we learned. Each note identifies the hypothesis, the data period, the tested variations, the cost assumptions, the validation design and the decision. When a check cannot be completed, we say so.
• Rejected hypotheses, published openly. A rejected idea, with the reason it was rejected, is often more useful than a winning one. Negative results are part of the record, and each study states the scope of the experiments it reports.
• Methodology deep dives. Practical explanations of the tools behind serious quantitative research, including holdout design, multiple-testing corrections, PBO and DSR in practice, and cost modeling.
• A transparent journey. We are building this lab in public, step by step.

What you will not get: hype, cherry-picked screenshots, trade signals or promises of returns. When we report results, we will also report the assumptions and limits that come with them, including when the honest conclusion is “this did not work.”

Why build in the open?

Because transparency makes our decisions easy to scrutinize, and scrutiny is the best quality control we know. Publishing the process, failures included, holds us to a standard we have to meet in public. It also lets readers learn from our mistakes without having to repeat every experiment themselves.

We believe the future of trading research belongs to those who measure honestly and say plainly what their measurements cannot tell them. Testrian Lab is our attempt to show what that looks like.

Join the lab

Subscribe for research notes on systematic trading. Each one covers the hypothesis, the tests, the assumptions and the decision, including when the decision is to stop. It’s free.

Welcome aboard.

The Testrian Lab team

Disclaimer: Testrian Lab publishes general research and educational content only. Nothing on this blog is financial, investment, legal or tax advice. Nor is it a personalized recommendation, or a solicitation to buy, sell or hold any instrument or to adopt any strategy.

Backtested, simulated and paper-trading results are hypothetical. They do not represent actual trading, and they have inherent limitations. These include the benefit of hindsight, modeling assumptions, and differences between simulated and actual execution, such as fills, slippage, liquidity and market impact. No representation is made that any strategy, account or reader will achieve, or is likely to achieve, results similar to those discussed.

Trading involves substantial risk of loss and is not suitable for everyone. Past performance, whether actual or hypothetical, does not guarantee future results. Applicable rules vary by jurisdiction. Consult a qualified professional before making any financial decision.