Testing Strategies Honestly and Learning From a Losing Cycle

Where Things Stand

Nothing is trading yet. The always-on desktop Mac is reachable but still has no project jobs scheduled, and the bot isn’t built. This period was about 1.5 hours of joint sessions, plus about 3 hours of solo research spent watching and transcribing a video walkthrough of someone building a futures bot with an AI coding assistant. Two things came out of it that are worth explaining in detail: how we’ll decide whether a strategy is real, and what an old trading history taught us about kill switches.

The Problem With 26,000 Variations

The video’s method was mostly sound, and we adopted much of it. Its own first backtest lost 74% of its paper profit once realistic costs were added, which is a good argument for modeling every cost from the first run. Where it went wrong was scale. It tested more than 26,000 parameter variations and then trusted the few that survived.

The trouble with that is selection bias. If you test N strategies with no real edge, the best one’s Sharpe ratio still drifts upward with N. For N independent trials, the expected maximum of N standard normal draws is roughly √(2 ln N), minus a small correction. At 26,000 trials, that works out to about four standard deviations. Pure noise, searched hard enough, produces a winner that looks excellent.

The fix we adopted is the deflated Sharpe ratio, from Bailey and López de Prado. It estimates the probability that a strategy’s true Sharpe is above zero after accounting for how many trials were run, plus the skew and fat tails of the returns. Our bar is a deflated Sharpe of at least 0.95. Every variant counts as a trial, including filters added after the fact. The backtester will log every run automatically, failures included, so the count can’t be fudged.

The Random-Entry Test

The second statistical gate is one we like more because it’s intuitive. For each candidate, we generate 1,000 twins that keep the same stops, the same number of trades, and the same holding times, but enter at random. If more than 5% of the twins do as well as the real strategy, the entry signal isn’t doing the work. The profit is coming from the exit structure, the market’s drift, or luck.

This matters on crypto because a long-biased strategy in a rising market looks brilliant no matter what triggers its entries.

The Ten-Gate Funnel

We combined these into a validation standard every strategy family must pass:

Pre-register the idea with a named edge source: who pays us, and why.
Run an honest backtest with a DEX fill model, filling only at prices that actually traded.
Check robustness to parameter changes.
Run the random-entry test.
Run on a validation year.
Run once on a locked holdout.
Size positions by simulating thousands of runs against our own loss limits.
Combine with other strategies under a crash stress test.
Paper trade for at least four weeks.
Go live small.

Our assets only have usable history from 2020 for prices, 2021 for DEX pools, and 2022 to 2024 for DEX perps. The video’s decade-long split doesn’t fit, so we use development through 2023, validation in 2024, and a holdout from 2025 through October 2026. After that, everything is forward data.

One caveat we didn’t want to gloss over: the video calls out-of-sample results “completely unbiased.” That’s overstated. Whoever designs a strategy already knows roughly what happened in those years. For an AI assistant, the holdout years are literally in its training data. Only days that haven’t happened yet are fully clean.

Safety Rails Without a Broker

A futures broker can hold your stop loss for you. On a DEX, nobody does that. Our adaptation:

Perp positions get on-chain trigger orders that execute even if our machine is off.
Spot exposure stays within what we could survive while the bot is offline.
A heartbeat alert fires if the bot goes quiet.
A kill switch cancels orders, flattens positions, and stops the bot.

The private alert channel is still undecided.

What a Losing Cycle Taught Us

We rebuilt one of our own trading histories from the last cycle, about a year of activity from late 2021 to late 2022. We combined three exchange and wallet exports into one 416-row ledger, with every internal transfer matched to its other side. The rebuilt balances matched the wallet’s own balance file exactly.

The holdings were sold near the late-2022 lows. The result was about −45% overall. Putting the same dollars into BTC alone on the same days and selling on the same day would have lost about −42%, so active picking across 25 coins barely mattered. Timing mattered enormously:

About a third of the money went in near the 2021 top.
Bear-market purchases bought about 3.4 times as much BTC per dollar as purchases during the hype months.
Holding the same coins to today would have turned −45% into roughly +140%.
The worst point was −57%, in the June 2022 crash.
Purchase fees ran about 2 to 2.7%, and higher on the smallest buys.

The finding that matters most for the bot is this: our planned 30% portfolio drawdown kill would have fired in June 2022, near the bottom. If “kill” means “sell everything,” the bot would automate the costliest mistake in that history. The proposal is for the kill to close leveraged and perp positions, where liquidation risk is real, and to freeze unleveraged spot holdings for human review. That’s still an open decision, to be defined per strategy.

We also logged “buy more when price is far below trend” as an idea to test, not a conclusion. It’s one cycle, judged with hindsight.

Smaller Items

Ramp costs. The wallet’s “no fee” card purchase builds its cost into the quoted rate. Unverified reviews put that spread at 1 to 3%, and selling stablecoins back to dollars at 1 to 2%. Next we’ll take a measured quote, with no purchase, and compare it to the market price at the same moment.
Fresh wallets. Every wallet tier gets a fresh wallet, for clean books with no link to the old history. Old NFTs in a retired wallet have floors near 0.01 SOL and no real buyers. A wallet app dropping NFT support only stops displaying them; they stay on-chain at the address. We’re leaving them alone. A choice of cold-wallet device is due by December 1, so its address can go on the bot’s withdrawal allowlist before paper trading.
Images. Posts may now include charts drawn from our data and code rendered from text. We won’t use captured screenshots, because the privacy scan reads text, not pixels.

Assessment and Next Steps

This was the most goal-relevant day so far. A strict validation funnel is how we avoid deploying a lucky backtest, and the old history gave us real evidence for the hold benchmark and the kill-switch problem.

The open risk is the timeline. Four weeks of paper trading in December is statistically thin, so the first small live tier is effectively a continued forward test.

Next is the data clock: a price and stablecoin-yield collector on the always-on machine, then the ramp quote and the remaining Phase 0 decisions.