Do AI trading bots actually work? An honest answer from someone who builds them

The year that ended at the index

Bloomberg recently profiled a software executive who spent more than a year of nights and weekends building his own automated trading system.¹ He wrote the Python by hand, ran simulations, and tracked hundreds of stocks. After a year, his results were roughly what an index fund would have handed him for free. He concluded the work wasn't worth it and stopped.

His experience is not the exception. It's close to the base rate. When the research lab Nof1 ran its Alpha Arena experiments, handing frontier AI models real $10,000 accounts and letting them trade autonomously, the aggregate portfolio lost roughly a third of its value, and across 32 model-contest results only six finished in profit.² In the first crypto season, four of six models lost money, with one major model down 63% in two weeks.³ Meanwhile, researchers studying the broader arms race find that as more investors adopt the same AI tools, trading signals that once stayed profitable for years now decay in months, because everyone converges on the same trades at the same time.⁴

So the honest answer to the title question is: mostly, no. But "no" is not interesting. What's interesting is why, because the failure reasons are specific, nameable, and mostly invisible to the people making them. I know, because I once watched all of them happen a thousand times in a single experiment.

Claude's thousand strategies

Last year I asked Claude, one of the leading AI models, to build 1,000 trading strategies. It obliged. The code ran. The backtests were beautiful. Every one of Claude's strategies looked like a winner, and I mean every one.

Then I did what I was trained to do at a hedge fund, which is to assume the result is wrong and read the code line by line until I found out how. Claude's strategies were riddled with subtle errors. Not crashes. Not anything a compiler or a unit test would flag. Quiet, plausible, statistically flattering mistakes: signals computed a day too early, returns credited a day too soon, thresholds tuned to the specific stretch of history they were tested on. I corrected the errors one at a time and re-ran everything.

Every single one of Claude's thousand strategies lost its edge. One thousand out of one thousand.

The lesson was not that AI can't build strategies. The lesson was that code that looks right and a strategy that survives contact with reality are two different things, and telling them apart is a discipline, not a feature. The rest of this piece is that discipline, organized as the five failures that killed the thousand.

Why it feels so buildable now

A language model will write you a working momentum strategy in thirty seconds. Platforms will host it. Brokerage APIs will execute it. The distance from idea to running code has collapsed from months to an afternoon, and that collapse is real.

What has not collapsed is the distance from running code to working strategy. That canyon is exactly as wide as it was in 1995. The five failures below live in it.

Failure one: the mirage machine

Test enough strategy variations against history and you will always find one that "won." This is not evidence of anything. Someone always wins the lottery, and that does not make ticket-buying a strategy.

The formal name is backtest overfitting, and the mathematics of it are unforgiving. Bailey, Borwein, López de Prado, and Zhu showed that with enough trials, a researcher is essentially guaranteed to find a strategy with a stellar simulated track record even when no strategy has any real edge, and they proposed measuring the probability of backtest overfitting directly.⁵ Harvey, Liu, and Zhu surveyed hundreds of published return "factors" and concluded that after accounting for how many things researchers had collectively tested, the conventional statistical bar was far too low.⁶ Their recommendation, a t-statistic hurdle of 3.0 rather than 2.0, exists because the finance literature itself had become a mirage machine. Bailey and López de Prado went further and built a deflated Sharpe ratio, which discounts a backtest's Sharpe by the number of trials that produced it.⁷

Now notice what an AI code generator is in this framing. It is an infinite trial machine. Claude's thousand strategies were a thousand lottery tickets, and of course some of the tickets looked like jackpots. More compute makes this problem worse, not better, because every additional variation is another draw from the same distribution of flukes.

The counter-discipline is austere: fix your hypotheses before testing, count every trial including the discarded ones, evaluate on data the strategy has never touched, penalize complexity, and expect to throw away nearly everything. A generation process that keeps most of its ideas is not a research process. It is a slot machine with extra steps.

Failure two: trading on tomorrow's newspaper

The most common category of error in Claude's thousand strategies was lookahead bias: using information at decision time that would not actually have existed yet.

The mechanical version is easy to state and easy to commit. Your strategy computes its signal from Monday's closing price. In a backtest, one indexing slip credits you with Monday-to-Tuesday's return. In live trading you cannot act on Monday's close until Tuesday, so the return you can actually earn starts Tuesday. That single day of misalignment, compounded across years of daily decisions, can manufacture an entire edge that never existed. Nothing about the error looks wrong. The code runs. The equity curve is gorgeous. You find it only by tracing what the strategy knew and when it knew it, line by line.

The insidious version is worse, because it lives in the data itself rather than in your code, and this is the part almost nobody building bots at home has confronted. Historical databases quietly rewrite history. Companies report earnings, then revise or restate them days or weeks later, and most commercial datasets overwrite the original figure with the final one. Banz and Breen documented this decades ago: strategies tested on the revised database showed effects that shrank or vanished when tested on the numbers as investors actually first saw them.⁸ Macroeconomic data is even more slippery. GDP, inflation, and employment figures are revised repeatedly for years, which is why the Federal Reserve Bank of Philadelphia maintains an entire real-time dataset preserving what each number looked like on each date.⁹ Orphanides showed the stakes: monetary policy rules that look sensible on revised data would have prescribed materially different actions on the data policymakers actually had at the time.¹⁰

So a backtest can be lookahead-clean in its code and still lookahead-poisoned in its inputs. The strategy "knew" the revised earnings number, the final inflation print, the cleaned-up price series, none of which existed on the simulated trade date. Genuinely point-in-time data, captured as first reported and never overwritten, exists, but it is institutional-grade, expensive, and raw. You then have to clean it, vet it, and align every field to the timestamp at which it became knowable. Most data services, and essentially all free and cheap ones, give you the rewritten history instead. Most people simply cannot access the real thing, and a strategy validated on rewritten history has been validated on a world that never happened.

One more relative in this family: survivorship bias. Test on today's index membership and you have tested only on the companies that lived. Brown, Goetzmann, Ibbotson, and Ross showed how conditioning on survival alone can conjure apparent performance persistence out of nothing.¹¹ The graveyard doesn't get a vote, so history looks safer than it was.

Failure three: the weather keeps changing

Here is the failure that separates finance from nearly every other machine learning domain, and the one I have spent more of my own research time on than any other. Markets are nonstationary. The data-generating process itself changes while you are modeling it.

A vision model can assume cats will keep looking like cats. A trading model gets no such courtesy. Israel, Kelly, and Moskowitz argue that this combination, vanishingly low signal-to-noise plus a distribution that shifts under your feet plus adversaries who adapt to your discoveries, is precisely what makes financial ML so much harder than the domains where machine learning earned its reputation.¹²

The formal literature on this is deep and, importantly, still open. Hamilton introduced the canonical framework: model the market as switching between hidden regimes, each with its own statistical behavior, where you never observe the regime directly and must infer it from the data.¹³ Ang and Timmermann review three decades of evidence that regimes are pervasive, showing up not just in average returns but in volatilities and correlations, with the cruel property that correlations tend to rise exactly when diversification is needed most.¹⁴ Pesaran and Timmermann quantified what ignoring structural breaks costs a forecaster,¹⁵ and the machine learning community studies the same beast under the name concept drift.¹⁶ Nystrup, Hansen, Madsen, and Lindström showed that allocation rules which adapt to inferred regimes behave very differently from static ones, for better and worse.¹⁷

Why does this matter for a homemade bot? Because a strategy tuned on the last two years learned the last two years' weather, and it embeds that weather as a silent assumption. Fixed thresholds age. A volatility filter calibrated to a calm market becomes noise in a turbulent one. And the hard part, the genuinely unsolved part, is real-time detection: regimes are obvious in hindsight and murky in the moment. Statistically, the regime probability you can compute with data through today (the filtered estimate) is far less certain than the one you can compute a year later looking back (the smoothed estimate), and trading happens on the first kind, not the second. Detection lag is not an implementation detail. It is often the whole ballgame, and it remains an active research frontier in quantitative finance. Anyone selling you a bot that has "solved" regime change is selling you a backtest that hasn't met one yet.

The practical discipline: test across deliberately different market environments rather than one friendly stretch, prefer rules that adapt to measured conditions over frozen constants, and treat any strategy that has only seen one regime as unvalidated by definition.

Failure four: the backtest trades for free

Every backtest is a fantasy in one specific way: it executes instantly, at the printed price, in unlimited size, for free. Real trading pays a toll at every step, and the tolls come in more denominations than most builders ever model.

Start with the visible ones. Commissions, exchange fees, and regulatory fees. The bid-ask spread, which you pay every time you cross it, and which is not a constant: spreads widen with volatility and narrow with liquidity, so the same strategy pays different tolls in different weather and different names. Slippage, the gap between the price your model decided on and the price your order actually got. Then the carrying costs: margin interest if you use leverage, borrow fees if you short, and the quiet drag of cash sitting idle between positions.

Then the toll most home backtests cannot even represent: market impact. Your own order moves the price against you. This is not a rounding error, it is a structural feature of markets that has been studied for forty years, from Kyle's foundational model of how trades move prices,¹⁸ to the empirical finding that impact grows roughly with the square root of order size relative to volume.^19,20 Impact is why institutional desks schedule executions across time instead of trading all at once,²¹ and why the same dollar order is cheap in a mega-cap and ruinous in a thin small-cap.

Here is the epistemic problem underneath all of it, and it deserves a moment of respect: the historical prices you backtest on were formed by a market that did not include your order. Your simulated fill is a counterfactual. You are not asking "what price did the market print," you are asking "what price would the market have printed if I had shown up," and that question cannot be answered from the tape alone. Every cost model is an approximation of an unobservable, which is an argument for modeling costs conservatively, conditioning them on each asset's liquidity and volatility, and stress-testing the strategy's survival under costs meaningfully worse than your estimate.

Does any of this actually change conclusions? Routinely. Novy-Marx and Velikov put dozens of published stock market anomalies through realistic cost models and found that many of the famous ones stop working, with high-turnover strategies hit hardest.²² Frazzini, Israel, and Moskowitz, using a trillion dollars of real institutional orders, found costs meaningful even for one of the world's most sophisticated executers.²³ And at retail, Barber and Odean showed decades ago that trading friction is a first-order determinant of investor outcomes.²⁴ A strategy that trades daily pays hundreds of tolls a year. Plenty of strategies are profitable right up until they pay for their own trading.

Failure five: the crowd in the machine

The current wave of AI bots has a problem the spreadsheet-era hobbyist never faced: everyone's bot has the same brain.

When thousands of people prompt the same model with the same framing on the same headlines, they receive variations of the same answer. Exchange data already shows retail automation clustering, with order flow spiking at the same minutes of the day as scheduled bots fire together.²⁵ The Bank of England devoted part of its July 2026 Financial Stability Report to the risk that widespread AI adoption in trading amplifies herding and correlated positioning.²⁶ And the research on what crowding does to profits is not speculative. McLean and Pontiff showed that merely publishing an anomaly in an academic journal erodes more than half of its subsequent returns, because capital piles in.²⁷ The 2026 studies extend the same logic to AI: as investors converge on similar models, edges that lasted years now last months.⁴ Herding is not just unprofitable, it is dangerous: Barber, Huang, Odean, and Schwarz found that the episodes when Robinhood users piled most intensely into the same stocks were followed by significantly negative abnormal returns.²⁸

If your bot buys what ten thousand identically prompted bots buy, you have not found an edge. You have joined one, and you are supplying the exit liquidity for whoever understands that.

The escape is heterogeneity by construction. Strategies built from an individual's specific objectives, constraints, and asset universe, optimized across thousands of instruments, differ from each other because their owners differ. Two users of the same underlying system should hold different portfolios because they want different things. A crowd can share a tool. It cannot share an edge.

What the fix actually costs

Look back across the five failures and notice what they share: not one is solved by a smarter model. All five are solved, to the extent they are solvable, by validation discipline, which is a different thing entirely, and an expensive one.

The institutional version of that discipline looks like this. Hypotheses fixed before testing, with every trial counted and Sharpe ratios deflated accordingly.⁷ Out-of-sample evaluation as a hard wall, and walk-forward testing across rolling windows rather than one lucky stretch.²⁹ Data that is genuinely point-in-time, sourced, cleaned, vetted, and aligned to what was knowable when. Test windows chosen to span different regimes on purpose, and rules that adapt to measured conditions instead of memorizing one era's weather. Cost models conditioned on each asset's liquidity and volatility, stress-tested pessimistically, because your fill is a counterfactual. And adversarial code review by people who have personally committed every subtle error on this list, because those are the only people who can see them.

Hedge funds run this gauntlet because careers end when they don't. That, honestly, is why a year of solo nights and weekends tends to end at the index: not a lack of intelligence or effort, but the practical impossibility of one person operating an institutional validation pipeline after their day job.

The final exam: live validation, and why most paper accounts fail it

After everything above, one test remains that cannot be gamed, because it has no answer key: run the strategy forward in time, live, with virtual money, and watch what it does in market conditions nobody could have tuned it for. A backtest asks "would this have worked?" Live forward validation asks "does this work?" Only the second question deserves real money. It is telling that the AI trading arenas reached the same conclusion from the opposite direction: because language models have already read history, backtesting them is meaningless, and live markets are the only honest exam left.²

But here is the caveat that almost nobody applies: not all paper trading is created equal, and a bad simulator quietly fails the same five ways a bad backtest does. Ask hard questions of any paper account before trusting it. How does it fill orders? A simulator that fills you at the last trade price, or at the midpoint, in unlimited size, has assumed away the spread, the depth, and the impact, which is to say it has assumed away failure four entirely. A credible simulation fills marketable orders against the real quote, at the ask when you buy and the bid when you sell, in real time, timestamped to the second, with size checked against what the market was actually displaying. Limit orders should rest until the real market genuinely reaches them, and even then an honest simulator admits what it cannot know, because your true position in the queue at a price level is unobservable. Halts, auctions, dividends, and corporate actions should be handled as they occur, not retrofitted. And the simulation should run at the same cadence and latency the real system would, because a fill that assumes zero delay is another small fantasy compounding daily.

A paper account that fails these tests is not validation. It is a slower backtest with a nicer interface. This is why we built Portfolio Lab's paper portfolios to execute against live market data, on the same daily cadence as real accounts: the entire point of a forward test is that reality, not the simulator, gets the last word.

Two honest paths

If you are technical and this problem genuinely pulls at you, everything above is learnable, and I would encourage you to try. The bar is now explicit: counted trials and deflated statistics, lookahead paranoia extending into the data itself, regime-spanning tests, costs modeled as if they hate you, and a live forward period with honest fills before a single real dollar moves. Run that gauntlet and you will be ahead of nearly everyone in the Bloomberg story, whatever your returns turn out to be.

The other path is to use a system that was built to run the gauntlet for you. That is what Portfolio Lab is: you define the objectives, a purpose-built optimization engine constructs strategies from validated components, and every strategy must then prove itself live, in a paper portfolio filling against real market data, before you commit anything. Building and validating are free.

Because after personally correcting a thousand beautiful, broken strategies, I run my entire professional life on one rule, and it is the same rule this article has been circling from the first paragraph: never trust a backtest you haven't tortured, and never fund a strategy you haven't watched run live.