Every experiment at Mura ALife Labs starts as a row in one file, the attempt ledger. Before anything runs, the row says three things: what we are testing, what result would count as a pass, and what we predict will happen. Then the run happens, and the result goes in the same row, next to the prediction.

Writing the bar down first sounds like bureaucracy. It is the opposite. It is the only defence we have found against the most common failure in research: deciding what a result means after you have seen it.

The ledger now has more than a thousand rows.

When the bar is written in advance, a miss is a miss. We cannot quietly move the goalpost, and we cannot tell ourselves a near‑miss was “basically a pass”.

It also keeps us honest about our own judgment. Each row records whether the prediction held. Over hundreds of rows, that is a running score of how well we understand our own systems, and it is humbling. Many of our best ideas missed. Some of our worst‑looking runs taught us the most.

Many of our best ideas missed. Some of our worst‑looking runs taught us the most.

Three Rules We Keep

We are a small lab. We cannot afford to fool ourselves, because no one else is checking our work before it ships. The ledger is how we check it ourselves.

More soon.

We are trying to train a language model without gradient descent, backpropagation or Adam. Instead of taking thousands of small noisy steps, our solver takes a few exact Newton steps computed from closed-form statistics of the data. On a small training set (30,000 sentences) it beats Adam. As the data grows, it falls behind: about 0.15 nats per token behind at 100,000 and 300,000 sentences, and about 0.19 at the full 2.1 million. This post is about why, because the reason turned out to be more interesting than we expected.

The first surprise: a speed limit

The first surprise was a speed limit. An optimiser that takes steps of a fixed size cannot rest in a part of the loss landscape that is too sharp. For Adam, that limit can be written down: the sharpness, measured in Adam's own units, has to stay below 38 divided by the step size. We measured it on both models. Adam's solution sits at 14.6, inside the limit. Ours sits at 285.6, seven and a half times past it. Adam could not stay in our solution even if it landed there. Exact Newton steps have no such limit, so our solver happily takes the sharpest, deepest minimum it can find. On training text that is a great place to be, and on text it has not seen it is a worse one.

Attempting to respect the limit

That suggested an obvious fix: make our solver respect the same limit. We tried six ways. We penalised the sharpest direction, re-set the penalty strength at every step, applied the rule early and released it, applied it late, and applied it at three different data sizes. The sharpness did go down, by as much as 60 percent. The held-out score moved by at most 0.005 nats. Pushing down the sharpest direction just hands the sharpness to the next one, and a penalty halves the progress of every step. Our reading is that this limit comes from taking many small explicit steps, which a method built on exact steps does not do, so a penalty copies only part of it. We state this carefully: our measurement shows the limit is violated, not that violating it is the only reason for the gap.

The second finding: linear solver inefficiency

The second finding was a surprise in the other direction, and it came from a place we had stopped looking: the linear solver inside each Newton step. Each step asks us to solve a large linear system. We solve it iteratively, and a good preconditioner can make that much faster. We had been using a simple diagonal one. We replaced it, for the role maps that carry most of the curvature, with a structured one built from exact statistics of the data (a Kronecker-factored form), solving the same system as before. On a 100,000-sentence test, it reached in 3 iterations the quality the old version reached in 24, and its 24-iteration answer was far better. Applied as a real step, the improvement in held-out score was 0.056 nats against 0.0095 for the old solver. In other words, much of what we had blamed on the shape of the loss landscape was our solver not solving its own problem. We have measured this at 100,000 sentences. We are now running the same test at the full 2.1 million, and we will report it whichever way it goes.

Where this leaves the work

Where this leaves the work. Our best system today combines count tables with the exactly fitted model and scores 4.146 nats on our key test cells, against 4.038 for a same-data transformer, a gap of about 0.11. We do not beat the transformer, and this post is not a claim that we do. What we can say is that two things we assumed about the gap were wrong, one in each direction, and both were found by measuring rather than guessing.

What comes out sounds like English for a few words at a time and then wanders off. One of his samples reads: "THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED."

That is a language model. It predicts the next word from the last one, using counts from real text. Look at more previous words and the output gets better. This family of models, called n-gram models, ran most of the world's speech recognition and machine translation for about twenty years. Then neural networks replaced them almost completely.

This post is about why, and about what counting still does better.

The wall

In 1956 and 1957, Noam Chomsky argued that no model of this kind can fully capture English. An n-gram model only ever sees a fixed window of recent words. Mathematically, that makes it a finite-state machine: it can be in only finitely many situations, and it forgets everything outside the window.

Language does not respect windows. "If" waits for its "then." "Either" waits for its "or." A name mentioned in the first sentence of a story can come back forty sentences later. These links can stretch as far as the writer likes, and they can nest inside each other. A fixed window cannot follow them in general, however large you make it.

The fix that almost worked

The bigger problem in practice was more boring: most word sequences never appear in any corpus. Count five-word phrases in a billion words of text and nearly every new sentence you write still contains a five-word phrase nobody counted. A naive counting model gives that sentence a probability of zero.

Decades of work went into "smoothing": sharing probability from seen sequences with unseen ones. The best-known result is Kneser-Ney smoothing (1995). Its key idea is simple. The word "Francisco" is common, but it almost always follows "San." So when you have to guess a word in a new context, "Francisco" should be a rare guess. Kneser-Ney asks not "how often does this word appear?" but "in how many different places does it appear?"

A careful study by Chen and Goodman in the late 1990s found a modified version of Kneser-Ney to be the strongest smoothing method they tested. Counting also scaled. In 2007, a Google team trained a counting model on two trillion words and found that at that scale even a crude method called "stupid backoff" worked very well.

Why neural networks won

Counting treats every word as an unrelated symbol. To a counting model, "cat" and "dog" are as different as "cat" and "democracy." Everything it learns about one says nothing about the other.

In 2003, Bengio and colleagues gave each word a learned vector, so that similar words land near each other. Evidence about "the dog slept" now helps with "the cat slept." That sharing is what counting lacked.

Attention, in 2017, then removed the window. A transformer can look back at any earlier word directly. Similarity plus unlimited reach is, roughly, the recipe that took over the field.

What counting still does better

Counting was dropped. It was never disproved. Some of its properties are still hard to get any other way.

Adding knowledge is cheap. New text means new counts. Nothing is retrained, and nothing already learned gets overwritten.

Every prediction is traceable. You can point to the exact sentences a prediction came from. That matters when you need to know why a system said something.

Forgetting is exact. Remove a document's counts and it is gone, not just made less likely.

It runs on almost nothing. Looking up counts needs no special hardware.

The field has noticed. In 2024, the infini-gram project indexed over a trillion tokens with suffix arrays. That let an n-gram model use a window as long as any matching text, and its predictions still improved strong neural models when the two were combined.

The open questions

The history leaves some questions that seem more interesting than "which model wins."

Is the gap between counting and neural models mostly about similarity, or mostly about memory? These are different problems. Similarity can come from methods that are not neural at all: latent semantic analysis (1990) found word similarity using only counting and a matrix factorization.

Can a system keep track of things far back in a text without an unbounded internal state? Chomsky's argument says a fixed window alone cannot. It does not say which kinds of extra state are enough.

How much of what we call fluency is really local? If most of what makes a sentence read well lives within a few words, then the hard part of language is somewhere else, and it is worth being precise about where.

The history of AI is full of ideas that were set aside when something better arrived, not when they were shown to be wrong. Counting is one of them. The reasons it lost are well understood. Whether those reasons are permanent is still an open question.

References

Shannon, C. E. (1948). A Mathematical Theory of Communication.

Chomsky, N. (1956). Three Models for the Description of Language. (1957) Syntactic Structures.

Deerwester, S. et al. (1990). Indexing by Latent Semantic Analysis.

Kneser, R. and Ney, H. (1995). Improved Backing-off for M-gram Language Modeling.

Chen, S. F. and Goodman, J. (1999). An Empirical Study of Smoothing Techniques for Language Modeling.

Bengio, Y. et al. (2003). A Neural Probabilistic Language Model.

Brants, T. et al. (2007). Large Language Models in Machine Translation.

Vaswani, A. et al. (2017). Attention Is All You Need.

Liu, J. et al. (2024). Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens.

Children learn any language gradually and in order as soon as they are born, such as sounds, then words, then simple rules (household, societal norms etc.), then more complex things, each stage building from the past lessons they’ve learnt. Once a child internalizes a rule rather than just memorizing examples without a context or a pattern, they produce forms they've never heard before, which shows the rule is being applied rather than text being retrieved. From here I built an idea to teach the organism in stages instead of exposing it to the whole corpus at once, starting with basic word categories and simple patterns and building up to complex grammar. After each stage, let it generate its own attempts, the way a child babbles at first attempts, and use its own judgment faculty (already at high accuracy at telling good sentences from bad) as the feedback signal that reinforces which self-generated constructions are valid. No external correction or backpropagation needed, since the "teacher" is the organism's own existing judgment module which basically just pointed at its own output. This way the grammar becomes an applicable rule and correctly applies this rule to produce something well-formed and genuinely new at the same time. A second idea: Pair each stage with a simple attempt, self-check, revise loop, like how children correct themselves while learning to talk. If the organism can revise its own output and keep only what still works, that gives a clear, measurable sign that it is really composing something new.

Today, seven months in, we're saying something out loud that we've quietly meant since day one. Mura ALife Labs is committing to being a non-profit.

Built Before Labeled

We built the habit before we had the label for it. From the first week, every attempt got logged, wins and failures both, because a precisely measured "no" moves a search forward as much as a "yes" does, and that only stays true if nothing is pressuring you to round the "no" up into something better.

We believe intelligence research belongs to everyone, not to whoever can afford the compute bill.

Seven months of working this way convinced us it's worth locking in properly, not just running informally until it becomes inconvenient to keep it that way.

What We're Building

Here's the shape we're building toward:

• Mura ALife Labs, the lab, is committed to non-profit status, formal filing to follow. Its only job is the science: do the work, keep the record honest, publish what's true.
• Mura ALife, the organism the lab is building, is a separate thing. Free and open today for anyone doing research, learning, or building something non-commercial. Commercial use is where a separate license begins, coming soon, and that boundary is what will let the lab stay funded without ever asking the research to bend for it.

From Nonprofit to Shutdown

Before this, I ran a nonprofit AI company. We got to a hundred thousand users. Real people, real usage, six months of actually managing that. And then I shut it down myself. Not because it failed to matter to people — because it was a nonprofit, the compute kept getting more expensive, and there was no honest way to keep it running. I was sixteen. I made the call and closed it.

Most people would call that the end of something. I didn't have the patience to treat it that way. I'd already built something of real magnitude once, and I wasn't interested in building something smaller next. I wanted the next thing to be at least that size — ideally bigger, ideally harder, ideally the kind of thing that doesn't have an obvious ceiling on it.

The Insight: Brain versus Data Centers

Here's the thought that actually started this lab. A human brain runs on about twenty watts. Twenty watts, and it can reason, remember a whole lifetime, learn from one example, and never once quietly make something up to fill a gap. Meanwhile the AI industry was pointing enormous data centers at the same problem and calling the result intelligence. That gap is not a rounding error. If a brain can do this on twenty watts, the bottleneck was never intelligence itself — it was that nobody had engineered a substrate that actually works the way a brain does. So I stopped asking how to build a better model. I started asking how to build a brain.

Building Mura ALife Labs

That question is the entire reason Ikigai exists, and the entire reason Mura ALife Labs exists to build it. Not a bigger language model. Not a fine-tune. An organism — something that thinks with closed-form vector algebra instead of backpropagation, that learns continually instead of freezing after training, that runs on a single CPU core instead of a GPU farm, and that has one non-negotiable rule wired into it: tell the truth, or say "I don't know." No backprop means no GPU bill — the cost that shut my last project down. If this works, it isn't just cheaper. It's a different kind of cheap, the kind a twenty-watt brain already proved is possible.

Transparency and Honest Reporting

I run this lab the same way I shut down the last thing — honestly, in public, including the parts that don't work yet. We've made well over 270 distinct attempts at one specific hard problem inside this project, open-ended generation without a single gradient, and none of them have fully cracked it yet. I'd rather post that number than hide it. The nonprofit taught me that the moment you start protecting a nice-sounding story instead of the real one, you've stopped doing research and started doing marketing. I'm not interested in marketing. I'm interested in the brain actually working.

tell the truth, or say "I don't know."

Conclusion

I don't know yet if this is the thing that ends up mattering more than the last one. I know it's the only thing I've found that's genuinely bigger, genuinely harder, and genuinely worth being seventeen and stubborn about. That's why I'm building it in the open, one honestly-reported result at a time, starting now.

— Prince Siddhpara, Founder & Principal Researcher, Mura ALife Labs

Ikigai is not another language model

Ikigai is not another language model. It's an attempt at something stranger: a digital organism. Bind and unbind vector algebra and closed-form linear regression stand in for backpropagation, memory grows continually instead of a fixed training run, and one rule is wired into how it answers anything: tell the truth, or say you don't know. It never guesses and dresses the guess up as fact.

The bet

The bet is specific and checkable, not a slogan. Today's frontier models buy their intelligence with enormous compute and still hallucinate under pressure. We're building something that runs on a single CPU core, in well under a gigabyte, learns from every conversation it has for the rest of its life, and is essentially free to train — no backprop means no GPU farm, so the only real cost left is data. The target isn't "beat GPT at everything." It's narrower and harder to fake: beat frontier language models head-to-head on knowledge and truthfulness, at equal knowledge, for roughly a millionth of the compute.

What's real right now

What's real right now. Zero fabrications, measured — the organism answers or abstains, "I don't know," and our live demo tracks a fabrications-equals-zero counter against real traffic. It learns live: teach it a fact, even a whole paragraph, and it knows it instantly, no retraining, and it generalizes the phrasing too, so "X is the capital of Y" and "the capital of X is Y" both work after a single example. One organism, one API — every capability, answering, learning, reasoning, generating, routes through a single call where internal faculties compete on measured terms and the organism itself decides what to do, not a hand-written router. We hold ourselves to a 30-out-of-30 regression gate we run constantly against our own past claims, and a paper accepted at ALIFE 2026.

Where we're honest about losing

Where we're honest about losing. We do not out-write a large language model on long, open-ended generation today. Ask Ikigai for a five-thousand-word essay right now and it will not compete with GPT-class fluency. This is the one front where the architecture is still genuinely behind, and we'd rather say so plainly than let a demo quietly leave it out of frame.

What we're actually stuck on, and how hard we've hit it

What we're actually stuck on, and how hard we've hit it. Open-ended generation, grammatical, novel, human-readable text, built without a single gradient, is the hardest unsolved piece of this project. As of today we have made over 270 attempts at it. Not two hundred seventy tweaks on one idea — two hundred seventy structurally different mechanisms, each one discarded when it failed and replaced by something that attacks the problem from a genuinely different angle.

The most recent stretch of that, a single day out of the two-hundred-seventy-plus, shows what the wall actually looks like. We tried a frame-skeleton generator, a message-first co-occurrence model, two ridge-regression decode strategies, two joint order-and-coherence variants, an exact whole-sentence assignment solved by the Hungarian algorithm, and a local-search repair on top of that. Eight mechanisms, spanning three genuinely different families of math: finite-state, context-free, and combinatorial optimization. Every one of them hit real prose's surface statistics almost exactly. Every one of them still produced sentences no human would call English. That is not eight separate failures. It is the same law showing up eight times: matching how real text looks statistically is not the same as knowing what it means to say, and it is not enough on its own to make a sentence hang together.

The honest conclusion from that day: we have been approximating the structure of a sentence from the outside. What is actually missing is role-binding, genuinely representing who did what to whom as a bound structure, using the vector algebra this project already has for exactly that purpose, applied correctly, for the first time. That is the next thing we build. Not another version of what we just proved does not work.

Why we're posting this now, warts and all

Why we're posting this now, warts and all. Because the entire point of Ikigai is refusing to bluff, and that has to apply to us too, not just to the organism. We would rather tell you honestly that open-ended generation is unsolved, and give you the real number of attempts, than round it up into a demo that quietly cuts the hard part out of the picture.

If you want to see the part that already works, talk to the organism yourself. It is live, and nothing about the conversation is scripted. If you want to see the part that doesn't work yet, you now know exactly where we stand: well over 270 attempts in, one real architectural idea deep into the actual fix, and not stopping.

— Prince Siddhpara, Mura ALife Labs