With our NLP++ textbook coming out world-wide this month (it may be already out by the time you read this), people are asking: what is the difference between LLMs and NLP++? Here is a first attempt of mine to explain.
NLP++ and LLMs aren’t two approaches to the same problem. They’re fundamentally different things.
NLP++ is a programming language. It gives developers the framework to write programs that read and understand text like humans. It comes with example analyzers that can be used as possible templates including a full English parser. Still, every text analyzer has to be built by hand, encoding all possible cases for a specific NLP task. That requires ingenuity and effort — real people power. The payoff is that it’s 100% deterministic code: it will always produce the same output, it can be inspected, and if something is wrong, it can be fixed. It runs cheaply and can run locally — it’s just executing code. And it builds on common knowledge, not copyrighted data. The task of writing NLP analyzers by hand is seen to be impossible today, but Amnon Meyers and I created our “dream programming language” for NLP that allows any programmer to write deterministic parsers without being a computational linguist.
LLMs work out of the box and require no programming, which makes them appealing. But they can never be fully trusted. Their behavior is rooted in training, not logic, so you can’t audit why they gave a particular answer, and you can’t guarantee they’ll give the same answer twice. They’re expensive to train and run — and they train on copyrighted data, which raises its own set of problems.
A good analogy for NLP++ is SQL. SQL is a programming language and framework for databases and also requires programmers to write code. It was a paradigm shift over flat files and once its advantages were understood, it became widely adopted and now runs over 80% of all databases worldwide. NLP++ is at that same early stage. The language exists. Now the world needs to know it exists and begin to use it to build trustworthy NLP. Thus the first NLP++ textbook.
The difference is that with SQL, the data is proprietary whereas with NLP++, the data and algorithms are common knowledge. With time, dictionaries, knowledge bases, and linguistic algorithms will eventually be constructed and shared so that like LLMs, NLP++ will be usable out of the box. The difference is one is deterministic code, the other an untrusted black box.
That’s why we’ve started the NLP Foundation — a decade-long, worldwide initiative to govern that process: building out the dictionaries, world knowledge, and NLP algorithms needed to make trustworthy, community-owned text understanding a reality in every language. This is possible given we now have the first universal programming language for natural language.
NLP++ and LLMs operate in a different categories entirely. If close enough is good enough, LLMs are convenient. If you need something correct, auditable, and trustworthy, you write the code.
