The Language That Parses Itself: Inside the Architecture of NLP++

Article, Blog, NLP++, Top

August 2026

People who encounter NLP++ usually meet it from the outside: a language for building glass-box, deterministic text analyzers, where every decision the system makes traces back to a line you can read. That’s the user’s view, and it’s the right one to start with.

Underneath it is an architecture that deserves its own telling, because Amnon Meyers built the thing on three foundations that are unusual on their own and remarkable together.

  • NLP++ parses itself. The thing that reads your NLP++ files is not a hand-written C++ parser. It is an NLP++ analyzer.
  • Knowledge is a grammar. A hierarchical knowledge base — the Conceptual Grammar — that an analyzer consults and extends as it reads, and that is itself a rule base.
  • None of it started in 1998. The design descends, through code you can still read in the repository, from a DARPA natural-language system Meyers built on LISP machines starting in 1983.

This post walks through all three, and then goes a layer deeper into the data structures that hold everything up: the string tables and hash tables. If you’ve ever wondered how a rule-based system with thousands of rules and a 191,000-word lexicon manages not to be hopelessly slow, that last part is the answer.

First: NLP++ is not just a rule language

It’s tempting to describe NLP++ as “a pattern-matching rule language,” and that description is wrong by a wide margin. An .nlp file can hold three different kinds of content, and real analyzers use all three heavily.

Rules — @RULES

Patterns over the parse tree, each with attached code that runs before (@PRE), during (@CHECK) or after (@POST) the match:

@POST
  group(1,2,"_clause");
  setunsealed(1,"true");
@RULES
_xNIL <-
	_clausestart [opt]
	_xWILD [plus fail=(\, _fnword _qEOS _dbldash _clause _clausestart)]
	@@

Code — @CODE

Straight procedural NLP++ that runs once when the pass executes. No pattern, no tree traversal — just a program. Here is an entire pass from the English analyzer, which builds an ontology in the knowledge base:

@CODE
G("ontology") = findconcept(findroot(),"ontology");
if (G("ontology"))
  exitpass();       # One-time load per VisualText session.

G("ontology") = makeconcept(findroot(),"ontology");

L("eventive") = makeconcept(G("ontology"),"eventive");
L("geo")      = makeconcept(G("ontology"),"geo");
L("body_of_water") = makeconcept(L("geo"),"body_of_water");

L("con") = makeconcept(L("body_of_water"),"ocean");
loadattr("aftermath","nounsem",L("eventive"));
@@CODE

Functions — @DECL

User-defined functions, with parameters, locals, control flow and return values, callable from anywhere in the analyzer. This region type arrived in December 2001 and changed how NLP++ analyzers get written:

@DECL
########
# FUNC:	DICTATTR
# SUBJ:	Add attribute for dictionary word.
########
dictattr(L("str"),L("attr"),L("val"))
{
if (!L("str") || !L("attr"))
  return;
L("str") = strtolower(L("str"));
L("con") = dictgetword(L("str"));
replaceval(L("con"),L("attr"),L("val"));
}

How much does this matter in practice? In the English analyzer that ships with the engine, 26 of the 141 pass files contain no rules whatsoever — they are pure code and function libraries. That’s 16,359 of the analyzer’s 63,589 lines, about 26%, with not a single pattern in them. funs.nlp alone is 5,922 lines holding 69 documented functions; across the analyzer there are roughly 190 named NLP++ functions, on top of the 356 builtins the engine supplies.

So NLP++ is better described as a programming language with pattern-matching rules as a first-class construct. The rules are what make it distinctive; the code and functions are what make it usable for real work. Keep that in mind, because the bootstrap has to handle all three.

The shape of an analyzer

An analyzer is an ordered sequence of passes, listed in analyzer.seq. Each pass is one file. Passes run strictly in order over a shared parse tree — some rewrite it, some just run code, some only declare functions for later passes to call.

The English analyzer has 136 passes across 141 files holding roughly 2,900 rules. Where rules do run, the matcher is greedy, leftmost and deterministic: at each position it tries rules in file order, the first match fires, its action code rewrites the tree, and the sweep resumes past the matched region. No chart, no ambiguity packing, no probabilities. One input, one tree, every time.

RFA: the grammar written by hand, in C++

Every bootstrap needs a version zero, and NLP++’s is a module called RFA — the Rule File Analyzer, in lite/rfa.cpp. Its comment header states the problem plainly:

“Define the rules-file analyzer! This will be an internal representation constructed by hand, that performs the parsing and internalization of the rules files themselves.”

RFA is an NLP++ analyzer that never existed as a text file. It is assembled in memory, object by object, by 4,635 lines of C++ — a function per pass: rfa_bigtok(), rfa_element(), rfa_rule(), rfa_code(), rfa_rulesfile().

Here’s a genuine excerpt from rfa_element(), building one single rule:

///////////////////////////////////////
// RULE  1:   _ELEMENT <- _NONLIT _PAIRS @@
///////////////////////////////////////

func = _T("rfaelement");
arg1 = new Iarg(_T("1"));
arg2 = new Iarg(_T("2"));
args = new Dlist<Iarg>();
args->rpush(arg1);
args->rpush(arg2);
posts = Iaction::makeDlist(func, args);

func = _T("single");
Iaction::addDelt(posts, func, 0);

phr = new Dlist<Ielt>();                        // Create rule phrase.
Ielt::addDelt(phr, _T("_NONLIT"), 0, 1, 1);     // Add element to phrase.
trig = Ielt::addDelt(phr, _T("_PAIRS"), 0, 1, 1);
Ielt *tr = trig->getData();
tr->setTrigger(true);

sugg = new Isugg(_T("_ELEMENT"));               // Create suggested elt.
sugg->setBase(true);

Roughly forty lines of C++ to express one rule. Notice the comment on the first line — Meyers wrote the NLP++ he wished he could write, as a comment, directly above the C++ that laboriously constructs it. That comment is a promissory note. He was about to cash it.

The other half of RFA lives in lite/postrfa.cpp — 8,618 lines of semantic actions named rfaelement, rfarule, rfbdecls, rfarulesfile. These turn a parse tree of an NLP++ file into live, executable objects. They are the only part of the machinery that had to stay in C++, and they are still there today.

Friday, November 5, 1999, 9:37 PM

There is a function in lite/ana.cpp called genAna(). Its job description is one line:

* FN:    GENANA
* CR:    11/04/99 AM.
* SUBJ:  Write the entire analyzer to files.
* NOTE:  Using internalized structures, not rulefile.

It walks a live in-memory analyzer and writes it back out as NLP++ source — the sequence file plus one .nlp file per pass. Point it at RFA, the analyzer built by hand in C++, and it emits an NLP++ program that parses NLP++.

That is exactly what happened. Open data/rfb/spec/element.nlp today:

###############################################
# FILE:     ELEMENT.PAT (pass 0)
# SUBJ:     Automatically generated RFA file.
# AUTH:     VisualText
# CREATED:  Fri Nov 05 21:37:45 1999
###############################################

@PATH _ROOT _RULES

@POST
	rfaelement(1, 2)
	single()
@RULES
_ELEMENT [base] <- _NONLIT _PAIRS @@
_ELEMENT [base] <- _LIT _PAIRS @@
_ELEMENT [base] <- _NUM _PAIRS @@

Forty lines of hand-written C++ became three lines of NLP++. And it wasn’t transcribed by hand — it was dumped, by the engine, from the structures the C++ had built. Twenty-three files in that directory carry the identical timestamp Fri Nov 05 21:37:45 1999. That single second is when the language became able to describe itself.

This is the classic compiler bootstrap — write version zero in the host language, use it to emit version one in the target language, then let version one take over. What makes it notable is the domain. Bootstraps are routine for programming languages; doing it for a text-analysis engine means the same machinery that recognizes noun phrases in a legal document recognizes @POST blocks in your rule file. There is one engine, not two.

RFB: the analyzer that replaces its own parent

The generated analyzer is RFB, living at data/rfb/spec/ — about 2,174 lines of NLP++ across 42 files. It’s loaded at startup by VTRun::init(), and the comment marking the handover is one of the best lines in the codebase:

rfa_ = create_rfa(htab_, logfile, nlpa_);
rfb_ = create_rfb(logfile, rfbspecdir, false, silent, rfa_, htab_, nlpb_);

// MAKE THE RFB ANALYZER.
// RFA is used to make RFB, which then replaces RFA!

From then on, every NLP++ file you write is parsed by RFB. RFA stays resident purely as the thing that can bring RFB up, and make_rfb() has a graceful fallback: if the spec directory is missing, the engine logs “No RFB. Using the RFA for parsing rules” and carries on. The ladder is never kicked away.

Here is RFB’s own 36-pass sequence:

tokenize        line
pat  retok      pat  bigtok     pat  x_white    pat  xvar
pat  nlppp      pat  un_mark    pat  list       pat  list1
pat  gram1      rec  gram2      pat  decls      pat  pair
pat  pairs      pat  element    pat  rule       pat  rules
pat  decl       pat  code       pat  pres       pat  checks
pat  posts      pat  tmp        pat  tmp1       pat  select
pat  region     pat  regions    pat  recurse    pat  recurses
pat  rulesfile  pat  finalerr

nintern         gen             hash            genhash

Read it and the architecture reveals itself. Early passes tokenize NLP++ source (bigtok handles comment and string collection). Middle passes recognize the language bottom-up, exactly the way the English analyzer recognizes clauses: pairs → elements → rules → rule files.

And notice how much of that sequence is not about rules. gram1 and gram2 are the expression and statement grammar for NLP++ code — gram2 is a rec pass, recursive, because expressions nest. Then decl, decls, code, pres, checks and posts collect the @DECL, @CODE, @PRE, @CHECK and @POST regions. A third of the pipeline exists to parse the programming-language half of NLP++.

The last four entries aren’t rule files at all — they’re engine-implemented pass types, dispatched by name in ana.cpp:

  • nintern — walks the finished parse tree and interns it into live objects the engine can execute.
  • gen — the code generator, for compiling analyzers to C++.
  • hash — builds the rule dispatch hash tables.
  • genhash — emits those tables as static C++ arrays.

The boundary is drawn exactly where it should be. Syntax is NLP++. Semantics and optimization are C++.

Why the bootstrap still earns its keep

A bootstrap that paid off once would be a curiosity. This one pays off every time the language grows. NLP++ recently gained C-style block comments — added by editing bigtok.nlp, an NLP++ file whose header still reads “CREATED: Fri Nov 05 21:37:45 1999”:

# C-STYLE BLOCK COMMENTS.  /* ... */
# Non-nesting, like C: the first "*/" closes the comment.
@POST
	excise(1, 3)
@RULES
_xNIL <- _BEGCOM _xWILD _ENDCOM @@

The xvar pass is another recent addition — support for the _xVAR("attribute") match-list special — and it too is a rule file, not a C++ change. And there’s a quieter payoff: RFB is a permanent, non-trivial, in-production test. Every startup runs a 36-pass NLP++ analyzer over real input. If the matcher regresses, the engine won’t boot.

The Conceptual Grammar: knowledge as a rule base

Now the second foundation — the one that most distinguishes NLP++ from every chunker and cascade it superficially resembles.

Alongside the analyzer sits a knowledge base management system whose headers introduce it like this:

* SUBJ:  API for VisualText KBMS.
*        KBMS = knowledge base management system. A hierarchical data base.
*        CG   = Conceptual Grammar. A knowledge representation scheme.

The name is the claim. It isn’t called a knowledge base or an ontology — it’s called a grammar, because in Meyers’ design knowledge and grammar are the same kind of thing. Rules recognize structure in text; the Conceptual Grammar holds structure about the world, in a form rules can read and write. Both are declarative, inspectable, and editable by hand.

What a concept is

The core structure is small and general. From xcon_s.h:

typedef enum xckind
   {
   cXPROXY,    // A con representing another con. Not in the hierarchy.
   cXBASIC,    // A concept that resides in the concept hierarchy.
   cXWORD,     // A concept representing an "atomic" piece of text.
   } XCKIND;

typedef struct xcon_s
   {
   int id;
   XCKIND kind;
   int flags;
   int attrs;      // Attribute list. Use for: skips, starts, ends, rules, etc.
   int up;         // up hierarchy
   int dn;         // down hierarchy
   int prev;       // prev sibling  /  if PROXY: previous elt of phrase
   int next;       // next sibling  /  if PROXY: next elt of phrase
   int atom;       // if PROXY: con that owns phrase/node
   int phrases;    // Setting up for concept to own multiple phrases.
   } XCON_S;

Four ideas, and each earns its place:

  • Hierarchy. up/dn/prev/next make concepts a tree, so inhierarchy(L("nsem"),"event") is an ancestry walk. Inheritance comes free.
  • Attributes. Named slots on a concept whose values can be strings, numbers, or other concepts. That last option is what makes it a semantic net rather than a property bag.
  • Word concepts. The lexicon isn’t a separate structure — a dictionary word is a concept. dictfindword() returns one, and its attributes are its lexical entry.
  • Phrases. A concept can own an ordered sequence of proxy nodes. A concept can hold a pattern.

That last one is the rule base. And note what the attrs comment says the attribute list is for: “skips, starts, ends, rules, etc.” Grammar knowledge is stored on concepts like any other knowledge.

The Gram hierarchy, and rules from samples

Open the shipped English knowledge base, hier.kb, and the third line is:

add hier "concept" "gram"
add hier "concept" "sys"
add hier "concept" "sys" "attrs" "rule_elt"
add hier "concept" "sys" "nlp" "pos" "noun"
add hier "concept" "sys" "nlp" "pos" "verb"

A gram branch, sitting as a peer of sys, right at the top of the world. And there’s an engine module that reads it — lite/mode.cpp:

“Mode data for Gram concept hierarchy. Mode data passes information down the Gram hierarchy. It includes flags for the levels of generalization of rules from samples. Rules for each flagged level will be written to rules files and retained internally.”

Read that carefully. You attach samples — concrete phrases from real text — to a concept in the Gram hierarchy. The rule generator (lite/literug.cpp, with its maxConstrains, fixConstrains, pruneConstrains) generalizes those samples into patterns at several levels of abstraction, governed by mode flags inherited down the hierarchy. The result is written out as NLP++ rule files.

So the Conceptual Grammar is a rule base in the full sense: rules live in it as knowledge, and rules are produced from it. And notice where that lands us — the KB generates NLP++ source, just as RFA generated NLP++ source. Meyers built two independent paths to the same destination, and both end in a text file a human can read and edit. That is the whole philosophy in one observation.

(This is VisualText IDE machinery. The shipped English analyzer’s gram branch is empty — its 2,900 rules were written by hand.)

Knowledge accumulating during a run

The KB isn’t read-only reference data. Rules and functions build it up while analyzing, through builtins like makeconcept, addconval, replaceval, findconcept, conval and inhierarchy. Later passes consult what earlier passes learned — which is how an analyzer reasons across sentence boundaries. As the VisualText documentation puts it:

“The real power of NLP++ is the ability to consolidate information about entities mentioned throughout a text. This is possible through the use of NLP++ rules, functions, the parse tree, and the knowledge base (conceptual grammar) all working together.”

Here’s what that actually produces. Run the English analyzer over a Department of Justice press release and dump the KB afterward:

currtext
  objects:
    count=[4]
    object1 = the former southeast asia chairman:
      type=[object]
      text=[the former southeast asia chairman]
      sem=[name]
      refs=["concept" "currtext" "parse" "sent1"  "clause3" "obj1 = ...",
            "concept" "currtext" "parse" "sent3"  "clause3" "obj1 = ...",
            "concept" "currtext" "parse" "sent5"  "clause3" "obj1 = ...", ...]

A concept per discourse entity, carrying its semantic type — and a refs attribute holding paths back into the parse tree, one per mention. Every place that entity appeared is recorded, addressable and inspectable. Coreference and entity consolidation as a knowledge structure you can print out and read, not as a similarity score.

The persistent knowledge is text too. The shipped English KB is four plain files — hier.kb (769 hierarchy commands), attr.kb, word.kb, phr.kb — plus a 191,542-entry dictionary. Every one is diffable, greppable and hand-editable. For very large lexicons the engine now binary-searches a sorted *-full.dict on disk one word at a time instead of loading it all, so the glass-box property survives scale.

The third foundation: VOX, and forty years of one idea

Here is where the repository turns into an archaeological site.

Open cs/libkbm/sym.cpp — the symbol table underneath the Conceptual Grammar. Above the Text Analysis International banner sits an older one:

Copyright (c) 1995 by Conceptual Systems.
*                                SYM.C
* FILE:	consh./sym.c
* SUBJ:	Symbol table manager for Conan.
* NOTE:	For the compiled analyzer (i.e., Conan), implementing highly optimized
*		(hopefully) (for Macintosh) primitive table managers from scratch.
*		As always (eg VOX), our symbol table is a simple hash table.
* CR:	5/02/95 AM.

“As always (eg VOX).” Four words, written in 1995, and they open onto another decade.

VOX was Meyers’ natural language system, built between 1983 and 1988 at the University of California, Irvine AI Laboratory, for DARPA and the Naval Ocean Systems Center. It was, in his description, “an extensible natural language processing system developed on LISP machines,” and its job was to read and understand naval message traffic. It has one more claim on the field’s history: the problem of evaluating VOX is what led to the creation of the Message Understanding Conferences — the benchmark series that defined information extraction as a discipline.

So the lineage runs like this, and every link is visible in the source tree:

  • 1983–1988 — VOX. LISP machines, UC Irvine, DARPA/NOSC. Naval message traffic. Symbol table: “a simple hash table.”
  • 1995 — Conch, Conan, Consh. Conceptual Systems. The ideas re-implemented in C, for the Macintosh, in 32K segments. st.c (May 1995) is the string table; sym.c (May 1995) is the symbol table; cc_gen.h (October 1995) is “Declarations for overall code generation.”
  • 1997–98 — Text Analysis International. Meyers and David de Hilster formalize a decade of work into NLP++, VisualText, and the Conceptual Grammar. The engine’s own stab.h and htab.h appear in November 1998.
  • 1999 — the bootstrap. RFA emits RFB; NLP++ starts parsing itself.
  • 2018 — open source. TAI dissolves after roughly twenty years and clients including NASDAQ and IBM Global Services UK; the engine, VisualText and NLP++ are released under the MIT license.

The 1995 names are not merely historical. Conan was the compiled analyzer; Consh was the interactive shell. Thirty years later the engine still ships that split as -COMPILED and -INTERP — and the library implementing the Conceptual Grammar is still called libconsh. Some files even carry both banners at once:

/*** CONAN: AUTOMATICALLY GENERATED! EDITS WILL BE LOST. ***/
/*** CONSH: HAND-EDITING OK ***/

And bind.cpp, from the same era, describes its own job as “Consh — Part of bootstrapping the knowledge base.” Bootstrapping, code generation, and a compiled/interpreted split were all present in 1995. What happened in November 1999 was that the same instincts were finally turned on the language.

One more symmetry is worth sitting with. VOX was built to read naval message traffic and pull out who did what to whom. Four decades later, the analyzer in this repository reads a Department of Justice press release and produces a concept per entity with every mention indexed back into the parse tree. Same task. Same conviction that the answer should be a structure you can read rather than a number you have to trust.

(Biographical details from the Amnon Meyers entry at the World Science Database. Meyers came to all of this sideways — a biology degree from MIT, then master’s degrees in organic chemistry and computer science from Berkeley.)

The string table: memory as a stack, not a heap

Text analysis is a string-shredding activity. Every token, node name, attribute value, concept name and dictionary word is a string, and a naive implementation drowns in malloc/free traffic and fragmentation.

Meyers wrote this allocator at least twice — for the KBMS in 1995, for the engine in 1998 — and reached the same answer both times. Here’s the engine version, lite/stab.h, whose header states the goal: “This will enable use of strings without worrying about allocation and freeing.”

#define STAB_SEG_SIZE  (((long) 8192 * (long) 64) + (long) 1)   // 524,289 chars
#define STAB_SEG_MAX   8000

_TCHAR *seg_[STAB_SEG_MAX];   // The array of segments.
int     curr_;                // The segment currently in use.
int     last_;                // The last segment allocated.
_TCHAR *ptr_;                 // First empty location in current segment.
int     perm_;                // The last permanent segment.

An arena allocator: adding a string copies it to ptr_ and advances. Nothing is individually freed. When a segment fills, nextSeg() takes the next, reusing an already-allocated segment when possible. Ceiling: eight thousand half-megabyte segments.

The 1995 ancestor is the same design at Macintosh scale, and its notes are wonderfully of their moment — “String table split into segments (Macintosh)”, and a wistful aside that “for UNIX, etc., bigger segment sizes than 32K are advisable.” The engine’s segments are exactly sixteen times larger. Nothing else changed. That’s a design that was right the first time.

The elegant part is perm_, and its comment gives away the whole idea:

int perm_;   // The last permanent segment.
   // eg, each new text analyzed starts using string space at perm_+1.

A generational split. Everything loaded once and needed forever — the grammar, the functions, the dictionary, the knowledge base — sits below perm_. Everything created while analyzing a document sits above. Then, between documents:

void Stab::resetStab()
{
if (perm_ == -1)
	{ curr_ = 0;         ptr_ = seg_[curr_]; }
else		// There is a permanent part.
	{ curr_ = perm_ + 1; ptr_ = seg_[curr_]; }
}

Two assignments. That’s the entire teardown of a document’s string memory — no traversal, no free list, no destructors, no fragmentation, and the segments stay allocated so the next document starts with the allocator warm. Per-document string cleanup is O(1).

The symbol table, and a lexicon hiding inside it

On top of the string table sits Htab — the hash table VOX’s descendants have always used — holding Sym objects, each pointing into the string table.

#define HTAB_SIZE   ((long) 250007)   // prime
#define HASH_MAX    25                // max chars to hash on

_TCHAR *Htab::getStr(_TCHAR *str)     // Get a hash table version of string.
{
Selt<Sym> *ptr;
if (!(ptr = hget(str)))               // hget = find, or add if absent
	return 0;
return ptr->getData()->getStr();
}

Textbook string interning: hand it a string, get back the canonical copy. Ask twice for "_noun", get the same pointer. Node names, rule element names, function names and dictionary words all flow through here, so the engine stores one copy of each distinct string and the parse tree becomes pointers into a compact arena.

The hash function is small, tuned pragmatism, refined across a decade of dated edits:

long Htab::hashfn(_TCHAR *str, long hsize)
{
_TUCHAR ch;
unsigned long val = 0, warp = 113, ii = 1;

while ((ch = *str++) != '\0')
	{
	warp += 22;
	val  += ((unsigned long) ch * warp * warp);
	if (val > (unsigned long) hsize)
		val = val % hsize;
	if (++ii > HASH_MAX)
		return val;
	}
return val % hsize;
}

Position-sensitive (warp grows as you walk, so anagrams don’t collide), reduced eagerly to stay in the register, and capped at 25 characters. That cap is a domain judgement: when your keys are node names and dictionary words, bounding hash cost beats perfect discrimination among long shared prefixes.

The symbol as a cache on the knowledge base

Then a move you won’t find in a general-purpose intern table. Look at what a Sym carries:

protected:
	_TCHAR *str_;      // POINTER TO STRING TABLE.
	int     flags_;    // isLooked(), isKnown(), isWord()
	Sym    *lcsym_;    // Lowercase sym, if any.
	int     use_;      // Use count, for garbage collection.

The comment above it is candid: “Can associate dictionary lookup, lexical information with the sym. Even concepts!” — then “Placing lexical information directly in the sym, for now.” That “for now” is dated January 1999 and it’s still there, because it was right.

Those flags make the symbol table a memoized bridge to the Conceptual Grammar. isLooked() records that the word has been through KB lookup; isKnown() records the answer:

if (sym && sym->isLooked() && !(sym->isKnown()))
	// Don't bother going to the knowledge base again.

Negative caching matters more than positive here. Real text is full of names, codes, typos and IDs that are in no lexicon, and re-querying the KB for each occurrence is exactly the cost you can’t afford. lcsym_ plays the same trick for case: the lowercase form is resolved once and linked, so case-insensitive lookup is a pointer hop rather than a per-occurrence transformation.

One more table is built at startup: htfunc_, holding NLP++’s 356 builtin function names, so resolving group() or makeconcept() in your code is a single probe.

Hashing the rules themselves

Here’s the optimization that makes large analyzers viable, and the cleverest use of hashing in the engine.

Recall the semantics: at each node, try every rule in the pass, in file order, fire the first that matches. Taken literally, a 300-rule pass over a 5,000-node tree is 1.5 million rule attempts.

The hash pass indexes each pass’s rules by the node names they could start with. From Ifile::rhash():

int fudge = 30;
len  = rules->getLength() * fudge + 1;
htab = new tHtab<Slist<Irule> > (Htab::getGstab(), len);
musts_ = new Slist<Irule>();          // Empty list.

for (drule = rules->getFirst(); drule; drule = drule->Right())
	{
	rule = drule->getData();
	rule->setNum(++num);              // Number the rules in FILE ORDER.

	if (!rule->rhash(htab, parse))    // Place into hash table if possible.
		musts_->rpush(rule);          // Couldn't hash: must always be tried.
	}

Three decisions are packed in there. The table is deliberately sparse — fudge = 30 gives thirty slots per rule, because a rule registers under many keys: every optional element at its head, plus every name in its match= list. Some rules can’t be indexed: a rule beginning with _xWILD has no usable key, so it lands on musts_ and is tried at every node forever — and the engine prints NOHASH rule: so you can see which of your rules opted out of the fast path. And rules are numbered in file order, which is what makes the whole thing sound.

Merging candidates without disturbing file order

At match time, Pat::resetRules() assembles the candidate list. It starts with musts_, then walks downward through the chain of single children beneath the node — stopping at a branch, a leaf, or a node marked base — pulling each name’s rule list from the hash:

while (node)
	{
	nname = node->getData()->getName();
	htab->hfind_lc(nname, /*DU*/ tmp);      // Get node's rules.
	if (tmp)
		Irule::mergeRules(rules, tmp);      // Merge rule lists.

	if (pn->getBase())          break;      // Bottommost singlet.
	if (!(node = node->Down())) break;      // Leaf.
	if (node->Right())          break;      // Branches out.
	}

That downward walk is what lets a rule written against _noun still fire at an _np node whose only child is that _noun — NLP++’s multi-tier matching, as a hash lookup per tier rather than a search. And mergeRules() merges by rule number, so the candidate list comes out in exactly the order those rules appear in your file.

The hashing is therefore a pure optimization: it changes which rules get attempted, never which rule wins. Delete the hash tables and the analyzer produces identical output, just slower. For a system whose central claim is determinism, that property isn’t a nicety — it’s the thing that lets the optimization exist at all.

Knowing when not to hash

Hashing appears a third time, inside rule elements. Set-membership tests like fail=(...) get tested at every node a wildcard crosses, so Ielt builds each one a private hash table — but only past a threshold:

#define HTHRESH 10

len = vals->getLength();
if (len < HTHRESH)      // Too short to bother with hash table.
	return;

hsize = Htab::makesize(len);          // 3*len + (len%2) + 1, odd
htab  = new Htab(Htab::getGstab(), hsize);

Under ten entries a linear scan wins — better cache behaviour, no table to build or free. Over ten, the hash wins. Small, but it’s the mark of someone who measured rather than assumed.

Freezing the tables into source code

One last turn, and it closes two loops at once.

NLP++ analyzers can be compiled to native shared libraries — and so can their knowledge bases. When they are, you’d rather not rebuild all those hash tables at startup, so the fourth engine pass, genhash, doesn’t build tables. It writes them out as C++ source:

// A table of tables of strings.
*ehead << _T("extern const _TCHAR **") << table << _T("[];");
*ehash << _T("const _TCHAR **")        << table << _T("[]={");

The hash table becomes a static array of arrays of string literals, laid out in the slots the runtime hash function will compute. The C++ compiler puts it in read-only data; the OS maps it in. A compiled analyzer starts with its dispatch tables already built, at zero cost.

Notice where that generator sits: it’s a pass in the sequence file of the analyzer that parses NLP++. The bootstrap doesn’t just give you a self-describing language — it gives the optimizer and the code generator a place to stand. And notice the date on its ancestor, cc_gen.h: October 1995. Meyers had been generating C from live knowledge structures for four years before there was an NLP++ to generate it for.

What the architecture adds up to

  • NLP++ is a programming language, not a rule notation. Rules, code and functions are co-equal. A quarter of the English analyzer contains no patterns at all.
  • The language is defined in itself. RFA bootstrapped RFB in 1999; RFB has parsed every NLP++ file since — rules, code and function bodies alike. New syntax arrives as new rules, not new C++.
  • Knowledge is a grammar. The Conceptual Grammar holds hierarchy, attributes, word concepts and phrases in one uniform structure — readable as text, writable during analysis, and capable of generating rules from samples.
  • There is one pattern matcher. The machinery that finds noun phrases finds @POST blocks. Every optimization makes both faster.
  • Strings are owned by the system. One arena, one canonical copy per string, O(1) reset between documents — a design proven on a Macintosh in 1995 and unchanged since.
  • Hashing never changes an answer. Rule dispatch is indexed but re-sorted into file order, so fast path and slow path are observably identical. Determinism survives optimization.
  • The engine tells you when it can’t help. NOHASH rule: is glass-box thinking applied to the optimizer itself.

Read the source and you can watch four decades of reasoning happen in place. Comment headers are initialed and dated — 5/02/95 AM, 10/18/98 AM, 11/05/99 AM, 06/24/03 AM — an engineering log left where it was written, arguing with itself, occasionally noting a fix that took four years to find. The _TCHARs date the code, and so does an honest // for now from January 1999 that turned out to be permanent.

But the architecture underneath has aged remarkably well, for a reason worth naming. Meyers treated language understanding as a systems problem — arenas, interning, symbol tables, indexed dispatch, a knowledge representation, a bootstrap, a code generator — and kept treating it that way while the field moved decisively toward statistics. That decision is why an NLP++ analyzer can be read, debugged, reproduced exactly, and compiled to a native library.

It began on a LISP machine in 1983, reading naval cables. Twenty-seven years after a Friday night in November 1999, extending the language still means writing NLP++.

The NLP++ engine is open source at github.com/VisualText/nlp-engine. The bootstrap analyzer is in data/rfb/spec/; the hand-built original is lite/rfa.cpp; the Conceptual Grammar KBMS — with its 1995 banners intact — is under cs/ and include/Api/; the string and hash tables are lite/stab.h and lite/htab.h. Biographical detail from the Amnon Meyers entry at the World Science Database. Join the discussion at nlp.discourse.group.

Loading