Experimental protocol
Which models the logic replicant was compared with, how they were configured, and how accuracy was measured.
Which methods the logic replicant was compared with, and how the exam was kept fair.
The rivals
The logic replicant was compared with four well-established methods, each used in its standard form:
- Random forest: a crowd of decision trees that vote.
- A small neural network: one hidden layer, sized as small as possible for each problem.
- XGBoost: a popular tree-based method that wins many data competitions.
- Self-organising map: a method that, like the replicant, produces a two-dimensional picture.
How the test was kept fair
Each dataset was shuffled and split ten times: 80% of the examples to learn from, 20% kept hidden for the exam. Every method took exactly the same ten exams. The reported score is the average share of hidden examples classified correctly. Two extra scores, precision and recall, check that small classes were not neglected. The four problems were chosen to range from a strict rule with no exceptions to a messy biological one, and to include one that is easy for everybody, so that the comparison shows where each method shines.
Baseline models
Four established classifiers serve as reference, covering easily interpretable models (SOM), ensembles of weak predictors (RF, XGBoost) and small feedforward networks (MLP):
- Random forest (RF). scikit-learn's
RandomForestClassifierwith default parameters: 100 estimators, all features, nodes grown as needed to reduce the training error as much as possible. Chosen because it almost always outperforms single decision trees and has documented good accuracy in many problems. - Multilayer perceptron (MLP). Keras on TensorFlow, one hidden layer with ReLU activation and a softmax output layer. The number of hidden neurons is the minimum required to learn the training set with the minimum error, chosen per experiment. A small MLP balances parameters against data size.
- XGBoost. The
xgboostlibrary with default configuration; onlymax_depthis tuned per experiment. Included for its record in competitions and its documented performance on many problems. - Self-organising map (SOM). MiniSom with a two-dimensional square grid, size varied per experiment, plus extra code to support classification. Included because it shares two design points with the replicant, dimensionality reduction and a graphical 2-D representation, while being internally very different.
Evaluation
The Akaike information criterion was considered, since the replicant needs fewer parameters than equivalent models, but the RF cannot be treated as a parametric model. Instead, prediction accuracy is estimated by non-exhaustive cross-validation with repeated random sub-sampling: 10 repetitions, each splitting the whole dataset at random into 80% for training and 20% for test, with the same 10 splits used for every model. Reported values are averages over the 10 repetitions: accuracy on training, and accuracy, macro precision and macro recall on test. For each class \(c\) the true positives \(\mathrm{TP}_c\), false positives \(\mathrm{FP}_c\) and false negatives \(\mathrm{FN}_c\) are counted, and
Accuracy is the share of correct answers. Precision and recall are computed per class and then averaged, so that small classes count as much as large ones.
| Problem | Training instances | Test instances |
|---|---|---|
| Parity function | 205 | 51 |
| Down syndrome in mice | 864 | 216 |
| Nucleosynthesis (augmented) | 204 | 5 |
| Handwritten digits | 1387 | 410 |
Sharing the splits removes data randomness from the comparison, but not the initialisation randomness of the MLP and the replicant, which rely on random initial parameters and completely different mathematical operations. This matters most for small replicants such as the 32-parameter parity model, where a poor initialisation is harder to recover from.
Why these four problems
The problems were chosen because (1) each is characterised by a dataset of modest size, (2) they range from a standard difficulty to extremely difficult, and (3) apart from the SOM, the baselines provide either reasonable accuracy or graphical interpretability but not both. Together they cover a compact deterministic logic (parity), an organic logic with secondary factors (Down syndrome in mice), a deterministic physical logic with many classes and very few samples (nucleosynthesis), and a mature perceptual task where mainstream models perform well (handwritten digits). The datasets.