3.3.2 General Results

After training, the training corpus and two test corpora were presented to the network again. (The test corpora also consisted of 4,000 sentences with lengths of up to 20 words.) This time, however, no modifications were made to the weights on the connections -- only the activation values of the 17 output units for each input pattern were saved so that the actual output of the network could be compared to the desired output. The three resulting files (one for each corpus) were 16 megabytes long and full of floating-point numbers. A Perl program was then used to process the files: a two-page summary of the error percentages for each of the output units was created, and the results files were rewritten to show the actual words of each input pattern, rather than the 85-bits long representations of these words. Figure 3.8 shows a sample session of the use of the program.

shivan:~/zin$ restore.input +-+-+ Starting CLASPnet's SNNS English Results Analyzer +-+-+ +-+-+ Copyright 1995-1996. Ezra Van Everbroeck +-+-+ Analyze which results file? demo.res Use which vocabulary file [demo.voc]? Name for RRF file (will overwrite) [demo.rrf]? ++ Writing RRF file: demo.rrf Name for error summary file (will overwrite) [demo.err]? ++ Writing error summary file: demo.err ++ Done. shivan:~/zin$

Figure 3.8: A sample session of how the simulator's results files were processed.

Before I provide more detailed results, it is worth comparing the overall performance of the network on the training corpus with how well it did on the test corpora. Figure 3.9 shows this comparison for various tolerance values. (These tolerance values are similar to the ones discussed above: if the actual activation value of an output unit was 0.85, and the desired output value was 1, then it would be considered to be an error for tolerances 0 and 0.1; with a tolerance of 0.2 an activation value of 0.8 instead of 1 suffices, so the output of the net would be considered correct. A tolerance of 0 is particularly harsh because networks use floating point numbers: even 0.99999 would not be considered correct if the target were 1. Usually, tolerance values of 0.2 and 0.3 are considered acceptable.) The precise numbers of these results can be found in Appendix 5.

There are three general observations I would like to make about Figure 3.9:

• The network has learned its task very well: ignoring the values for a tolerance of 0, we find that more than 90% of the input patterns are analyzed correctly -- and this on the basis of the averages over the 17 output units. At first glance, connectionist networks seem capable of learning to detect clausal properties.

• The network generalizes very well: the performance on the two test corpora is nearly on a par with that on the training corpus. Given, first, that there are millions of possible sentences which the grammar can generate, and, second, that the training corpus only contains 4,000 of these sentences, this is a somewhat surprising finding.

• Even a low tolerance gives very good results: increasing the tolerance above 0.1 does not yield much better performance. This shows that if the network recognizes an input pattern, it can also process it with near perfection. Conversely, if the net errs, it also makes large errors.

Word	Ind	Imp	Int	ID1	ID2	ID3	Sta	Dec	Ord	WH	YN	Rel	Con	Com	Inf	Voi	Pol
the	1	0	0	1	0	0	1	1	0	0	0	0	0	0	0	1	1
	1	0.04	0	0.98	0.01	0.01	1	0.92	0.01	0	0	0	0.09	0.01	0	0.54	0.60
child	1	0	0	1	0	0	1	1	0	0	0	0	0	0	0	1	1
	0.98	0.01	0	1	0	0.01	1	1	0	0	0	0	0	0	0	0.61	0.62
proclaim	1	0	0	1	0	0	1	1	0	0	0	0	0	0	0	1	1
	0.99	0	0	1	0	0.03	1	1	0	0	0	0	0	0	0	1	0.99
which	1	0	0	0	1	0	0	0	0	1	0	0	0	0	0	1	1
	1	0	0	0.01	1	0	0	0.01	0	1	0	0	0	0	0	0.81	0.81
pilot	1	0	0	0	1	0	0	0	0	1	0	0	0	0	0	1	1
	1	0	0	0	1	0.01	0	0	0	0.96	0	0	0	0.01	0	0.62	0.49
the	1	0	0	0	1	0	0	0	0	1	0	0	0	0	0	1	1
	1	0	0	0	1	0.01	0	0	0	0.90	0	0.06	0	0	0	0.87	0.45
great	1	0	0	0	1	0	0	0	0	1	0	0	0	0	0	1	1
	1	0	0	0	1	0	0	0	0	0.94	0	0.05	0	0	0	0.90	0.78
horses	1	0	0	0	1	0	0	0	0	1	0	0	0	0	0	1	1
	1	0	0	0	1	0	0	0.02	0	0.97	0	0	0	0	0	0.98	0.86
,	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
which	1	0	0	1	1	0	0	0	0	0	0	1	0	0	0	1	0
	1	0	0.02	1	1	0	0	0.01	0	0	0	0.98	0	0	0	0.28	0.11
can	1	0	0	1	1	0	0	0	0	0	0	1	0	0	0	1	0
	0.99	0	0	1	1	0.02	0	0	0	0	0	0.99	0	0	0	0.81	0.03
never	1	0	0	1	1	0	0	0	0	0	0	1	0	0	0	1	0
	0.96	0	0	1	0.98	0.01	0.02	0	0	0	0	1	0	0	0	0.99	0
eat	1	0	0	1	1	0	0	0	0	0	0	1	0	0	0	1	0
	1	0	0	1	1	0	0.01	0	0	0	0	1	0	0	0	1	0
,	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
can	1	0	0	0	1	0	0	0	0	1	0	0	0	0	0	1	1
	1	0	0	0.12	0.91	0.01	0.36	0.55	0	0.02	0.01	0.05	0.06	0.01	0.03	0.89	0.38
see	1	0	0	0	1	0	0	0	0	1	0	0	0	0	0	1	1
	1	0	0	0.06	0.94	0.02	0.89	0.13	0	0.01	0.02	0	0.04	0.17	0.01	1	0.95
.	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0

Table 3.3: The sentence 'the child proclaim which pilot the great horses, which can never eat, can see.' with the desired activation values for each word (top row), and the actual values (bottom row).

Let's focus on each group of output units in turn:

• Looking at the first three output units, we find that the mood of the sentence is determined correctly from the first word onwards -- imperatives do not begin with a determiner, nor do WH-Questions or the Yes/No-Questions in the corpus;
• The three ID units are also always very close to the mark. Of special interest is the fact that these units change to the correct value again when the embedded relative clause 'which can never eat' ends;
• The performance on the Status unit, however, is incorrect towards the end. This is a case in which the network seems to be expecting some extra words for the matrix clause;
• The units for the clausal types behave correctly up to the last two words: at the beginning of the WH-clause and the relative clause (both of which start with 'which'), the activation values of the relevant output units suddenly increase, while those of the previous types drop to zero; but when it gets to the final 'can see', the network finds itself at a loss;
• The Infinity unit never receives a significant activation value;
• The Voice unit only reaches a high activation value when the net is absolutely certain that the clause it is processing cannot be passive;
• The Polarity unit similarly begins each clause with an ambivalent activation value, and then changes to either a very high or a very low value when it is confronted with definite evidence (i.e. the presence or absence of a negative element in the position in which such elements tend to occur).