4.2 Discussion

As the results which have just been summarized have already been discussed when they were first mentioned in Chapter 3, I will limit myself here to a few general remarks about the implications of CLASPnet. First, the positive aspects:

• CLASPnet is small: With its nearly 400 units and 20,000 modifiable connection weights, the network is not small by connectionist standards -- interesting behavior has been exhibited by much smaller networks. But from a neuroscientific point of view, 20,000 synapses is infinitesimal. (The brain has approximately 10E11 neurons and 10E14 synapses.) Naturally, the comparison is not entirely fair, because the brain can not be trained using efficient supervised learning techniques like the backpropagation algorithm.

  The fact that a small artificial neural network can learn to detect clausal properties with a very high degree of accuracy on the basis of local, word-level input suggests that the task itself can not be that difficult. Hence, it also suggests that detecting properties like voice and polarity, or recognizing the general clausal type is not very difficult for humans either. If we are dealing with very robust overt phenomena, it is only natural that they can be picked up quite easily by neural nets, as pattern spotters par excellence. Psycholinguistic experiments are needed to confirm this hypothesis, but recall that a sentence like 'the bilsam will never eeph the lolums in a tum.' already provides us with weak evidence that humans can indeed understand a lot about the properties of a sentence and clause, even when there is not much in this sentence or clause that makes sense.

• CLASPnet is a connectionist model: In this respect, it provides yet another piece of evidence that artificial neural networks can be successfully used to construct computational models of how the mind may work. As most of the theoretical motivations for doing this research have sprung from the desire to show that the traditional formal approaches to mind (see 1.2.1) and language (see 1.2.2) are not the most appropriate ones, the fact that an inherently 'un-formal' approach like that of CLASPnet can be shown to be able to deal with matters it supposedly could not is rewarding. As soon as the words in the input leave the input layer behind, the distributed representations which they create in the hidden layers are no longer symbolic -- there's no systematicity, productivity, or compositionality in the sense defended by Fodor & Pylyshyn (1988).

  As far as the acquisition of syntactic elements is concerned, CLASPnet illustrates that precise innate knowledge of any kind is not required. As long as the regularities which have to be learned are somehow present in the input, and as soon as there is an incentive to search for these regularities -- backpropagation in this case, but in human beings probably the desire to communicate -- then a neural network has a fair chance of learning to find them.

  As a connectionist model, CLASPnet is also useful because it proves that the findings of other models which used more limited grammars and vocabularies do indeed scale reasonably well to more complex, and more natural-language-like input. The only area in which this has not been the case are center-embedded relative clauses, but even these may become tractable when more of them are presented to the network during training.

• CLASPnet likes semantics: In my opinion, one of the most interesting results of all the simulations is that the network uses the semantic information which is presented to it, and this despite the fact that the semantic units were explicitly chosen with the aim not to give high-level information about word classes. As the scores of the network which was only trained on semantic representations show, there is a lot of information in there -- it may be largely redundant, and it may not be as easy to detect as the information stored in the orthographic representations, but it is nonetheless used. (This conclusion meshes well with that of first language acquisition research by Bates et al. (1988): they found that there was no evidence, at least until the children were 28 months old, of separate syntactic and semantic processes.)

  With this result, the two practical motivations for CLASPnet (see 1.2.3 and 1.2.4) have also linked up: not only do they show that typically linguistic concepts can be integrated into connectionist models, they are also an indication of the mileage which is to be had from combining cognitive linguistic ideas with neural network implementations. As such, one can only hope that more linguists will become involved in running connectionist simulations.

The good aspects of CLASPnet notwithstanding, a number of caveats have to be presented as well:

• It is just English!: There are two points to this criticism, and they both sting. First, making any theoretical linguistic claims on the basis of data from a single language is unwise. Hence, for 'independent' validation similar models for other languages should be developed. Second, English happens to be a very untypical language (Bates & MacWhinney 1989): its rigidly enforced word order is a very rare property. As CLASPnet had to become sensitive to sequences of words in order to determine their clausal properties, it might have had a far more difficult task if the word order had been not as strict. On the other hand, languages with a looser word order tend to have a richer case system, and case markings are usually overt; so, for these languages too, valuable cues would be available to the network. A few CLASPnet-like simulations for such languages should provide more objective data about which system can be learnt more easily.

• It is not English!: The corpora used to train and test CLASPnet are not real English. More particularly, the absence of inflectional morphemes for tense and agreement should be considered a major omission, because these morphemes tend to carry a lot of information which is relevant to clausal properties. The positive side of this criticism is that adding such inflectional information to the input will also provide the network with additional useful cues. But even if the context-free grammar were enriched to generate inflectional morphemes as well, the linguist in me knows that it would never generate real English. So, the only one correct answer to this criticism is to bite the bullet and manually code thousands of real English sentences, preferably with extensive samples of all the various types of spoken and written English.

• What about the brain?: Like most connectionist models, CLASPnet is but a feeble copy of the way a group of real neurons in a real brain behaves. There are too many links, only one type of cell, no neurotransmitters, no glial cells, the recurrent neural network architecture is implausible, and backpropagation most definitely does not accurately model the way in which real synapses change their behavior. Supervised learning techniques in general are a no-no if one wants to stay close to what is known about the grey matter.

• What about the mind?: Even ignoring questions about consciousness and the like, there are known psychological and psycholinguistic data which CLASPnet can not or does not model: for example, once training has stopped, the network can not really learn any new words unless the entire old training corpus is used, but this time with the new words added; CLASPnet currently does not react to semantic anomalies in the surprised way it should, but parses them as if they were normal (versus the so-called N400 effect -- see Kutas 1993); and it is not self-evident how the model could be expanded to be able to analyze two clauses at once (e.g. when one is reading but also listening to spoken language) -- at the moment, the analyses of the two clauses would interfere in a catastrophic manner, resulting in a hybrid mixture of both.

• It is a black box!: Even if the current version of CLASPnet were able to parse real utterances spoken by real Welshmen, one could still argue that it would still be a collection of links and units, not an explanatory model. This is largely true, of course, but I have tried to show a few times that one can open the box and look at how the connections are wired. And by comparing one black box with others like it, as was done in the experiments, one can learn a lot about how it is functioning.

Note 1
Cf. the 'put up or shut up' challenge which MacWhinney & Leinbach (1991) have recently used successfully to evoke the creation of another, competitive model (Ling & Marinov 1993).