3.1.2 The Hidden Layers

Recall from Figure 3.1 above that CLASPnet has two separate layers of hidden units, with the first layer being subdivided in orthographic and semantic groups. I have three reasons for the choice of two layers of hidden units. First, and perhaps somewhat surprisingly, it keeps the total number of connections down: an input layer with 85 units which is fully interconnected with a hidden layer of 100 units has approximately 10% more connections than the present architecture, because the semantic and orthographic input representations, of 25 and 60 units respectively, are now compressed into much smaller representations of 15 and 30 units. (Note 6)

Second, having access to two hidden layers with recurrent layers of their own boosts the performance of the network in yet another way: the second hidden layer can profit from the regularities discovered at the first hidden layer and use this information to construct second-order regularities. The higher the order, the closer the network probably gets to the right level for detecting the properties of entire clauses and sentences. (To test this hypothesis, the experiment reported on in 3.4.1 has compared the performance of CLASPnet with that of a recurrent network with only a single hidden layer, but about the same amount of connections.)

Third, having two layers makes it possible to construct a modular neural network in which the semantic and orthographic representations are kept separate at first and only integrated later. Modular neural nets are not only more plausible from a neurophysiological point of view (Murre & Sturdy in press), they also tend to perform better than their non-modular cousins (Jacobs, Jordan & Barto 1991; Schyns 1991), because separate modules can learn different aspects of the task without interfering with one another (i.e. 'crosstalk') (Note 7). An additional advantage in the context of CLASPnet is that the separate semantic module makes it possible to fairly easily investigate which of the units in the semantic input representation are most important for the network, as these units have strong excitatory or inhibitory connections to the 15 units in the hidden layer (see 3.4.4). Finally, even from a theoretical linguistics perspective, the division between formal information (i.e. the orthographic representation) and conceptual information (i.e. the semantic representation) seems acceptable: like most frameworks, Cognitive Linguistics recognizes a signifiant and a signifié in each linguistic symbol (Langacker 1987). But unlike many of those frameworks, Cognitive Linguistics also posits that the two are actually integrated quite closely in human natural language processing -- a position which is reflected here in the integration of both types of information in the single large hidden layer.

Note 6
I have considered adding an extra hidden layer in between the large hidden layer and the output layer, but the computational gains from such an extra layer appear limited -- the output layer only has 17 units, so there is not much to be gained from compression.

Note 7
One of the most interesting possibilities offered by modular nets is to simply create a net with a large number of modules, train the network on a task, and then find out how this task was decomposed into subtasks by the net. Such an approach was, however, not possible in the case of CLASPnet: the software package used to build the networks does not offer the option of automatically creating modules. (The current net has been constructed by manually creating extra units and then adding all the required links one by one -- a method which is not particularly practical or pleasant, and which is prone to minor errors.)

Modular nets also make it easier to investigate the effects of lesioning certain groups of links and/or units in a trained neural network (see e.g. Hinton, Plaut & Shallice 1993).