3.1.3 The Output Layer

The output layer of CLASPnet has 17 units, which, using a localist representation, represent properties about the sentence and the clause which the network is processing at any given moment. Every time a new word is presented to the input units, the activation values of the output units are updated to reflect this change. In this manner, one can 'see' the analysis which the net gives to a sentence or clause evolve word per word. Sequential processing of this kind is naturally interesting from a psycholinguistic point of view, as one can compare results of experiments with humans with those with neural networks. (Whether a close fit between the two is of any scientific relevance, however, depends on a lot of factors -- e.g. whether any time has been spent on getting the network results to correspond with the human data.)

Before I discuss the individual output units, I briefly want to motivate my choice to focus on clausal properties. Hudson's (1984) work on Word Grammar proves that it is not entirely obvious to do so: in Word Grammar, sentences, clauses, and even nominal or verbal phrases are abandoned in favor of a 'companion' relationship between words, and the dependency relation between heads and their modifiers. Still, even Hudson (1984: 81) mentions in passing that there is a language, Kalkatungu, in which words can occur just about anywhere in a clause, but they never cross clause boundaries. Similar data have been reported by Taylor (1995: 180) for the Zulu clitic 'ke': it is used for topicalization of various linguistic structures, but only one can appear in each clause. Developmental evidence for accepting clauses as useful linguistic entities is given in Slobin (1985: 222f.): he reports that, cross-linguistically, children prefer separate negation morphemes which are put just outside of the normal clause, and without the words in this clause being reordered. And Langacker (1990b: 212) has written: "The finite clause is a pivotal unit of grammatical organization". Taken together, all this seems to make a strong case for the existence of the clause as a mental entity of some sorts: the words in a clause can be thought of constructing a single conceptual scene, in which -- at least typically -- a single action or state is taking place. At the center of each clause then is a verb phrase which -- at least partly -- describes this action or state. Or, turning the tables, each verb phrase indicates the presence of a separate clause. It is this last rule which I have followed for the definition of clauses in CLASPnet. (The definition of a sentence is even less controversial: all the words before a full stop, question mark, or exclamation mark.)

Figure 3.2 summarizes the interpretation of the output units of CLASPnet:

• The first three units are used to indicate the mood of the entire sentence (Note 8):
  • Indicative: for sentences which end in a full stop;
  • Imperative: for sentences which end in an exclamation mark;
  • Interrogative: for sentences which end in a question mark.

• The next three units indicate the number of the clause in the sentence through a binary pattern: 100 is the first clause; 010 the second; 110 the third; 001 the fourth; etc. Although highly abstract in character and not very relevant to the task -- a relative clause is a relative clause, whether it is the first or the fifth clause in the sentence --, these units have been introduced because the traditional linguistic distinction between the matrix clause and any subordinate clauses does not make it possible to distinguish between different subordinate clauses in the same sentence. Especially when processing sentences containing embedded clauses, it is very useful to be able to see precisely which clause the network thinks the word it is currently processing belongs to.

• The seventh unit represents the traditional linguistic notions of matrix clause versus subordinate clause. It is mainly useful as a backup indicator when the network has lost count of the number of the clause.

• The next seven units each stand for a different clause type. The classification used for CLASPnet is to some degree idiosyncratic, as it has been developed for the current project. (However, the classification has also never been changed or optimized in any way in order to improve the performance of the network. My aim has been to find out whether the network could learn a classification which I deem acceptable on linguistic grounds, not to find out which classification the network finds easiest to learn -- though that would be an interesting project in itself.) One approach to categorizing clause types is to give each syntactical construction a separate unit. But even if it would always be easy to determine the exact syntactical pattern, this solution would still be unsatisfactory, because it fails to acknowledge that constructions carry meaning (cf. Goldberg 1995), and that different constructions can carry a very similar meaning. The classification used here tries to combine elements from both the formal and the semantic sides, and also looks at the function of the clause with respect to other phrases and clauses in the same sentence (the details about the analysis can be found in the grammar and the parser -- see Appendices A.1 and A.3):
  • Declarative clause: only used for matrix-clause indicative statements, as this clause type is conceptually independent and has a purely informative function (e.g. 'the happy men chase the sad tigers.', 'this man be a good president for the united kingdom.') (Note 9);
  • Order: used both for true imperatives (e.g. 'be happy!', 'sleep!') and for the complements of verbs of ordering (e.g. 'they ordered them not to paint on the boat.';
  • WH-Question: used both for true WH-Questions (e.g. 'which falcons can the pilots chase?', 'who is too tired to sleep?') and for some complements of verbs of asking (e.g. 'the doctors ask when they eat.'
  • YN-Question: used both for true Yes/No-Questions (e.g. 'will the group of teachers never cry?', 'can he explain to the doctor when he work?') and for some complements of verbs of asking (e.g. 'the girls inquire whether the doctor will be sad.');
  • Relative clause: used for full and compressed relative clauses (e.g. 'the boy, who be painting on the barn, drink.', 'the doctors, being asked when they eat, smile.');
  • Connector clause: this type is used for all clauses which begin with a real connector ('the eagles chase the falcons, because they be angry.'), and for some other types which also begin with a special word indicating that this clause can only be interpreted in combination with another clause (e.g. 'the more they drink, the less they sleep.', 'in order to love her, he hate her.', 'the wolf can not wash the tiger, however hard he tease her.');
  • Complement clause: used for complements of verbs of saying (e.g. 'she told the nurse that the postmen were sad.', 'can you tell them to sleep in the boat?').

• The fifteenth unit represents infinity and is only active for clauses built round non-finite verbs (i.e. infinitives and -'ing' forms).

• The penultimate unit represents voice. It has an activation value of 1 in active clauses, and 0 in passive clauses.

• The final unit codes the polarity of the clause, and is 1 for positive clauses, and 0 for negative clauses (i.e. clauses containing 'no', 'not', 'never', 'the fewer', or 'the less').

  While the combination of the 17 output units certainly does not express everything that could to be said about any English clause -- aspect, tense, and pragmatic functions come to mind -- it does provide a lot of information about such a clause. Enough so, I feel, that a recurrent neural network like CLASPnet could be expected to have a hard time coming up with the desired values on the output units. Before I discuss the actual results of the simulations, however, it is useful to take a closer look at how the training and testing corpora were created.

  
  

  Note 8
  There are of course many more moods than the three used here. Bybee (1985: 23) gives the following list: "imperative, indicative, negative, probable, interrogative, optative, conjunctive, conditional, and dubitative" (Pawnee has eight!), and Hudson (1984) also mentions the exclamation as a separate mood. Implementing more moods in CLASPnet would be easy because of the type of grammar used to generate sentences and the type of parser used to analyze them (see 3.2).

  Note 9
  The 'be' in the latter sentence is not a mistake -- see 3.2.2 below for more details.