Segment-Holistic Speech Recognition System Based
on Model of Brain Hemispheric Interaction in Speech Perception

O.I.Fedyayev 1), I.Yu.Bondarenko 2) 
1) Donetsk National Technical University, Ukraine, fedyaev@r5.dgtu.edu.ua 
2) Donetsk National Technical University, Ukraine, bond005@yandex.ru

Abstract: This paper is devoted to the development of two-channel segment-holistic system of medium-vocabulary speech recognition. Theoretical basis of the given system is the concept about functional asymmetry of cerebral hemispheres in human speech perception. The structure of automatic speech recognition system based on interaction of segment (phoneme-by-phoneme) and holistic (word-by-word) recognition subsystems is proposed. The segment subsystem is realized by means of neural networks and dynamic programming combination, and in the holistic subsystem the fuzzy matching of speech patterns is used.

Keywords: automatic speech recognition, neural networks, fuzzy pattern matching, collective recognition.

1. INTRODUTION 
There exist two approaches to the problem solution of automatic recognition of oral speech, and namely: information (pragmatic) and bionic.
The first approach includes a great number of methods which common feature is phenomenologic modeling of the process of speech recognition regardless the structure of a biologic prototype that is a human auditory system. One of the most popular methods within the frames of informational approach is the method of Dynamic Time Warping based on the principles of dynamic programming [1], and the method of Hidden Markov Models [2]. This method is applied in numerous modern systems of speech recognition. However supposition of the probability organization of human speech processes that is the ground of the method of Hidden Markov Models is not evident. Some experimental tests of recognition systems working accuracy where this method is applied do not provide better then 5% WER. Besides, for spontaneously pronounced texts the probability of the words correctly pronounced douse not exceed one third [3].
Another approach, a bionic one, is based on structural modeling of central nervous system regions dealing with analysis and perception of speech signal. However it should be noted that the modern level of biologic sciences development does not guarantee an comprehensive knowledge in the sphere of speech and hearing physiology. In the author’s opinion, the solution of the speech recognition problem within the frames of information and bionic approaches combination looks prospective ,when reproduction of the range of familiar structures and mechanisms of the brain activity for speech perception is combined with phenomenologic modeling of human auditory system elements that are not sufficiently investigated by physiological science.
This paper represents the description of speech patterns recognition system development based on the concept physiologists of functional asymmetry of human cerebral hemispheres. In the process of speech perception this symmetry is embodied in the fact that the brains operate like a two-channel segment-holistic system for oral speech recognition. The channels of segment and holistic recognition corresponding to the left and right hemispheres work in parallel providing high speed and reliability to functioning of the whole system [4]. This paper which is the further development of the work [5] proposes the first channel realization in the form of neuronet phoneme-by-phoneme recognizer, and the second in the form of a fuzzy classifier of integral patterns (words). Besides, the problem of collective decision making of final recognition of a speech pattern on the ground of integration of the channels working results is considered.

2. THE TWO-CHANNEL SYSTEM STRUCTURE OF AUTOMATIC SPEECH RECOGNITION 
The modern concept of the human speech activity mechanism forms the fundamentals for two-channel system of speech patterns recognition [4]. The block diagram of the two-channel recognition system is represented in Fig.1.
In the input sound signal the borders of speech frame as a supposed speech pattern are determined on the ground of the short-time energy and zero-crossing rate. Then the emphasized pattern is analyzed simultaneously by the segment and holistic channel. The segment channel is based on the method of neuronet phonemes approximation [6], and the holistic channel on the method of fuzzy pattern matching with optimal temporal alignment [7]. These channels form independent lists of candidate words to any of which a speech pattern under recognition could be related with a definite certainty factor.
At the last system level coordination of approximate decisions of segment and holistic channel is done using the lists of candidate words and corresponding certainty factors, with a final decision making on the speech pattern under recognition.

3. HOLISTIC RECOGNITION CHANNEL 
For the holistic recognition channel development the method of fuzzy pattern matching with optimal temporal alignment is proposed which is the further development of the fuzzy pattern matching method [8]. Application of optimal temporal alignment procedure for speech patterns normalization under matching along the time axis allows on the one hand, improve the recognition quality level (compared with linear alignment), and on the other hand, to reduce the calculations volume (compared with alignment based on dynamic programming).
As speech units the words the set of which determines 

the holistic channel vocabulary are considered. The speech signal is represented in the form of a binary flat spectral temporal pattern (BSTP) allowing to determine the resonance frequencies (local discharges) location which is the constitutive feature of the speech pattern [8].
BSTP of the speech signal is understood as a binary ration of F set (of f frequency numbers) to T set (of t time interval numbers). Using the average of teaching speech signals BSTP of the specified class a binary fuzzy relation typical for the given class is formed:

,	(1)	

where R is fuzzy relation which brings the value of  membership function to correspond each pair of elements . The set of fuzzy relations R = {r1, r2, … rn} determines the recognition vocabulary of n size.
For BSTP of input speech signal y the degrees of similarity Sj are determined with every fuzzy relation rj, and the recognition result for which , where

.	(2)	

Patterns normalization along the time axis before their matching is executed by means of the optimal temporal alignment algorithm. The problem of optimal temporal alignment is considered to be a non-linear optimization of an integer-valued argument function (the number of time intervals added to a lower length pattern) while the optimization criterion of the alignment is similarity degree maximization of patterns under matching [7].

4. SEGMENT RECOGNITION CHANNEL 
A segment approach to speech recognition is based on the phonetic analysis of speech signals. The method of neuronet phonemes approximation based on the similarity determination between the fragments of speech signal with each of the phonemes followed by the selection of the most authentic phonetic chain is applied [6]. The method allows, with some error, to determine whether the phoneme described with Fk(t) is the fragment of Aw(t) expression where Aw(t) is acoustic representation of the expression w; Fk(t) is acoustic representation of a phoneme k. For this purpose the function Fk(t) on [t0, t1] segment is represented in the form of pairs set:

,	(3)	

where , m=const; . The function Aw(t) is similarly represented in the form of pairs {X(t), Y(t)}.
Representation of Fk(t) in the form (3) allows to form a neuronet function NET: . Then the degree of difference Errk of the segment Aw(t) with  from Fk(t) is determined as follows:

.	(4)	

Thus, a new parametric description on an initial signal is formed:

,	(5)	

where Errk(t) is the degree of difference between the segment of a signal Aw(t) and k-th phoneme on the signal fragment of m duration.
A new parametric description of the initial signal possesses advantages connected with higher stability of description on stationery sectors as well as interpretability of the obtained values. But the complicated form and sufficient instability of the speech signal doesn’t allow to make a conclusion about the phoneme on the ground of instant values of Errk(t) difference degrees. That is why fecognition results were averaged during quite a long period of time. The derived parametric description of the signal is used for further context processing as it is represented in the segment channel recognition scheme (Fig.1).
The first level of the segment channel contains the set of neural networks each of which is taught to recognize separate phoneme. The networks outputs are interpreted as the next signal values forecasts provided the corresponding phoneme is available. At the second level the forecast error is accumulated on the whole length of the speech segment window. Integral error gets the third level where the best phonemes are selected from the whole number. The obtained set participates in phonetic formation being hypotheses of a pronounced word. The pronounced word is determined through the chain with the highest degree of authenticity. 

5. DECISIONS INTEGRATOR OF SEGMENT AND HOLISTIC CHANNEL 
At the final stage of the recognition system operation there appears the need to form a collective decision on the ground of work results integration of fuzzy and neuronet channels. For this purpose an integrator is introduced for a final decision formation in the event of uncertainty information. Taking into consideration inaccuracy of conclusion each of the channels the methods of collective recognition were applied for find decision [9]. One of the first approaches to decision making in the environment of uncertainty knowledge there was the one based on the Bayesian probability [10]. As a consequence of failure to provide correct probabilistic reasoning the method of certainty factors was proposed later [11] which became a classic example of uncertainty knowledge processing. There are other well-known decision making methods in the environment of uncertainty knowledge: Dempster-Schaffer method, methods of fuzzy logics, subjective Bayesian inference, neuronet methods, and voting method [9, 11].
An integrator block implements the method of fuzzy reasoning based on the certainty  factors proposed for MYCIN system [11]. The method was selected due to its simplicity and good capacity for compromise with consolidation of viewpoints by independent experts (recognition channels). The integrator block represented in Fig.2 produced forward chaining based on statements of recognition channels with CF (Certainty Factors) and rules of collective decision making by the team of independent experts.

Combined connection indicated as COMB [11] is of great importance for the issues of collective recognition with uncertainty data. In the independent manner it proves or refutes a proposed hypothesis related to a pronounced word on the ground of two or more opinions of the experts. So the knowledge which is introduced in the form of production rules (Fig.3) provides a combined connection between decisions proposed by recognition channels. Suppose, one of the experts of each channel has already determined the certainty factors X and Y as the result of preliminary recognition, and one  should make a conclusion (calculate the certainty factor of V conclusion) using rules from the knowledge base of the decision integrator. Forward chaining is based on the well-known method  application of rules antecedents evaluation and COMB connection consideration according to MYCIN method. The certainty factor covers an inference hierarchic system (Fig.4) formed by production rules.

In MYCIN method facts uncertainty is represented through CF – Certainty Factor accepting values from +1 (when the fact is knowingly true) to 0 (for knowingly false factors). The record of CF[V, Z] is treated as certainty factor for the validity of V consequent provided the antecedent Z is satisfied. In the process of inference with COMB connection CF[Vi, Xi] and CF[Vi, Yi] are calculated separately using the following formula: 

,	(6)	

where ; ;  is i-th expert’s certainty factor about j-th recognized word.
Consolidation of experts views on corresponding words of vocabulary is done using the following combined function

.	(7)	

Recognition result (the number of the recognized word) is formed as the number of maximum element in CF[Vi] array (Fig.4).

Figure 5 represents the results of collective recognition of the Russian word “markers” matching with other words of vocabulary. In the given example a segment (neurnet) channel evidently does not prefer any word, but taking into account the view of another expert – the holistic (fuzzy) channel – the integrator forms a collective decision that is more certain in identifying the input word as “markers”.

6. EXPERIMENTAL RESULTS 
The single-speaker and multi-speaker recognition experiments using the set of Russian words are described. The set including 105 commands of the text editor control, numerals from one to ten and ordinary words recorded by 8-bit samples in PCM format with 11,025 Hz frequency. 9 speakers, 6 men and 3 women, took part in the set composition each pronounced all words thrice (see Table 1).
Two of three deliveries of each word were used in single-speaker recognition for membership functions formation and one delivery for testing purposes.  Experiments were conducted for each of nine speakers individually the results of recognition having been averaged for all speakers.

For multi-speaker recognition two of three deliveries of each word by each speaker were used for training (18 speech deliveries total) for testing one speech delivery by each speaker (9 speech deliveries total).
Table 1. Speech database structure applied in experiments
	AGE	
GENDER	18 – 40	41 – 60	∑	
male	5	1	6	
female	2	1	3	
∑	7	2	9	
Each of 105 words was pronounced by each speaker 3 times.
Total: 2835 records of the isolated words and word-combinations.	
The single-speaker recognition results are shown in Table 2, and the multi-speaker recognition results are shown in Table 3.
Table 2. Results of single-speaker recognition experiments
Recognition system name	Rate of correctly recognized words, %	
Holistic recognition channel	99,05	
Segment recognition channel	98,73	
Segment-holistic recognition system	99,70	

Table 3. Results of multi-speaker recognition experiments
Recognition system name	Rate of correctly recognized words, %	
Holistic recognition channel	87,62	
Segment recognition channel	88,71	
Segment-holistic recognition system	91,43	

7. CONCLUSION
The general structure of segment-holistic system of speech patterns recognition based on physiologic understanding of functional cerebral asymmetry at speech perception is developed.
Possible realizations of segment and holistic channels are considered. Positive results were obtained due to implementation of the neuronet phonemes approximation method for recognition segment channel modeling, and the method of fuzzy pattern matching with optimal temporal alignment for recognition holistic channel modeling.
The method for decisions coordination of segment and holistic channels in two-channel model based on the method of certainty factors is proposed. The application of this method allowed to take into account a collective viewpoint of recognition channels as independent experts and make a more definite decision about the word under recognition. The consolidation of channels decisions is performed by the integrator represented by expert system making an inference on the ground of uncertainty knowledge. Experiments showed that the given method allows to settle effectively conflict situations arising from experts recognition channels opinions disagreement.

8. REFERENCES
[1]    Taras K. Vintsiuk. Generative Phoneme-Triphone Model for ASR. Proceedings of the International Conference “Text, Speech and Dialogue”, Zelena Ruda, Czech Respublic, 2001, pp.201-207.
[2]    Xuedong Huang. Spoken language processing: a guide to theory, algorithm, and system development. Prentice-Hall. 2001. p. 504.
[3]    V.I. Galunov, A.N.Soloviev. Black patches in the field of automatic speech recognition. Proceedings of the 15th Session of Russian Acoustical Socienty, Nizhny Novgorod, Russia, 2004.
[4]    V.I. Galunov, A.N.Soloviev, V.K.Uvarov. Models of speech perceptions, speech production and problem of automatic speech recognition. Proceedings of the 9th International Conference “Speech and Computer (SPECOM’2004)”, St.Peterburg, Russia, 2004.
[5]    Oleg. I. Fedyaev, Ivan Yu. Bondarenko. Segmentally-integral model for spoken pattern recognition based on functional brain asymmetry conceptions. Proceedings of the 9th Ukrainian Conference on Signal/Image Processing and Pattern Recognition “UkrObraz’2008”, Kyjiv, Ukraine, 2008, pp.80-84. [in Ukrainian]
[6]    Oleg. I. Fedyaev, Sergey A. Gladunov. Organizing a speech input of information based on neural phonemes approximation. Proceedings of the International Conference “Interactive Systems: Problems of Human-Computer Interaction”, Ulyanovsk, Russia, 2003, pp.198-206.
[7]    Oleg. I. Fedyaev, Ivan Yu. Bondarenko. Isolated Soeech Word Recognition Based on Fuzzy Pattern Matching with Optimal Temporal Alignment. Proceedings the 13th International Conference “Speech and Computer (SPECOM’2009)”, St.Peterburg, Russia, 2009.
[8]    K.Asai, D.Vatada, S.Ivai, and others. Speech recognition, in Applied Fuzzy Systems. Mir. Moscow, Russia, 1993. pp. 157-170. [translated into Russian]
[9]    V.Gorodetsky, S.Serebryakov. Methods and algorithms of collective recognition. Automation and Remote Control 11 (2008). pp. 3-40. [in Russian]
[10]    Gary D. Riley, Jospeph C. Giarratano. Expert Systems: Principles and Programming (4th edition). Course Technology. 2004. p. 624.
[11]    George F. Luger. Artificial Intelligence: Structures and Strategies for Complex Problem Solving (5th edition).
Addison Wesley. 2004. p. 928.