A Corpus-based Approach to the Chinese Word Segmentation

For a society based upon laws and reason, it has become too easy for us to believe

that we live in a world without them. And given that our linguistics wisdom was

originally motivated by the search for rules, it seems strange that we now consider

these rules to be the exceptions and take exceptions as the norm.

The current task of contemporary computational linguistics is to describe these

exceptions. In particular, it suffices for most language processing needs, to just

describe the argument and predicate within an elementary sentence, under the

framework of local grammar. Therefore, a corpus-based approach to the Chinese

Word Segmentation problem is proposed, as the first step towards a local grammar

for the Chinese language.

The two main issues with existing lexicon-based approaches are (a) the classification

of unknown character sequences, i.e. sequences that are not listed in

the lexicon, and (b) the disambiguation of situations where two candidate words

overlap.

For (a), we propose an automatic method of enriching the lexicon by comparing

candidate sequences to occurrences of the same strings in a manually segmented

reference corpus, and using methods of machine learning to select the optimal

segmentation for them. These methods are developed in the course of the thesis

specifically for this task. The possibility of applying these machine learning

method will be discussed in NP-extraction and alignment domain.

(b) is approached by designing a general processing framework for Chinese text,

which will be called multi-level processing. Under this framework, sentences are

recursively split into fragments, according to a language-specific, but domainindependent

heuristics. The resulting fragments then define the ultimate boundaries

between candidate words and therefore resolve any segmentation ambiguity

caused by overlapping sequences. A new shallow semantical annotation is also

proposed under the frame work of multi-level processing.

A word segmentation algorithm based on these principles has been implemented

and tested; results of the evaluation are given and compared to the performance of

previous approaches as reported in the literature.

The first chapter of this thesis discusses the goals of segmentation and introduces

some background concepts. The second chapter analyses the current state-of-theart

approach to Chinese language segmentation. Chapter 3 proposes a new corpusbased

approach to the identification of unknown words. In chapter 4, a new shallow

semantical annotation is also proposed under the framework of multi-level

processing.

Weitere Episoden von „Fakultät für Sprach- und Literaturwissenschaften - Digitale Hochschulschriften der LMU“