=========================================================== Documentation for OpinionFinder 1.4 Annotations on Penn Treebank III =========================================================== Contents: 1. Introduction 2. Extracting the Annotations 3. Annotations and their Format A. Source Annotations B. Direct Subjective Expression and Speech Event Annotations C. Polarity Annotations D. Sentence Subjectivity Annotations 4. References ----------------------------------------------------------- 1. Introduction This package includes automatic OpinionFinder annotations that have been generated for the Penn Treebank III - Wall Street Journal sections 00 through 24. Note that this package includes the annotations ONLY. Using the OpinionFinder annotations requires that you install the Penn Treebank III on your system and then run a program to merge the annotations with the Penn Treebank. The following annotations were produced by OpinionFinder: a) sources b) direct subjective expressions and speech events c) polarity d) sentence subjectivity (both from the high accuracy classifier and the high precision classifier) The format used for OpinionFinder annotations on the Penn Treebank is the same as the Start-End format used for the CoNLL-2005 shared task. ----------------------------------------------------------- 2. Extracting the Annotations To retrieve the annotations for the Penn Treebank III, do the following: 1) Edit the doclist file merged.doclist, replacing the "/YOURPATHTOPTB3WSJ/" with the WSJ directory that contains the section folders 00 through 24, each of which have .mrg files. For example, if your path is "/ptb_v3/parsed/mrg/wsj/" replace the line /YOURPATHTOPTB3WSJ/00/wsj_0001.mrg with /ptb_v3/parsed/mrg/wsj/00/wsj_0001.mrg 2) Run the program ptbOpinionFinderAnnotations.py to produce the annotations. The program should be called as follows: python ptbOpinionFinderAnnotations.py -f -o - the full path to the directory of where you want the output to be put. For example, if you have the merged doclist "/annotations/merged.doclist" and want to output the annotations to the folder "/annotations/output", you would execute the following command: python ptbOpinionFinderAnnotations.py -f /annotations/merged.doclist -o /annotations/output ----------------------------------------------------------- 3. Annotations and their Format Here is an example of an annotated sentence: But * * * (A[subj,8.9]* (P[unk]* with * * * * * the * * * * * end * * * * * of * * * * * the * * * * * year * * * * * in * * * * * sight * * * * * , * * * * * money (S* * * * * managers *) * * * * are * * * * * eager * (D*) (P[pos]*) * * to * * * * * take * * * * * profits * * * * * and * * * * * cut * * * * * their * * * * * risks * * * * * of * * * * * losing * * (P[neg]*) * * what * * * * * for * * * * * many * * * * * have * * * * * been * * * * * exceptionally * * (P[pos]*) * * good * * (P[pos]*) * * returns * * * * * in * * * *) *) Each token of the sentence is on a separate line, and sentences are separated by spaces. The tokens shown are the same as those in the Penn Treebank, with the following exceptions: - the tokens -LRB-, -RRB-, -LCB-, -RCB-, -LSB-, and -RSB- have been replaced with the brackets they represent - tokens labeled as -NONE- are not included Each column is separated by whitespace and shows the OpinionFinder annotations in the following order: a) sources b) direct subjective expressions and speech events c) polarity d) sentence subjectivity (high accuracy classifier) e) sentence subjectivity (high precision classifier) The Start-End format used for the OpinionFinder annotations is the same as the format used for the CoNLL-2005 shared task. Each annotation is of the form START*END, where the word at the START of an annotation is marked by a parenthesis followed the annotation type and possible attributes, e.g., "(A[subj,8.9]", and the last word in the annotation is marked by a closing parenthesis ")". === A. Source Annotations SourceFinder identifies the sources of speech events and private states (i.e., opinions, emotions, and speculations), as described in Choi et al. (2005). That is, it identifies speakers of speech events and writers of writing events. For opinions, emotions, and so on, it identifies the person who holds the opinion, who experiences the emotion, and so on. OpinionFinder uses the extraction pattern source identifier from Choi et al. (2005). Source annotations are in the first column of annotations, labeled with an "S" as shown in the example sentence above. === B. Direct Subjective Expression and Speech Event (DSESE) Annotations The DSESE identifier uses a maximum entropy model to identify direct subjective expressions (e.g., "fears," "is happy") and speech events (e.g., "said," "according to"). Direct subjective expressions are words or phrases where an opinion, emotion, sentiment, etc. is directly described. Speech events include both speaking and writing events. The DSESEs in the sentences below are "said," "hate," "thought," and "hoped." (1) Jill said, "I hate Bill." (2) John thought he won the race. (3) Mary hoped her presentation would go well. The DSESE identifier is similar to the opinion expression classifier used in Choi et al. (2006), but it uses a simpler feature set. In addition to features based on words and their contexts, the model includes features based on verbs from Levin's verb classes (1993), verbs and adjectives from FrameNet (Baker et al. 1998) with frame "experiencer," and information about synsets and hypernyms from WordNet 1.6 (ed., Fellbaum 1998). The DSESE model was trained on 437 documents from the MPQA opinion corpus and evaluated on the remaining 98 documents, yielding an 82% precision and 79% recall for identifying DSESEs. The DSESE identifier uses PyWordNet to access WordNet. In the output, DSESEs are given in the second column of annotations, labeled with a "D". === C. Polarity Annotations The polarity classifier takes clues consisting of words with a prior polarity of "positive", "negative", "neutral" or "both" (for example, "love", "hate", "think", and "brag", respectively) and then uses a modified version of the classifier described in Wilson et al. (2005) to determine the contextual polarity of the clues. Heuristics were used to improve the speed of the classifier so it no longer needs the dependency parse output. The contextual polarity of the clues is then written to files in the auto_anns directory in MPQA format. When evaluated on the MPQA opinion corpus, the overall accuracy is 74.3%. The clues used by the polarity classifier come from several sources: 1) Subjectivity clues from Riloff and Wiebe (2003). 2) General Inquirer Dictionary - those words that are marked with category "Positiv" and "Negativ" (http://www.wjh.harvard.edu/~inquirer/homecat.htm) Part-of-speech is as marked in the dictionary; duplicates have been removed, and clues in these lists were manually filtered in order to remove those clues judged to be highly ambiguous or not positive or negative. 3) Adjectives that were manually identified to have positive or minus (negative) polarity from Vasileios Hatzivassiloglou's dissertation work and for his paper with McKeown in ACL-97. Clues in these lists were manually filtered in order to remove those clues judged to be highly ambiguous or not positive or negative. 4) Other words added from dictionaries and thesauri. Polarity expressions are marked in the third column of annotations with the "P[pol]" tag, where pol can be "pos" (positive), "neg" (negative), or "both." === D. Sentence Subjectivity Annotations There are two subjectivity classifiers included with OpinionFinder for identifying subjective and objective sentences (Riloff and Wiebe, 2004; Wiebe and Riloff, 2005). The first classifier tags each sentence as either subjective or objective. This classifier uses the strategy that yields our highest overall accuracy. By "accuracy," we simply mean the percentage of the answers (according to manual annotations) that the system gets right. Evaluated on a 9732 sentences from the MPQA Opinion Corpus, this classifier has an accuracy of 74%, subjective precision of 78.4%, subjective recall of 73.2%, and subjective F-measure of 75.7%. The baseline accuracy is 55.3%. Subjective annotations for the high accuracy classifier are marked in the next to last column with the label "A[class,diff]" where class can be "subj" (subjective) or "obj" (objective) and diff is a real number indicating the classifier's confidence. The second classifier optimizes precision at the expense of recall. That is, it classifies a sentence as subjective or objective only if it can do so with confidence. Otherwise, it labels the sentence as "unknown." Evaluated on the same 9732 sentences from the MPQA Corpus (4352 objective and 5380 subjective sentences), this strategy yields about 91.7% subjective precision (91.7% of the sentences the system classifies as subjective are indeed subjective, according to the manual annotations) and 30.9% subjective recall (of all the subjective sentences, 30.9% are automatically classified as subjective, rather than objective, or unknown). Objective precision is 83.0% and objective recall is 32.8%). Subjective annotations for the high precision classifier are given in the last column of the output, labeled with "P[class]" where class can be "subj" (subjective), "obj" (objective), or "unk" (unknown). The clues used by the subjectivity classifiers include ones identified from following sources: 1) "Speech activity" verbs from pages 71--167 of a book by Th. Ballmer and W. Brennenstuhl (1981). Bruce Fraser identified the book as a good source for speech events, Dee DeLorenzo typed all of the verbs in so that an electronic version could be created. Paul Davis did processing to put it in the MPQA file format. Our group made additional changes. 2) Psychological verbs and verbs from some other verb classes from Beth Levin's "English Verb Classes and Alternations" (1993). 3) FrameNet verbs and adjectives with frame "experiencer" (Baker et. al, 1998). FrameNet can be found on the web at: http://www.icsi.berkeley.edu/~framenet/ 4) Clues taken from Janyce Wiebe's dissertation (1990) and other papers. For more information of the sentence subjectivity classifiers, please see Riloff and Wiebe (2004) and Wiebe and Riloff (2005). ----------------------------------------------------------- 4. References CoNLL 2005 shared task: http://www.lsi.upc.es/~srlconll/ Penn TreeBank Project: http://www.cis.upenn.edu/~treebank Collin F. Baker, Charles J. Fillmore, and John B. Lowe (1998). The Berkeley FrameNet project. In Proceedings of COLING/ACL. Pages 86-90. Thomas Ballmer and Waltraud Brennenstuhl "Speech Act Classification: A Study in the Lexical Analysis of English Speech Activity Verbs." Springer-Verlag, 1981. Yejin Choi, Eric Breck, and Claire Cardie (2006). Joint Extraction of Entities and Relations for Opinion Recognition. Conference on Empirical Methods in Natural Language Processiong (EMNLP-2006). Yejin Choi, Claire Cardie, Ellen Riloff, and Siddharth Patwardhan (2005). Identifying Sources of Opinions with Conditional Random Fields and Extraction Patterns. In the Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, Vancouver, Canada. Vasileios Hatzivassiloglou, Kathleen McKeown: Predicting the Semantic Orientation of Adjectives. ACL 1997: 174-181. Beth Levin. English Verb Classes and Alternations: A Preliminary Investigation. University of Chicago Press, Chicago, 1993. Ellen Riloff and Janyce Wiebe (2003). Learning Extraction Patterns for Subjective Expressions. Conference on Empirical Methods in Natural Language Processing (EMNLP-03). ACL SIGDAT. Pages 105-112. Philip J. Stone, Dexter C. Dunphy, Marshall S. Smith, Daniel M. Ogilvie, and associates. "The General Inquirer: A Computer Approach to Content Analysis," The MIT Press, 1966. Janyce Wiebe (2002). Instructions for Annotating Opinions in Newspaper Articles. Department of Computer Science Technical Report TR-02-101, University of Pittsburgh, Pittsburgh, PA. Janyce Wiebe (1990). Recognizing Subjective Sentences: A Computational Investigation of Narrative Text. (Ph.D. dissertation) Technical Report 90-03. (Buffalo: SUNY Buffalo Dept. of Computer Science). Janyce Wiebe and Ellen Riloff (2005). Creating subjective and objective sentence classifiers from unannotated texts. Sixth International Conference on Intelligent Text Processing and Computational Linguistics (CICLing-2005). Janyce Wiebe, Theresa Wilson, and Claire Cardie (2005). Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, volume 39, issue 2-3, pp. 165-210. Theresa Wilson, Janyce Wiebe and Paul Hoffmann (2005). Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. Proceedings of Human Language Technologies Conference/Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP 2005), Vancouver, Canada. WordNet: An electronic lexical database (1998). Christiane Fellbaum, editor. Cambridge: MIT Press.