+ Chunker Experiments

Table of Contents

Description

These Experiments where done to explore the influence of chunking information on the cdg parser.

Chunker Evaluation

As has already been reported at TreeChunkerReport, the TreeTagger is not doing so well for our corpus as it has been reported elsewhere. The evaluation of all 1894 sentences of our corpus is shown in the following table:

recall precision without fault anticipation 87.47% 82.32% with fault anticipation 87.10% 82.93%

Note that the TreeTagger is 5% chunkier than the annotation, that is splits chunks where we would like to have them as one chunk. Note further that the used constraint is not so affected by having more chunks but by having the wrong stuff chunked together.

Grammars

The following grammar versions have been tested:

• heise grammar v0.6: 24, October 2002, without fault anticipation)
• heise grammar v0.7: 1, November 2002, with fault anticipation)
• heise grammar v0.7: 1, November 2002, without fault anticipation

The published results should all be derived from one grammar version, i.e. v0.7 .

Aspects

This is the list of aspects which have been combined:

1. parser flavour
   • frobbing dynamic: Spliting sentences into subproblems allong punctation boundaries. We decided to only publish results using frobbin dynamic as this gives us best numbers right now, and cuts down the evaluation computation.
2. partial parsers (tagger, chunker)
   • using no tagging information
   • using a single tagger
   • using a multi tagger
   • using the ideal tagger, cheating at the POS-tags of the annotation
   • using the ideal chunker: Chunking information was derived from the annotations
   • using a real chunker: Use chunking information from the TreeTagger
   • using a real chunker augmented with a fault anticipation: Find out reasonable rules to fix most obvious errors which is done by the help of a perl wrapper script and by encoding exceptions into the chunker constraints
3. chunk constraint penalization
   • hard integration of chunking information: chunk constraint with penalty 0.0
   • soft integration of chunking information: chunk constraint with penalty 0.1 to 0.9
4. time limit
   • 3 minutes have been chosen on a quite adhoc decision
   • christmas configuration: for a closing comparison we evaluate the "all nodes on" configuration, that is using the best setting that will show up plus a 6 minutes timelimit

The Results

All results measure the labelled and lexical recall of the SYN level. See TermsAndConventions for an explanation of the evaluation measures.

Problem size

The Experiment have been tests on 1845 of 1894 sentences of the annotated heise corpus with sentence lenghts as shown below. The table below shows the labelled recall for the specified problem sizes for the best configurations.

Nr Words	Nr Sentences	no tagger	single tagger	multi tagger	ideal chunker 0.0	real chunker 0.4	augmented chunker 0.0
03-10	160	65.57%	85.55%	86.13%	90.40%	87.39%	86.71%
11-20	650	54.29%	76.82%	77.59%	81.47%	79.43%	78.80%
21-30	651	49.78%	72.18%	74.69%	78.98%	76.58%	70.63%
31-40	274	48.70%	69.23%	70.43%	75.70%	72.44%	72.81%
41-50	81	48.06%	67.10%	68.50%	73.81%	70.56%	71.06%
51-60	19	43.83%	63.25%	64.31%	70.91%	66.60%	67.18%
>60	9	47.80%	28.73%	59.68%	66.28%	64.22%	63.78%

Evaluation of all 1845 Sentences

No	Partial Parser	Penalty	Anticipation	Struct.Recall	Lab.Recall	Lex.Recall	Time	Limit	Grammar
1	no tagger	-	-	58.24%	50.71%	40.44%	2 d 20 h 36 m 25 s 240 ms	3 m	v0.7
2	single tagger	-	-	75.73%	71.07%	65.01%	1 d 8 m 43 s 880 ms	3 m	v0.7
3	multi tagger	-	-	78.24%	73.71%	67.92%	1 d 6 h 14 m 9 s 310 ms	3 m	v0.7
4	ideal tagger	-	-	80.22%	75.76%	70.34%	21 h 14 m 42 s 930 ms	3 m	v0.7
5	ideal tagger & chunker	-	-	83.87%	79.74%	74.08%	15 h 23 m 16 s 910 ms	3 m	v0.7
6	ideal chunker	0.0	no	82.65%	78.30%	72.25%	22 h 54 m 31 s 350 ms	3 m	v0.7
7	ideal chunker	0.1	no	82.17%	77.81%	71.80%	1 d 1 h 29 m 38 s 910 ms	3 m	v0.7
8	ideal chunker	0.2	no	82.07%	77.77%	71.81%	1 d 2 h 5 m 630 ms	3 m	v0.7
9	ideal chunker	0.3	no	81.70%	77.38%	71.43%	1 d 1 h 56 m 2 s 520 ms	3 m	v0.7
10	ideal chunker	0.4	no	81.58%	77.25%	71.33%	1 d 1 h 54 m 41 s 430 ms	3 m	v0.7
11	ideal chunker	0.5	no	81.48%	77.17%	71.25%	1 d 2 h 13 m 49 s 220 ms	3 m	v0.7
12	ideal chunker	0.6	no	81.29%	76.94%	71.01%	1 d 2 h 11 m 38 s 450 ms	3 m	v0.7
13	ideal chunker	0.7	no	81.32%	76.98%	71.06%	1 d 2 h 47 m 56 s 440 ms	3 m	v0.7
14	ideal chunker	0.8	no	81.34%	77.03%	71.10%	1 d 3 h 2 m 1 s 860 ms	3 m	v0.7
15	ideal chunker	0.9	no	81.26%	76.95	70.98%	1 d 3 h 45 m 11 s 570 ms	3 m	v0.7
16	ideal chunker	0.0	yes		77.74%	71.76%	23 h 29 m 39 s 100 ms	3 m	v0.7
17	ideal chunker	0.1	yes		77.27%	71.27%	1 d 2 h 7 m 24 s 570 ms	3 m	v0.7
18	ideal chunker	0.2	yes		76.95%	71.02%	1 d 2 h 15 m 24 s 890 ms	3 m	v0.7
19	ideal chunker	0.3	yes		76.81%	70.92%	1 d 2 h 15 m 490 ms	3 m	v0.7
20	ideal chunker	0.4	yes		76.75%	70.86%	1 d 2 h 33 m 10 s 480 ms	3 m	v0.7
21	ideal chunker	0.5	yes		76.56%	70.65%	1 d 2 h 28 m 5 s 420 ms	3 m	v0.7
22	ideal chunker	0.6	yes		76.45%	70.57%	1 d 3 h 4 m 58 s 320 ms	3 m	v0.7
23	ideal chunker	0.7	yes		76.48%	70.64%	1 d 3 h 24 m 47 s 740 ms	3 m	v0.7
24	ideal chunker	0.8	yes		76.29%	70.40%	1 d 3 h 12 m 13 s 990 ms	3 m	v0.7
25	ideal chunker	0.9	yes		76.36%	70.47%	1 d 4 h 2 m 43 s 560 ms	3 m	v0.7
26	real chunker	0.0	no	79.71%	75.29%	69.34%	1 d 46 m 38 s 960 ms	3 m	v0.7
27	real chunker	0.1	no	79.77%	75.34%	69.33%	1 d 2 h 38 m 44 s 350 ms	3 m	v0.7
28	real chunker	0.2	no	79.92%	75.50%	69.60%	1 d 3 h 20 m 11 s 420 ms	3 m	v0.7
29	real chunker	0.3	no	79.99%	75.64%	69.68%	1 d 3 h 5 m 45 s 940 ms	3 m	v0.7
30	real chunker	0.4	no	80.03%	75.67%	69.72%	1 d 3 h 2 m 38 s 390 ms	3 m	v0.7
31	real chunker	0.5	no	79.94%	75.52%	69.60%	1 d 3 h 11 m 24 s 230 ms	3 m	v0.7
32	real chunker	0.6	no	79.90%	75.50%	69.56%	1 d 3 h 34 m 49 s 30 ms	3 m	v0.7
33	real chunker	0.7	no	79.80%	75.43%	69.48%	1 d 3 h 40 m 14 s 630 ms	3 m	v0.7
34	real chunker	0.8	no	79.79%	75.42%	69.46%	1 d 3 h 40 m 31 s 300 ms	3 m	v0.7
35	real chunker	0.9	no	79.73%	75.32%	69.42%	1 d 4 h 7 m 26 s 950 ms	3 m	v0.7
36	real chunker	0.0	yes	79.90%	75.51%	69.61%	1 d 49 m 24 s 410 ms	3 m	v0.7
37	real chunker	0.1	yes	79.68%	75.24%	69.28%	1 d 2 h 44 m 8 s 60 ms	3 m	v0.7
38	real chunker	0.2	yes	79.63%	75.22%	69.37%	1 d 3 h 13 m 6 s 980 ms	3 m	v0.7
39	real chunker	0.3	yes	79.76%	75.35%	69.50%	1 d 2 h 55 m 54 s 520 ms	3 m	v0.7
40	real chunker	0.4	yes	79.76%	75.35%	69.47%	1 d 3 h 18 m 200 ms	3 m	v0.7
41	real chunker	0.5	yes	79.57%	75.16%	69.30%	1 d 3 h 14 m 39 s 370 ms	3 m	v0.7
42	real chunker	0.6	yes	79.52%	75.09%	69.23%	1 d 3 h 47 m 43 s 580 ms	3 m	v0.7
43	real chunker	0.7	yes	79.38%	74.95%	69.11%	1 d 3 h 42 m 56 s 280 ms	3 m	v0.7
44	real chunker	0.8	yes	79.40%	75.00%	69.17%	1 d 4 h 8 m 6 s 550 ms	3 m	v0.7
45	real chunker	0.9	yes	79.49%	75.13%	69.32%	1 d 4 h 6 m 58 s 780 ms	3 m	v0.7
46	ideal chunker	0.0	no	82.98%	78.68%	72.70%	1 d 4 h 14 m 47 s 680 ms	6 m	v0.7
47	real chunker	0.4	no	80.57%	76.23%	70.36%	1 d 10 h 31 m 11 s 580 ms	6 m	v0.7

Related Topics: EaclPaper2003, TreeChunkerReport

-- MichaelDaum - 31 Oct 2002