Monday, July 24, 2006
Some basic queries about RTE 2 Dataset
loading RTE Dataset...
avg. number of tokens in text / hypothesis:
entailment: | NO YES all
task: |
--------------+--------------------------------------------------
SUM | 39.32 / 27.25 38.47 / 18.62 38.90 / 22.93
QA | 37.53 / 11.99 32.89 / 11.66 35.21 / 11.82
IR | 33.92 / 11.52 35.06 / 11.17 34.49 / 11.35
IE | 45.85 / 12.02 40.23 / 11.95 43.04 / 11.98
all | 39.16 / 15.70 36.66 / 13.35 37.91 / 14.52
avg. number of sentences in text / hypothesis:
entailment: | NO YES all
task: |
--------------+--------------------------------------------------
SUM | 1.02 / 1.00 1.00 / 1.00 1.01 / 1.00
QA | 1.23 / 1.00 1.09 / 1.00 1.16 / 1.00
IR | 1.05 / 1.00 1.08 / 1.00 1.06 / 1.00
IE | 1.06 / 1.01 1.05 / 1.00 1.05 / 1.00
all | 1.09 / 1.00 1.05 / 1.00 1.07 / 1.00
avg. number of tokens in a sentence of text / hypothesis:
entailment: | NO YES all
task: |
--------------+--------------------------------------------------
SUM | 38.55 / 27.25 38.47 / 18.62 38.90 / 22.93
QA | 30.51 / 11.99 30.17 / 11.66 35.21 / 11.82
IR | 32.30 / 11.52 32.46 / 11.17 34.49 / 11.35
IE | 43.25 / 11.90 38.31 / 11.95 40.80 / 11.93
all | 35.92 / 15.66 34.75 / 13.35 35.35 / 14.50
avg. number of exactly matching words in a pair:
entailment: | NO YES all
task: |
--------------+--------------------------------------
SUM | 8.57 9.57 9.07
QA | 6.70 7.10 6.90
IR | 3.95 4.53 4.24
IE | 6.59 6.21 6.40
all | 6.46 6.86 6.66
avg. standard deviation in the alignment of the exacly matching words in a pair (T-H):
EXAMPLE 1:
Text: u1 u2 x y z u6 u7 u8 u9 u10 u11 u12 u13 u 14 u15 u16
| | |
| | |
Hypothesis: v1 v2 v3 v4 x y z v8
[Perfectly aligned: StdDev = 0]
EXAMPLE 2:
Text: u1 u2 x u4 u5 u6 u7 u8 y u10 u11 u12 u13 z u15 u16
| | |
| | |
Hypothesis: v1 z v3 y v5 v6 x v8
[Not a good alignment:
number of shift for x = s_x = +4
number of shift for y = s_y = -5
number of shift for z = s_z = -12
number of exact matches = n = 3
mean shift = m = (4 - 5 - 12) / 3 = -7
stddev = sqrt (((m - s_x)^2 + (m - s_y)^2 + (m - s_z)^2) / n)
stddev = sqrt (((4 - 7)^2 + (-5 - 7)^2 + (-12 - 7)^2) / 3)
stddev = 13.08]
entailment: | NO YES all
task: |
--------------+-------------------------------------
SUM | 6.99 5.46 6.23
QA | 6.21 5.02 5.62
IR | 3.88 4.31 4.09
IE | 7.47 5.85 6.66
all | 6.15 5.16 5.65
avg. distance between the first and last exactly matching words in a text (T):
entailment: | NO YES all
task: |
--------------+--------------------------------
SUM | 19.96 19.52 19.74
QA | 17.72 14.38 16.05
IR | 10.20 12.06 11.13
IE | 20.78 16.31 18.55
all | 17.18 15.58 16.38
avg. distance between exactly matching words in a text (T):
entailment: | NO YES all
task: |
--------------+------------------------------
SUM | 2.11 1.77 1.94
QA | 2.36 1.70 2.03
IR | 1.89 2.09 1.99
IE | 2.91 2.25 2.58
all | 2.32 1.95 2.14
avg. number of tokens in text / hypothesis:
entailment: | NO YES all
task: |
--------------+--------------------------------------------------
SUM | 39.32 / 27.25 38.47 / 18.62 38.90 / 22.93
QA | 37.53 / 11.99 32.89 / 11.66 35.21 / 11.82
IR | 33.92 / 11.52 35.06 / 11.17 34.49 / 11.35
IE | 45.85 / 12.02 40.23 / 11.95 43.04 / 11.98
all | 39.16 / 15.70 36.66 / 13.35 37.91 / 14.52
avg. number of sentences in text / hypothesis:
entailment: | NO YES all
task: |
--------------+--------------------------------------------------
SUM | 1.02 / 1.00 1.00 / 1.00 1.01 / 1.00
QA | 1.23 / 1.00 1.09 / 1.00 1.16 / 1.00
IR | 1.05 / 1.00 1.08 / 1.00 1.06 / 1.00
IE | 1.06 / 1.01 1.05 / 1.00 1.05 / 1.00
all | 1.09 / 1.00 1.05 / 1.00 1.07 / 1.00
avg. number of tokens in a sentence of text / hypothesis:
entailment: | NO YES all
task: |
--------------+--------------------------------------------------
SUM | 38.55 / 27.25 38.47 / 18.62 38.90 / 22.93
QA | 30.51 / 11.99 30.17 / 11.66 35.21 / 11.82
IR | 32.30 / 11.52 32.46 / 11.17 34.49 / 11.35
IE | 43.25 / 11.90 38.31 / 11.95 40.80 / 11.93
all | 35.92 / 15.66 34.75 / 13.35 35.35 / 14.50
avg. number of exactly matching words in a pair:
entailment: | NO YES all
task: |
--------------+--------------------------------------
SUM | 8.57 9.57 9.07
QA | 6.70 7.10 6.90
IR | 3.95 4.53 4.24
IE | 6.59 6.21 6.40
all | 6.46 6.86 6.66
avg. standard deviation in the alignment of the exacly matching words in a pair (T-H):
EXAMPLE 1:
Text: u1 u2 x y z u6 u7 u8 u9 u10 u11 u12 u13 u 14 u15 u16
| | |
| | |
Hypothesis: v1 v2 v3 v4 x y z v8
[Perfectly aligned: StdDev = 0]
EXAMPLE 2:
Text: u1 u2 x u4 u5 u6 u7 u8 y u10 u11 u12 u13 z u15 u16
| | |
| | |
Hypothesis: v1 z v3 y v5 v6 x v8
[Not a good alignment:
number of shift for x = s_x = +4
number of shift for y = s_y = -5
number of shift for z = s_z = -12
number of exact matches = n = 3
mean shift = m = (4 - 5 - 12) / 3 = -7
stddev = sqrt (((m - s_x)^2 + (m - s_y)^2 + (m - s_z)^2) / n)
stddev = sqrt (((4 - 7)^2 + (-5 - 7)^2 + (-12 - 7)^2) / 3)
stddev = 13.08]
entailment: | NO YES all
task: |
--------------+-------------------------------------
SUM | 6.99 5.46 6.23
QA | 6.21 5.02 5.62
IR | 3.88 4.31 4.09
IE | 7.47 5.85 6.66
all | 6.15 5.16 5.65
avg. distance between the first and last exactly matching words in a text (T):
entailment: | NO YES all
task: |
--------------+--------------------------------
SUM | 19.96 19.52 19.74
QA | 17.72 14.38 16.05
IR | 10.20 12.06 11.13
IE | 20.78 16.31 18.55
all | 17.18 15.58 16.38
avg. distance between exactly matching words in a text (T):
entailment: | NO YES all
task: |
--------------+------------------------------
SUM | 2.11 1.77 1.94
QA | 2.36 1.70 2.03
IR | 1.89 2.09 1.99
IE | 2.91 2.25 2.58
all | 2.32 1.95 2.14
# posted by Bengi Mizrahi @ 9:15 PM 
