Note: this is no longer a wiki, only a static archive of the orginal!

Home

Frequently Asked Questions

How does the directory structure work?

All languages have the following directory structure:

All languages except English are extracted from a single treebank. For english, the structure is as follows:

Why is the number of tokens excluded with the -p flag in the evaluation script not the same as the number of tokens with the characteristic tag for punctuation (e.g., Punc in Turkish and PU in Italian)?

The definition of punctuation in the evaluation script is based on the UTF-8 category of punctuation, applied to the FORM field. So, for example, the reason that the number of punctuation tokens does not coincide with the number of occurrences of the Punc tag in Turkish is simply that not all punctuation tokens (according to the UTF-8 definition) are tagged as Punc.

The decision to use the UTF-8 definition was taken by last year's organization committee in order not to have to rely on treebank specific definitions, sometimes based on POSTAGs, sometimes on other criteria. We decided to stick to this definition, although we realize that it may not be perfect in every detail.

Will the official scoring include punctuation or not this year?

As stated in the task definition of the MultiLingualTrack (to which the task definition of the DomainAdaptationTrack refers) the evaluation policy differs from that of last year in that all tokens are included, specifically punctuation tokens. This is why we have modified the evaluation script so that the default is that all tokens are included, while the -p flag can be used to exclude all punctuation tokens. (For more information, see the SoftwarePage.)

What additional resources (both annotated and not) are allowed in the shared task?

For the multilingual track no annotated resources are allowed. The idea behind the shared task is that all systems should learn from the same data sets.

For the domain adaptation track, users may submit to either a closed class or an open class, which is described below.

Closed class: Systems can lean and be developed **only** on the data provided by the organizers. This also prohibits the use of any additional taggers or other components that have either been trained or hand developed on another set of data.

Open class: Resources permitted include: additional annotations for the data provided, additional data, or additional system components that have been trained or developed on data not provided by the organizers. However, in the spirit of the shared task, systems should only use resources from WSJ like domains (i.e., news) to avoid training or developing systems on the test domains. The only exception is for unlabeled data, which may come from any domain. If a participant is unsure of whether a particular resource is allowed, they can forward their inquiry to the organizers.

Participants can submit to either the closed class or the open class or both.

FrequentlyAskedQuestions (last edited 2007-03-26 15:04:58 by RyanMcDonald)