Make token merging optional during token annotation in each PipelineComponent. #197

New issue

Open

opened 2025-10-14 16:32:06 -06:00 by navan · 0 comments

navan commented

2025-10-14 16:32:06 -06:00

Owner

Originally created by @AndriyMulyar on 11/6/2018

Currently, tokens are merged by default in components such as the MetaMap annotator or the various UnitAnnotators. This is so that annotated groups of tokens are seen as individual block by the end classifier. This functionality is often wanted and should be default but still the option of turning off this merging should be provided to the end developer of a pipeline. This should be made de-facto for any new components but the re-factoring of the MetaMap and individual unit annotation components will be required.

*Originally created by @AndriyMulyar on 11/6/2018* Currently, tokens are merged by default in components such as the MetaMap annotator or the various UnitAnnotators. This is so that annotated groups of tokens are seen as individual block by the end classifier. This functionality is often wanted and should be default but still the option of turning off this merging should be provided to the end developer of a pipeline. This should be made de-facto for any new components but the re-factoring of the MetaMap and individual unit annotation components will be required.