Improve performance of maps with many labeled tokens

[johncs]

This was done as part of investigating some performance issues that @Learuis reported.