Feature/jobbertjobnorm onet

[workforcedeveloper]

Description

• Adding a new benchmark which allows individuals to evaluate language models on the TechWolf/JobBERT-evaluation-dataset benchmark but using the ONET taxonomy, a dataset stored in HuggingFace (workforcedeveloper/JobBERT-ONET-evaluation-dataset). Each ESCO job title has been mapped to a ONET code using the crosswalk from here (https://esco.ec.europa.eu/system/files/2023-08/ONET_%28Occupations%29_0_updated.csv).

Checklist

• Added new tests for new functionality
• Tested locally with example tasks
• Code follows project style guidelines
• Documentation updated
• No new warnings introduced
• If the rankings artifact schema changed: bumped SCHEMA_VERSION in workrb/rankings.py and updated SUPPORTED_SCHEMA_VERSIONS

[warreveys]

Hi!

Nice contribution, thanks a lot!

Some thoughts on this:

• Your indexing inside the loop is causing an unnecessary O(k*n), which we can bring down to O(n) by creating a mapping dictionary before the loop ({title: i for i, title in enumerate(TARGET_SPACE)}).
• We should consider creating an ONET class to allow for more tasks with a target space derived from ONET. We can start small here. To check whether this would work: is the JSON file from a specific ONET version that can still be retrieved online? I'd like to move toward a design similar to the ESCO class here, but I am happy to contribute this myself later.
• We should think twice about the name of this task. 'ONETJobNormRanking' is very generic, and it sounds like the ground-truth normalisation test for ONET, of which there could be loads. Could we at least call it something like 'JobBERTONetJobNormRanking'?
• I'd like to see a little docstring in the added task, just briefly explaining what the queries and targets are exactly.
• I have approved the CI workflow, so please check this when it finished

Excited to hear what you think!

[blackmlengineer]

Hi @warreveys, I accidentally pushed this under the incorrect GH account and was trying to figure out how to have the merge request moved over to this account.

All of your thoughts make sense and I will be implementing them and submitting a new request soon.