pretty_name: Evaluation run of T145/ZEUS-8B-V17-abliterated
dataset_summary: "Dataset automatically created during the evaluation run of model
\ T145/ZEUS-8B-V17-abliterated\n
The dataset is composed of 38 configuration(s), each one corresponding to one of
\ the evaluated task.\n\nThe dataset has been created from 1 run(s). Each run can
\ be found as a specific split in each configuration, the split being named using
\ the timestamp of the run.The "train" split is always pointing to the latest
\ results.\n\nAn additional configuration "results" store all the aggregated results
\ of the run.\n\nTo load the details from a run, you can for instance do the following:\n\
,\n\tname=\"T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_boolean_expressions\"\
,\n\tsplit=\"latest\"\n)\n```\n\n## Latest results\n\nThese are the [latest results\
\ from run 2025-01-02T01-08-47.205010](https://huggingface.co/datasets/open-llm-leaderboard/T145__ZEUS-8B-V17-abliterated-details/blob/main/T145__ZEUS-8B-V17-abliterated/results_2025-01-02T01-08-47.205010.json)\
\ (note that there might be results for other tasks in the repos if successive evals\
\ didn't cover the same tasks. You find each in the results and the \"latest\" split\
\ for each eval):\n\n```python\n{\n \"all\": {\n \"leaderboard\": {\n\
\ \"prompt_level_loose_acc,none\": 0.7467652495378928,\n \"\
prompt_level_loose_acc_stderr,none\": 0.018713577543655574,\n \"inst_level_strict_acc,none\"\
: 0.7961630695443646,\n \"inst_level_strict_acc_stderr,none\": \"N/A\"\
,\n \"acc_norm,none\": 0.4759372162407576,\n \"acc_norm_stderr,none\"\
: 0.005429024748052275,\n \"exact_match,none\": 0.022658610271903322,\n\
\ \"exact_match_stderr,none\": 0.004049863901189202,\n \"\
acc,none\": 0.36220079787234044,\n \"acc_stderr,none\": 0.004381933916687772,\n\
\ \"prompt_level_strict_acc,none\": 0.7190388170055453,\n \
\ \"prompt_level_strict_acc_stderr,none\": 0.019342047683776974,\n \"\
inst_level_loose_acc,none\": 0.8201438848920863,\n \"inst_level_loose_acc_stderr,none\"\
: \"N/A\",\n \"alias\": \"leaderboard\"\n },\n \"leaderboard_bbh\"\
: {\n \"acc_norm,none\": 0.5181392119423711,\n \"acc_norm_stderr,none\"\
: 0.006319610424018335,\n \"alias\": \" - leaderboard_bbh\"\n \
\ },\n \"leaderboard_bbh_boolean_expressions\": {\n \"alias\"\
: \" - leaderboard_bbh_boolean_expressions\",\n \"acc_norm,none\": 0.748,\n\
\ \"acc_norm_stderr,none\": 0.027513851933031318\n },\n \
\ \"leaderboard_bbh_causal_judgement\": {\n \"alias\": \" - leaderboard_bbh_causal_judgement\"\
,\n \"acc_norm,none\": 0.5668449197860963,\n \"acc_norm_stderr,none\"\
: 0.03633267411102591\n },\n \"leaderboard_bbh_date_understanding\"\
: {\n \"alias\": \" - leaderboard_bbh_date_understanding\",\n \
\ \"acc_norm,none\": 0.472,\n \"acc_norm_stderr,none\": 0.031636489531544396\n\
\ },\n \"leaderboard_bbh_disambiguation_qa\": {\n \"alias\"\
: \" - leaderboard_bbh_disambiguation_qa\",\n \"acc_norm,none\": 0.568,\n\
\ \"acc_norm_stderr,none\": 0.03139181076542941\n },\n \
\ \"leaderboard_bbh_formal_fallacies\": {\n \"alias\": \" - leaderboard_bbh_formal_fallacies\"\
,\n \"acc_norm,none\": 0.588,\n \"acc_norm_stderr,none\":\
\ 0.031191596026022818\n },\n \"leaderboard_bbh_geometric_shapes\"\
: {\n \"alias\": \" - leaderboard_bbh_geometric_shapes\",\n \
\ \"acc_norm,none\": 0.364,\n \"acc_norm_stderr,none\": 0.030491555220405475\n\
\ },\n \"leaderboard_bbh_hyperbaton\": {\n \"alias\": \"\
\ - leaderboard_bbh_hyperbaton\",\n \"acc_norm,none\": 0.684,\n \
\ \"acc_norm_stderr,none\": 0.02946265759857865\n },\n \"leaderboard_bbh_logical_deduction_five_objects\"\
: {\n \"alias\": \" - leaderboard_bbh_logical_deduction_five_objects\"\
,\n \"acc_norm,none\": 0.38,\n \"acc_norm_stderr,none\": 0.030760116042626098\n\
\ },\n \"leaderboard_bbh_logical_deduction_seven_objects\": {\n \
\ \"alias\": \" - leaderboard_bbh_logical_deduction_seven_objects\",\n\
\ \"acc_norm,none\": 0.332,\n \"acc_norm_stderr,none\": 0.029844039047465857\n\
\ },\n \"leaderboard_bbh_logical_deduction_three_objects\": {\n \
\ \"alias\": \" - leaderboard_bbh_logical_deduction_three_objects\",\n\
\ \"acc_norm,none\": 0.62,\n \"acc_norm_stderr,none\": 0.030760116042626098\n\
\ },\n \"leaderboard_bbh_movie_recommendation\": {\n \"\
alias\": \" - leaderboard_bbh_movie_recommendation\",\n \"acc_norm,none\"\
: 0.528,\n \"acc_norm_stderr,none\": 0.031636489531544396\n },\n\
\ \"leaderboard_bbh_navigate\": {\n \"alias\": \" - leaderboard_bbh_navigate\"\
,\n \"acc_norm,none\": 0.628,\n \"acc_norm_stderr,none\":\
\ 0.03063032594455827\n },\n \"leaderboard_bbh_object_counting\":\
\ {\n \"alias\": \" - leaderboard_bbh_object_counting\",\n \
\ \"acc_norm,none\": 0.312,\n \"acc_norm_stderr,none\": 0.02936106757521985\n\
\ },\n \"leaderboard_bbh_penguins_in_a_table\": {\n \"\
alias\": \" - leaderboard_bbh_penguins_in_a_table\",\n \"acc_norm,none\"\
: 0.5136986301369864,\n \"acc_norm_stderr,none\": 0.04150715331223415\n\
\ },\n \"leaderboard_bbh_reasoning_about_colored_objects\": {\n \
\ \"alias\": \" - leaderboard_bbh_reasoning_about_colored_objects\",\n\
\ \"acc_norm,none\": 0.604,\n \"acc_norm_stderr,none\": 0.030993197854577898\n\
\ },\n \"leaderboard_bbh_ruin_names\": {\n \"alias\": \"\
\ - leaderboard_bbh_ruin_names\",\n \"acc_norm,none\": 0.696,\n \
\ \"acc_norm_stderr,none\": 0.029150213374159652\n },\n \"\
leaderboard_bbh_salient_translation_error_detection\": {\n \"alias\"\
: \" - leaderboard_bbh_salient_translation_error_detection\",\n \"acc_norm,none\"\
: 0.536,\n \"acc_norm_stderr,none\": 0.031603975145223735\n },\n\
\ \"leaderboard_bbh_snarks\": {\n \"alias\": \" - leaderboard_bbh_snarks\"\
,\n \"acc_norm,none\": 0.6404494382022472,\n \"acc_norm_stderr,none\"\
: 0.03606913914074032\n },\n \"leaderboard_bbh_sports_understanding\"\
: {\n \"alias\": \" - leaderboard_bbh_sports_understanding\",\n \
\ \"acc_norm,none\": 0.768,\n \"acc_norm_stderr,none\": 0.026750070374865202\n\
\ },\n \"leaderboard_bbh_temporal_sequences\": {\n \"alias\"\
: \" - leaderboard_bbh_temporal_sequences\",\n \"acc_norm,none\": 0.448,\n\
\ \"acc_norm_stderr,none\": 0.03151438761115349\n },\n \
\ \"leaderboard_bbh_tracking_shuffled_objects_five_objects\": {\n \"\
alias\": \" - leaderboard_bbh_tracking_shuffled_objects_five_objects\",\n \
\ \"acc_norm,none\": 0.276,\n \"acc_norm_stderr,none\": 0.02832853727421142\n\
\ },\n \"leaderboard_bbh_tracking_shuffled_objects_seven_objects\"\
: {\n \"alias\": \" - leaderboard_bbh_tracking_shuffled_objects_seven_objects\"\
,\n \"acc_norm,none\": 0.284,\n \"acc_norm_stderr,none\":\
\ 0.02857695873043744\n },\n \"leaderboard_bbh_tracking_shuffled_objects_three_objects\"\
: {\n \"alias\": \" - leaderboard_bbh_tracking_shuffled_objects_three_objects\"\
,\n \"acc_norm,none\": 0.428,\n \"acc_norm_stderr,none\":\
\ 0.031355968923772626\n },\n \"leaderboard_bbh_web_of_lies\": {\n\
\ \"alias\": \" - leaderboard_bbh_web_of_lies\",\n \"acc_norm,none\"\
: 0.496,\n \"acc_norm_stderr,none\": 0.0316851985511992\n },\n\
\ \"leaderboard_gpqa\": {\n \"acc_norm,none\": 0.3036912751677852,\n\
\ \"acc_norm_stderr,none\": 0.013325416200909214,\n \"alias\"\
: \" - leaderboard_gpqa\"\n },\n \"leaderboard_gpqa_diamond\": {\n\
\ \"alias\": \" - leaderboard_gpqa_diamond\",\n \"acc_norm,none\"\
: 0.3434343434343434,\n \"acc_norm_stderr,none\": 0.03383201223244441\n\
\ },\n \"leaderboard_gpqa_extended\": {\n \"alias\": \"\
\ - leaderboard_gpqa_extended\",\n \"acc_norm,none\": 0.30036630036630035,\n\
\ \"acc_norm_stderr,none\": 0.019636438043304946\n },\n \
\ \"leaderboard_gpqa_main\": {\n \"alias\": \" - leaderboard_gpqa_main\"\
,\n \"acc_norm,none\": 0.29017857142857145,\n \"acc_norm_stderr,none\"\
: 0.021466115440571226\n },\n \"leaderboard_ifeval\": {\n \
\ \"alias\": \" - leaderboard_ifeval\",\n \"prompt_level_strict_acc,none\"\
: 0.7190388170055453,\n \"prompt_level_strict_acc_stderr,none\": 0.019342047683776974,\n\
\ \"inst_level_strict_acc,none\": 0.7961630695443646,\n \"\
inst_level_strict_acc_stderr,none\": \"N/A\",\n \"prompt_level_loose_acc,none\"\
: 0.7467652495378928,\n \"prompt_level_loose_acc_stderr,none\": 0.018713577543655574,\n\
\ \"inst_level_loose_acc,none\": 0.8201438848920863,\n \"\
inst_level_loose_acc_stderr,none\": \"N/A\"\n },\n \"leaderboard_math_hard\"\
: {\n \"exact_match,none\": 0.022658610271903322,\n \"exact_match_stderr,none\"\
: 0.004049863901189202,\n \"alias\": \" - leaderboard_math_hard\"\n \
\ },\n \"leaderboard_math_algebra_hard\": {\n \"alias\"\
: \" - leaderboard_math_algebra_hard\",\n \"exact_match,none\": 0.05537459283387622,\n\
\ \"exact_match_stderr,none\": 0.01307447837002421\n },\n \
\ \"leaderboard_math_counting_and_prob_hard\": {\n \"alias\": \" \
\ - leaderboard_math_counting_and_prob_hard\",\n \"exact_match,none\"\
: 0.0,\n \"exact_match_stderr,none\": 0.0\n },\n \"leaderboard_math_geometry_hard\"\
: {\n \"alias\": \" - leaderboard_math_geometry_hard\",\n \
\ \"exact_match,none\": 0.0,\n \"exact_match_stderr,none\": 0.0\n \
\ },\n \"leaderboard_math_intermediate_algebra_hard\": {\n \
\ \"alias\": \" - leaderboard_math_intermediate_algebra_hard\",\n \
\ \"exact_match,none\": 0.0035714285714285713,\n \"exact_match_stderr,none\"\
: 0.0035714285714285713\n },\n \"leaderboard_math_num_theory_hard\"\
: {\n \"alias\": \" - leaderboard_math_num_theory_hard\",\n \
\ \"exact_match,none\": 0.006493506493506494,\n \"exact_match_stderr,none\"\
: 0.006493506493506494\n },\n \"leaderboard_math_prealgebra_hard\"\
: {\n \"alias\": \" - leaderboard_math_prealgebra_hard\",\n \
\ \"exact_match,none\": 0.046632124352331605,\n \"exact_match_stderr,none\"\
: 0.015216761819262606\n },\n \"leaderboard_math_precalculus_hard\"\
: {\n \"alias\": \" - leaderboard_math_precalculus_hard\",\n \
\ \"exact_match,none\": 0.014814814814814815,\n \"exact_match_stderr,none\"\
: 0.010436494549594376\n },\n \"leaderboard_mmlu_pro\": {\n \
\ \"alias\": \" - leaderboard_mmlu_pro\",\n \"acc,none\": 0.36220079787234044,\n\
\ \"acc_stderr,none\": 0.004381933916687772\n },\n \"leaderboard_musr\"\
: {\n \"acc_norm,none\": 0.42592592592592593,\n \"acc_norm_stderr,none\"\
: 0.01744135631778622,\n \"alias\": \" - leaderboard_musr\"\n \
\ },\n \"leaderboard_musr_murder_mysteries\": {\n \"alias\": \"\
\ - leaderboard_musr_murder_mysteries\",\n \"acc_norm,none\": 0.596,\n\
\ \"acc_norm_stderr,none\": 0.03109668818482536\n },\n \
\ \"leaderboard_musr_object_placements\": {\n \"alias\": \" - leaderboard_musr_object_placements\"\
,\n \"acc_norm,none\": 0.30078125,\n \"acc_norm_stderr,none\"\
: 0.02871850463421181\n },\n \"leaderboard_musr_team_allocation\"\
: {\n \"alias\": \" - leaderboard_musr_team_allocation\",\n \
\ \"acc_norm,none\": 0.384,\n \"acc_norm_stderr,none\": 0.030821679117375447\n\
\ }\n },\n \"leaderboard\": {\n \"prompt_level_loose_acc,none\"\
: 0.7467652495378928,\n \"prompt_level_loose_acc_stderr,none\": 0.018713577543655574,\n\
\ \"inst_level_strict_acc,none\": 0.7961630695443646,\n \"inst_level_strict_acc_stderr,none\"\
: \"N/A\",\n \"acc_norm,none\": 0.4759372162407576,\n \"acc_norm_stderr,none\"\
: 0.005429024748052275,\n \"exact_match,none\": 0.022658610271903322,\n \
\ \"exact_match_stderr,none\": 0.004049863901189202,\n \"acc,none\"\
: 0.36220079787234044,\n \"acc_stderr,none\": 0.004381933916687772,\n \
\ \"prompt_level_strict_acc,none\": 0.7190388170055453,\n \"prompt_level_strict_acc_stderr,none\"\
: 0.019342047683776974,\n \"inst_level_loose_acc,none\": 0.8201438848920863,\n\
\ \"inst_level_loose_acc_stderr,none\": \"N/A\",\n \"alias\": \"leaderboard\"\
\n },\n \"leaderboard_bbh\": {\n \"acc_norm,none\": 0.5181392119423711,\n\
\ \"acc_norm_stderr,none\": 0.006319610424018335,\n \"alias\": \"\
\ - leaderboard_bbh\"\n },\n \"leaderboard_bbh_boolean_expressions\": {\n\
\ \"alias\": \" - leaderboard_bbh_boolean_expressions\",\n \"acc_norm,none\"\
: 0.748,\n \"acc_norm_stderr,none\": 0.027513851933031318\n },\n \"\
leaderboard_bbh_causal_judgement\": {\n \"alias\": \" - leaderboard_bbh_causal_judgement\"\
,\n \"acc_norm,none\": 0.5668449197860963,\n \"acc_norm_stderr,none\"\
: 0.03633267411102591\n },\n \"leaderboard_bbh_date_understanding\": {\n \
\ \"alias\": \" - leaderboard_bbh_date_understanding\",\n \"acc_norm,none\"\
: 0.472,\n \"acc_norm_stderr,none\": 0.031636489531544396\n },\n \"\
leaderboard_bbh_disambiguation_qa\": {\n \"alias\": \" - leaderboard_bbh_disambiguation_qa\"\
,\n \"acc_norm,none\": 0.568,\n \"acc_norm_stderr,none\": 0.03139181076542941\n\
\ },\n \"leaderboard_bbh_formal_fallacies\": {\n \"alias\": \" - leaderboard_bbh_formal_fallacies\"\
,\n \"acc_norm,none\": 0.588,\n \"acc_norm_stderr,none\": 0.031191596026022818\n\
\ },\n \"leaderboard_bbh_geometric_shapes\": {\n \"alias\": \" - leaderboard_bbh_geometric_shapes\"\
,\n \"acc_norm,none\": 0.364,\n \"acc_norm_stderr,none\": 0.030491555220405475\n\
\ },\n \"leaderboard_bbh_hyperbaton\": {\n \"alias\": \" - leaderboard_bbh_hyperbaton\"\
,\n \"acc_norm,none\": 0.684,\n \"acc_norm_stderr,none\": 0.02946265759857865\n\
\ },\n \"leaderboard_bbh_logical_deduction_five_objects\": {\n \"alias\"\
: \" - leaderboard_bbh_logical_deduction_five_objects\",\n \"acc_norm,none\"\
: 0.38,\n \"acc_norm_stderr,none\": 0.030760116042626098\n },\n \"\
leaderboard_bbh_logical_deduction_seven_objects\": {\n \"alias\": \" - leaderboard_bbh_logical_deduction_seven_objects\"\
,\n \"acc_norm,none\": 0.332,\n \"acc_norm_stderr,none\": 0.029844039047465857\n\
\ },\n \"leaderboard_bbh_logical_deduction_three_objects\": {\n \"\
alias\": \" - leaderboard_bbh_logical_deduction_three_objects\",\n \"acc_norm,none\"\
: 0.62,\n \"acc_norm_stderr,none\": 0.030760116042626098\n },\n \"\
leaderboard_bbh_movie_recommendation\": {\n \"alias\": \" - leaderboard_bbh_movie_recommendation\"\
,\n \"acc_norm,none\": 0.528,\n \"acc_norm_stderr,none\": 0.031636489531544396\n\
\ },\n \"leaderboard_bbh_navigate\": {\n \"alias\": \" - leaderboard_bbh_navigate\"\
,\n \"acc_norm,none\": 0.628,\n \"acc_norm_stderr,none\": 0.03063032594455827\n\
\ },\n \"leaderboard_bbh_object_counting\": {\n \"alias\": \" - leaderboard_bbh_object_counting\"\
,\n \"acc_norm,none\": 0.312,\n \"acc_norm_stderr,none\": 0.02936106757521985\n\
\ },\n \"leaderboard_bbh_penguins_in_a_table\": {\n \"alias\": \" \
\ - leaderboard_bbh_penguins_in_a_table\",\n \"acc_norm,none\": 0.5136986301369864,\n\
\ \"acc_norm_stderr,none\": 0.04150715331223415\n },\n \"leaderboard_bbh_reasoning_about_colored_objects\"\
: {\n \"alias\": \" - leaderboard_bbh_reasoning_about_colored_objects\"\
,\n \"acc_norm,none\": 0.604,\n \"acc_norm_stderr,none\": 0.030993197854577898\n\
\ },\n \"leaderboard_bbh_ruin_names\": {\n \"alias\": \" - leaderboard_bbh_ruin_names\"\
,\n \"acc_norm,none\": 0.696,\n \"acc_norm_stderr,none\": 0.029150213374159652\n\
\ },\n \"leaderboard_bbh_salient_translation_error_detection\": {\n \
\ \"alias\": \" - leaderboard_bbh_salient_translation_error_detection\",\n \
\ \"acc_norm,none\": 0.536,\n \"acc_norm_stderr,none\": 0.031603975145223735\n\
\ },\n \"leaderboard_bbh_snarks\": {\n \"alias\": \" - leaderboard_bbh_snarks\"\
,\n \"acc_norm,none\": 0.6404494382022472,\n \"acc_norm_stderr,none\"\
: 0.03606913914074032\n },\n \"leaderboard_bbh_sports_understanding\": {\n\
\ \"alias\": \" - leaderboard_bbh_sports_understanding\",\n \"acc_norm,none\"\
: 0.768,\n \"acc_norm_stderr,none\": 0.026750070374865202\n },\n \"\
leaderboard_bbh_temporal_sequences\": {\n \"alias\": \" - leaderboard_bbh_temporal_sequences\"\
,\n \"acc_norm,none\": 0.448,\n \"acc_norm_stderr,none\": 0.03151438761115349\n\
\ },\n \"leaderboard_bbh_tracking_shuffled_objects_five_objects\": {\n \
\ \"alias\": \" - leaderboard_bbh_tracking_shuffled_objects_five_objects\"\
,\n \"acc_norm,none\": 0.276,\n \"acc_norm_stderr,none\": 0.02832853727421142\n\
\ },\n \"leaderboard_bbh_tracking_shuffled_objects_seven_objects\": {\n \
\ \"alias\": \" - leaderboard_bbh_tracking_shuffled_objects_seven_objects\"\
,\n \"acc_norm,none\": 0.284,\n \"acc_norm_stderr,none\": 0.02857695873043744\n\
\ },\n \"leaderboard_bbh_tracking_shuffled_objects_three_objects\": {\n \
\ \"alias\": \" - leaderboard_bbh_tracking_shuffled_objects_three_objects\"\
,\n \"acc_norm,none\": 0.428,\n \"acc_norm_stderr,none\": 0.031355968923772626\n\
\ },\n \"leaderboard_bbh_web_of_lies\": {\n \"alias\": \" - leaderboard_bbh_web_of_lies\"\
,\n \"acc_norm,none\": 0.496,\n \"acc_norm_stderr,none\": 0.0316851985511992\n\
\ },\n \"leaderboard_gpqa\": {\n \"acc_norm,none\": 0.3036912751677852,\n\
\ \"acc_norm_stderr,none\": 0.013325416200909214,\n \"alias\": \"\
\ - leaderboard_gpqa\"\n },\n \"leaderboard_gpqa_diamond\": {\n \"\
alias\": \" - leaderboard_gpqa_diamond\",\n \"acc_norm,none\": 0.3434343434343434,\n\
\ \"acc_norm_stderr,none\": 0.03383201223244441\n },\n \"leaderboard_gpqa_extended\"\
: {\n \"alias\": \" - leaderboard_gpqa_extended\",\n \"acc_norm,none\"\
: 0.30036630036630035,\n \"acc_norm_stderr,none\": 0.019636438043304946\n\
\ },\n \"leaderboard_gpqa_main\": {\n \"alias\": \" - leaderboard_gpqa_main\"\
,\n \"acc_norm,none\": 0.29017857142857145,\n \"acc_norm_stderr,none\"\
: 0.021466115440571226\n },\n \"leaderboard_ifeval\": {\n \"alias\"\
: \" - leaderboard_ifeval\",\n \"prompt_level_strict_acc,none\": 0.7190388170055453,\n\
\ \"prompt_level_strict_acc_stderr,none\": 0.019342047683776974,\n \
\ \"inst_level_strict_acc,none\": 0.7961630695443646,\n \"inst_level_strict_acc_stderr,none\"\
: \"N/A\",\n \"prompt_level_loose_acc,none\": 0.7467652495378928,\n \
\ \"prompt_level_loose_acc_stderr,none\": 0.018713577543655574,\n \"inst_level_loose_acc,none\"\
: 0.8201438848920863,\n \"inst_level_loose_acc_stderr,none\": \"N/A\"\n \
\ },\n \"leaderboard_math_hard\": {\n \"exact_match,none\": 0.022658610271903322,\n\
\ \"exact_match_stderr,none\": 0.004049863901189202,\n \"alias\":\
\ \" - leaderboard_math_hard\"\n },\n \"leaderboard_math_algebra_hard\": {\n\
\ \"alias\": \" - leaderboard_math_algebra_hard\",\n \"exact_match,none\"\
: 0.05537459283387622,\n \"exact_match_stderr,none\": 0.01307447837002421\n\
\ },\n \"leaderboard_math_counting_and_prob_hard\": {\n \"alias\":\
\ \" - leaderboard_math_counting_and_prob_hard\",\n \"exact_match,none\"\
: 0.0,\n \"exact_match_stderr,none\": 0.0\n },\n \"leaderboard_math_geometry_hard\"\
: {\n \"alias\": \" - leaderboard_math_geometry_hard\",\n \"exact_match,none\"\
: 0.0,\n \"exact_match_stderr,none\": 0.0\n },\n \"leaderboard_math_intermediate_algebra_hard\"\
: {\n \"alias\": \" - leaderboard_math_intermediate_algebra_hard\",\n \
\ \"exact_match,none\": 0.0035714285714285713,\n \"exact_match_stderr,none\"\
: 0.0035714285714285713\n },\n \"leaderboard_math_num_theory_hard\": {\n \
\ \"alias\": \" - leaderboard_math_num_theory_hard\",\n \"exact_match,none\"\
: 0.006493506493506494,\n \"exact_match_stderr,none\": 0.006493506493506494\n\
\ },\n \"leaderboard_math_prealgebra_hard\": {\n \"alias\": \" - leaderboard_math_prealgebra_hard\"\
,\n \"exact_match,none\": 0.046632124352331605,\n \"exact_match_stderr,none\"\
: 0.015216761819262606\n },\n \"leaderboard_math_precalculus_hard\": {\n \
\ \"alias\": \" - leaderboard_math_precalculus_hard\",\n \"exact_match,none\"\
: 0.014814814814814815,\n \"exact_match_stderr,none\": 0.010436494549594376\n\
\ },\n \"leaderboard_mmlu_pro\": {\n \"alias\": \" - leaderboard_mmlu_pro\"\
,\n \"acc,none\": 0.36220079787234044,\n \"acc_stderr,none\": 0.004381933916687772\n\
\ },\n \"leaderboard_musr\": {\n \"acc_norm,none\": 0.42592592592592593,\n\
\ \"acc_norm_stderr,none\": 0.01744135631778622,\n \"alias\": \" -\
\ leaderboard_musr\"\n },\n \"leaderboard_musr_murder_mysteries\": {\n \
\ \"alias\": \" - leaderboard_musr_murder_mysteries\",\n \"acc_norm,none\"\
: 0.596,\n \"acc_norm_stderr,none\": 0.03109668818482536\n },\n \"\
leaderboard_musr_object_placements\": {\n \"alias\": \" - leaderboard_musr_object_placements\"\
,\n \"acc_norm,none\": 0.30078125,\n \"acc_norm_stderr,none\": 0.02871850463421181\n\
\ },\n \"leaderboard_musr_team_allocation\": {\n \"alias\": \" - leaderboard_musr_team_allocation\"\
,\n \"acc_norm,none\": 0.384,\n \"acc_norm_stderr,none\": 0.030821679117375447\n\
\ }\n}\n```"
repo_url: https://huggingface.co/T145/ZEUS-8B-V17-abliterated
leaderboard_url: ''
point_of_contact: ''
configs:
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_boolean_expressions
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_boolean_expressions_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_boolean_expressions_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_causal_judgement
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_causal_judgement_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_causal_judgement_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_date_understanding
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_date_understanding_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_date_understanding_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_disambiguation_qa
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_disambiguation_qa_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_disambiguation_qa_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_formal_fallacies
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_formal_fallacies_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_formal_fallacies_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_geometric_shapes
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_geometric_shapes_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_geometric_shapes_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_hyperbaton
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_hyperbaton_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_hyperbaton_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_logical_deduction_five_objects
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_logical_deduction_five_objects_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_logical_deduction_five_objects_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_logical_deduction_seven_objects
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_logical_deduction_seven_objects_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_logical_deduction_seven_objects_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_logical_deduction_three_objects
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_logical_deduction_three_objects_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_logical_deduction_three_objects_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_movie_recommendation
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_movie_recommendation_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_movie_recommendation_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_navigate
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_navigate_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_navigate_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_object_counting
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_object_counting_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_object_counting_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_penguins_in_a_table
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_penguins_in_a_table_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_penguins_in_a_table_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_reasoning_about_colored_objects
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_reasoning_about_colored_objects_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_reasoning_about_colored_objects_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_ruin_names
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_ruin_names_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_ruin_names_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_salient_translation_error_detection
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_salient_translation_error_detection_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_salient_translation_error_detection_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_snarks
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_snarks_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_snarks_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_sports_understanding
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_sports_understanding_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_sports_understanding_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_temporal_sequences
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_temporal_sequences_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_temporal_sequences_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_tracking_shuffled_objects_five_objects
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_tracking_shuffled_objects_five_objects_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_tracking_shuffled_objects_five_objects_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_tracking_shuffled_objects_seven_objects
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_tracking_shuffled_objects_seven_objects_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_tracking_shuffled_objects_seven_objects_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_tracking_shuffled_objects_three_objects
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_tracking_shuffled_objects_three_objects_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_tracking_shuffled_objects_three_objects_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_web_of_lies
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_bbh_web_of_lies_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_bbh_web_of_lies_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_gpqa_diamond
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_gpqa_diamond_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_gpqa_diamond_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_gpqa_extended
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_gpqa_extended_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_gpqa_extended_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_gpqa_main
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_gpqa_main_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_gpqa_main_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_ifeval
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_ifeval_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_ifeval_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_math_algebra_hard
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_math_algebra_hard_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_math_algebra_hard_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_math_counting_and_prob_hard
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_math_counting_and_prob_hard_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_math_counting_and_prob_hard_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_math_geometry_hard
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_math_geometry_hard_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_math_geometry_hard_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_math_intermediate_algebra_hard
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_math_intermediate_algebra_hard_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_math_intermediate_algebra_hard_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_math_num_theory_hard
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_math_num_theory_hard_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_math_num_theory_hard_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_math_prealgebra_hard
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_math_prealgebra_hard_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_math_prealgebra_hard_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_math_precalculus_hard
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_math_precalculus_hard_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_math_precalculus_hard_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_mmlu_pro
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_mmlu_pro_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_mmlu_pro_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_musr_murder_mysteries
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_musr_murder_mysteries_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_musr_murder_mysteries_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_musr_object_placements
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_musr_object_placements_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_musr_object_placements_2025-01-02T01-08-47.205010.jsonl'
- config_name: T145__ZEUS-8B-V17-abliterated__leaderboard_musr_team_allocation
data_files:
- split: 2025_01_02T01_08_47.205010
path:
- '**/samples_leaderboard_musr_team_allocation_2025-01-02T01-08-47.205010.jsonl'
- split: latest
path:
- '**/samples_leaderboard_musr_team_allocation_2025-01-02T01-08-47.205010.jsonl'
---
# Dataset Card for Evaluation run of T145/ZEUS-8B-V17-abliterated
<!-- Provide a quick summary of the dataset. -->
Dataset automatically created during the evaluation run of model [T145/ZEUS-8B-V17-abliterated](https://huggingface.co/T145/ZEUS-8B-V17-abliterated)
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.
An additional configuration "results" store all the aggregated results of the run.
To load the details from a run, you can for instance do the following:
```python
from datasets import load_dataset
data = load_dataset(
"open-llm-leaderboard/T145__ZEUS-8B-V17-abliterated-details",
name="T145__ZEUS-8B-V17-abliterated__leaderboard_bbh_boolean_expressions",
split="latest"
)
Latest results
These are the latest results from run 2025-01-02T01-08-47.205010 (note that there might be results for other tasks in the repos if successive evals didn't cover the same tasks. You find each in the results and the "latest" split for each eval):
{
"all": {
"leaderboard": {
"prompt_level_loose_acc,none": 0.7467652495378928,
"prompt_level_loose_acc_stderr,none": 0.018713577543655574,
"inst_level_strict_acc,none": 0.7961630695443646,
"inst_level_strict_acc_stderr,none": "N/A",
"acc_norm,none": 0.4759372162407576,
"acc_norm_stderr,none": 0.005429024748052275,
"exact_match,none": 0.022658610271903322,
"exact_match_stderr,none": 0.004049863901189202,
"acc,none": 0.36220079787234044,
"acc_stderr,none": 0.004381933916687772,
"prompt_level_strict_acc,none": 0.7190388170055453,
"prompt_level_strict_acc_stderr,none": 0.019342047683776974,
"inst_level_loose_acc,none": 0.8201438848920863,
"inst_level_loose_acc_stderr,none": "N/A",
"alias": "leaderboard"
},
"leaderboard_bbh": {
"acc_norm,none": 0.5181392119423711,
"acc_norm_stderr,none": 0.006319610424018335,
"alias": " - leaderboard_bbh"
},
"leaderboard_bbh_boolean_expressions": {
"alias": " - leaderboard_bbh_boolean_expressions",
"acc_norm,none": 0.748,
"acc_norm_stderr,none": 0.027513851933031318
},
"leaderboard_bbh_causal_judgement": {
"alias": " - leaderboard_bbh_causal_judgement",
"acc_norm,none": 0.5668449197860963,
"acc_norm_stderr,none": 0.03633267411102591
},
"leaderboard_bbh_date_understanding": {
"alias": " - leaderboard_bbh_date_understanding",
"acc_norm,none": 0.472,
"acc_norm_stderr,none": 0.031636489531544396
},
"leaderboard_bbh_disambiguation_qa": {
"alias": " - leaderboard_bbh_disambiguation_qa",
"acc_norm,none": 0.568,
"acc_norm_stderr,none": 0.03139181076542941
},
"leaderboard_bbh_formal_fallacies": {
"alias": " - leaderboard_bbh_formal_fallacies",
"acc_norm,none": 0.588,
"acc_norm_stderr,none": 0.031191596026022818
},
"leaderboard_bbh_geometric_shapes": {
"alias": " - leaderboard_bbh_geometric_shapes",
"acc_norm,none": 0.364,
"acc_norm_stderr,none": 0.030491555220405475
},
"leaderboard_bbh_hyperbaton": {
"alias": " - leaderboard_bbh_hyperbaton",
"acc_norm,none": 0.684,
"acc_norm_stderr,none": 0.02946265759857865
},
"leaderboard_bbh_logical_deduction_five_objects": {
"alias": " - leaderboard_bbh_logical_deduction_five_objects",
"acc_norm,none": 0.38,
"acc_norm_stderr,none": 0.030760116042626098
},
"leaderboard_bbh_logical_deduction_seven_objects": {
"alias": " - leaderboard_bbh_logical_deduction_seven_objects",
"acc_norm,none": 0.332,
"acc_norm_stderr,none": 0.029844039047465857
},
"leaderboard_bbh_logical_deduction_three_objects": {
"alias": " - leaderboard_bbh_logical_deduction_three_objects",
"acc_norm,none": 0.62,
"acc_norm_stderr,none": 0.030760116042626098
},
"leaderboard_bbh_movie_recommendation": {
"alias": " - leaderboard_bbh_movie_recommendation",
"acc_norm,none": 0.528,
"acc_norm_stderr,none": 0.031636489531544396
},
"leaderboard_bbh_navigate": {
"alias": " - leaderboard_bbh_navigate",
"acc_norm,none": 0.628,
"acc_norm_stderr,none": 0.03063032594455827
},
"leaderboard_bbh_object_counting": {
"alias": " - leaderboard_bbh_object_counting",
"acc_norm,none": 0.312,
"acc_norm_stderr,none": 0.02936106757521985
},
"leaderboard_bbh_penguins_in_a_table": {
"alias": " - leaderboard_bbh_penguins_in_a_table",
"acc_norm,none": 0.5136986301369864,
"acc_norm_stderr,none": 0.04150715331223415
},
"leaderboard_bbh_reasoning_about_colored_objects": {
"alias": " - leaderboard_bbh_reasoning_about_colored_objects",
"acc_norm,none": 0.604,
"acc_norm_stderr,none": 0.030993197854577898
},
"leaderboard_bbh_ruin_names": {
"alias": " - leaderboard_bbh_ruin_names",
"acc_norm,none": 0.696,
"acc_norm_stderr,none": 0.029150213374159652
},
"leaderboard_bbh_salient_translation_error_detection": {
"alias": " - leaderboard_bbh_salient_translation_error_detection",
"acc_norm,none": 0.536,
"acc_norm_stderr,none": 0.031603975145223735
},
"leaderboard_bbh_snarks": {
"alias": " - leaderboard_bbh_snarks",
"acc_norm,none": 0.6404494382022472,
"acc_norm_stderr,none": 0.03606913914074032
},
"leaderboard_bbh_sports_understanding": {
"alias": " - leaderboard_bbh_sports_understanding",
"acc_norm,none": 0.768,
"acc_norm_stderr,none": 0.026750070374865202
},
"leaderboard_bbh_temporal_sequences": {
"alias": " - leaderboard_bbh_temporal_sequences",
"acc_norm,none": 0.448,
"acc_norm_stderr,none": 0.03151438761115349
},
"leaderboard_bbh_tracking_shuffled_objects_five_objects": {
"alias": " - leaderboard_bbh_tracking_shuffled_objects_five_objects",
"acc_norm,none": 0.276,
"acc_norm_stderr,none": 0.02832853727421142
},
"leaderboard_bbh_tracking_shuffled_objects_seven_objects": {
"alias": " - leaderboard_bbh_tracking_shuffled_objects_seven_objects",
"acc_norm,none": 0.284,
"acc_norm_stderr,none": 0.02857695873043744
},
"leaderboard_bbh_tracking_shuffled_objects_three_objects": {
"alias": " - leaderboard_bbh_tracking_shuffled_objects_three_objects",
"acc_norm,none": 0.428,
"acc_norm_stderr,none": 0.031355968923772626
},
"leaderboard_bbh_web_of_lies": {
"alias": " - leaderboard_bbh_web_of_lies",
"acc_norm,none": 0.496,
"acc_norm_stderr,none": 0.0316851985511992
},
"leaderboard_gpqa": {
"acc_norm,none": 0.3036912751677852,
"acc_norm_stderr,none": 0.013325416200909214,
"alias": " - leaderboard_gpqa"
},
"leaderboard_gpqa_diamond": {
"alias": " - leaderboard_gpqa_diamond",
"acc_norm,none": 0.3434343434343434,
"acc_norm_stderr,none": 0.03383201223244441
},
"leaderboard_gpqa_extended": {
"alias": " - leaderboard_gpqa_extended",
"acc_norm,none": 0.30036630036630035,
"acc_norm_stderr,none": 0.019636438043304946
},
"leaderboard_gpqa_main": {
"alias": " - leaderboard_gpqa_main",
"acc_norm,none": 0.29017857142857145,
"acc_norm_stderr,none": 0.021466115440571226
},
"leaderboard_ifeval": {
"alias": " - leaderboard_ifeval",
"prompt_level_strict_acc,none": 0.7190388170055453,
"prompt_level_strict_acc_stderr,none": 0.019342047683776974,
"inst_level_strict_acc,none": 0.7961630695443646,
"inst_level_strict_acc_stderr,none": "N/A",
"prompt_level_loose_acc,none": 0.7467652495378928,
"prompt_level_loose_acc_stderr,none": 0.018713577543655574,
"inst_level_loose_acc,none": 0.8201438848920863,
"inst_level_loose_acc_stderr,none": "N/A"
},
"leaderboard_math_hard": {
"exact_match,none": 0.022658610271903322,
"exact_match_stderr,none": 0.004049863901189202,
"alias": " - leaderboard_math_hard"
},
"leaderboard_math_algebra_hard": {
"alias": " - leaderboard_math_algebra_hard",
"exact_match,none": 0.05537459283387622,
"exact_match_stderr,none": 0.01307447837002421
},
"leaderboard_math_counting_and_prob_hard": {
"alias": " - leaderboard_math_counting_and_prob_hard",
"exact_match,none": 0.0,
"exact_match_stderr,none": 0.0
},
"leaderboard_math_geometry_hard": {
"alias": " - leaderboard_math_geometry_hard",
"exact_match,none": 0.0,
"exact_match_stderr,none": 0.0
},
"leaderboard_math_intermediate_algebra_hard": {
"alias": " - leaderboard_math_intermediate_algebra_hard",
"exact_match,none": 0.0035714285714285713,
"exact_match_stderr,none": 0.0035714285714285713
},
"leaderboard_math_num_theory_hard": {
"alias": " - leaderboard_math_num_theory_hard",
"exact_match,none": 0.006493506493506494,
"exact_match_stderr,none": 0.006493506493506494
},
"leaderboard_math_prealgebra_hard": {
"alias": " - leaderboard_math_prealgebra_hard",
"exact_match,none": 0.046632124352331605,
"exact_match_stderr,none": 0.015216761819262606
},
"leaderboard_math_precalculus_hard": {
"alias": " - leaderboard_math_precalculus_hard",
"exact_match,none": 0.014814814814814815,
"exact_match_stderr,none": 0.010436494549594376
},
"leaderboard_mmlu_pro": {
"alias": " - leaderboard_mmlu_pro",
"acc,none": 0.36220079787234044,
"acc_stderr,none": 0.004381933916687772
},
"leaderboard_musr": {
"acc_norm,none": 0.42592592592592593,
"acc_norm_stderr,none": 0.01744135631778622,
"alias": " - leaderboard_musr"
},
"leaderboard_musr_murder_mysteries": {
"alias": " - leaderboard_musr_murder_mysteries",
"acc_norm,none": 0.596,
"acc_norm_stderr,none": 0.03109668818482536
},
"leaderboard_musr_object_placements": {
"alias": " - leaderboard_musr_object_placements",
"acc_norm,none": 0.30078125,
"acc_norm_stderr,none": 0.02871850463421181
},
"leaderboard_musr_team_allocation": {
"alias": " - leaderboard_musr_team_allocation",
"acc_norm,none": 0.384,
"acc_norm_stderr,none": 0.030821679117375447
}
},
"leaderboard": {
"prompt_level_loose_acc,none": 0.7467652495378928,
"prompt_level_loose_acc_stderr,none": 0.018713577543655574,
"inst_level_strict_acc,none": 0.7961630695443646,
"inst_level_strict_acc_stderr,none": "N/A",
"acc_norm,none": 0.4759372162407576,
"acc_norm_stderr,none": 0.005429024748052275,
"exact_match,none": 0.022658610271903322,
"exact_match_stderr,none": 0.004049863901189202,
"acc,none": 0.36220079787234044,
"acc_stderr,none": 0.004381933916687772,
"prompt_level_strict_acc,none": 0.7190388170055453,
"prompt_level_strict_acc_stderr,none": 0.019342047683776974,
"inst_level_loose_acc,none": 0.8201438848920863,
"inst_level_loose_acc_stderr,none": "N/A",
"alias": "leaderboard"
},
"leaderboard_bbh": {
"acc_norm,none": 0.5181392119423711,
"acc_norm_stderr,none": 0.006319610424018335,
"alias": " - leaderboard_bbh"
},
"leaderboard_bbh_boolean_expressions": {
"alias": " - leaderboard_bbh_boolean_expressions",
"acc_norm,none": 0.748,
"acc_norm_stderr,none": 0.027513851933031318
},
"leaderboard_bbh_causal_judgement": {
"alias": " - leaderboard_bbh_causal_judgement",
"acc_norm,none": 0.5668449197860963,
"acc_norm_stderr,none": 0.03633267411102591
},
"leaderboard_bbh_date_understanding": {
"alias": " - leaderboard_bbh_date_understanding",
"acc_norm,none": 0.472,
"acc_norm_stderr,none": 0.031636489531544396
},
"leaderboard_bbh_disambiguation_qa": {
"alias": " - leaderboard_bbh_disambiguation_qa",
"acc_norm,none": 0.568,
"acc_norm_stderr,none": 0.03139181076542941
},
"leaderboard_bbh_formal_fallacies": {
"alias": " - leaderboard_bbh_formal_fallacies",
"acc_norm,none": 0.588,
"acc_norm_stderr,none": 0.031191596026022818
},
"leaderboard_bbh_geometric_shapes": {
"alias": " - leaderboard_bbh_geometric_shapes",
"acc_norm,none": 0.364,
"acc_norm_stderr,none": 0.030491555220405475
},
"leaderboard_bbh_hyperbaton": {
"alias": " - leaderboard_bbh_hyperbaton",
"acc_norm,none": 0.684,
"acc_norm_stderr,none": 0.02946265759857865
},
"leaderboard_bbh_logical_deduction_five_objects": {
"alias": " - leaderboard_bbh_logical_deduction_five_objects",
"acc_norm,none": 0.38,
"acc_norm_stderr,none": 0.030760116042626098
},
"leaderboard_bbh_logical_deduction_seven_objects": {
"alias": " - leaderboard_bbh_logical_deduction_seven_objects",
"acc_norm,none": 0.332,
"acc_norm_stderr,none": 0.029844039047465857
},
"leaderboard_bbh_logical_deduction_three_objects": {
"alias": " - leaderboard_bbh_logical_deduction_three_objects",
"acc_norm,none": 0.62,
"acc_norm_stderr,none": 0.030760116042626098
},
"leaderboard_bbh_movie_recommendation": {
"alias": " - leaderboard_bbh_movie_recommendation",
"acc_norm,none": 0.528,
"acc_norm_stderr,none": 0.031636489531544396
},
"leaderboard_bbh_navigate": {
"alias": " - leaderboard_bbh_navigate",
"acc_norm,none": 0.628,
"acc_norm_stderr,none": 0.03063032594455827
},
"leaderboard_bbh_object_counting": {
"alias": " - leaderboard_bbh_object_counting",
"acc_norm,none": 0.312,
"acc_norm_stderr,none": 0.02936106757521985
},
"leaderboard_bbh_penguins_in_a_table": {
"alias": " - leaderboard_bbh_penguins_in_a_table",
"acc_norm,none": 0.5136986301369864,
"acc_norm_stderr,none": 0.04150715331223415
},
"leaderboard_bbh_reasoning_about_colored_objects": {
"alias": " - leaderboard_bbh_reasoning_about_colored_objects",
"acc_norm,none": 0.604,
"acc_norm_stderr,none": 0.030993197854577898
},
"leaderboard_bbh_ruin_names": {
"alias": " - leaderboard_bbh_ruin_names",
"acc_norm,none": 0.696,
"acc_norm_stderr,none": 0.029150213374159652
},
"leaderboard_bbh_salient_translation_error_detection": {
"alias": " - leaderboard_bbh_salient_translation_error_detection",
"acc_norm,none": 0.536,
"acc_norm_stderr,none": 0.031603975145223735
},
"leaderboard_bbh_snarks": {
"alias": " - leaderboard_bbh_snarks",
"acc_norm,none": 0.6404494382022472,
"acc_norm_stderr,none": 0.03606913914074032
},
"leaderboard_bbh_sports_understanding": {
"alias": " - leaderboard_bbh_sports_understanding",
"acc_norm,none": 0.768,
"acc_norm_stderr,none": 0.026750070374865202
},
"leaderboard_bbh_temporal_sequences": {
"alias": " - leaderboard_bbh_temporal_sequences",
"acc_norm,none": 0.448,
"acc_norm_stderr,none": 0.03151438761115349
},
"leaderboard_bbh_tracking_shuffled_objects_five_objects": {
"alias": " - leaderboard_bbh_tracking_shuffled_objects_five_objects",
"acc_norm,none": 0.276,
"acc_norm_stderr,none": 0.02832853727421142
},
"leaderboard_bbh_tracking_shuffled_objects_seven_objects": {
"alias": " - leaderboard_bbh_tracking_shuffled_objects_seven_objects",
"acc_norm,none": 0.284,
"acc_norm_stderr,none": 0.02857695873043744
},
"leaderboard_bbh_tracking_shuffled_objects_three_objects": {
"alias": " - leaderboard_bbh_tracking_shuffled_objects_three_objects",
"acc_norm,none": 0.428,
"acc_norm_stderr,none": 0.031355968923772626
},
"leaderboard_bbh_web_of_lies": {
"alias": " - leaderboard_bbh_web_of_lies",
"acc_norm,none": 0.496,
"acc_norm_stderr,none": 0.0316851985511992
},
"leaderboard_gpqa": {
"acc_norm,none": 0.3036912751677852,
"acc_norm_stderr,none": 0.013325416200909214,
"alias": " - leaderboard_gpqa"
},
"leaderboard_gpqa_diamond": {
"alias": " - leaderboard_gpqa_diamond",
"acc_norm,none": 0.3434343434343434,
"acc_norm_stderr,none": 0.03383201223244441
},
"leaderboard_gpqa_extended": {
"alias": " - leaderboard_gpqa_extended",
"acc_norm,none": 0.30036630036630035,
"acc_norm_stderr,none": 0.019636438043304946
},
"leaderboard_gpqa_main": {
"alias": " - leaderboard_gpqa_main",
"acc_norm,none": 0.29017857142857145,
"acc_norm_stderr,none": 0.021466115440571226
},
"leaderboard_ifeval": {
"alias": " - leaderboard_ifeval",
"prompt_level_strict_acc,none": 0.7190388170055453,
"prompt_level_strict_acc_stderr,none": 0.019342047683776974,
"inst_level_strict_acc,none": 0.7961630695443646,
"inst_level_strict_acc_stderr,none": "N/A",
"prompt_level_loose_acc,none": 0.7467652495378928,
"prompt_level_loose_acc_stderr,none": 0.018713577543655574,
"inst_level_loose_acc,none": 0.8201438848920863,
"inst_level_loose_acc_stderr,none": "N/A"
},
"leaderboard_math_hard": {
"exact_match,none": 0.022658610271903322,
"exact_match_stderr,none": 0.004049863901189202,
"alias": " - leaderboard_math_hard"
},
"leaderboard_math_algebra_hard": {
"alias": " - leaderboard_math_algebra_hard",
"exact_match,none": 0.05537459283387622,
"exact_match_stderr,none": 0.01307447837002421
},
"leaderboard_math_counting_and_prob_hard": {
"alias": " - leaderboard_math_counting_and_prob_hard",
"exact_match,none": 0.0,
"exact_match_stderr,none": 0.0
},
"leaderboard_math_geometry_hard": {
"alias": " - leaderboard_math_geometry_hard",
"exact_match,none": 0.0,
"exact_match_stderr,none": 0.0
},
"leaderboard_math_intermediate_algebra_hard": {
"alias": " - leaderboard_math_intermediate_algebra_hard",
"exact_match,none": 0.0035714285714285713,
"exact_match_stderr,none": 0.0035714285714285713
},
"leaderboard_math_num_theory_hard": {
"alias": " - leaderboard_math_num_theory_hard",
"exact_match,none": 0.006493506493506494,
"exact_match_stderr,none": 0.006493506493506494
},
"leaderboard_math_prealgebra_hard": {
"alias": " - leaderboard_math_prealgebra_hard",
"exact_match,none": 0.046632124352331605,
"exact_match_stderr,none": 0.015216761819262606
},
"leaderboard_math_precalculus_hard": {
"alias": " - leaderboard_math_precalculus_hard",
"exact_match,none": 0.014814814814814815,
"exact_match_stderr,none": 0.010436494549594376
},
"leaderboard_mmlu_pro": {
"alias": " - leaderboard_mmlu_pro",
"acc,none": 0.36220079787234044,
"acc_stderr,none": 0.004381933916687772
},
"leaderboard_musr": {
"acc_norm,none": 0.42592592592592593,
"acc_norm_stderr,none": 0.01744135631778622,
"alias": " - leaderboard_musr"
},
"leaderboard_musr_murder_mysteries": {
"alias": " - leaderboard_musr_murder_mysteries",
"acc_norm,none": 0.596,
"acc_norm_stderr,none": 0.03109668818482536
},
"leaderboard_musr_object_placements": {
"alias": " - leaderboard_musr_object_placements",
"acc_norm,none": 0.30078125,
"acc_norm_stderr,none": 0.02871850463421181
},
"leaderboard_musr_team_allocation": {
"alias": " - leaderboard_musr_team_allocation",
"acc_norm,none": 0.384,
"acc_norm_stderr,none": 0.030821679117375447
}
}
Dataset Details
Dataset Description
- Curated by: [More Information Needed]
- Funded by [optional]: [More Information Needed]
- Shared by [optional]: [More Information Needed]
- Language(s) (NLP): [More Information Needed]
- License: [More Information Needed]
Dataset Sources [optional]
- Repository: [More Information Needed]
- Paper [optional]: [More Information Needed]
- Demo [optional]: [More Information Needed]
Uses
Direct Use
[More Information Needed]
Out-of-Scope Use
[More Information Needed]
Dataset Structure
[More Information Needed]
Dataset Creation
Curation Rationale
[More Information Needed]
Source Data
Data Collection and Processing
[More Information Needed]
Who are the source data producers?
[More Information Needed]
Annotations [optional]
Annotation process
[More Information Needed]
Who are the annotators?
[More Information Needed]
Personal and Sensitive Information
[More Information Needed]
Bias, Risks, and Limitations
[More Information Needed]
Recommendations
Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations.
Citation [optional]
BibTeX:
[More Information Needed]
APA:
[More Information Needed]
Glossary [optional]
[More Information Needed]
More Information [optional]
[More Information Needed]
Dataset Card Authors [optional]
[More Information Needed]
Dataset Card Contact
[More Information Needed]