2bit and 4bit Aider Polyglot scores 60/62.7

#3
by Fernanda24 - opened

UD-Q4_K_XL:

  • dirname: 2025-07-31-09-05-03--DeepSeek-Tng-R1T2-Chimera-UD-Q4_K_XL-de
    test_cases: 225
    model: openai/Deepseek-Tng-R1T2-Chimera
    edit_format: diff
    commit_hash: f38200c
    pass_rate_1: 27.6
    pass_rate_2: 62.7
    pass_num_1: 62
    pass_num_2: 141
    percent_cases_well_formed: 95.6
    error_outputs: 13
    num_malformed_responses: 13
    num_with_malformed_responses: 10
    user_asks: 84
    lazy_comments: 0
    syntax_errors: 0
    indentation_errors: 0
    exhausted_context_windows: 0
    prompt_tokens: 2325961
    completion_tokens: 953907
    test_timeouts: 3
    total_tests: 225
    command: aider --model openai/Deepseek-Tng-R1T2-Chimera
    date: 2025-07-31
    versions: 0.85.3.dev
    seconds_per_case: 210.6

UD-Q2_K_XL:

  • dirname: 2025-07-04-06-01-08--DeepSeek-TNG-R1T2-Chimera-UD-IQ2_M
    test_cases: 225
    model: openai/DeepSeek-TNG-R1T2-Chimera-UD-IQ2_M
    edit_format: diff
    commit_hash: 3db4d37
    pass_rate_1: 21.3
    pass_rate_2: 60.0
    pass_num_1: 48
    pass_num_2: 135
    percent_cases_well_formed: 97.3
    error_outputs: 9
    num_malformed_responses: 9
    num_with_malformed_responses: 6
    user_asks: 106
    lazy_comments: 0
    syntax_errors: 0
    indentation_errors: 0
    exhausted_context_windows: 0
    prompt_tokens: 2929500
    completion_tokens: 968958
    test_timeouts: 4
    total_tests: 225
    command: aider --model openai/DeepSeek-TNG-R1T2-Chimera-UD-IQ2_M
    date: 2025-07-04
    versions: 0.85.2.dev
    seconds_per_case: 177.5

Sign up or log in to comment