Model tests
When the major providers release new models, we test them on three fixed production tasks: a 3D flight over NMBU, an analysis of the Norwegian Oil Fund and a promotional film. Every model gets the same starting point, so the results can be compared over time.
Other benchmarks Arena has measured all 6 models we have tested, and Epoch AI 5 of them. See what they say.
First: how
A model that writes good answers in a chat window is not necessarily good at finishing a whole piece of work. So we give the models larger tasks of the kind a teacher or student could actually commission, and look at what comes out.
What we add
International benchmarks are useful, but they measure something other than what we test. With us, the task text, the material and the delivery are Norwegian, and it is the finished work that is assessed.
| What | International benchmarks, typically | Our tests |
|---|---|---|
| Language | Mostly English. | A Norwegian task text and Norwegian material, and the model has to write in Norwegian Bokmål. |
| The task | Bounded questions with an answer key, such as the multiple-choice questions in GPQA Diamond, or coding tasks from GitHub, as in SWE-bench Verified. | A whole production task: a 3D experience, an interactive report or a film with slides. |
| The assessment | Automatic against the answer key, or user votes: in Arena, users pick the better of two anonymous answers. | A person opens and uses the delivery, recalculates the key figures and checks every point in the task. |
| The result | A score or a place in a ranking. | Four scores, a separate score for honesty, and the actual time, number of steps and cost at the provider. |
Every model gets the same task text, the same material and the same limits: 30 minutes and 200 steps per task.
The model works in its own container with no network and no access to our systems. It can run code, take screenshots, render video and make presentations.
The run is automatic from start to finish. We log the time, the number of steps and the actual cost at the provider.
We go through every delivery before it is published, and assess whether it works, follows the task, is of good quality and how much follow-up work it needs.
The tasks
The tasks are the same in every round, so a new model can be compared with those tested before.
An interactive 3D flight over Storplenen, Urbygningen and Tårnbygningen in Ås, built with PlayCanvas.
An interactive analysis of the fund's portfolio as at 30 June 2026, built on NBIM's original data.
A finished film with Norwegian voice-over and subtitles about Virtual AI Corp's student content, plus three slides that present the website to a student.
The assessment
We read the whole delivery, recalculate the key figures, open the result as a user would and check every point in the task. A fault is deducted only where it belongs most. Each model gets one attempt per task.
Four scores from 1 to 5. The average of them is the number you see in the comparison.
Does the delivery start, and does it do what it promises? 5 means everything works without faults, 1 means it does not start.
Was everything that was ordered delivered within the limits? Invented figures, facts or sources give 2 at most.
Academic, visual and creative level. 5 means it could be shown to students as it is.
How much we would have to do before we could use it. 5 is nothing, 1 means it has to be redone.
Comparison
A low list price is not the same as a low bill. Gemini 3.8 Flash has the lowest list price of the models, but used the most tokens of them in the runner (18.1 million in) and became the most expensive of those that delivered all three tasks (USD 14.44).
| Model | Drone flight over NMBU | The Oil Fund as a learning resource | 30-second promotional film | Average | Cost |
|---|---|---|---|---|---|
| GPT-6.1 SolOpenAI | 4.75honesty 5 | 4.75honesty 5 | 4.75honesty 5 | 4.75 | USD 2.42 |
| Claude Opus 5.5Anthropic | 4.75honesty 5 | 4.75honesty 5 | 4.75honesty 5 | 4.75 | USD 5.39 |
| GPT-6 SolOpenAI | 4.75honesty 5 | 4.75honesty 5 | 4.75honesty 5 | 4.75 | USD 12.96 |
| Claude Sonnet 5.5Anthropic | 4.0honesty 4 | 4.75honesty 5 | 4.75honesty 5 | 4.5 | USD 3.67 |
| Gemini 3.8 FlashGoogle | 4.25honesty 4 | 4.0honesty 3 | 3.0honesty 2 | 3.75 | USD 14.44 |
| Gemini 3.1 ProGoogle | 2.75honesty 4 | 3.25honesty 4 | 3.0honesty 3 | 3.0 | USD 1.53 |
| Mistral Medium 3.5Mistral AI | 1.5honesty 2 | 2.25honesty 2 | Not delivered within the budget (USD 23)* | 1.88 (2 of 3) | USD 11.93for the two delivered |
| Manual test in the provider's own app | |||||
| Claude Fable 5.1Anthropic · manual test (Claude Code on the Mac, Fable 5.1 (High)) | 4.75honesty 5 | 4.5honesty 5 | 4.5honesty 4 | 4.58 | ≈ USD 44.90estimate, paid through a subscription |
* Not delivered within the budget (USD 23): Stopped on 28 September 2026 after 8.5 minutes without a delivery: the model's total budget of USD 23 for the three tasks had been used up. The average for Mistral Medium 3.5 therefore covers only two tasks.
Manual tests in the providers' own apps have different tools and no fixed time limit, so they have a row of their own. A cost marked ≈ is an estimate: what the token usage in the session would have cost at the provider's list price. The tests were paid for through a subscription.
Higher is better, further left is cheaper. The cost is the average per delivered task. The hollow dot is a manual test with an estimated cost.
The results
Each result has a picture of the delivery, the scores and what we thought was best and weakest. Open a result to see the whole delivery, the check report and the full assessment.
Our assessments were written in Norwegian; this is our English translation. Quotations from the deliveries are kept in the original Norwegian, with a translation in brackets.
Task 1 of 3 · 8 results
An interactive 3D flight over Storplenen, Urbygningen and Tårnbygningen in Ås, built with PlayCanvas.
4.75 out of 5
Best The most detailed drone in the test: Urbygningen with round-arched windows, gable pediments and a double main staircase down towards Speildammen, Tårnbygningen with a green lantern, and a striped lawn with trees, benches and lamp posts.
Weakest No touch controls for free flight on mobile (stated in the README).
4.75 out of 5
Best The most recognisable drone: arched windows and a wide main staircase on Urbygningen, the lantern in the middle of Tårnbygningen, fan-shaped flower beds, Speildammen and the sunken lawn, with name signs at the places.
Weakest No touch controls for mobile.
4.75 out of 5
Best Clearly recognisable buildings: the gables, round windows and dormer windows of Urbygningen, and Tårnbygningen with the lantern in the middle.
Weakest No touch controls, so it does not work on mobile.
4.25 out of 5
Best Recognisable buildings: Tårnbygningen with the lantern in the middle, Urbygningen with its clock and gables, and the fan-shaped flower beds on Storplenen.
Weakest The check report calls the buildings “nøyaktig modellert” (“accurately modelled”) and says that “samtlige krav” (“all requirements”) are met, before calling it a stylised reconstruction further down.
4.0 out of 5
Best A recognisable campus: Urbygningen with projecting gabled bays and dormer windows, Tårnbygningen with a green lantern and spire, the sunken lawn with a ring path, fan-shaped flower beds and Speildammen, plus a minimap and place names.
Weakest The “Start kameratur” (“Start camera tour”) button does nothing: it has two click listeners, one of which starts the tour while the other stops it immediately. The same applies to “Kjør kameraturen på nytt” (“Run the camera tour again”). The tour can only be started with T.
2.75 out of 5
Best Starts without console errors; flying, the height limit, reset and the camera tour work; 60 fps on the Mac.
Weakest The buildings are windowless blocks, and the places are hard to recognise.
1.5 out of 5
Best Starts without console errors, with the engine copied from the starter project.
Weakest The buildings are blocks with severe shadow artefacts, and the places cannot be recognised.
4.75 out of 5
Best Recognisable buildings: the pavilion gables, the clock gable and the portal on Urbygningen, and the lantern in the middle of Tårnbygningen. The places are in the right position because the Kartverket map is georeferenced.
Weakest The main staircase stands loose in the grass well in front of the façade, and the terrace in front of Urbygningen is grass.
Task 2 of 3 · 8 results
An interactive analysis of the fund's portfolio as at 30 June 2026, built on NBIM's original data.
4.75 out of 5
Best All key figures are correct when recalculated: equities NOK 16,435.35 billion and fixed income NOK 6,087.66 billion, USA 54.47%, technology 32.09%, three largest sectors 60.68%, top 10 20.88%, top 100 47.84%, NVIDIA 3.72%, HHI 65.14 and effective number of holdings 153.5; real estate and infrastructure with one country total per country (371.2 and 83.4 billion).
Weakest Country names are left in English (NBIM's), while sector names are translated into Norwegian.
4.75 out of 5
Best All key figures are correct when recalculated from the NBIM files: equities NOK 16,435.3 billion and fixed income NOK 6,087.7 billion, USA 54.5%, technology 32.1%, ten largest 20.9%, 115 companies for half the value, effective number of holdings 154, US technology 22.9% and 538 companies with a different country of incorporation.
Weakest The page scrolls sideways on mobile (577 px of content in a 375 px width); the model itself states that mobile has not been tested.
4.75 out of 5
Best All file totals match the control figures, and the reconciliation against both Table 1 and the balance sheet shows each discrepancy (+77, +228 and −21 billion) without forcing them to agree.
Weakest Country names are left in English, while the sectors are translated into Norwegian.
4.75 out of 5
Best All key figures are correct, and the reconciliation shows each discrepancy against Table 1 and the balance sheet (equities +77.35 and fixed income +227.66 billion) without forcing them to agree.
Weakest Sector and country names are left in English.
4.0 out of 5
Best All key figures are correct (equities NOK 16,435.3 billion, top 10 20.88%, technology 32.09%), and the reconciliation covers all four asset classes against both Table 1 and the balance sheet.
Weakest Brings in a fact that is not in the package (a strategic equity share of about 70 per cent).
3.25 out of 5
Best All key figures are correct when recalculated (equities NOK 16,435 billion, USA 54.5%, technology 32.1%, real estate and infrastructure against the balance sheet).
Weakest The source code for the data processing is outside the delivery, and there is no separate description of the method.
2.25 out of 5
Best The figures in the text are correct: USA 54.47%, technology 32.09% and the top 10 at 20.88%.
Weakest None of the charts are drawn, because the Chart.js file is the wrong variant for the way it is loaded.
4.5 out of 5
Best All key figures and report figures are correct when recalculated (equities NOK 16,435 billion against 16,358 in Table 1 and 16,505 in Note 5.1; USA 54.5%, technology 32.1%, top 10 20.9%, 115 companies for half the value; sector discrepancies under 0.2 pp against Table 5).
Weakest The introduction states as fact that the files leave out cash, derivatives and unsettled trades, which the package says is not documented and which the model elsewhere calls an interpretation.
Task 3 of 3 · 7 results
A finished film with Norwegian voice-over and subtitles about Virtual AI Corp's student content, plus three slides that present the website to a student.
4.75 out of 5
Best A strong hook: “Hvem vil du spørre først?” (“Who will you ask first?”) with a colleague graph that grows from the first frame, then the office, “Still spørsmål. Vurder svarene. Ta en beslutning.” (“Ask questions. Assess the answers. Make a decision.”) and an end card with the address that stays for over six seconds.
Weakest The address is shown but not read aloud.
4.75 out of 5
Best Technically exact: H.264 High + AAC, 1920 × 1080, 900 frames = 30.000 s, −17.1 LUFS and true peak −4.2 dBTP; the subtitles start within 0.02 s of the voice-over (measured with silencedetect).
Weakest The script is 52 words, slightly under the brief's “ca. 55–70” (“about 55–70”).
4.75 out of 5
Best Every claim in the script is traced back to its point in the approved brief, and the call to action is exactly “Prøv en simulering – uten konto og uten kode” (“Try a simulation – no account and no code”).
Weakest The voice-over ends at 24.9 s, so the end card stands still for five seconds without sound.
4.75 out of 5
Best The script sticks strictly to the approved brief and ends with “Prøv en simulering uten konto” (“Try a simulation without an account”) and the address on screen.
Weakest In some scenes the subtitles sit right on top of the bottom bar reading “Virtual AI Corp / Simuleringer” (“Simulations”).
3.0 out of 5
Best The script sticks to the approved brief (64 words) and ends with “Prøv en simulering på virtualaicorp.no/forhandsvisning” (“Try a simulation at virtualaicorp.no/forhandsvisning”).
Weakest Wrong resolution: 1280 × 720.
3.0 out of 5
Best Technically exact: 900 frames = 30.000 s, H.264 High + AAC, 1920 × 1080; the voice-over (24.2 s) fits, and the subtitles follow it to within 0.1 s.
Weakest Invented case content: three “kollegaens innspill” (“colleague's input”) in scene 1 (budget overrun, data discrepancy, deadline) and the roles and attitudes on slide 2 are not in the package; the case in the student video is a different one.
4.5 out of 5
Best Factually accurate: every claim is traced back to the student brief, and the call to action “Prøv en simulering” (“Try a simulation”) and the address virtualaicorp.no/forhandsvisning are the approved ones.
Weakest The serif typeface loses the crossbar in H and A: the hook reads “I lvem ville du spurt?” and the wordmark “Virtual Λ/ Corp”.
not assessed
Not delivered within the budget (USD 23)
Stopped on 28 September 2026 after 8.5 minutes without a delivery: the model's total budget of USD 23 for the three tasks had been used up.
Other benchmarks
Our own tests show how the models solve three concrete tasks. Here is what independent, international benchmarks say about the same models. The figures are measured by others, with other tasks and settings, and have not been checked by us.
International benchmarks of the models we have tested, with the providers' list prices at the end. The highest value among our models is highlighted.
| Model | Arena textArena · 02/10/2026 | Arena codeArena · 02/10/2026 | Arena agentArena · 02/10/2026 | ECIEpoch AI · 04/10/2026 | List priceUSD per 1M tokens, in / out |
|---|---|---|---|---|---|
| Claude Opus 5.5Anthropic | 1,504high · 4,552 votes | 1,815max · 2,062 votes | 0.138high · 5,359 sessions | 167.4 | 4 / 20 |
| GPT-6 SolOpenAI | 1,457max · 8,787 votes | 1,689max · 3,411 votes | 0.097max · 6,106 sessions | not measured | 2 / 10 |
| Claude Fable 5.1Anthropic | 1,501max · 11,800 votes | 1,749max · 6,318 votes | 0.143max · 15,130 sessions | 164.8 | 10 / 50 |
| Gemini 3.8 FlashGoogle | 1,495high · 26,298 votes | 1,583high · 8,524 votes | 0.030high · 34,396 sessions | 156.9 | 0.75 / 3.75 |
| Gemini 3.1 ProGoogle | 1,487121,806 votes | 1,44624,347 votes | -0.07786,483 sessions | 154.8 | 2 / 12 |
| Mistral Medium 3.5Mistral AI | 1,42711,757 votes | 1,2632,322 votes | -0.12411,156 sessions | 141.4 | 1.65 / 8.25 |
Higher is better in all the benchmarks, and for list prices lower is better. The variant the model was measured with is shown below the figure, because the sources often measure a different reasoning level from the one we use (“high”). Arena scores with few votes are uncertain.
Last updated: Epoch AI 04/10/2026 · Arena 02/10/2026. Sources: Epoch AI (CC BY 4.0, fetched 4 October 2026); Arena (LMArena) (CC BY 4.0, fetched 4 October 2026). Selected and rounded by us.
The table shows how the models fare outside our three tasks. Arena is based on user votes, and Epoch AI combines many tests with answer keys into one index.
The benchmarks use other tasks, in English and often at a different reasoning level from ours. Use them alongside our results, not instead of them.
All international benchmarksThe models
The models are run through the providers' own APIs. The tests contain only public material and no information about students or users of the tool. The table shows where each provider processes what is sent. More models are added as they are tested.
| Model | Provider | List priceUSD per 1M tokens, in / out | Where data is processed |
|---|---|---|---|
| Claude Fable 5.1 | Anthropic | 10 / 50 | USA (Anthropic). Requires 30 days of retention by the provider. |
| Claude Opus 5.5 | Anthropic | 4 / 20 | USA (Anthropic). |
| Gemini 3.1 Pro (preview) | 2 / 12 | Google (no EU region in the Developer API). Abuse logs kept for 55 days. A paid key is required in the EEA. | |
| Mistral Medium 3.5 | Mistral AI | 1.65 / 8.25EU endpoint, 10% more expensive | EU EU (Mistral AI, France), via the EU endpoint. |
| Claude Sonnet 5.5 | Anthropic | 2 / 10 | USA (Anthropic). |
| GPT-6.1 Sol | OpenAI | 2 / 10 | USA (OpenAI). Abuse logs kept for 30 days. Nothing is stored in the API (store: false). |
| GPT-6 Sol | OpenAI | 2 / 10 | USA (OpenAI). Abuse logs kept for 30 days. Nothing is stored in the API (store: false). |
| Gemini 3.8 Flash | 0.75 / 3.75introductory price until 31 Dec 2026 | Google (no EU region in the Developer API). Abuse logs kept for 55 days. A paid key is required in the EEA. |
Security
The models get no login and no keys. The calls to the providers and the speech generation are made by our runner outside the model's working environment. The deliveries are shown from the storage service's own address in a restricted frame, not from virtualaicorp.no, so the code the models have written never runs side by side with our login.
