I Gave GPT-6.1 Sol and GPT-6 Astra the Same Room and the Same Brief. They Tied on Every Graded Test, and the Test That Finally Told Them Apart Was the One I Stopped Grading.

Note: No affiliate links and no sponsor. One paid account, mine: ChatGPT Plus, on 14 October 2026. Both models live in the same place, the Work tab, and both showed the same effort setting, Light, which I did not change. Every file below came off those runs and is saved. The prices below are OpenAI's published figures. I cannot measure token cost from inside a subscription.

OpenAI's own page for GPT-6.1 Sol puts the claim in the headline: "Near-Astra intelligence for a fifth of the price." The body says Sol "nearly matches GPT-6 Astra's intelligence on agentic coding, computer use, and professional work at one-fifth of Astra's standard input and output token prices." The cards underneath give the numbers: Astra at $10.00 input and $50.00 output per million tokens, Sol at $2.00 and $10.00.

The claim, from the vendor's own page. One-fifth is exact: $10 → $2 in, $50 → $10 out.

"Nearly matches" is the kind of sentence that sounds measurable. So I measured it, badly, and then measured it again.

First I ran the test I already knew was worthless

I wrote three tasks with answer keys. A daylight calculator with six numbers to hit. An area schedule with seven answers to compute. A window schedule with two impossible dimensions buried in it, to see whether either model would notice.

Then I ran each task twice per model and graded the output by machine. Not by reading it: by opening each generated file in a browser and comparing the numbers on screen against a key I had written first.

Task 1  daylight tool     Sol 6/6, 6/6     Astra 6/6, 6/6
Task 2  area schedule     Sol 7/7, 7/7     Astra 7/7, 7/7
Task 3  trap task         Sol 8/8, 8/8     Astra 8/8, 8/8

Twelve runs, twelve full marks.

On task 2 it was worse than a tie. I sliced the four answers out of the four conversations and hashed them. All four produce the same SHA-256: 35e43dff183372eb…. Not similar. Identical, character for character:

1507.4
183.83%
542.6
4
6
10
6, 7, 8

The trap task was the one I had most hope for. I told both models the ceiling was 2.70 m and then told them the window head was at 2.90 m, which is above the ceiling, and which also disagrees with the sill at 0.90 m plus a window 1.50 m tall. Both models caught both contradictions, in writing, on screen, and both resolved them the same way — keep the sill and the window, derive the head, use 2.40 m.

Astra's version of the trap task. Sol's wording differs; the decision does not.

A month ago I published a post that opens with this exact failure. Both models scored six out of six on a tightly constrained brief, I wrote it up as a finding, and then I had to say in print that a test both sides ace has measured nothing. I wrote that sentence, published it, and then built this test the same way anyway.

So the twelve runs above are not the result. They are the control, and the lesson they carry is about me.

Then I took the grading away

The second brief fixes the room and nothing else:

I have one rectangular room: 7.20 m long, 4.80 m wide, 2.70 m floor-to-ceiling.
There is one window opening, 2.40 m wide and 1.50 m high, and I have not
decided where to put it yet.

Build me something that helps me decide where that window should go.

The only things fixed are those dimensions. Everything else is your call: what
the thing does, how it looks, what it measures, whether it is interactive. Make
it a single self-contained HTML file I can open straight from disk, and load any
library you want from a CDN if that makes it better.

Two runs each, four files, all four opened on my own machine.

Sol builds a light meter

Both of Sol's runs came back with the same title, Window Lab — placement explorer, and the same idea. You slide the window along a wall, and a heatmap of the floor repaints to show how much sky each point can see.

Sol, run 1. The numbers on screen: mean sky-light fraction 4.04%, minimum/mean 0.02, 4.46% at the inspection point.

It gives you a Find highest mean button that searches for the best position, a table to save alternatives against each other, and, the part I did not expect, a section headed Assumptions you can inspect that tells you the model behind the picture: the whole 3.60 m² opening transmits light, no glass loss, no frame, no sun, and the integration uses 24 × 16 area samples per window against a 36 × 24 grid of cell centres.

It also tells you what it is not: "This is a placement study, not a daylight-factor, lux, glare or compliance calculation."

Astra builds a sightline study

Astra's two runs are the same instrument as each other and a different instrument from Sol's. You are not a light meter here. You are a person sitting in the room.

Astra, run 1. Pick a seat, pick an eye height, read the geometry.

It puts a seat in the plan, draws the view cone from that seat through the opening, and reports what that costs you: 5.37 m from the farthest floor point to the opening, 41.11° of your field of view filled by the window, 2.40 m of unbroken wall left beside it. Eye height is a slider with two labels, 1.20 m seated and 1.60 m standing. The elevation underneath is dimensioned like a drawing: sill 0.90, head 2.40, 0.30 m of wall above the head.

Both files can print a comparison. The difference is what they hand you before that: Sol searches for the best position, Astra asks where you will be sitting.

The difference that repeated

The two second runs look like the first two, and the split holds in the source:

Sol reached for a canvas both times. Astra reached for SVG both times. That is not a style note; it is the whole difference in one line. A canvas is for painting a field of values, which is what a light map is. SVG is for drawing lines you can measure, which is what a plan and an elevation are. Each model picked the surface that matches the question it decided to answer, and each picked the same one twice.

Two runs per model is two runs per model. I am not telling you this is a law. I am telling you it held in all four files, and that the files are saved.

Sol, run 2. Same title as run 1, down to the words.

Astra, run 2. New name, same instrument.

Three things neither of them did

Neither used the CDN I offered. The brief says to load any library you want. All four files contain zero external src or href links. Both models chose to ship something that works with the network off, and both said so unprompted. Sol's phrasing was "Works entirely offline", Astra's was "It works fully offline."

Three of the four admitted they had not seen it run. Sol's second run: "JavaScript and calculation checks passed. Browser testing was unavailable." Astra's two: "Geometry and control checks passed; browser visual verification wasn't available" and "Calculations and placement limits checked; visual browser verification was unavailable." Only Sol's first run skipped the disclaimer. They wrote the file, verified the arithmetic, and handed it over without having seen it. Opening the file is the part that is still mine.

Neither drifted off the dimensions. 7.20 × 4.80 × 2.70 and a 2.40 × 1.50 opening are in all four files, and visible on all four screens.

What about speed

This is where my own instrument is the weakest thing in the test, so here is how it broke.

My script polls the page every five seconds and then waits another twenty seconds of silence before calling a run finished. That inflates every number by up to 25 seconds. On the graded tasks it was worse: a stale selector returned zero characters, so one Sol run sat in the loop until the cap and logged 421 seconds for work that did not take 421 seconds. I threw that number away.

Then I noticed ChatGPT prints its own timer above each answer: "3분 31초 동안 작업함", worked for 3 minutes 31 seconds. That is the vendor's number, not mine, and it is the one to use:

open brief, vendor's own timer
  Sol    211 s, 204 s     mean 207.5
  Astra  191 s, 229 s     mean 210.0

They overlap. On the graded tasks my inflated instrument had Astra ahead in every pair (75 s and 80 s against 100 s; 105 s and 90 s against 121 s and 105 s), and I was ready to write that Astra is faster. The open brief does not reproduce it. One instrument, two tasks, two directions. I am not claiming a speed difference in either direction.

So is "near-Astra" true

On what I can measure, from inside a subscription, on three graded tasks and one open one: I could not find the gap. Twelve graded runs scored the same. Four open runs produced two coherent tools, one per model, both honouring the brief, neither broken.

What I cannot measure is the half of the sentence that matters to a developer. Price lives in the API, and a Plus subscription shows no tokens and no bill. If you are choosing between $2 and $10 per million input tokens, nothing above is your test. It is a test of whether the cheaper model embarrasses itself on ordinary professional work, and across sixteen runs in one afternoon I could not make it.

The difference I did find is not on the price card. Ask the two models the same open question and you get a light meter from one and a sightline study from the other. If you have a view problem, the cheap model will hand you a beautiful answer to a different question, and so will the expensive one, if your problem is light.

Where I was wrong

Three of my own errors are in this run, and two of them changed numbers I had already written down.

I wrote the answer key wrong. My daylight key used floor area, 34.56 m², as the denominator A and expected a daylight factor of 7.65%. Both models used the total internal surface area (floor, ceiling and four walls, 133.92 m²) and got 1.97%. The standard formula uses the surface area. The models were right and my key was wrong. My prompt never defined A, and both models printed their interpretation on screen rather than hiding it. I corrected the key and kept my wrong version in the file beside it.

My grader marked a right answer wrong. On the trap task it reported that Sol had not flagged the contradiction. Sol had, in the words "the entered head is above the ceiling" and "does not equal sill + window height", and my regular expression was only looking for words like impossible and conflict. A grader that misses the answer because of vocabulary is not a grader. I widened it and re-ran all three files.

I said I had fixed something that was not fixed. Mid-run I patched the model-selector logic, reported it as done, and a later patch quietly restored the old code. The next run opened the wrong menu, which means it would have asked Astra a question and filed the answer under Sol. It did not, because the script checks the model name in the composer before it types anything and aborted. What saved that run was a check in the script. My memory of having fixed it would not have.

If you want to run this yourself

Fix your dimensions. Write your answer key before you look at any output, and define every symbol in it. Mine failed on a single undefined letter. Then write a second brief with the constraints taken out, because the graded one will probably tie.

And open the file. Three of the four runs told me, in their own words, that they had not.