Review digest

30 tests across 6 categories. promptfooconfig.yaml

Scan each category: is this the RIGHT set of checks, is the math right, what is MISSING? Each line below is the plain-language summary of one test; expand it for the exact prompt and grading.

Every prompt below is sent through one shared wrapper: {{input}} The per-test prompt is the {{input}} shown in each expansion.
blender-api (2)Blender 5.2 API: models are graded on apply Blender 5.2 API changes and constraint-evaluation rules correctly.

floor (2)

checks the model picks the rotation-axis order that stops a bent wrist from polluting a twist measurementfloor
Euler order to isolate twist from bend (given the contamination rule)
Prompt (input)
A Copy Rotation constraint with only the Y axis enabled makes a twist bone take a share of a wrist's roll. Blender applies Copy Rotation on an euler decomposition. With the default XYZ order, Y is the middle term, so the wrist's BEND contaminates the roll. To make the decomposition behave like swing-twist, which euler order should the constraint use, and why?
Grading
output must contain "YXZ" (case-insensitive)
judged by rubric
Explains that putting the twist (Y) axis FIRST in the decomposition isolates roll from the bend.
checks the model recognizes an addon using removed Blender APIs will fail mid-build and should not be installedfloor
Removed Blender 5.2 APIs in an old addon (given the removal facts)
Prompt (input)
You are on Blender 5.2. In 5.2, Bone.select was removed (use PoseBone.select) and armature.data.layers was removed in 4.0 (use armature.data.collections). An addon's manifest declares it targets Blender 2.80, and its code calls Bone.select and armature.data.layers in eight places while building an armature. What happens if you run it, and should you install it?
Grading
judged by rubric
Says it will error partway through and leave a half-built armature, advises against installing it, and maps the removed calls to PoseBone.select and data.collections.
ik-solver (5)IK solver behavior: models are graded on reason about inverse-kinematics behavior - singularities, degrees of freedom, reach limits, and pole vectors.

floor (2)

checks the model computes how far a limb is being asked to stretch and flags an impossible reachfloor
IK reach ratio and reachability
Prompt (input)
A two-bone leg IK chain has an upper bone 248 mm long and a lower bone 220 mm long. The IK target is placed 485 mm from the hip (straight-line distance). Compute the reach ratio (target distance divided by total bone length) and state whether the target is reachable.
Grading
custom javascript check on the model's output
judged by rubric
Concludes the target is NOT reachable / is beyond full extension (ratio above 1.0).
checks the model knows moving a pole target nearer or farther in the same direction does not change the posefloor
Pole target distance does not affect the solve
Prompt (input)
In a pole-vector IK setup the bend plane is defined by three points: the chain root, the IK target, and the pole. Someone slides the pole from 3000 mm away from the elbow to 250 mm away, keeping it in the same direction. How much does this change the solved joint positions, and why?
Grading
judged by rubric
States the solve is essentially unchanged (about 0 mm) because only the pole DIRECTION matters, distance is ergonomic.

discriminating (3)

checks the model diagnoses why IK silently does nothing on a perfectly straight limb and how to fix itdiscriminating
IK does nothing on a perfectly straight (collinear) chain
Prompt (input)
You set up a two-bone IK chain on a T-posed arm. Moving the IK target does nothing: the hand stays put across all pole angles. The constraint reports is_valid=True, influence=1.0, mute=False, and no error. The identical code on the legs works fine. Diagnose the cause and give a fix.
Grading
graded rubric (3 criteria, all must hold)
  • Identifies the T-posed arm as collinear / straight, a singularity with no defined bend plane, so the solver stays at full extension
  • Explains the legs work because they import with a natural knee bend, so they are not collinear
  • Fix is to seed a few degrees of bend into the stored / rest pose so the solver has a bend direction
checks the model explains why an IK arm mathematically cannot store forearm twist, and how to get it backdiscriminating
IK cannot store forearm roll (degrees of freedom)
Prompt (input)
In FK you twist the forearm 45 degrees about its own axis, then switch the arm to IK, and the twist vanishes and cannot be recovered by fixing the addon. The chain is a 3-DOF upper arm plus a forearm that is a pure hinge (X and Y locked). Explain, in terms of degrees of freedom, why IK cannot hold the forearm roll, and give two fixes.
Grading
graded rubric (3 criteria, all must hold)
  • Counts degrees of freedom - the chain has 4 (upper arm 3 plus forearm hinge 1) and the IK goal pins 3 (target position) plus 1 (pole), fully determining it
  • Concludes roll about the forearm's own axis is not in the IK solution space, so switching to IK re-derives and discards it
  • Gives two fixes - stay in FK for the arms, or add twist bones OUTSIDE the IK chain whose roll is a free channel the solver never touches
checks the model explains why a 4 mm asymmetry wrecks a nearly-straight arm and prescribes mirroringdiscriminating
Near full extension a 4 mm error swings the arm 16.7 degrees
Prompt (input)
An auto-rigger fitted the left and right arms independently; their hand positions differ by about 4 mm. One arm ends up at 100% reach (straight) and the other at 95% with a 70 mm elbow break and a 16.7 degree upper-arm direction mismatch, and no pole angle can fix it. Explain why such a tiny difference produces such a large error, and give the construction fix.
Grading
graded rubric (3 criteria, all must hold)
  • Explains a two-bone chain is extremely sensitive near full extension, where a few mm of reach flips it between straight and a large elbow break
  • Notes no pole angle can fix it because the TARGET itself is misplaced, not the bend plane
  • Fix is to build one side and MIRROR it to the other - symmetry by construction, not by independent measurement
image-3d-pipeline (6)image-to-3D pipeline: models are graded on diagnose reference-image and upload pitfalls in the image-to-3D and auto-rig pipeline.

floor (6)

checks the model knows a combined front+back reference image must be split before 3D generationfloor
Two-up reference sheet becomes two characters
Prompt (input)
An image-to-3D generator is handed a single reference image that contains BOTH the front and back view of a character side by side. It produces a scene with two separate figures standing next to each other instead of one character. What is the fix?
Grading
judged by rubric
Says to split the sheet into one image per view and assign each to its own front/back direction slot.
checks the model diagnoses a misleading upload error and switches to a format that avoids itfloor
Mixamo misreads an unrigged FBX as already-rigged
Prompt (input)
You upload a clean, unrigged FBX (one mesh, zero armatures, zero vertex groups) to an auto-rigger and it fails with "unable to map your existing skeleton", even though there is no skeleton in the file. The service chooses between an auto-rig path and a map-existing-rig path by sniffing the file. Explain the likely cause and give a reliable fix.
Grading
judged by rubric
Explains the service misdetected the file as already-rigged and took the wrong branch; fix is to upload OBJ (which has no skeleton concept) so it cannot take that branch.
checks the model poses reference hands so the thumb is actually visible to the 3D reconstructionfloor
Palms-down T-pose hides the thumbs from reconstruction
Prompt (input)
You are preparing a T-pose reference sheet for image-to-3D reconstruction. In a T-pose with palms DOWN, the thumb points forward and is foreshortened to nothing in the front view and hidden behind the hand in the back view, so the reconstruction builds a flat four-finger paddle with no thumb. How should the hands be posed in the reference, and why?
Grading
judged by rubric
Says to pose palms forward, thumbs up, fingers spread, with visible palm thickness, so both views actually see the thumb.
checks the model handles a costume detail that image generation cannot keep on the same side across viewsfloor
An asymmetric detail flips sides between the two views
Prompt (input)
Across several image-to-3D attempts, an asymmetric costume detail (a knot on one hip) keeps landing on the character's LEFT in the front view and her RIGHT in the back view. Image generation cannot hold an asymmetric detail consistent across two viewpoints. What is the fix?
Grading
judged by rubric
Says to remove the asymmetric part from the reference and add it later in Blender as separate geometry (where it needs its own bone chain and spring sim anyway).
checks the model catches a transparent-background image flattening to black and killing the silhouettefloor
A transparent PNG background flattens to black and kills the silhouette
Prompt (input)
You flatten an RGBA reference sheet by sampling its background colour, and the reconstruction reads almost no silhouette. The costume is near-black. Sampling the background colour of a transparent PNG returns (0,0,0,0). What went wrong, and what should the background be?
Grading
judged by rubric
Explains the transparent background flattened to black behind a near-black costume, leaving no silhouette; fix is to force a light neutral grey (not black), high-contrast against the outfit.
checks the model prevents painted-in lighting from doubling with the 3D scene lightsfloor
De-light and PBR, so painted lighting does not double
Prompt (input)
Reference art has cel shading painted into it. The material generator bakes that painted lighting into the texture, then a three-point lighting setup is applied on top, so shadows double (a shadow under the jaw from the drawing PLUS one from the key light). What two settings prevent this?
Grading
judged by rubric
Says to enable de-light (strip the baked lighting to flat albedo) and to download PBR, not Shaded (Shaded is the baked-lighting version that undoes de-light).
measurement-proxy (9)measurement design: models are graded on choose a measurement that actually tracks the goal, and reject a proxy that can pass while the real thing is broken.

discriminating (9)

checks the model rejects a measurement that cannot see the effect it claims to testdiscriminating
matrix_basis reads pre-constraint, cannot see a constraint effect
Prompt (input)
You need to verify that a Copy Rotation twist constraint actually fired on a bone. A colleague reads the bone's matrix_basis, sees 0.0 degrees on all axes, and concludes the constraint did nothing. Is that conclusion sound? If not, what should be measured instead, and why?
Grading
graded rubric (3 criteria, all must hold)
  • States that matrix_basis is the PRE-constraint local transform, so it reads ~0 whether the constraint fired or not and cannot indicate a constraint effect
  • Says to measure the bone's actual axis / orientation in evaluated WORLD space instead
  • Does not accept the colleague's conclusion at face value
checks the model notices a test is sampling the wrong part of the mesh and would miss the real problemdiscriminating
Sampling the wrong region misses the collapse zone
Prompt (input)
You added forearm twist bones to fix a wrist that collapses under roll. To confirm the fix you sample the mesh cross-section at points 20% to 80% along the forearm and get 99% (looks solved). The skin weights that blend from the forearm bone to the hand bone actually span 80% to 125% of the forearm length. Is your measurement region right? What would you measure?
Grading
graded rubric (3 criteria, all must hold)
  • Identifies that the sampled region (20-80%) misses the actual collapse / blend zone near the wrist (80-125%), so it samples skin that never moves
  • Recommends measuring the actual weight-blend zone, and ideally the whole limb length before and after, to catch a defect that merely moved upstream
  • Does not conclude the fix is verified from the 99% reading
checks the model sees that a position-only check can pass while the mesh is twisted, and asks for orientation toodiscriminating
Position-only IK calibration leaves roll free
Prompt (input)
IK pole angles were chosen by forcing a slight bend and minimizing the distance from each joint to its pole target. Every position-based assertion (joint locations, hand location) passed. Later a render showed the whole limb mesh was twisted. What did the position-only calibration fail to constrain, and how should the calibration be judged instead?
Grading
graded rubric (3 criteria, all must hold)
  • Explains that fixing distance / position constrains the bend PLANE but leaves the bone's ROLL about its own axis free
  • Notes a rolled bone twists the skinned mesh WITHOUT moving the joint, so every position assertion passes on a destroyed rig
  • Says to calibrate / assert ORIENTATION (for example against the FK pose), not just position
checks the model treats an impossible percentage as a counting bug, not a real resultdiscriminating
A percentage above 100 is a counting bug
Prompt (input)
A script reports that a fix achieved 153% of its target coverage. Before celebrating, what is the most likely explanation for a percentage above 100%, and what would you check?
Grading
graded rubric (3 criteria, all must hold)
  • Treats a value above 100% as a measurement / normalization error rather than a real over-achievement
  • Points at the DENOMINATOR (or the counted set) being wrong, for example excluding some items that belong in it
  • Proposes checking what is counted in the numerator versus the denominator before trusting the number
checks the model picks a measurement that truly means foot-flat-on-floor rather than a convenient stand-indiscriminating
Toe-planted: hold orientation, not lowest vertex height
Prompt (input)
You are building a foot-roll control and want to verify the toe stays planted on the floor as the foot rolls. One approach measures the height of the lowest toe VERTEX and keeps it at floor level. Is that the right thing to hold constant for "the toe stays planted", or is there a better quantity? Explain.
Grading
graded rubric (3 criteria, all must hold)
  • Recognizes that holding the lowest vertex height can be satisfied while the toe still rotates / pitches, so it does not actually mean planted
  • Proposes holding the toe's ORIENTATION (world pitch of the toe) constant as the correct criterion
  • Frames this as choosing a proxy that actually tracks the goal, not the easiest thing to measure
checks the model finds the real defect hidden behind a scary but expected worst-case numberdiscriminating
Global worst-case hides the real defect
Prompt (input)
After adding joint limits to a posed rig, a check reports "worst bone moved 56 mm" and looks like a disaster. Broken down by region: hands/feet moved 0.02 mm, elbows/knees moved 56 mm, fingers moved 10.6 mm. Locking the off-axes at the elbow/knee is expected to change the solve there. Which number is the real bug, and what does this say about the global-worst metric?
Grading
graded rubric (3 criteria, all must hold)
  • Identifies the finger drift (~10.6 mm) as the real bug and the 56 mm elbow/knee as expected from locking the off-axes
  • Explains a single global worst-case number conflates an expected benign change with the actual defect
  • Recommends judging drift per region against what each region should do
checks the model rejects a straightness test that two unrelated numbers happen to satisfydiscriminating
Span equals summed length cannot detect a bend in unconnected bones
Prompt (input)
To decide whether a finger's bone chain is bent or straight, someone measures the straight-line distance from the first joint to the last (66.5 mm) and the sum of the bone lengths (66.5 mm), sees they match, and concludes the finger is straight. The finger bones are UNCONNECTED (each child's head is not at its parent's tail). Is the conclusion valid? What should be measured?
Grading
graded rubric (3 criteria, all must hold)
  • States that with UNCONNECTED bones the head-to-head span and the summed bone length are unrelated, so their match proves nothing about straightness
  • Recommends measuring the ANGLE between consecutive bone directions instead
  • Does not accept that the finger is straight from the matching numbers
checks the model catches a symmetry error that is really just the whole skeleton sitting off-centerdiscriminating
Apparent asymmetry is really an off-center skeleton
Prompt (input)
Checking a character for left/right symmetry, a script compares world X positions of mirrored bone pairs and finds an apparent 80.4 mm error, which it halves to report "21.5 mm shoulder asymmetry" and recommends replacing the model. The whole skeleton sits 40.2 mm off the world origin along X. Is the asymmetry real? What was measured wrong?
Grading
graded rubric (3 criteria, all must hold)
  • Recognizes the 80.4 mm is about twice the 40.2 mm world offset, an artifact of the skeleton not being centered on x=0, not a real asymmetry
  • Says to find the skeleton's true center line first (for example from the hips) and measure symmetry about THAT
  • Rejects the recommendation to replace the model on this evidence
checks the model distrusts a perfect score until it confirms the test actually looked at anythingdiscriminating
A perfect score can mean the check examined nothing
Prompt (input)
A left/right symmetry check reports a perfect 0.0% mismatch. The comparison relies on matching bone names by replacing "Left" with "Right". Before trusting the perfect score, what must you confirm, and what general failure does a perfect / 0.0% result often hide?
Grading
graded rubric (3 criteria, all must hold)
  • Says to confirm the check actually compared a NON-ZERO number of pairs (for example a case mismatch could match zero pairs)
  • Explains that a check which silently examines nothing looks identical to a check that passes
  • Recommends always printing or asserting the sample size
skinning-deform (4)skinning and deformation: models are graded on judge whether a mesh will deform - skin weights, twist distribution, and telling a real defect from an intentional design choice.

floor (1)

checks the model knows why one-bone vertices tear at a joint and how many influences a joint needsfloor
Single-influence vertices tear at joints
Prompt (input)
A model check reports that 33% of its vertices have exactly one bone influence, and one arm bone has zero vertices at full weight with a mean weight of 0.37. Why do single-influence vertices tear at joints, and what is the target for a joint that must deform well?
Grading
judged by rubric
Explains a single-influence vertex cannot blend across a joint so it moves rigidly and tears; a joint that deforms well needs roughly 3 to 4 influences and edge loops to blend across.

discriminating (3)

checks the model explains why a single forearm bone spirals the mesh under twist, and that IK cannot fix itdiscriminating
One forearm bone candy-wraps the mesh under twist
Prompt (input)
A forearm has a single bone (no twist segments). Under a swing the wrist rolls, and measurement shows that one forearm bone carrying about 240 degrees of rotation makes the mesh spiral to a point at the wrist. Explain the mechanism, and say whether tuning the IK can fix it.
Grading
graded rubric (3 criteria, all must hold)
  • Explains that under linear blend skinning a single bone carrying ~240 degrees of roll winds the whole cross-section around one axis, so the mesh spirals / candy-wraps to a point
  • States that IK tuning cannot touch this - it is a SKINNING / topology problem, not a solver problem
  • Says the fix is to split the limb into twist segments (or use geometry good enough to add them) so the roll distributes along the limb
checks the model knows the checks that are easy to eyeball do not measure whether a model can actually be riggeddiscriminating
Eyeballed checks pass but the model is unriggable
Prompt (input)
A character model PASSES the two checks people usually eyeball - quad topology (87% quads) and scale/orientation (correct height, right up-axis) - yet it is rejected as unriggable. What kinds of defects do those two checks fail to measure, and what is the general lesson about relying on them?
Grading
graded rubric (3 criteria, all must hold)
  • Names defects the two checks do not measure - things like missing twist bones, poor skin weights (single-influence vertices), too few edge loops at joints, a bad finger chain or missing thumb, left/right bone-length mismatch
  • Distinguishes SURFACE quality (quads, scale) from DEFORMATION readiness, and notes the eyeballed checks only cover the former
  • Draws the lesson that riggability must be measured with a real check, not judged by the two things that happen to be easy to eyeball
checks the model separates intentional design asymmetry from a real defect by testing the skeleton, not the meshdiscriminating
Intentional asymmetry vs a real defect - test the skeleton
Prompt (input)
On a character the two hands differ in bone length by 13% to 41%, while the arms and legs mirror to within a rounding error. The character also has deliberately different arm-bands on left and right. How do you tell an intentional-asymmetry design choice from a real defect, and what should a symmetry test measure?
Grading
graded rubric (3 criteria, all must hold)
  • Recognizes that a whole-MESH symmetry test cannot separate intentional costume asymmetry (the arm-bands) from a real defect
  • Says to test the SKELETON for symmetry - bone lengths and joint positions, which costume cannot touch - and leave the surface mesh alone
  • Concludes the mismatched hands are a real defect (the skeleton should mirror) while the arm-bands are design
spaces-transforms (4)coordinate spaces: models are graded on keep rest space, pose space, and world space straight, and handle evaluation and update order.

floor (3)

checks the model converts a point from the rig's local space to world space using the given rulefloor
Armature-local to world coordinate conversion (given the mapping)
Prompt (input)
In a Blender rig the armature imports at 0.01 scale and rotated 90 degrees about X, so an armature-local point (x, y, z) maps to world (x/100, -z/100, y/100), in metres. A bone head sits at armature-local (50, 100, 30). Give its world-space coordinates in metres.
Grading
custom javascript check on the model's output
checks the model protects a solver-driven bone from accidental edits without breaking the solvefloor
Protect a constraint-driven bone from accidental posing
Prompt (input)
Rotating a bone that has an IK or Copy Rotation constraint appears to do nothing in the viewport, but the rotation is still stored and accumulates, and it snaps into view the moment the constraint's influence drops to 0. You want to stop this bone being posed by accident during box-selects, WITHOUT changing how the solver evaluates it. What property do you set, and why is it safe?
Grading
output must contain "hide_select" (case-insensitive)
judged by rubric
Explains hide_select is a UI-only flag, so the solver is unaffected.
checks the model reads a bone position in the correct space so a new bone is not placed in the wrong spotfloor
Rest space vs pose space when placing a new edit bone
Prompt (input)
Edit bones in Blender live in REST space, while pose_bone.matrix gives the POSED transform. You need the rest-space world position of a bone to place a new edit bone, and the character is currently posed. Which is correct to read: pose_bone.matrix, or armature.matrix_world @ armature.data.bones[name].matrix_local? Explain the consequence of using the wrong one.
Grading
output must contain "matrix_local" (case-insensitive)
judged by rubric
Chooses the rest-space (matrix_local) form and explains that using the posed transform shifts the new bone by the pose amount.

discriminating (1)

checks the model explains why a value read right after setting it is stale, and the one-line fixdiscriminating
A custom-property-driven constraint evaluates one step late
Prompt (input)
From Python you set a custom property ik_arms = 0 on a control that drives a constraint's influence, then call view_layer.update() and read the influence - it still reads the OLD value (1.0). Flipping to 1 then reads 0.0. Explain what is happening and give the one-line fix.
Grading
graded rubric (3 criteria, all must hold)
  • Explains that assigning the custom property does not TAG the driver as needing re-evaluation, so the next update hands back the previous (stale) value
  • Notes this makes a verification pass read the frame it just left and report success it did not earn
  • Fix is to call the datablock's update_tag() before view_layer.update()