30 tests across 6 categories. promptfooconfig.yaml
Scan each category: is this the RIGHT set of checks, is the math right, what is MISSING? Each line below is the plain-language summary of one test; expand it for the exact prompt and grading.
{{input}} The per-test prompt is the {{input}} shown in each expansion.A Copy Rotation constraint with only the Y axis enabled makes a twist bone take a share of a wrist's roll. Blender applies Copy Rotation on an euler decomposition. With the default XYZ order, Y is the middle term, so the wrist's BEND contaminates the roll. To make the decomposition behave like swing-twist, which euler order should the constraint use, and why?
output must contain "YXZ" (case-insensitive)judged by rubricExplains that putting the twist (Y) axis FIRST in the decomposition isolates roll from the bend.
You are on Blender 5.2. In 5.2, Bone.select was removed (use PoseBone.select) and armature.data.layers was removed in 4.0 (use armature.data.collections). An addon's manifest declares it targets Blender 2.80, and its code calls Bone.select and armature.data.layers in eight places while building an armature. What happens if you run it, and should you install it?
judged by rubricSays it will error partway through and leave a half-built armature, advises against installing it, and maps the removed calls to PoseBone.select and data.collections.
A two-bone leg IK chain has an upper bone 248 mm long and a lower bone 220 mm long. The IK target is placed 485 mm from the hip (straight-line distance). Compute the reach ratio (target distance divided by total bone length) and state whether the target is reachable.
custom javascript check on the model's outputjudged by rubricConcludes the target is NOT reachable / is beyond full extension (ratio above 1.0).
In a pole-vector IK setup the bend plane is defined by three points: the chain root, the IK target, and the pole. Someone slides the pole from 3000 mm away from the elbow to 250 mm away, keeping it in the same direction. How much does this change the solved joint positions, and why?
judged by rubricStates the solve is essentially unchanged (about 0 mm) because only the pole DIRECTION matters, distance is ergonomic.
You set up a two-bone IK chain on a T-posed arm. Moving the IK target does nothing: the hand stays put across all pole angles. The constraint reports is_valid=True, influence=1.0, mute=False, and no error. The identical code on the legs works fine. Diagnose the cause and give a fix.
graded rubric (3 criteria, all must hold)In FK you twist the forearm 45 degrees about its own axis, then switch the arm to IK, and the twist vanishes and cannot be recovered by fixing the addon. The chain is a 3-DOF upper arm plus a forearm that is a pure hinge (X and Y locked). Explain, in terms of degrees of freedom, why IK cannot hold the forearm roll, and give two fixes.
graded rubric (3 criteria, all must hold)An auto-rigger fitted the left and right arms independently; their hand positions differ by about 4 mm. One arm ends up at 100% reach (straight) and the other at 95% with a 70 mm elbow break and a 16.7 degree upper-arm direction mismatch, and no pole angle can fix it. Explain why such a tiny difference produces such a large error, and give the construction fix.
graded rubric (3 criteria, all must hold)An image-to-3D generator is handed a single reference image that contains BOTH the front and back view of a character side by side. It produces a scene with two separate figures standing next to each other instead of one character. What is the fix?
judged by rubricSays to split the sheet into one image per view and assign each to its own front/back direction slot.
You upload a clean, unrigged FBX (one mesh, zero armatures, zero vertex groups) to an auto-rigger and it fails with "unable to map your existing skeleton", even though there is no skeleton in the file. The service chooses between an auto-rig path and a map-existing-rig path by sniffing the file. Explain the likely cause and give a reliable fix.
judged by rubricExplains the service misdetected the file as already-rigged and took the wrong branch; fix is to upload OBJ (which has no skeleton concept) so it cannot take that branch.
You are preparing a T-pose reference sheet for image-to-3D reconstruction. In a T-pose with palms DOWN, the thumb points forward and is foreshortened to nothing in the front view and hidden behind the hand in the back view, so the reconstruction builds a flat four-finger paddle with no thumb. How should the hands be posed in the reference, and why?
judged by rubricSays to pose palms forward, thumbs up, fingers spread, with visible palm thickness, so both views actually see the thumb.
Across several image-to-3D attempts, an asymmetric costume detail (a knot on one hip) keeps landing on the character's LEFT in the front view and her RIGHT in the back view. Image generation cannot hold an asymmetric detail consistent across two viewpoints. What is the fix?
judged by rubricSays to remove the asymmetric part from the reference and add it later in Blender as separate geometry (where it needs its own bone chain and spring sim anyway).
You flatten an RGBA reference sheet by sampling its background colour, and the reconstruction reads almost no silhouette. The costume is near-black. Sampling the background colour of a transparent PNG returns (0,0,0,0). What went wrong, and what should the background be?
judged by rubricExplains the transparent background flattened to black behind a near-black costume, leaving no silhouette; fix is to force a light neutral grey (not black), high-contrast against the outfit.
Reference art has cel shading painted into it. The material generator bakes that painted lighting into the texture, then a three-point lighting setup is applied on top, so shadows double (a shadow under the jaw from the drawing PLUS one from the key light). What two settings prevent this?
judged by rubricSays to enable de-light (strip the baked lighting to flat albedo) and to download PBR, not Shaded (Shaded is the baked-lighting version that undoes de-light).
You need to verify that a Copy Rotation twist constraint actually fired on a bone. A colleague reads the bone's matrix_basis, sees 0.0 degrees on all axes, and concludes the constraint did nothing. Is that conclusion sound? If not, what should be measured instead, and why?
graded rubric (3 criteria, all must hold)You added forearm twist bones to fix a wrist that collapses under roll. To confirm the fix you sample the mesh cross-section at points 20% to 80% along the forearm and get 99% (looks solved). The skin weights that blend from the forearm bone to the hand bone actually span 80% to 125% of the forearm length. Is your measurement region right? What would you measure?
graded rubric (3 criteria, all must hold)IK pole angles were chosen by forcing a slight bend and minimizing the distance from each joint to its pole target. Every position-based assertion (joint locations, hand location) passed. Later a render showed the whole limb mesh was twisted. What did the position-only calibration fail to constrain, and how should the calibration be judged instead?
graded rubric (3 criteria, all must hold)A script reports that a fix achieved 153% of its target coverage. Before celebrating, what is the most likely explanation for a percentage above 100%, and what would you check?
graded rubric (3 criteria, all must hold)You are building a foot-roll control and want to verify the toe stays planted on the floor as the foot rolls. One approach measures the height of the lowest toe VERTEX and keeps it at floor level. Is that the right thing to hold constant for "the toe stays planted", or is there a better quantity? Explain.
graded rubric (3 criteria, all must hold)After adding joint limits to a posed rig, a check reports "worst bone moved 56 mm" and looks like a disaster. Broken down by region: hands/feet moved 0.02 mm, elbows/knees moved 56 mm, fingers moved 10.6 mm. Locking the off-axes at the elbow/knee is expected to change the solve there. Which number is the real bug, and what does this say about the global-worst metric?
graded rubric (3 criteria, all must hold)To decide whether a finger's bone chain is bent or straight, someone measures the straight-line distance from the first joint to the last (66.5 mm) and the sum of the bone lengths (66.5 mm), sees they match, and concludes the finger is straight. The finger bones are UNCONNECTED (each child's head is not at its parent's tail). Is the conclusion valid? What should be measured?
graded rubric (3 criteria, all must hold)Checking a character for left/right symmetry, a script compares world X positions of mirrored bone pairs and finds an apparent 80.4 mm error, which it halves to report "21.5 mm shoulder asymmetry" and recommends replacing the model. The whole skeleton sits 40.2 mm off the world origin along X. Is the asymmetry real? What was measured wrong?
graded rubric (3 criteria, all must hold)A left/right symmetry check reports a perfect 0.0% mismatch. The comparison relies on matching bone names by replacing "Left" with "Right". Before trusting the perfect score, what must you confirm, and what general failure does a perfect / 0.0% result often hide?
graded rubric (3 criteria, all must hold)A model check reports that 33% of its vertices have exactly one bone influence, and one arm bone has zero vertices at full weight with a mean weight of 0.37. Why do single-influence vertices tear at joints, and what is the target for a joint that must deform well?
judged by rubricExplains a single-influence vertex cannot blend across a joint so it moves rigidly and tears; a joint that deforms well needs roughly 3 to 4 influences and edge loops to blend across.
A forearm has a single bone (no twist segments). Under a swing the wrist rolls, and measurement shows that one forearm bone carrying about 240 degrees of rotation makes the mesh spiral to a point at the wrist. Explain the mechanism, and say whether tuning the IK can fix it.
graded rubric (3 criteria, all must hold)A character model PASSES the two checks people usually eyeball - quad topology (87% quads) and scale/orientation (correct height, right up-axis) - yet it is rejected as unriggable. What kinds of defects do those two checks fail to measure, and what is the general lesson about relying on them?
graded rubric (3 criteria, all must hold)On a character the two hands differ in bone length by 13% to 41%, while the arms and legs mirror to within a rounding error. The character also has deliberately different arm-bands on left and right. How do you tell an intentional-asymmetry design choice from a real defect, and what should a symmetry test measure?
graded rubric (3 criteria, all must hold)In a Blender rig the armature imports at 0.01 scale and rotated 90 degrees about X, so an armature-local point (x, y, z) maps to world (x/100, -z/100, y/100), in metres. A bone head sits at armature-local (50, 100, 30). Give its world-space coordinates in metres.
custom javascript check on the model's outputRotating a bone that has an IK or Copy Rotation constraint appears to do nothing in the viewport, but the rotation is still stored and accumulates, and it snaps into view the moment the constraint's influence drops to 0. You want to stop this bone being posed by accident during box-selects, WITHOUT changing how the solver evaluates it. What property do you set, and why is it safe?
output must contain "hide_select" (case-insensitive)judged by rubricExplains hide_select is a UI-only flag, so the solver is unaffected.
Edit bones in Blender live in REST space, while pose_bone.matrix gives the POSED transform. You need the rest-space world position of a bone to place a new edit bone, and the character is currently posed. Which is correct to read: pose_bone.matrix, or armature.matrix_world @ armature.data.bones[name].matrix_local? Explain the consequence of using the wrong one.
output must contain "matrix_local" (case-insensitive)judged by rubricChooses the rest-space (matrix_local) form and explains that using the posed transform shifts the new bone by the pose amount.
From Python you set a custom property ik_arms = 0 on a control that drives a constraint's influence, then call view_layer.update() and read the influence - it still reads the OLD value (1.0). Flipping to 1 then reads 0.0. Explain what is happening and give the one-line fix.
graded rubric (3 criteria, all must hold)