Robotics evidence and evaluation

Editorial guide

Start assistive robotics with a bounded human need

Choose the assistance task with the people affected and define what useful support would mean to them. A humanoid demonstration is not enough to establish a beneficial service.

Editorial guide

Choose a robotics benchmark for the decision it informs

Choose evaluation conditions that test the capability you need, not the leaderboard on which a system looks strongest. A benchmark is a measurement context, not a universal ranking of robots.

Synthetic prototype

pick place example

Inspect a deterministic grid-world experiment. Simulation is not a hardware test.

Editorial guide

Robot embodiment is not a capability claim

Describe a robot by the tasks and conditions it has demonstrated, not just by whether its body resembles a person. Maha’s proposed evidence unit is a bounded task attempt with an identified configuration.

Editorial guide

Write a robot task contract before collecting results

A task contract states the initial conditions, permitted actions, endpoint, timeout and outcome rules before the first evaluation attempt. Otherwise a successful demonstration can quietly redefine what counted as success.

Editorial guide

Separate perception evidence from task completion

A completed movement does not prove the robot perceived the scene correctly. Record what the perception system estimated, what reference established the object state and how that reference was obtained.

Editorial guide

What a pick-and-place result actually demonstrates

Report the object, grasp conditions, transfer and release outcome together. A single successful placement supports a claim about that attempt, not general dexterity across objects or environments.

Editorial guide

Mobility results need localization context

Record the frame and localization behavior behind a reported route. Reaching a coordinate means little if the reference frame changed or the position estimate drifted without being recorded.

Editorial guide

Separate plans, control and authority

A task plan, an actuator command and permission to execute are distinct records. Maha’s first robotics integration should observe and verify evidence offline, not sit inside a motion-control or emergency-stop loop.

Editorial guide

Keep human assistance visible in robot evaluations

Record interventions as part of the outcome, not as a hidden reset. A task finished with assistance is useful evidence, but it is not the same result as autonomous completion.

Editorial guide

Robot evidence needs explicit frames and units

Attach units and frame identifiers to values before combining them. A correct number in the wrong convention can support an incorrect action or an invalid comparison.

Editorial guide

Count all planned robot evaluation attempts

Publish the denominator with the numerator. Register expected attempts before execution, preserve failures and assistance, and distinguish missing records from unsuccessful recorded trials.

Editorial guide

Keep simulation and hardware evidence separate

A simulation result supports a claim about the simulated model and conditions. Moving that claim to hardware requires new evidence about the physical task, not a change of label on the same result.

Editorial guide

Replaying robot logs is not reproducing an experiment

Replay can supply recorded messages again. Reproduction additionally requires enough configuration and execution context to determine which behavior is expected to recur and how equivalence will be judged.

Editorial guide

Compare controller revisions without moving the goalposts

Keep the task, cases and scoring rules fixed while comparing controller revisions. Report regressions and improvements per case before deciding whether an aggregate difference matters.

Editorial guide

Choose a robotics benchmark for the decision it informs

Choose evaluation conditions that test the capability you need, not the leaderboard on which a system looks strongest. A benchmark is a measurement context, not a universal ranking of robots.

Editorial guide

Report uncertainty in robot measurements

A reported placement error needs a measurement method and uncertainty context. More decimal places do not establish a more accurate robot.

Editorial guide

Classify robot failures without inventing a cause

Separate the observed failure stage from its suspected cause. “Object not released” is an observation; “bad perception” is a causal explanation that needs additional evidence.

Editorial guide

Keep training exposure out of robot evaluation claims

Record how evaluation episodes differ from training and tuning material. A different filename is not evidence of an independent test.

Editorial guide

Document how a robot outcome was judged

Ground truth is a reference with its own method and limitations. Identify who or what judged success and whether that judgement is independent of the controller being evaluated.

Editorial guide

Turn a robot demonstration into an evidence-gap register

List the untested conditions between a demonstration and the intended application. A gap register should describe what remains to be learned, not convert a demonstration into deployment approval.

Editorial guide

Trace robot datasets from collection to evaluation

Preserve the relationship between source recordings, transformations and the dataset used in an experiment. A dataset name alone cannot tell a reviewer which observations actually contributed.

Editorial guide

Align robot sensor time without hiding assumptions

Name the clock and timestamp meaning for every stream before comparing events. Message arrival order and physical event order are not automatically the same.

Editorial guide

Bind robot measurements to calibration context

Attach calibration evidence to the exact sensor configuration and measurement interval it describes. A calibration file with the right name is not enough.

Editorial guide

Identify the model that actually ran

Bind an evaluation to the model artifact, preprocessing and runtime configuration actually used. A model family name or mutable download URL is not an execution identity.

Editorial guide

Record the robot configuration behind each trial

Describe the hardware and software configuration actually present during an evaluation, including substitutions. A product model name hides too many variables for reproducible comparison.

Editorial guide

Separate access to robot episodes from permission to reuse them

Public availability, technical access and permission for a proposed use are different questions. Record the reuse basis before importing an episode into a commercial evidence product.

Editorial guide

Retain enough robot evidence without keeping everything

Define why each artifact is retained, who can access it and when its disposition is reviewed. Indefinite video retention is not a necessary consequence of traceability.

Editorial guide

What a robotics artifact digest can and cannot prove

A matching digest establishes agreement with a referenced byte representation under the chosen algorithm. It does not show who created the artifact, whether a sensor told the truth or whether all trials were supplied.

Editorial guide

Start assistive robotics with a bounded human need

Choose the assistance task with the people affected and define what useful support would mean to them. A humanoid demonstration is not enough to establish a beneficial service.

Editorial guide

Evaluate access to the whole assistance workflow

Accessibility includes starting, understanding, pausing and recovering a task—not merely reaching an object. Evaluate the interface and workflow with the people for whom the assistance is intended.

Editorial guide

Measure supervised robotics as supervised robotics

Describe what the supervisor must observe, decide and do. A supervised system can be useful without being mislabelled autonomous.

Editorial guide

Treat home robot recordings as more than task data

A home recording may reveal information unrelated to the task. Design the evidence flow to minimize what is collected and shared, rather than assuming a robot log is impersonal.

Editorial guide

Record recovery without erasing the original failure

A recovered task contains at least two important facts: something failed, and a subsequent action restored progress. Keep both instead of replacing the original attempt with a successful ending.

Editorial guide

Distinguish requesting a handoff from completing one

A handoff has separate request, acknowledgement and observed-transfer stages. Software acknowledgement alone does not establish that a person received the object or responsibility.

Editorial guide

Give robotics evidence a clear decision owner

Governance identifies who can request an action, approve a change, assess evidence and accept responsibility for a decision. It must not collapse those roles into an automated pass flag.

Editorial guide

Determine which robotics requirements need specialist review

Start standards review with the intended use, system boundary and jurisdiction. Do not infer that a requirement applies—or is satisfied—because a robot resembles a familiar product category.

Editorial guide

Preserve robotics incident evidence without premature conclusions

Record what was observed, what is missing and which artifacts were preserved. An incident record should support investigation rather than decide blame from an incomplete trace.

Editorial guide

Review a robot change against its exact evidence revision

Approve a defined change for a defined scope. A green regression test or an old review should not automatically authorize a new controller, environment or use.

Synthetic prototype

evidence package

Inspect the required fields, replay contract and limits of an evidence package.

Synthetic prototype

pick place example

Inspect a deterministic grid-world experiment. Simulation is not a hardware test.

Editorial guide

Worked example: reviewing a ROS bag intake manifest

Begin with an offline inventory of a recording’s streams, time basis and gaps. A ROS bag is a recording container, not a complete robotics evidence package.

Editorial guide

Worked example: checking a LeRobot episode reference

Resolve the episode through its metadata and preserve its data, video and transformation references. Do not assume that one video file is one evaluation attempt.

Physical AI

Explore learned models, training and control.

Nanotechnology

Inspect materials and measurement evidence.

Computational Architecture

Connect physical systems to building models and reproducible design evidence.

Mathematics

Explore the formal foundations behind calculations and evaluations.

Governed workflows

Keep execution evidence separate from permission and substantive correctness.