Generator-option reliability

Option combinations grow quickly as a generator gains targets and features. Sampling lets each build compile a changing selection while recording enough information to replay it. That is a contract with three parts: an identity for the run, a gate the output has to pass, and a report that lets someone else replay a failure.

Identity

A generation run publishes its identity next to the generated output:

{
  "schema": "nav-relationships",
  "language": "php-sql",
  "seed": -447741108,
  "options": {
    "WithNavigationProperties": true,
    "WithValidationAnnotations": true,
    "InitializeProperties": true,
    "UseStrictTypes": false
  }
}

Schema, language, seed, applied options, plus the build number that names the engine version: that report is the identity. Replaying it means applying the recorded options to the same schema with the same engine build and running the same gate.

Build a sampler for your own generator

A small .NET example is enough to show the mechanism:

var seed = Random.Shared.Next();
var random = new Random(seed);
var options = new
{
    IncludeRelations = random.Next(2) == 1,
    UseStrictTypes = random.Next(2) == 1
};

Record seed and the resulting options before generation. Reusing the seed with the same runtime, option code and call order reproduces the sample. Applying the recorded options directly is more durable when the sampler evolves.

If a sampler derives its seed from names or build identifiers, its hash function matters. .NET initializes HashCode with a process-specific seed, so recomputing that hash in another process will not recover the original sample. Store the chosen seed and options instead of relying on the inputs to a hash.

Gate

Generate, compile, probe. The pipeline review snapshot records the commands and reporting contract reviewed for this article. It describes MetaEngine’s CI; the public incident pack below reproduces the PHP compiler rule.

1. Generate. One xUnit test per target generates every reference schema with its sampled options, in memory, and zips the output as a build artifact. Zero files is a failure, not an empty success.
2. Compile. A separate job per target unpacks the archive and runs the language's own compiler or analyzer over everything at once.
3. Probe. A failure the sampler finds becomes a probe test: a fixed schema, a fixed option combination, an assertion on the emitted shape. Probes run on every build regardless of what the sampler rolls, so a fixed corner stays fixed.

One failure, end to end

Open the sanitized incident pack for the recorded report, relationship schema and a one-command PHP lint probe. It replays the compiler boundary — failing ?mixed, fixed mixed — rather than regenerating the model.

• Engine. CI build 6021, May 2026. Target: PHP models from SQL DDL.
• Schema. nav-relationships.sql — tables with one-to-many and many-to-many relations, self-references, and two foreign keys to the same parent table.
• Identity. Seed -447741108; options WithNavigationProperties: true, WithValidationAnnotations: true, InitializeProperties: true, UseStrictTypes: false.
• Gate. php -l on every generated class.
• Failure. Type mixed cannot be marked as nullable since mixed already includes null. A navigation property whose target type had no PHP mapping fell back to mixed, and the renderer prepended ? to every nullable property without exception. ?mixed is a parse error in PHP 8, whose type documentation includes null in mixed.
• Why sampling found it. The combination needs both toggles to land the wrong way — navigation properties on, strict types off — which happens on about a quarter of rolls. No hand-written fixture had combined them.
• Fix. Nullable rendering is centralised so mixed is never prefixed; a probe test pins the emitted shape for exactly that combination; the combination went back into the sampler.

Which question each technique answers

These are different questions. Calling one by another's name is how a testing story overstates itself.

What the contract does not give you

• Sampling is not coverage. A combination that fails on one roll in four is found quickly; one that needs five toggles aligned may go unrolled for a long time. Coverage accrues across builds and is never complete.
• A compile gate says nothing about behavior. php -l accepts a class that parses and does the wrong thing at runtime.
• The identity pins options, schema and engine build. It does not pin the toolchain on the build agent; a compiler upgrade can turn a passing artifact red with no change in the generator.