Custom properties are applied during pull request

[wernerb]

Problem Description

When in PR mode, adding a custom property results in the property added before being merged into main.

What is actually happening

Can see the changes in the PR, once i did a different unrelated change, i could see autolinks still waiting to be applied but the custom properties were applied.

What is the expected behavior

Custom properties are only applied in main

Error output, if available

Context

Are you using the hosted instance of probot/settings or running your own?

Running our own in PR mode

If running your own instance, are you using it with github.com or GitHub Enterprise?

Github.com

Version of probot/settings

latest

Version of GitHub Enterprise

[decyjphr]

The custom_properties plugin doesn't have the code for nop flag and so it applies the change. It would be a pretty simple fix to resolve this.

[wernerb]

The same problem is also with environments unfortunately. These are both from a security perspective, critical components as we use custom properties for targetting rulesets being applied now on branches.

[wernerb]

Is then dry-run is effectively unsupported for various plugins?
The documentation says that:

The settings in the default branch are applied. If the settings are changed on a non-default branch and a PR is created to merge the changes, the app runs in a dry-run mode to evaluate and validate the changes. Checks pass or fail based on the dry-run results.

Which led me to believe all changes will be dry-run on PR.

[PendaGTP]

The custom_properties plugin doesn't have the code for nop flag and so it applies the change. It would be a pretty simple fix to resolve this.

@decyjphr I would be happy to contribute on this fix if you're interested. I could prepare and open a PR today or in the next few days.

[decyjphr]

That'd be great. Thanks @PendaGTP

[PendaGTP]

@wernerb

Is then dry-run is effectively unsupported for various plugins? The documentation says that:

The settings in the default branch are applied. If the settings are changed on a non-default branch and a PR is created to merge the changes, the app runs in a dry-run mode to evaluate and validate the changes. Checks pass or fail based on the dry-run results.

Which led me to believe all changes will be dry-run on PR.

Based on my experience reviewing the codebase, the dry-run/nop mode support varies across plugins:

Currently supported:

• Autolinks
• Branches
• Collaborators
• Issue labels
• Repository
• Rulesets
• Teams
• Repository name validator

Not supported:

• Custom properties
• Milestones

Partially supported:

• Environments