The regression threshold is the percentage of performance degradation relative to your default branch that is considered acceptable.
If the overall performance degradation is less than the threshold, the status check is considered successful. Otherwise, the status check will fail and the only way to make it pass is either to fix the performance issues or to acknowledge the regression.
By default, the regression threshold is set to 10% but you can change it in the Settings tab of your repository:
Pull Request comments
You can customize when the Pull Request reports will be sent:
Always: performance will be reported on every commit. If a comment already exists, it will be updated.
On Change: performance will be reported if there is a significant improvement(more than the regression threshold) or a failure(a regression or a drop). If the comment is created after a change, it will be updated later on even if the performance is in standard bounds.
Never: the performance will never be reported in the comments of your Pull Requests. The status check will still be sent though.