Configure CI/CD Pipeline Quality Gates
Configuring the CI/CD Pipeline Tool Quality Gates
OverOps provides six configurable quality gates that are designed to mark a build as unstable. Using the configuration screen (shown below), each gate can be configured to meet your specific needs. To skip a gate, leave the values blank or zero as described in the table that follows.
Quality Gates (closeup)
Next, configure the quality gates according to the following information:
Gate | Description | Comments | Defaults |
|---|---|---|---|
New Error Gate | If any new error is detected, the build will be marked as unstable. | TRUE | |
Resurfaced Error Gate | If any resurfaced error is detected, the build will be marked as unstable. | TRUE | |
Total Error Volume Gate | Set the max total error volume allowed. If the volume is equal to or exceeds this value, the build will be marked as unstable, and top errors will be listed in the report. | A setting of 0 (the default) will tell OverOps to ignore this quality gate, and it will be omitted from the report. | 0 |
Unique Error Volume Gate | Set the max unique error volume allowed. f the volume is equal to or exceeds this value, the build will be marked as unstable, and the top errors will be listed in the report. | A setting of 0 (the default) will tell OverOps to ignore this quality gate, and it will be omitted from the report. | 0 |
Critical Exception Types Gate | A comma delimited list of exception types that are deemed as severe regardless of their volume. If any events of any exceptions listed have a count greater than zero, the build will be marked as unstable. | 'NullPointerException,IndexOut OfBoundsException,InvalidCast Exception,AssertionError' | |
Increasing Errors Gate* | Combines the following parameters:
| minVolumeThreshold: 20, minErrorRateThreshold: 0.1, regressionDelta: 0.5, criticalRegressionDelta: 1.0, applySeasonality: true, | |
| Active Time Window (d - day, h - hour, m - minute) | The time window inspected to search for new issues and regressions. Set to zero to use the Deployment Name (which would be the current build). Example: 1d would be one day active time window. | '12h' | |
| Baseline Time Window (d - day, h - hour, m - minute) | The time window (in minutes) against which events in the active window are compared to test for regressions. Must be set to a non zero value Example: 14d would be a two week baseline time window. | '7d' | |
| Event Volume Threshold | The minimal number of times an event of a non-critical type (e.g. uncaught) must take place to be considered severe. | If a New event has a count greater than the set value, it will be evaluated as severe and could break the build if its event rate is above the Event Rate Threshold.If an Existing event has a count greater than the set value, it will be evaluated as severe and could break the build if its event rate is above the Event Rate Threshold and the Critical Regression Threshold.If any event has a count less than the set value, it will not be evaluated as severe and will not break the build. | |
| Event Rate Threshold (0-1) | The minimum rate at which event of a non-critical type (e.g. uncaught) must take place to be considered severe. A rate of 0.1 means the events is allowed to take place <= 10% of the time. | If a New event has a rate greater than the set value, it will be evaluated as severe and could break the build if its event volume is above the Event Volume Threshold.If an Existing event has a rate greater than the set value, it will be evaluated as severe and could break the build if its event volume is above the Event Volume Threshold and the Critical Regression Threshold.If an event has a rate less than the set value, it will not be evaluated as severe and will not break the build. | |
| Regression Delta (0-1) | The change in percentage between an event's rate in the active time span compared to the baseline to be considered a regression. The active time span is the Active Time Window or the Deployment Name (whichever is populated). A rate of 0.1 means the events is allowed to take place <= 10% of the time. | If an Existing event has an error rate delta (active window compared to baseline) greater than the set value, it will be marked as a regression, but will not break the build. | 0 |
| Critical Regression Threshold (0-1) | The change in percentage between an event's rate in the active time span compared to the baseline to be considered a critical regression. The active time span is the Active Time Window or the Deployment Name (whichever is populated). A rate of 0.1 means the events is allowed to take place <= 10% of the time. | If an Existing event has an error rate delta (active window compared to baseline) greater than the set value, it will be marked as a severe regression and will break the build. | 0 |
| Apply Seasonality | If peaks have been seen in baseline window, then this would be considered normal and not a regression. Should the plugin identify an equal or matching peak in the baseline time window, or two peaks of greater than 50% of the volume seen in the active window, the event will not be marked as a regression. | FALSE | |
| Debug Mode | If checked, all query and results will be displayed in the OverOps reliability report. For advanced debugging purposes only. | FALSE |
Regression Testing Examples
New Issues combines both Event Volume Threshold and Event Rate Threshold:
- Event Volume Threshold: Used to identify the minimum number of occurrence of an event to be considered in the algorithm. If an error happens only once or twice, it is likely not as important as one that happens 10k times.
- Event Rate Threshold: Used to identify a minimum error rate minimum of an event to be considered in the algorithm. If an error happens at .0001%, it is likely not as important as one that happens at 2%.
- Example combining the two thresholds
- Event Volume Threshold set to 20
- Event Rate Threshold set to .05 (5%)
- If an event occurs 10 times, it will be excluded
- If an event occurs 100 times at a rate of 5%, it will be marked as severe and break the build
- Example combining the two thresholds
Existing Issues combines Event Volume Threshold, Event Rate Threshold, Regression Delta, Critical Regression Threshold and Apply Seasonality:
- Event Volume Threshold: Used to identify the minimum number of occurrence of an event to be considered in the algorithm. If an error happens only once or twice, it is likely not as important as one that happens 10k times.
- Event Rate Threshold: Used to identify a minimum error rate minimum of an event to be considered in the algorithm. If an error happens at .0001%, it is likely not as important as one that happens at 2%.
- Regression Delta: Used to measure the the delta of the active time window’s error rate to the baseline time window’s error rate. OverOps will measure the difference to determine if the error rate is increasing over time. Any event that has an error rate in the active time window that is above the defined value, will be marked as a regression. These will not break the build.
- Critical Regression Threshold: Used to measure the the delta of the active time window’s error rate to the baseline time window’s error rate. OverOps will measure the difference to determine if the error rate is increasing over time. Any event that has an error rate in the active time window that is above the defined value, will be marked as a severe regression. These will break the build.
- Apply Seasonality: Used to identify previous peaks in the baseline time window to rule out previous peaks compared to the active time window. Should the plugin identify an equal or matching peak in the baseline time window, or two peaks of greater than 50% of the volume seen in the active window, the event will not be marked as a regression.
Example combining all thresholds: - Event Volume Threshold set to 20
- Event Rate Threshold set to .05 (5%)
- Regression Delta set to .1 (10%)
- Critical Regression Threshold set to .2 (20%)
- Apply Seasonality checked
Stable example: If an event occurs more than 20 times at an error rate of 10% in the active time window, it will be compared to the baseline time window. When compared to the baseline time window, if the event was occurring at 9% versus the current 10%, that would be a 11% increase. If no previous spikes the in the baseline window are detected the event would mark the event as a regression.
Unstable example: If an event occurs more than 20 times at an error rate of 12% in the active time window, it will be compared to the baseline time window. When compared to the baseline time window, if the event was occurring at 9% versus the current 12%, that would be a 33% increase. If no previous spikes the in the baseline window are detected the event would mark the event as a severe regression.
Troubleshooting Tips
When setting up and configuring the plugin, make sure that the deployment name you use when configuring OverOps to monitor the test is the same deployment name you use to configure the quality gates.
Related Documentation
Integrating OverOps with the CI/CD Pipeline
OverOps Quality Gates
Updated almost 2 years ago