Understanding Results

Bug0 provides detailed results for every test run, helping you understand what happened and why.

Result Components

Step Results

Each step shows:

Field	Description
Status	Pass (✓) or Fail (✗)
Duration	How long the step took
Reasoning	AI's decision-making explanation
Action taken	What the AI did (click, type, etc.)

Assertion Results

Each assertion shows:

Field	Description
Status	Pass or Fail
Confidence	How certain the AI is (0-100%)
Reasoning	Why the assertion passed or failed

Overall Result

• Passed - All steps and assertions succeeded
• Failed - At least one step or assertion failed

Interpreting Step Results

Successful Step

✓ Click the "Add to Cart" button
  Duration: 1.2s

  Reasoning: Located button with text "Add to Cart"
  below the product price. Clicked successfully and
  observed cart icon update.

Failed Step

✗ Click the "Checkout" button
  Duration: 5.3s (timeout)

  Error: Could not find element matching description

  Reasoning: Searched for button with text "Checkout"
  but found "Proceed to Checkout" and "Express Checkout".
  Unable to determine which one matches the step
  description.

How to fix: Make the description more specific:

• "Click the 'Proceed to Checkout' button"

Interpreting Assertion Results

Passed Assertion

✓ User should see the dashboard
  Confidence: 95%

  Reasoning: The page displays a heading "Dashboard"
  and shows user-specific content including the
  user's name and recent activity.

Failed Assertion

✗ Cart should show 3 items
  Confidence: 88%

  Reasoning: The cart displays 2 items. The expected
  count was 3, but only 2 products are visible in
  the cart summary.

Low Confidence

? Product was added to cart
  Confidence: 62%

  Reasoning: A toast notification appeared briefly,
  but it's unclear if it confirmed the add action
  or was an unrelated message.

Low confidence indicates the AI is uncertain. Consider:

• Making assertions more specific
• Adding screenshots at key moments
• Verifying the expected state is achievable

HTML Reports

For CI/CD runs, Bug0 generates detailed HTML reports:

Report Contents

• Summary - Overall pass/fail status
• Test list - All tests with status
• Screenshots - Captured during execution
• Videos - Full execution recording
• Logs - Detailed step-by-step logs

Accessing Reports

• From notifications - Email and Slack include report links
• From GitHub - PR comments include report links
• From execution history - View past runs in the UI

Common Failure Patterns

Element Not Found

✗ Click the submit button
Error: Could not find element

Causes:

• Element doesn't exist on page
• Element has different text
• Element not visible (needs scroll)
• Element loaded asynchronously

Solutions:

• Verify element exists in browser
• Update description to match actual text
• Add wait condition before the step
• Check for typos in description

Timeout

✗ Wait for dashboard to load
Error: Timeout waiting for condition (30s)

Causes:

• Condition never becomes true
• Page loads differently than expected
• Slow network or server response

Solutions:

• Verify the condition text matches reality
• Check if page actually reaches that state
• Simplify the wait condition

Wrong Element

✗ Click the delete button
Error: Clicked wrong element

Causes:

• Multiple similar elements on page
• Description too generic

Solutions:

• Add location context ("in the sidebar")
• Include surrounding element info
• Use unique identifiers or labels

Assertion Failed

✗ User should be logged in

Causes:

• Previous steps didn't complete correctly
• Application behavior changed
• Assertion text doesn't match state

Solutions:

• Review step execution first
• Verify expected state manually
• Update assertion text

Debugging Workflow

1. Identify the failure - Which step or assertion?
2. Read the reasoning - What did AI try?
3. Check screenshots - What was the page state?
4. Reproduce manually - Does the flow work?
5. Adjust and rerun - Fix and verify

Success Metrics

Track these metrics across runs:

Metric	Meaning
Pass rate	% of successful runs
Flaky tests	Tests that sometimes pass, sometimes fail
Average duration	How long tests take
Failure patterns	Common failure types

Aim for:

• 95%+ pass rate
• Zero flaky tests
• Consistent durations