The Pros and Cons of End-to-End Testing

How to perform E2E tests for an Online E-Commerce Store?

Objective:

To conduct a thorough End-to-End (E2E) testing on an online shopping platform to ensure a seamless shopping experience from account registration to order confirmation.

Test Strategy Development:

• Aim: To validate the complete functionality of the e-commerce platform, ensuring that all user actions lead to the expected outcomes without errors.

  
  

• Key Customer Journey to Test: The process from creating a new account, finding products, adding items to the cart, checking out, making payment, to receiving an order confirmation.

Testing Environment Configuration:

Set up a staging environment that closely mirrors the production environment, including web servers, databases, and mock services for external integrations like payment gateways.

Test Cases Development:

1. Account Registration

   • Purpose: To confirm that users can successfully register on the platform.

   • Procedure: Navigate to the signup page, fill out the registration form with details (username, email, password), and submit.

   • Expected Result: The user is registered and receives a confirmation email.

     
     

2. Login Functionality

   • Purpose: To ensure that the login mechanism works correctly with valid user credentials.

   • Procedure: Go to the login page, enter valid email and password, and submit.

   • Expected Result: The user is logged into their account and directed to the homepage.

     
     

3. Product Browsing and Selection

   • Purpose: To verify that users can browse through the product listings and access product details.

   • Procedure: Visit the product listing section, choose a category, and select a product to view its details.

   • Expected Result: The product's details page loads with all the relevant information.

     
     

4. Adding Product to Cart

   • Purpose: To test the functionality of adding products to the shopping cart.

   • Procedure: From a product's details page, click the "Add to Cart" button.

   • Expected Result: The product is added to the cart, and the cart's item count is updated.

     
     

5. Checkout Process

   • Purpose: To confirm the checkout process is intuitive and error-free.

   • Procedure: Access the shopping cart, click "Proceed to Checkout," enter necessary shipping and billing information, and submit.

   • Expected Result: The user is navigated to the payment page.

     
     

6. Payment Transaction

   • Purpose: To ensure the payment process is secure and processes transactions correctly using mock payment details.

   • Procedure: Input mock payment information and submit.

   • Expected Result: The payment is processed, and an order confirmation screen is shown.

     
     

7. Order Confirmation

   • Purpose: To verify that the order confirmation details are accurate and an email confirmation is sent.

   • Procedure: After payment, confirm the details on the order confirmation page and check for an email confirmation.

   • Expected Result: The order details are correct, and an email confirmation is received.

     
     

Preparation of Test Data:

Data Needed: User credentials for login tests, product details for browsing and selection, and mock payment information for checkout.

Perform End-to-end testing without the need to prepare test data, learn how?

Execution of Tests:

Automated testing scripts (e.g., using Selenium WebDriver) will simulate the user journey from registration to order confirmation, asserting the expected outcomes at each step.

# Example of a Python script using Selenium for automated E2E testing
from selenium import webdriver
import unittest

class E2ETesting(unittest.TestCase):
    def setUp(self):
        self.browser = webdriver.Chrome('path/to/chromedriver')
        self.addCleanup(self.browser.quit)

    def testCompleteUserJourney(self):
        # Detailed steps for each part of the test go here, including:
        # - Navigating to the site
        # - Registering a new account
        # - Logging in
        # - Browsing products and adding to cart
        # - Checking out
        # - Verifying order confirmation
        pass

if __name__ == '__main__':
    unittest.main()

Analysis of Test Results:

After executing tests, analyze logs and outputs to identify any bugs or issues with the platform.

Test Reporting:

Compile a detailed report of the testing process, findings, and recommendations. This report should include test coverage details, success rates, bugs identified, and screenshots or logs as evidence.

This comprehensive approach to E2E testing ensures the online shopping platform functions correctly across all user interactions, offering stakeholders confidence in the platform's reliability and user satisfaction.

The Pros of E2E testing

E2E tests offers the full picture of the test scenario, offering advantages like:

1. Replicates Real-User Experience: E2E testing evaluates the system's overall functionality and its interaction with external interfaces, databases, and other systems, mirroring real-world user scenarios and behaviors.

   
   

Scenario: Testing a login feature in an application.

describe('Login Feature', () => {
  it('successfully logs in the user', () => {
    cy.visit('/login') // Navigate to the login page
      .get('input[name="email"]').type('user@example.com') // Enter email
      .get('input[name="password"]').type('password123') // Enter password
      .get('form').submit() // Submit the login form
      .get('.welcome-message').should('contain', 'Welcome back, user!'); // Verify login success
  });
});

Real-User Experience: This code simulates a user navigating to the login page, entering their credentials, and submitting the form, closely mirroring a real user's actions.

Increases Confidence: Verifying the presence of a welcome message after login actions ensures the application behaves as expected, boosting confidence in deployment.

• 2. Identifies System-wide Issues: It helps uncover issues related to data integrity, services integration, and user interface, which might not be detected during unit or integration testing phases.

  
  

  3. Facilitates Compliance with Requirements: For applications in regulated sectors, E2E testing ensures that the software meets necessary compliance standards, including security protocols and data handling practices.

  
  

  4. Supports Continuous Integration/Continuous Deployment (CI/CD): Automated E2E tests can be integrated into CI/CD pipelines, enabling regular testing at various stages of development, which helps in identifying and addressing issues promptly.

  
  

The Cons of E2E testing

This test pyramid approach needs to be modified for testing microservices. E2E tests need to be completely dropped. Apart from taking a long time to build and maintain, E2E tests execute complete user-flows every time on the entire application, with every test.

This requires all services under the hood to be simultaneously brought up (including upstream) even when it is possible to catch the same kind and the same number of failures by testing only a selected group of services; only the ones that have undergone a change.

1. Resource Intensive: E2E testing can be time-consuming and expensive due to the need for comprehensive test cases, the setup of testing environments that mimic production, and potentially longer execution times for tests.

   
   

Scenario: Setting up a Selenium test environment for the same login feature.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

# Setup WebDriver
driver = webdriver.Chrome()

# Navigate to the login page
driver.get("http://example.com/login")

# Enter login details and submit
driver.find_element(By.NAME, "email").send_keys("user@example.com")
driver.find_element(By.NAME, "password").send_keys("password123")
driver.find_element(By.NAME, "submit").click()

# Verification
assert "Welcome back, user!" in driver.page_source

# Teardown
driver.close()

Resource Intensiveness: Setting up Selenium, managing WebDriver instances, and ensuring the environment matches the production settings can be time-consuming and resource-heavy.

Complexity in Maintenance: The Selenium example requires explicit browser management (setup and teardown), which adds to the complexity, especially when scaling across different browsers and environments.

1. Flakiness and Reliability Issues: E2E tests can sometimes produce inconsistent results due to their reliance on multiple external systems and networks, leading to flakiness in test outcomes.

   
   

2. Slow Feedback Loop: Due to the extensive nature of E2E tests, there can be a significant delay in getting feedback, which can slow down the development process, particularly in agile environments that prioritize quick iterations.

   
   

3. Not Suited for All Types of Testing: E2E testing is not always the best choice for detecting specific, low-level code issues, which are better identified through unit testing or integration testing.

   
   

Perform E2E Testing without test data preparation

The flakiness and complexity of End-to-End (E2E) tasks often stem from the need for test data preparation. For E2E scenarios to run smoothly, it's essential to create and maintain relevant test data.

In the context of app testing, particularly for e-commerce platforms like Nykaa or Flipkart, the process is akin to testing different states of the app.

For example, verifying if a user can apply loyalty points for a discount involves specific state testing.

• Requirements for Test Data: To test the aforementioned scenario, a QA engineer must prepare several pieces of test data, including:

  • A valid user account

  • A valid product listing

  • Sufficient inventory for the product

  • The addition of the product to a shopping cart

  This setup is necessary before the app reaches the state where the discount via loyalty points can be applied.

  
  

The scenario described is relatively straightforward. However, an e-commerce app may contain hundreds of such flows requiring test data preparation. Managing the test data and app states for numerous scenarios significantly increases the workload and stress for QA engineers.

Fortunately, there exists a straightforward approach that allows QA engineers to test the functionality of an application without the need for extensive test data creation and management. This method focuses on testing the core functions directly, alleviating the burden of test data preparation. Click here to learn more now.

Conclusion

Concluding our discussion on the pros and cons of end-to-end (E2E) testing, it's evident that E2E testing is a critical tool in the software development but it comes at the cost of time, money and effort. They’re extremely difficult to write, maintain and update. An E2E test that actually invokes the inter service communication like a real user would catch this issue. But cost of catching this issue with a test that could involve many services would be very high, given the time and effort spent creating it.

• imprecise because they've such a broad scope

• needs the entire system up & running, making it slower and difficult to identify the error initiation point

  
  

The essence of navigating E2E testing successfully is choosing the right tools, automating where possible, and continuously refining testing processes to align with project needs and goals. Get in touch with us if you want to test E2E scenario’s without needing to spend any time creating and managing test data.