bringwookietome-blog - Tumblr blog

bringwookietome-blog · 8 years ago

Text

on E2E Terminology - from existence to value

Working on my thesis on quality assurance, being a kind of on the midway now and clearing issues related to terminologies. For instance something very popular like End-to-end testing, very thrilling to put that concept in paper exactly in the concept of my work.

Moving on, naturally online sources gives some insight what kind of activity this would be: Here are my best sources to this day:

End-toEnd testing guide - Guru99 Tech Pvt Ltd

What is End-to-End Testing - Exforsys Inc.

End-to-End Test - technopedia

Definitions may vary...

As a summary of above sources, E2E test is defined through several separate views:

A test scope definition: “e2e test is wider than a SUT(when test is a system test), then an interface or accompanying system is tested too”.

A Process flow definition: “A user logs in to web portal, searches a product, moves a product to chart, actuates buying of the product by going to register, clears out buying, logs out from portal”.

A reference to a end user: Test is executed as an end user of the application - making it an end to end user scenario, a.k.a. end to end test.

Genuinely I’d say I was left a bit vague, there are nothing wrong with those definitions. But due working with Information Technology thesis, my goal is look something more exact to a central work artefact of it.

Lets take another look what the sources say - End-to-end testing is a methodology used...

To test whether the flow of an application is performing as designed from start to finish.

To identify system dependencies.

To ensure that the right information is passed between various system components and systems.

To ensure that the integrated components of an application function as expected.

To test entire application in a real-world scenario (e.g. communicating with the database, network, hardware and other applications).

To know and understand the exact status of the system.

To find out in what manner system behaves in a real-time ambiance.

To find out how the underlying data that causes the user interface to act and function the way it does.

To check that the scenarios that cause the system and its subsystems to behave the way it does, happen as expected.

So it looks like there are actually quite many objectives, can it actually be that definitive testing?

Two Dimension of E2E make testing more interesting

There seems to be a separation in two methods.

Horizontal method: Method is that occurs horizontally, and across multiple applications’ context. i.e. A Web-based order-taking system might relate to the back-office accounts, inventory, and shipping sections. System has separate functionality by all those components of that system while all that functionality manages the same shared business data flow inside the system.

Verifying and evaluating transactions that each web order application does, or checking the correctivines of the web forms that are generated for user interface - both would be end-to-end testing for the system.

Vertical method: Vertical end-to-end testing means testing each individual layer of single applications’ built from top to the bottom.

In a typical web order-entry system that uses HTML codes to reach a web server, you will need API situated on the transaction server where it will generate SQL transaction codes against the mainframe database. Other applications might share the same API and SQL components, those are integrated to be a part of the system.

When test uses understanding these technical details of the system and its functionality in the test, it contributes widely used testing technique 'the white box testing technique'. This is end-to-end testing of that technology stack.

That being said, it becomes suddenly very clear that Vertical method contributes to 'the black box testing' methodology, where only functions of the system are tested in order to validate and check the inputs and outputs of the end-to-end test scenario.

And E2E is a combination

End to end testing combines the benefits of both the black box testing and white box testing. It approaches the program from both the functional point of view and the architectural perspective.

The steps required to perform an end-to-end test include the test design, that makes an appropriate test that will validate the functions of the system. What steps are needed to take to make the SUT to desired state may require white box techniques and a lot of technical understanding of the system functionality.

On the other hand, end-to-end test suite can be totally functional testing(Black box testing) in nature. So it does not have to be that way - but it can.

Good End-to-Ed test need to be focused

Most importantly, designing end-to-end tests it should be very clear on the scope of the test. Major guidelines for E2E -test design are:

It should focus on the designing a test from the end user’s perspective,

It should have a comprehensive scenario that must be used to create multiple cases,

It should focus on testing some of the features of the existing system.

Whats wrong with the E2E Tests?

A very famous critique to end 2 end testing is google’s blog(just say no to more end-to-end tests) about it destroying the effectiveness of the test procedure in order to find and correct software defects fast. In order the test to deliver any value, it should at least isolate the defect to make it pretty easy to analyse what is the problem in the product.

Number two attribute to test is if it is reliable. Fail a build and everything is ok(should not fail) - congratulations, you have just started an investigation for some people who’s contributed work is a total waste of their time.

Thirdly any given test should be fast to execute to make it useful. if a suite of very robust and useful test take over night to execute, that assurance is coming too late, there are already another half a dozen commits merged in, and fixing the issue on top of that its another mess why your test won’t be liked.

Critique states that all these three features are in fact properties of end-to-end tests. So why not write your e2e- automated tests just opposite of that? - oh yeah ;)

Test Objective approach

FIGURE(a system under test).

Here’s a SUT that has 3 modules A,B and C. It also has interfaces that may or may connect some of the modules with business data flow. Say we need to plan and execute e2e testing for that (without knowing the specifications). It doesn’t make much sense to test each module with any business data on e2e level, that kind of testing should be done on module(sub system) level(with checking the specifications).

It is needed to look for the business processes that exist in the system, then track them down and design tests to verify that each process flow is correct. It does not matter if the end user is involved in every test. Every e2e test might have different scope according the business function. But it may also be that several e2e test have same scope, and it should be considered that each business rule of that flow is investigated on e2e level.

Take a look module C, there are three entry points in that system: two black holes as inputs and one exit arrow to its right, something is not right - what are the business rules of this module? It’s looks like in this configuration whoever uses that system, doesn’t use all it’s features in production.

TDD and E2E

There is a good illustration that i picked from a very nice book: Growing Object-Oriented Software Guided By Tests by Steve Freeman, Nat Pryce. It describes this issue that google testing blog wrote about e2e tests not giving reasonable feedback, those tests are just not meant to do that.

Keep in mind when reading these curves, you are looking at the information that is a measure of the software quality, that information is captured each time when you change something and run ALL the tests. Every tests can reveal some aspects of the quality(they still don’t necessarily do that). General idea of this value of each test type is seen on x -axis for Unit-, integration and end-to-end tests.

FIGURE(feedback from tests)

Amount of information that you may receive from failing tests results gets fewer and fewer the larger the test scope is. In the book that information is called as “internal quality” that basically is a measure “is the software build right”.

The term “external quality” is reserved for other type of testing, that can be measured from unit test as well, those tests just don’t do that well. External quality is a measure to “did we build the right software”. These test give information if the business functions of the product is working as it was promised. Test may even be designed to cover several products and works processes to work together in some business meaningful way.

Summary

Make sure what is the test objective. How the SUT is supposed to do what is expected, plan your e2e test to confirm those assumptions.

After you have your test objectives cleared, desing and implement separete tests for each objective, othervice you can never tell what that test is about - go after clarity!

When designing E2E test, keep in mind you are testing the business rule of the product not some technicality or quality attribute of a software component. That kind of test must be written to unit or integration test level.

When designing e2e -tests look for verification of external quality attributes. Business rules are not about for example how you use page objects, those are part of products business workflow that is inside the test. Don’t make those things as separate tests, they are steps inside test. Any page object itself is part of technical implementation of the test, that exist in web browser memory. Don’t implement it as a part of test step(workflow) either - try to abstract page objects to user terms or workflow items.

Look for test objective of how the system function implements a business rule, a separate test objective leads to separate business rule-set and those lead to separate tests --> C-L-A-R-I-T-Y!

An E2E test doesn’t really care if it is written in unit, integration or end user level. You can definitely save a lot of execution time implementing your E2E tests on lowers test levels, and making those more robust too.

#end-to-end #testing #end-to-end-testing

0 notes