Open AI developed Chat GPT, an auto-generative technology for AI chatbots to use in providing online customer support. It employs Natural Language Processing (NLP) and has been trained to generate conversational responses. Textbooks, webpages, and other materials serve as its data source, from which it models its own language for reacting to human contact.
When it comes to the IT sector, software testing is one area where Chat GPT is predicted to thrive. In this post you may find out what ChatGPT is and how it may be an excellent tool for software testing.
What's the purpose of Chat GPT?
The primary function of Chat GPT is to provide text box replies that are convincingly human. This means it may be used for chats with chatbots, AI systems, and virtual assistants. In addition, it may provide conversational responses to inquiries and produce original works of fiction, poetry, and more. Not only that, but it may also:
- Develop and Debug software
- Translate
- Suggest some tunes and lyrics
- Do some creative writing.
Al-powered equipment are becoming more common in our daily lives, and Chat GPT represents a major breakthrough in this area. This model is cutting-edge and powerful because of its use of Deep Learning and Natural Language Processing (NLP). In the end, it may provide responses that seem natural and make sense to humans. Just because you think something doesn't make it so.
OpenAI's Chat GPT latest offering is a program based on the "GPT 3.5 language model." OpenAI has just published their fresh new massive multimodal model, GPT-4. Furthermore, in the next weeks, this new model will be included into Chat GPT. Meaning that users may now provide the bot with visual content (images or videos) in order to get a written answer.
Creating a Test Scenario with Chat GPT
Taking the example of creating a test case using Chat-GPT and see how it performs compared to a human tester so we can learn more but also, the efficacy and limits of Chat-GPT as an AI model contribution for software testing may be better understood via this comparison.
1) STEPS: Let's set up a Chat-GPT login test scenario for a predefined website.
Our given prompt was "Create a test scenario for testing the login of https://web.testquality.com " and it's shown below:
Our given prompt was "Create a test scenario for testing the login of https://web.testquality.com " and it's shown below:
2) RESULT: The result is acceptable, and the test scenario is created in an organized and appropriate way, covering all the steps.
3) ANALYZING THE OUTCOME
- Only considers the best possible outcome
We find that this scenario simply considers the best possible outcome. But if you ask a QA or a software tester the same thing, the answer given would take into account not only the best but also the worst case scenarios, plus any special circumstances that may arise in this login process.
- It provides the action asked
Chat GPT provides alternative answers but it will not provide a negative test scenarios, but a software tester can provide an immediate response with superior test coverage.
- Chat GPT works with limitations
The test login page which is shown below offers different possibilites with Single Sign-On (SSO) Login which is an authentication process that allows users to access multiple enterprise applications with a single set of login credentials. This combines several application logins into one, where you use credentials once and get access to GitHub without individually logging into this application.
It's sure, these kinds of errors are less likely to be found by a human software tester.
It's sure, these kinds of errors are less likely to be found by a human software tester.
Using Single Sign On with applications that support OpenID
As explained, this is an example where a software tester will think creatively in order to devise scenarios and tests that evaluate the user-friendliness of the user interface, as well as logos and graphics, all of which are challenging for an artificial intelligence to comprehend and analyze.
The facts presented here make it abundantly evident that insights, efficacy, and some kinds of limits of dificulties using Chat GPT for software testing as well as the reasons why it cannot replace a software tester.
Although Chat GPT and other AI technologies can be used as part of manual and automated testing frameworks and enhance the capabilities of software testers in a positive way, it is not possible for these technologies to fully replace the technical skills, standard quality, and expertise that a skilled QA tester brings to a project. No matter how worthless and meaningless the human experience may eventually become with the evolution of time, ChatGPT's mission is that it is destined to replicate human behavior and as of today, it is unable of ever having an actual human experience.
The facts presented here make it abundantly evident that insights, efficacy, and some kinds of limits of dificulties using Chat GPT for software testing as well as the reasons why it cannot replace a software tester.
Although Chat GPT and other AI technologies can be used as part of manual and automated testing frameworks and enhance the capabilities of software testers in a positive way, it is not possible for these technologies to fully replace the technical skills, standard quality, and expertise that a skilled QA tester brings to a project. No matter how worthless and meaningless the human experience may eventually become with the evolution of time, ChatGPT's mission is that it is destined to replicate human behavior and as of today, it is unable of ever having an actual human experience.
Amazing Automated Testing Made Possible Thanks to ChatGPT
Nevertheless we think that the fundamental AI models (GPT4, Sage, Claude or Bloom, an open-source alternative), which are used to construct tools like Chat GPT, will have the most significant influence, which will arrive in waves, and will become incorporated across a variety of products.
The most apparent use case is in the development and editing of test scripts, followed by the usage of test data.
For example, it is possible to convert selenium code from java to javascript
It's possible that it's not completely correct, and it could require. some assistance from humans to mend and improve... Yet, it was responsible for much of the work.
Inside TestQuality, one of the things that we are focusing on is developing an artificial intelligence that can correctly produce test scripts and change them. And from our point of view, the final aim is one in which you essentially have a conversation with a bot about how to build test scripts and how they may be improved. (Similar to the code translation described above, but applied to your complete test code).
As a result of our recent trials, we have determined that it has a great deal of inaccuracy in a variety of respects as explained before; hence, its function will consist mostly of that of an assistance to the human testers. And is confused by certain odd things, namely the fact that some flaws may be considered features. In essence, the elements of the input that are more subjective.
The most apparent use case is in the development and editing of test scripts, followed by the usage of test data.
For example, it is possible to convert selenium code from java to javascript
It's possible that it's not completely correct, and it could require. some assistance from humans to mend and improve... Yet, it was responsible for much of the work.
Inside TestQuality, one of the things that we are focusing on is developing an artificial intelligence that can correctly produce test scripts and change them. And from our point of view, the final aim is one in which you essentially have a conversation with a bot about how to build test scripts and how they may be improved. (Similar to the code translation described above, but applied to your complete test code).
As a result of our recent trials, we have determined that it has a great deal of inaccuracy in a variety of respects as explained before; hence, its function will consist mostly of that of an assistance to the human testers. And is confused by certain odd things, namely the fact that some flaws may be considered features. In essence, the elements of the input that are more subjective.
These are some of the common ways through which you can use Chat GPT for automation examples.
1) TEST DATA CREATION
Chat GPT may be trained by professionals to operate on massive amounts of data. Because of this, it is able to create test data that are identical to the data from the actual world. In the end, it automates the whole procedure and optimizes the amount of time that is necessary. A few instances of client data include a customer's address, name, and other contact details.
Our given prompt was "Create an array of 3 users consisting of firstname, city and date of birth in json format" and it's shown below:
[ { "firstName": "John", "city": "New York", "dateOfBirth": "1985-02-15" }, { "firstName": "Maria", "city": "Los Angeles", "dateOfBirth": "1992-07-24" }, { "firstName": "David", "city": "Chicago", "dateOfBirth": "1978-11-03" }]2) CREATE A GHERKIN CUCUMBER SCENARIO WITH CHAT GPT
Moreover, Chat GPT is able to efficiently build test scenarios in Gherkin by first assessing the requirements and then translating those requirements into comprehensive methods for carrying out the scenarios.
Hence, the highest level of accuracy is achieved when test cases are generated using Chat GPT. The easiness with which users may enter the system using a username or email address is something that the testers might stipulate as a criteria. As a result, Chat GPT is able to provide test cases such as "test login with username," "Test login with an email address," etc.
We assumed that Chat GPT would be capable of creating Gherkin Cucumber scenarios, but we were interested about the writing style that it would adopt.
Since we were curious in seeing how Chat GPT would go about writing Cucumber scenarios to test the login page of the TestQuality app, we decided to use this prompt: "Create a cucumber example to login to https://web.testquality.com"
Since we were curious in seeing how Chat GPT would go about writing Cucumber scenarios to test the login page of the TestQuality app, we decided to use this prompt: "Create a cucumber example to login to https://web.testquality.com"
Feature: Login to TestQuality website
As a registered user
I want to login to the TestQuality website
So that I can access my account
Scenario: Login with valid credentials
Given I am on the TestQuality login page
When I enter valid login credentials
And click the login button
Then I should be redirected to the TestQuality dashboard page
Scenario: Login with invalid credentials
Given I am on the TestQuality login page
When I enter invalid login credentials
And click the login button
Then I should see an error message displayed on the pageAnd here's an example of step definitions for the above scenario:
const { Given, When, Then } = require('cucumber');
const { Builder, By, Key, until } = require('selenium-webdriver');
const driver = new Builder().forBrowser('chrome').build();
Given('I am on the TestQuality login page', async function () {
await driver.get('https://web.testquality.com/login');
});
When('I enter valid login credentials', async function () {
await driver.findElement(By.name('email')).sendKeys('user@example.com');
await driver.findElement(By.name('password')).sendKeys('password123', Key.RETURN);
});
When('I enter invalid login credentials', async function () {
await driver.findElement(By.name('email')).sendKeys('user@example.com');
await driver.findElement(By.name('password')).sendKeys('invalidpassword', Key.RETURN);
});
Then('I should be redirected to the TestQuality dashboard page', async function () {
await driver.wait(until.urlContains('dashboard'));
await driver.quit();
});
Then('I should see an error message displayed on the page', async function () {
const errorMessage = await driver.findElement(By.css('.error-message')).getText();
expect(errorMessage).to.equal('Invalid email or password.');
await driver.quit();
});Note that in the above example, we're using the "selenium-webdriver" library to interact with the Chrome browser and perform actions like clicking buttons and entering text in input fields. We're also using the "chai" library to make assertions about the results of our test steps.
The preceding example, might still be enhanced a bit, but it serves as an excellent starting point for the creation of several situations. It is important to take into consideration that Chat GPT reacts according on the training it has received. But, the advantage of Chat GPT is that you may iterate on the replies, so even if it has been trained on public Github projects, you will still likely to receive whichever style is predominant by default.
3) CREATE A CYPRESS TEST CASE WITH CHAT GPT
After seeing what Chat GPT was capable of, we wondered how it would be capable of creating a Cypress test case. However, we were more interested in knowing the generated script to test the login page of the TestQuality app after using this promp: "Can you write the commands for the login test of https://web.testquality.com using the Cypress test tool?"
describe('Login Test', () => {
it('Should successfully log in', () => {
cy.visit('https://web.testquality.com')
cy.get('input[name=email]').type('your_email')
cy.get('input[name=password]').type('your_password')
cy.get('button[type=submit]').click()
cy.url().should('include', '/dashboard')
})
it('Should fail to log in with incorrect credentials', () => {
cy.visit('https://web.testquality.com')
cy.get('input[name=email]').type('incorrect_email')
cy.get('input[name=password]').type('incorrect_password')
cy.get('button[type=submit]').click()
cy.contains('Invalid email or password').should('be.visible')
})
})The first test case logs in with valid credentials and checks that the URL contains '/dashboard' to ensure that the login was successful. The second test case attempts to log in with incorrect credentials and checks that an error message containing the text 'Invalid email or password' is displayed on the page. (Note that you'll need to replace 'your_email' and 'your_password' with your actual login credentials to run these tests.)
What are the disadvantages of ChatGPT when testing software?
The following are the current restrictions of ChatGPT that prevent it from being used for comprehensive software testing.
1. There Is a Lack of quality of Test Cases to Use
While creating test cases, inputs are crucial. Unfortunately, accuracy and precision cannot be assured in the final product. It may not be able to create the specialized tests and cases needed to cover edge situations in software testing.
1. There Is a Lack of quality of Test Cases to Use
While creating test cases, inputs are crucial. Unfortunately, accuracy and precision cannot be assured in the final product. It may not be able to create the specialized tests and cases needed to cover edge situations in software testing.
2. There Is a Lack of test context awareness
One of the main drawbacks of utilizing Chat GPT for programming is that it does not have context awareness. This signifies that the resulting code applied to testing scenarios may not be suitable for the intended purpose. Before utilizing the model to create code, developers should double-check that the code produced is suitable for their testing needs and the environment in which it will be utilized.
One of the main drawbacks of utilizing Chat GPT for programming is that it does not have context awareness. This signifies that the resulting code applied to testing scenarios may not be suitable for the intended purpose. Before utilizing the model to create code, developers should double-check that the code produced is suitable for their testing needs and the environment in which it will be utilized.
3. Need to depend on training data
The quality and relevance of the training data used to train ChatGPT for coding is closely correlated with the accuracy and dependability of the resulting model. The produced code may not be up-to-date or correct if the model was trained using stale or low-quality data.
The quality and relevance of the training data used to train ChatGPT for coding is closely correlated with the accuracy and dependability of the resulting model. The produced code may not be up-to-date or correct if the model was trained using stale or low-quality data.
4. Human review and test validation
While Chat GPT can create testing script rapidly and correctly, it still requires human review and validation by QA engineers or Testers. This is because the model may not always provide software code that is suitable for a certain project's needs or circumstances.
While Chat GPT can create testing script rapidly and correctly, it still requires human review and validation by QA engineers or Testers. This is because the model may not always provide software code that is suitable for a certain project's needs or circumstances.
5. Limited Creativity producing test scripts
Although Chat GPT may produce code that is syntactically valid and makes sense semantically, it falls short when it comes to producing test script that is innovative or creative. It is very improbable that the model would come up with new, effective solutions to complicated issues, given it has been taught to write code based on current patterns and practices.
Although Chat GPT may produce code that is syntactically valid and makes sense semantically, it falls short when it comes to producing test script that is innovative or creative. It is very improbable that the model would come up with new, effective solutions to complicated issues, given it has been taught to write code based on current patterns and practices.
In conclusion
Chat GPT might be useful in certain circumstances, however at the moment the test script needs a QA professional or Test developer to evaluate and validate it. Needless to say that Chat GPT has the potential to be efficient. Time savings may be lost if this doesn't happen. If the development team fails to catch an issue, fixing it might take longer and cost more money.
Chat GPT is only viable because generative AI, NLP, and other related technologies have matured to the point where they can provide value in a variety of contexts. But keep in mind that we're still talking about their beginnings. There is benefit in using Chat GPT for coding right now, and it will only become better as time goes on, and more AI tools are almost certainly in the future.
Chat GPT is only viable because generative AI, NLP, and other related technologies have matured to the point where they can provide value in a variety of contexts. But keep in mind that we're still talking about their beginnings. There is benefit in using Chat GPT for coding right now, and it will only become better as time goes on, and more AI tools are almost certainly in the future.
TestQuality can simplify test case creation and organization, it offers a very competitive price used within your GitHub workflows or Jira Test Management providing Rich and flexible reporting that can help you to visualize and understand where you and your dev or QA Team are at in your project's quality lifecycle. But also look for analytics that can help identify the quality and effectiveness of your test cases and testing efforts to ensure you're building and executing the most effective tests for your efforts.
