Skip to content

Project 5: Testing

Learning Goals

  • Learn how to write unit tests using multi-shot prompting.
  • Learn how to run tests locally.
  • Learn how to automate test runs using GitHub continuous integration.

Project Context

In P1, you defined requirements and user stories. In P2, you expanded some of those stories into detailed development specifications. In P3, you implemented two user stories for the front end of your application. In P4, you implemented two dev specs for the back end of your application. In P5, you will write unit tests for two core frontend and two core backend files/classes.

By the end of this sprint, you'll create four test files that unit test your application's features. You will also create a test script to run the tests on your local machine. Finally, you will automate test execution so that it runs on GitHub every time someone commits code to the repository. You will learn to look at the results of those tests to make sure your code doesn't break the build without you knowing about it right away.

Remember to use the LLMs as much as possible to generate your deliverables. You may not modify any generated code directly, only by prompting the LLM. You should use formalized LLM prompts, such as those we introduced in class. These will make the LLM output more reliable.

Deliverables

You will begin by writing a test specification for two code files in the frontend of your codebase and unit tests for code files in the backend of your codebase. You should choose the most important core code files that enable your user stories to run.

A test specification is an English-language document that describes the purpose of each function to be tested along with every program path that should be tested with a unique unit test. For each unit test, the document should describe the inputs to the function that are required to engage the desired program path and the expected output. Your goal should be 80% code coverage. This means that for each function to be tested, your unit tests will exercise at least 80% of the function's possible execution paths. Don't forget about exceptions!

Next, you will implement the spec by creating Javascript (or Typescript) unit tests for each function. Ensure that your tests are isolated to the frontend or backend; they should not test functionality that requires connecting across the network from the frontend to the backend or vice versa. If the function under test requires connecting to the other end, you must create mock objects that simulate the other end's public interface.

You will then create scripts to setup the application and run the unit tests on your own machine. Finally, you will create a GitHub Action that sets up and runs all of your unit tests whenever anyone commits code to the GitHub repository.

1. Choose the files you want to test

Look through the source code of your frontend and identify two code files that contain the most core functionality that implements the two frontend user stories. There must be at least 5 functions in each file.

Look through the source code of your backend and identify two code files that contain the most core functionality that implements the two backend user stories. There must be at least 5 functions in each file.

2. Write the test specification

Ask your LLM to write one English-language test specification for each code file you plan to test. Each specification should contain a list of all functions in the code file, followed by a table of tests. Each row of the table should describe the purpose of the test, the test inputs to the function, and the test output that is expected if the test passes. You must write at least one unit test for every function.

For example, you have a validateEmail(string address) function to test. One possible test may check whether GMail addresses are considered valid. The input address would be "realemailaddress@gmail.com" and the expected output would be the boolean "true".

2. Create your unit tests

Most projects that are written in Javascript or TypeScript should use the Jest unit testing framework. You may use another framework, but you may not simply manually test the code. You must use a testing framework.

If you are building a VS Code extension, you must use VS Code's preferred testing framework, Mocha. See VS Code's Extension Testing Instructions on how to set it up.

First, install the test framework into your application. Create a tests/ folder and keep all of your test files there. Next, have your LLM read the test specification document and generate the required unit tests. If a mock is needed, have your test framework do it for you.

  1. Jest
  2. Mocha uses SinonJS to generate mocks.

Generate unit tests for each test specification using your LLM. At the end, you should have four test files.

Note

We strongly suggest that you use the LLM to generate only one unit test at a time. We have learned from experience that trying to get the LLM to do the entire thing in one prompt will lead to incorrect output.

Note

You will be graded on how well you prevent the LLM from hallucinating nonsensical test cases or creating duplicate or significantly overlapping test cases. The LLM must not generate test cases for functions and functionalities that do not exist.

3. Run the tests on your own machine

Ask the LLM to generate an npm script to setup the frontend or backend of your application, as needed, and then execute the tests with your testing framework.

How did it go? Did every test pass? If not, use your LLM to give you a plan on how it wants to fix the bugs (ask it for three alternative fixes). Choose the bug fix you like and have the LLM make the change. Did your test case pass? Congratulations! If not, try again.

4. Check code coverage

Check to make sure that you have achieved at least 80% code coverage in each test file. Use your test framework to do this --- the LLM will not be able to check this for you.

  1. For Jest, read through this blog post to see how to check coverage.
  2. For Mocha, read through this blog post.

Check in your test code to the GitHub repository for your project.

5. Automate your tests

Adding continuous integration for quality assurance is a critical part of software development. Although you have tested your system manually, you are now setting out to establish sustained practices that can be used moving forward as you iterate over and continue to improve your system.

Create GitHub Actions that run your tests on every commit. You will need to create two YAML workflow files in the .github/workflows directory. The first, for your frontend code, should be named run-frontend-tests.yml and the second, for your backend code, named run-backend-tests.yml. Each YAML file should check out your code, set up the application environment (e.g., install Node.js if needed), install dependencies, and then execute your tests.

  1. For CI instructions to run tests with Jest, check out Dennis O'Keefe's blog.
  2. For CI instructions with VS Code extensions, check out these instructions.

Check in your YAML files to the GitHub repository.

Test out your CI code by making a change to one of the source code files in the frontend and one backend source code file. Commit the change to Git, push to the remote repository, create a pull request on GitHub, and accept the pull request. Finally, go to the GitHub repository on the web and click on teh Actions tab on the navigation bar. You'll see all the workflows on the left and workflow runs on the right. If you have a green checkmark next to a workflow run, that means it worked! If there is a red cross, then it did not. Click on the workflow run to see exactly what got executed in GitHub's "terminal window" and find out what went wrong. Fix the problem and try again until each of your two GitHub actions run successfully.

Wrap it up

Edit the README.md file for your project.

  1. Provide all the instructions needed for how to manually run the frontend tests on a developer's local machine. Don't forget to explain what frameworks and libraries need to be installed!
  2. Provide all the instructions needed for how to manually run the backend tests on a developer's local machine. Don't forget to explain what frameworks and libraries need to be installed!

6. Reflection

Write a 500-word (i.e., one-page) reflection on:

  1. How effective was the LLM in generating the test specification? What did you like about the result? What was wrong with what the LLM first generated? What were you able to fix it easily? What problems were more difficult to fix?
  2. How effective was the LLM in generating unit tests from the test specification? What did you like about the result? What was wrong with what the LLM first generated? What were you able to fix it easily? What problems were more difficult to fix?
  3. How did you verify that the LLM correctly did what you asked? How did you use the test framework or the LLM to help you understand if everything was done correctly?

Turn-in Instructions

Please turn in a single document that contains these parts:

  1. The two code files you chose to test for your frontend.
  2. The frontend test specifications.
  3. Provide a Git repository link to the test code files that implement your frontend test specs.
  4. Copy-paste in the test output from running your frontend tests.
  5. Copy-paste in the code coverage report for your frontend test files.
  6. The two code files you chose to test for your backend.
  7. The backend test specifications.
  8. Provide a Git repository link to the test code files that implement your backend test specs.
  9. Copy-paste in the test output from running your backend tests.
  10. Copy-paste in the code coverage report for your backend test files.
  11. Provide a Git repository link to your run-frontend-tests.yml file.
  12. Provide a Git repository link to your run-backend-tests.yml file.
  13. Provide a GitHub link to a workflow run of run-frontend-tests that shows it ran successfully after frontend code was committed.
  14. Provide a GitHub link to a workflow run of run-backend-tests that shows it ran successfully after backend code was committed.
  15. Provide a link to your project's README in GitHub.
  16. A 1-page reflection as in the Reflection section.
  17. Copy-paste logs of all LLM interactions you used during this sprint. Identify the name and version of the LLM used.