Software Testing

Testing

The driving force behind our ability to provide our auction software faster and cheaper than our competitors is our robust and complete testing infrastructure. The sheer volume of tests our software contains allows us to always know our software is working properly, and allows us to focus our time, and our clients money, on creating their auctions with their auction rules. We don't need to spend time testing our core code, and we can spend a minimal amount of time creating new tests specifically for our clients and their unique auction designs. Our competitors have a bad, and expensive, habit of making their clients pay exorbitant amounts to test their auctions - in essense, their new clients are paying the same "testing fees" that all the previous clients have also paid. Our competitors are double, even triple, dipping on their fees. We prefer to keep our costs low, and only make our clients pay for their new unique tests. All the thousands of existing tests in our software are included for free!

Our software has a total and complete unit test framework that is run against the software every single day, assuring that our software is always working and always capable of running auctions properly. In addition to our complete unit test framework, we also have created a robust system test framework, which allows our team of economists to write tests against our software. These tests run every one of our auction clients' tests against the software - both current clients and all our past clients. In essence, our software today is capable of running every single one of our past client's auctions. New clients can feel assured that they aren't getting a new untested version of the software.

3 Types of Tests

A "unit test" is defined as a simple and straighforward test that tests the correctness of a small block of code. These tests don't test any auction logic in our software, but instead test all the underlying code used in the software. Tests like "does rounding this number to the nearest penny work correctly?" and "will a new password get saved to the database correctly?". It is a goal of any unit test suite to test, quite literally, every single line of code in a piece of software. There are certain tools available that let you measure this number, called "code coverage". Our code coverage is discussed more below. It's a general rule of software development that the more unit tests within a software product, the less bugs exist in the product. But, the unit tests can only test if code runs correctly without having errors - it can't test whether auction rules and auctions perform correctly.

In order to test the auction logic and auction rules in our software, we have created "system tests". System tests are much more involved tests that don't focus on the individual lines of code in the software, but instead focus on the individual auction rules and auction situations that may occur in typical auctions. In addition to having a general suite of system tests for our various auction configurations (SMR, Clock, CCA, Dutch, Dynamic), we also create a suite for each client. In other words, each client that works with Optimal Auctions has a dedicated suite of system tests created for them, tests that are able to confirm the auction rules and configurations that are unique to their auction design. Our system tests are able to test these auction rules by literally running dozens, even hundreds, of fake auctions through our auction software, checking everything along the way. For example, a 10 round auction can be simulated, and the software can check all the prices, all the bids, the winners, and everything else that takes place in an auction. Our system tests allow us to simulate far more auctions than could ever be run manually, and allows us to run these tests consistenly and quickly at any time.

To test our user interface, ane ensure that what every bidder sees in their browser is correct, and ensure that their bids are processed properly, we have a suite of "user interface tests". These tests use the popular UI testing tool called Selenium. These tests mimic the actions of a bidder, using the browser and pressing buttons, clicking checkboxes, and doing everything exactly like a real bidder would do in the auction. These tests are the perfect way to simulate how an auction works from a bidder's perspective.

Finally, other types of tests are run against our software, and these are discussed on other pages - Scalability Tests and Performance Tests.

Test Suite Statistics

  • Our unit test suite has 2,995 tests in it.
  • Our unit tests cover an amazingly high 96% of our code, as measured by the popular code coverage tool Cobertura. 96% is a staggering high coverage, especially considering in general 75% is considered "well covered" when discussing software.
  • Our unit test suite is run every day to ensure the software is always in perfect working condition. If a single test fails, our software team doesn't go home.
  • Our system test suite has 528 tests in it. Essentially this means we run 528 simulated auctions against our software.
  • Our 528 system tests simulate a total of 3,154 rounds of auction.
  • Our system test suite is run every time a new version of the software is created, to ensure the software always runs every client's auction, past and present, correctly.
  • Our software has 71 user interface tests.
  • Our user interface tests cover every button and text field in the software.
  • Our total test suite (unit tests, system tests, and user interface tests) takes 30 minutes to complete.

Unit Test Suite

The unit test suite tests the core of the auction software. It tests all the underlying code used to make the auctions run properly and essentially ensures that the fundamental framework on which we build auctions is working properly. The importance of this can't be understated - knowing without a doubt that the core code is working properly saves a tremendous amount of time when it comes to creating auctions for new clients. It allows us to focus exclusively on their auction rules without having to worry about other parts of the software. It's what allows us to deliver our software cheaper and faster than our competitors.

The best practice of unit tests is to keep them easy - write lots of tests, but keep them easy and straightforward. Here is an example of our unit tests. These tests, which come directly from our code, test whether a number is rounded correctly in the code. Notice how each test is simple, testing one number. There are then many tests to test all the possible scenarios. This approach allows for a more robust and better-covering unit test suite.

Unit Test Example

System Test Suite

The most important aspect of our system tests is that they allow our economists to write tests without having to know computer code! The advantages of this design are tremendous - economists know best about tricky auction situations, auction rules, and auction configurations, and naturally it makes the most sense for economists to write the tests to make sure the software handles these situations properly. Our economists are the experts in auction theory, and we want them to be able to test the code. It's not only important, it's critical, to have economists check every aspect of the design.

Optimal Auctions is proud of our custom designed and custom coded solution to the this problem of how to get economists to write auction tests without making them learn any computer code. We've created an extremely powerful internal software product that allows our team of economists to write tests using only Excel. Excel! They can write the tests in Excel, lay out auction rules they want to test, and our unique testing solution can translate those Excel files into tests to be run against the software.

The importance of this feature can't be stressed enough - our system tests are written by economists using Excel. Our software converts the Excel files to tests and preserves them forever.

Here's a quick view at an Excel file which shows just how easy it is to create system tests for our team of economists.

System Test Example

Optimal Auctions' test suites allow us to deliver superior tests, written by our team of economists. As a result, our software is better tested than our competitors. It also allows us to deliver our software faster and cheaper than our competitors. Our robust test suite has given us a distinct competitive advantage in the auction software industry.