WTST 12 2013

JANUARY 25-27, 2013


The Workshop on Teaching Software Testing is concerned with the practical aspects of teaching university-caliber software testing courses to academic or commercial students.

WTST 2013 is focused on high volume automated testing (HiVAT). Our goal is to bring together instructors who have experience teaching high-volume techniques or who have given serious consideration to how to teach these techniques. We also welcome participants focused on the teaching of complex cognitive concepts and the transfer of what was learned to industrial practice.

As at all WTST workshops, we reserve some seats for senior students who are strongly interested in teaching and for faculty who are starting their careers in this area or beginning a research program connected with teaching this type of material.

There is no fee to attend this meeting. You pay for your seat through the value of your participation. Participation in the workshop is by invitation based on a proposal. We expect to accept 15 participants with an absolute upper bound of 25.


High volume automated testing involves automated generation, execution and evaluation of the results of a large set of tests. This contrasts with more traditional “automated” testing that involves automated execution of a relatively small number of human-created tests.

Here are four examples of the types of problems that underlie the need for HiVAT:

  1. Many types of code weakness (such as timing-related problems) yield intermittent failures and are hard to detect with traditional testing techniques
  2. There are immense numbers of possible combination tests of several variables together. Some combinations are unique (a failure appears only on a particular combination)
  3. Some failures occur primarily when a system under test is under load and so detection and characterization is essentially a statistical challenge.
  4. Characterizing the reliability of software requires a statistically useful set of tests.

In the academic community, the most commonly discussed HiVAT family of techniques is called “fuzzing” – but fuzzing as we know it involves very simplistic evaluation of the test results—essentially run the software until it crashes or fails in some other very obvious way. Other HiVAT techniques rely on more powerful oracles and can therefore find other kinds of bugs.

WTST is about teaching testing, not creating new techniques. The challenge we are trying to address in this WTST is that many of these techniques are known but not widely applied. We believe this is because they are poorly taught. As far as we can tell, most testing courses don’t even mention these techniques. Of those that do (and go beyond fuzzing), our impression is that students come out baffled about how to actually DO that type of testing in their work.

At Florida Tech, we’re trying to address this by creating “reference implementations” for several techniques—open source demonstrations of them, with commentary on the design and implementation. We’re hoping that WTST will provide examples of other good approaches.

The hosts of the meeting are:

Confirmed attendees include:

Name Affiliation Presentation
Kaner, Cem Florida Tech An Overview of High Volume Automated Testing
Karena, Catherine
Kelly, Michael DeveloperTown Workshop Facilitator
Fiedler, Rebecca Kaner, Fiedler & Associates
Balasooriyi, Janaka Arizona State University
Bedran, Thomas Progressive Insurance
Demott, Jared
Gallagher, Keith Florida Tech
Hoffman, Doug Software Quality Methods Results Oriented Teaching of High Volume Automation
Hoffman, Dan University of Victoria Key Tradeoffs in High Volume Test Automation
Robinson, Harry Microsoft Experience report from Bing
Sabourin, Rob McGill University Implementing Multi-thread Testing in Fitnesse
Tinkham, Andy Magenic Technologies
Vaniotis, Thomas LiquidNet
Xie, Tao North Carolina State University Teaching High Volume Automated Testing via Interactive Gaming
Doran, Casey Florida Tech student A Survey of the State of Testing in the Open Source Community
Fiorvanti, Mark Florida Tech student High Volume Automated Testing in Security Testing
Frystacky, Michal Florida Tech student
Fuller, Scott Florida Tech student
Kabbani, Nawwar Florida Tech student
Oliver, Carol Florida Tech student Reference implementations for high volume automated testing
Tereshchenko, Vadym McGill University student Implementing Multi-thread Testing in Fitnesse


WTST is a workshop, not a typical conference. It is a peer conference in the tradition of The Los Altos Workshops on Software Testing (http://lawst.com). Our presentations serve to drive discussion. The target readers of workshop papers are the other participants, not archival readers. We are glad to start from already-published papers, if they are presented by the author and they would serve as a strong focus for valuable discussion.

In a typical presentation, the presenter speaks 10 to 90 minutes, followed by discussion. There is no fixed time for discussion. Past sessions’ discussions have run from 1 minute to 4 hours. During the discussion, a participant might ask the presenter simple or detailed questions, describe consistent or contrary experiences or data, present a different approach to the same problem, or (respectfully and collegially) argue with the presenter. In 20 hours of formal sessions, we expect to cover six to eight presentations. Some of our sessions will be activities, such as brainstorming sessions, collaborative searching for information, creating examples, evaluating ideas or work products. We also have lightning presentations, time-limited to 5 minutes (plus discussion). These are fun and they often stimulate extended discussions over lunch and at night.

Presenters must provide materials that they share with the workshop under a Creative Commons license, allowing reuse by other teachers. Such materials will be posted at http://wtst.org.

Our agenda will evolve during the workshop. If we start making significant progress on something, we are likely to stick with it even if that means cutting or time boxing some other activities or presentations.


We will hold the meetings at

Harris Center for Assured Information, Room 327

Florida Institute of Technology

150 W University Blvd

Melbourne, FL 32901


Melbourne International Airport is 3 miles from the hotel and the meeting site. It is served by Delta Airlines and US Airways. Alternatively, the Orlando International Airport offers more flights and more non-stops but is 65 miles from the meeting location.


We recommend the Courtyard by Marriott – West Melbourne located at 2101 W. New Haven Avenue in Melbourne, FL.

Please call 1-800-321-2211 or 321-724-6400 to book your room by December 24, 2012. Be sure to ask for the special WTST rates of $93 per night. Tax is an additional 11%. All reservations must be guaranteed with a credit card by Tuesday, December 24, 2013. If rooms are not reserved, they will be released for general sale. Following that date reservations can only be made based upon availability.

For additional hotel information, please visit the hotel website at http://www.marriott.com/hotels/travel/mlbch-courtyard-melbourne-west/


We expect to publish some outcomes of this meeting. Each of us will probably have our own take on what was learned. Participants (all people in the room) agree to the following:

  • Any of us can publish the results as we see them. None of us is the official reporter of the meeting unless we decide at the meeting that we want a reporter.
  • Any materials initially presented at the meeting or developed at the meeting may be posted to any of our web sites or quoted in any other of our publications, without further permission. That is, if I write a paper, you can put it on your web site. If you write a problem, I can put it on my web site. If we make flipchart notes, those can go up on the web sites too. None of us has exclusive control over this material. Restrictions of rights must be identified on the paper itself.
    • NOTE: Some papers are circulated that are already published or are headed to another publisher. If you want to limit republication of a paper or slide set, please note the rights you are reserving on your document. The shared license to republish is our default rule, which applies in the absence of an asserted restriction.
  • The usual rules of attribution apply. If you write a paper or develop an idea or method, anyone who quotes or summarizes you work should attribute it to you. However, many ideas will develop in discussion and will be hard (and not necessary) to attribute to one person.
  • Any publication of the material from this meeting will list all attendees as contributors to the ideas published as well as the hosting organization.
  • Articles should be circulated to WTST-2012 attendees before being published when possible. At a minimum, notification of publication will be circulated.
  • Any attendee may request that his or her name be removed from the list of attendees identified on a specific paper.
  • If you have information which you consider proprietary or otherwise shouldn’t be disclosed in light of these publication rules, please do not reveal that information to the group.


Support for this meeting comes from the Harris Institute for Assured Information at the Florida Institute of Technology, and Kaner, Fiedler & Associates, LLC.

Funding for WTST 1-5 came primarily from the National Science Foundation , under grant EIA-0113539 ITR/SY+PE “Improving the Education of Software Testers.” Partical funding for the Advisory Board meetings in WTST 6-10 came from the the National Science Foundation, under grant CCLI-0717613 “Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing”.

Opinions expressed at WTST or published in connection with WTST do not necessarily reflect the views of NSF.