Software assessment

1. Meta

1. Does the software have a good name?
   • All software projects should start with a good name. Bonus points for animals or (recursive) acronyms! ;)
2. What is the license? Is it open source? Is the license clearly stated? Is it a good choice?
3. Are author(s) and affiliation(s) clearly stated?
4. Is the software maintained in proper versioning control? (If not, flog the programmer)
   • Is the versioning control public? Is the version control system a good choice?
5. Is the purpose of the software clear?
6. Does the software have a website? Does the site properly explain what it does?
7. Is the software easily downloadable?
8. Is the intended audience clear?

2. Installation

1. Does the software have a clear and generic installer?
   • Is it an ad-hoc or a platform-standard installer?
   • Does the installer make rigid assumptions about where libraries/data are located?
2. Does it depend on many other libraries and how hard is it install these necessary dependencies?
3. Does it compile/install?
   • How clean is compiler/installer output, does it generate many warnings?
4. How does it fare on other platforms, such as Mac OS X or shiver Windows?

3. Status

1. How mature is the software? Is it alpha, beta, release candidate, stable?
2. What can be expected at this stage?
3. Does it do what it claims/promises to do?
4. What parts need more development?

4. Documentation

1. Is there any documentation? (If not, flog the programmer)
   • Is it clear enough for the intended audience?
   • Is it complete?
2. If the sofware offers an API, is there an API reference?
   • Is it clear and verbose enough?
   • Is it complete?
   • Are there enough examples?
   • Is the API documentation generated from the code itself?
3. Is there a hand-on tutorial for working with the sofware?
   • ..as an end-user?
   • ..as a programmer?

5. Software Architecture

1. Programming Language

   • What programming language(s) is/are used? Are they good choices for the task? Is the language well-supported and expected to last?
   • What libraries and frameworks are used? Are they good choices for the task? Are there libraries/frameworks missing which could have been used?

2. Form

   • What is the form of the software: Web-based application, Webservice, Command line tool, Programming Library, GUI? Is this a suitable choice (considering the intended audience)? Should it be expanded?

3. Algorithms & Techniques

   • What algorithms and techniques are used?
   • Are these good for the task at hand? Can they be improved?

4. User Interface

   • Is there a GUI or web-interface? If not, could/should there be?
   • Is the interface intuitive?
   • Does the interface expose enough of the underlying functionality?
   • Is the interface visually appealing?
   • For web-based interfaces: How is cross-browser support?

5. Application Programming Interface

   • Is there an API interface? If not could/should there be? For what languages?
   • Are classes/methods etc suitably named?

6. Data

   • Does the software have sufficient facilities for reading and storing data?
   • Does the software define any new formats for its data?
     • Are these sufficiently documented?
     • Does it follow standards (XML), or is it too ad-hoc? Is there a scheme? (XSD, RelaxNG)
     • Is it a good choice?
   • Are there enough import/export facilities for data in other formats? Should there be?
   • How is the visualisation of the data?
   • Are the data metrics used by the software good choices?

7. Unit tests

   • Are there unit tests/regression tests?
   • Are they any good?
   • Do they pass?
   • Do they cover enough of the software?

8. Security

   • How is the security of the software, are there any obvious security holes?
   • Is there, or should there be, proper user authentication? Are user credentials properly encrypted and transferred over the net as such?
   • Is user input in the interface properly validated?
   • Is the software sufficiently hardened against SQL injection, shell injection and cross-site scripting?
   • Is any sensitive data leaking? Is there anonimisation or data encryption, should there be?

9. Modularity, see also 6.1

   • Is the code split into proper modules? Are these reusable for other ends? Should the software be split into parts?

10. Performance, see also 6.3

    • How is the memory usage of the sofware? Does this seem right?
    • How quickly does the software perform its task? Does this seem right?

11. Exception Handling, see also 6.4

    • Is the error output sane and understandable? Can the user easily remedy the problem?

12. Maintainability

    • How maintainable is the software?
    • Is it build to last?
    • Can it survive if the original programmer(s) abandon it?
    • Is the software easily extendible with new functionality?
    • Are there needless complexities or over-simplifications? Should parts be simplified, or generalised?

6. Source review

1. Modularity, see also 5.9
   • Is the code object oriented, should it be? Are the choices logical?
   • Think about: seperation into classes, proper constructors,
   • method/operator overloading, class hierarchy, polymorphism / multiple inheritance
   • Is the code split into clear functions? Is code repetition kept to a strict minimum?
   • Are global variables kept to a strict minimum?
   • Are settings properly configurable and not hard-coded (paths,addresses)
2. Style & Comments
   • Is the coding style and naming clear and clean?
   • Does the coding style and naming follow standard conventions for the language?
   • Are there comments to explain what each function/class does?
   • Are there enough comments to explain difficult parts?
   • Are standard or third-party libraries used properly and enough, or is the wheel reinvented needlessly?
   • Are there compatibility issues for different versions of the language?
3. Performance, see also 5.10
   • Is allocated memory properly freed when no longer needed? Think about: Dangling references/pointers
   • Are the algorithms efficient enough for the task? Are there better options? Think about: Nested loops
   • Is the speed/memory trade-off decent?
4. Exception Handling, see also 5.11
   • Is there enough exception handling? Are obvious chokepoints ignored?
   • Is user input properly validated?
   • Is the software verbose enough in raising errors of its own?
   • Is there any obvious danger of segmentation faults, null pointer exceptions, double free or overflowing data types?
5. Bugs
   • Are there known bugs?
   • Did you encounter any bugs? (If not, you didn't look hard enough)
   • What should be prioritised when fixing?

7. Case Study

1. Do you see any use for the software in your own project(s)?
2. Any interesting ideas for projects that could be conducted with this software?