LIS 505 - Software Evaluation Criteria

• Hardware compatibility. A lot of this is the kind of thing that you can check for on the box (or in the basic requirements published on a vendor's Web site), but there may be particular problems that you have to find out about elsewhere.
  • Computer. Mostly, this amounts to an operating system question (Windows, Mac, Linux?), but the specific CPU may be sometimes be important (a newer program may not run correctly on an older CPU, or an old program may fail on a system with too high a clock speed).
  • Memory. Software will typically have a minimum memory requirement to run and a recommended amount of memory to run efficiently. Sometimes, certain features of the software will work only if more memory is available.
  • Graphics cards. Software may not work correctly with certain graphics cards (not just display incorrectly, but also malfunction in other ways). You may be able to avoid these problems by changing the graphic acceleration settings (possibly affecting the functioning of other software in the process) or by updating the drivers.
  • Disks. The most common piece of information is how much space the software takes up on the hard disk. There may be different installation options which require different amounts of hard disk space. If the software comes on disk, you also need to have the right kind of drive to read the distribution disks. In addition, some software is designed not to run unless one of the distribution disks is present. If the software is download in compressed form from a Web site, it will quite often require additional space on the hard disk for temporary files created during installation.
  • Printer. This is not much of a concern with Windows machines, since software almost always uses the respective printer drivers. You may want to check that the software can accommodate the paper sizes that you are going to use and will avoid printing to unprintable areas.
  • Other peripherals. Many packages require the use of a pointing device such as a mouse (or a software alternative, such as the Accessibility Options in Windows) for at least some operations. Some packages require an Internet connection, either for registration or to access additional resources such as clip art. Certain keyboard shortcuts may not work with compact keyboards like those on notebook computers. Specialized packages may be designed to work with type-specific peripherals; for example, a speech generator with a sound card, or a character recognition software with a scanner.
• Software compatibility
  • Operating environment. Apart from the general question of operating system family, you may have to consider the specific version. Some programs that run perfectly well under Windows 98 fail completely under Windows XP, and vice versa. There are occasional reports of software that ceases to work after an operating system patch has been installed. In addition, other installed programs such as virus checkers may cause some software to malfunction. Having the wrong version of a device driver installed is another common source of woe.
  • Other software required. This may mean a runtime system, such as that for Visual Basic applications; or it may be that the software package being considered is just a plug-in or add-in to some other software package. Some software borrows DLLs from other packages (for example, from Internet Explorer for displaying Web pages), or requires a standard database management engine to be running on a server.
• Software quality
  • Capabilities. Does the software do the kinds of things that you want it to do? Are additional features also likely to be useful, or will they just get in the way?
  • Stability. Does the software tend to crash or hang, or crash the operating system? Does it suffer from memory leaks (not releasing memory when it is done with it, so that the machine eventually ends up low on memory)? Do its settings seem to change mysteriously for no reason?
  • Correctness and adherence to standards. Does the software do what it says it does? Does it use interface elements such as text boxes in a standard way? When it creates files, do they adhere to the relevant standards, or do other applications experience problems in reading them?
  • Ease for all users. Beginners should find it easy to get started using the software. Experienced users should have plenty of shortcuts to increase their efficiency. There should not be any cliffs in the learning curve, where learning the next thing suddenly becomes much harder than learning the previous thing.
  • Adaptability. Are there different options available for installation? Does the software adjust well to different screen dimensions, color depths, and font sizes? Can different interfaces be chosen to suit beginners and more advanced users? How many of the default settings can be modified (for example, can you tell to program to start browsing for files somewhere else than in My Documents)?
  • Use of keys, mouse, and menus. Package functions should be accessible both via the menus and via shortcut keys. Menu items should show the associated shortcut key values, so that users can find out easily what they are. Menus should be well organized, with clear captions. Mouse actions should be standard (left click to select or execute, right click for a pop-up menu, drag to move or resize). Keys should be used in conventional ways (Tab to move between fields, F1 for help, etc.).
  • Online help. Text should be clear and use language correctly, with appropriate headings and subheadings. Unfamiliar terms should be defined and explaned. Organization should be logical. All information should be readily accessible for reference; for longer help files, this means having both a topical index and a search function that works correctly. Charts, diagrams, and screenshots should illustrate where needed. What is said in the help files should reflect exactly what is the case in the software itself. Any links to other documentation should be correct, complete, and useful.
• Training and documentation
  • Documentation. Many of the same points apply to printed documentation as are noted above for online help. The availability of guides produced by third parties should also be considered.
  • Tutorials. If the package includes tutorials, consider how much they cover and whether they provide real interactive learning or simply encourage the user to click through a series of steps without taking in what they are doing.
  • Training. This may be provided by vendors or by third parties such as educational institutions. Relatively simple software may not require much user training.
• Vendor relations.
  • Charging. You need to know what the local cost is to you or your organization; this may mean taking account of taxes, shipping and handling, currency exchange, and customs charges. While less expensive software often has a fairly simple sticker price, more expensive or specialized packages may have more complex pricing systems; price may depend on whether the buyer is a non-profit institution or not, or, in the latter case, on something like annual sales revenue. Competitive or upgrade prices may also be available, as may special rates for multiple licenses. Some software licenses are not sold, but leased, requiring ongoing expenditure if you want to continue using the product.
  • Copy protection. Technical copy protection (as against legal sanctions on unauthorized copying) is found especially on games, educational software, and expensive niche products. One method is to require the presence of a physical object (such as an installation disk, or a "dongle" attached to one of the computer's ports) in order for the software to run. Another method is to require activation after the software has been installed in order to access all its features or so that it will continue to run after a given date; depending on the activation scheme, this may mean that the software cannot later be transferred to another machine (or even run on the current machine if it is substantially upgraded) without contacting the vendor and perhaps paying an additional fee.
  • Warranty. About all you usually get here is a limited time period in which you can ask for a refund. Software developers are notorious for avoiding providing guarantees of the quality of their products.
  • Maintenance. There may be a limited period during which you can download free updates from an Internet site. Various fees may be applicable for more extended maintenance plans.
  • New releases. Consider how often new versions of the software are being released and whether you will wish to upgrade at each new release, at every second new release, etc. Is there a special upgrade price? Sometimes, uggrading to a new release is made free of charge to those who bought an old version very recently.
  • Assistance with problems. This might be by telephone, fax, or e-mail, or though an online user discussion board. Larger vendors may provide a range of support options from free for very bare-bones support through various prices for different degrees of premium support, with the highest levels covering, for example, response every hour of every day.
• Other.
  • Reviews. Read what independent reviewers have to say about the product. Be aware that reviewers for commercial magazines may pull their punches to some extent to avoid offending advertisers. You may also see product announcements, which are not real reviews, but just repeat information supplied by the vendor. Postings to discussion groups may range from "astroturfing" by corporate shills to malicious attacks; look for discussions that seem to provide a good balance of judicious comments.
  • References. For major software purchases, such as library automation packages, you should ask vendors to provide references, whom you should contact independently. For any software selection, you can contact others who have some experience with the software to get their reactions.
  • Market share. Market share is not always easy to determine, though surveys reported on various Web sites may be of assistance for certain kinds of software. Market share is an important consideration for several reasons: a product with a high market share is likely to meet some minimum standard of quality (though it is not necessarily the best product in its class); if the product has a high market share, the company is more likely to stay in business and so continue to supply support and upgrades (you can also check on the prospects of the company as a whole, of course); even if the company goes out of business or ceases to support the product, there will be a large group of other users out there who may be able to provide assistance; file formats used are more likely to be supported by other products; third-party documentation will be more plentiful and varied; new and prospective employees are more likely to be familiar with the software.

Home