O'Reilly: WindowsDevCenter.com - March 16, 2004
Windows Error Reporting Under the Covers

By Sheryl Canter

Many people feel alarmed when a dialog box appears after a program crash that asks for permission to send a snapshot of their system to Microsoft. They have questions about what that dialog means: What's in this error report? What will Microsoft do with the information? Can it be used to track my activities? Why is it going to Microsoft rather than the company whose program crashed? How come I've reported this same error 20 times and it's still not fixed?

In this article, we'll take a closer look at the Windows Error Reporting (WER) system, answer those questions, and provide more information about it for both developers and users.

A Look Inside WER

Microsoft created WER to help itself and other developers find and fix the bugs that are inevitable in complex software. Many bugs are caused by simple oversights and are easily fixed. The programmer's bane is a bug that occurs on a customer's system, but not his own. If the programmer can't reproduce the bug, it's very hard to fix. The solution is to obtain a snapshot of the user's system at the time of the crash, but historically this has been difficult to do. Worse yet, customers often don't go to the trouble of reporting a crash; they just sigh and reboot. Programmers can't fix problems they don't know about.

Figure 1. Initial WER dialog that pops up after a crash.

WER was introduced with Windows XP and has three components:

  1. An easy and secure method for users to provide crash reports.
  2. A database that categorizes and prioritizes the reports for vendors, and
  3. A means for vendors to respond with status reports, requests for additional information, links to the vendor's site, or hotfixes distributed through the Windows Update system.

In the programming world, the term for a bug that causes a crash is an "exception." The portion of a program that says what to do when an exception occurs is called an "exception handler." WER is a "global exception handler," meaning that all exceptions occurring within the program are, by default, handled by WER.

WER is called automatically when a crash occurs unless a program has its own global exception handler. Programs with their own global exception handlers can still use WER by calling the ReportFault API within their handlers. The API is contained in a system file named faultrep.dll. When WER is invoked, an operating system utility called dwwin.exe creates the crash report and prompts the user to send it to Microsoft.

Advantages of a Central Repository

You may wonder why WER sends the error reports to Microsoft rather than to the publisher of the program that crashed. The reason is that the bug causing the crash may not actually be in the program that crashed. Programs running on Windows are highly interdependent. Your new adventure game may be crashing because of a bug in your video driver. Explorer may be crashing because of a third party Browser Helper Object (BHO). The cause of blue screen crashes can't be determined without backend analysis of the report files. With centralized reporting, Microsoft can provide copies of the error report to all vendors with products involved in the crash.

Microsoft's system for organizing the error reports is another significant benefit. In a product with a large user-base, one bad bug can generate a huge number of crash reports. Microsoft condenses the information by categorizing the reports into "buckets." For user-mode crashes, buckets are defined by application name and version, module name and version, and offset into module. For kernel-mode errors, buckets consist of stop codes and associated parameters.

Figure 2. Vendor interface for WER data.

The WER database sorts the buckets by report frequency, which lets developers see at a glance which bugs need to be fixed first. This categorizing and sorting revealed an unexpected pattern: 80% of crashes are caused by only 20% of the reported bugs. The ability to fix bugs in order of user impact lets developers solve a lot of user problems quickly.

Developers can click on a bucket to receive a .CAB file containing up to 50 mini-dumps for that error. It's useful to know if 10,000 people have reported the same error, but you really don't need all 10,000 mini-dumps.

WER was introduced with Windows XP, and also runs on Windows Server 2003. The system files that implement WER are not redistributable because they rely on kernel and global-exception handling code that is specific to XP. Microsoft will support WER on future versions of Windows, but has no plans to retrofit it for older versions. The reason that Office XP and Explorer can produce crash reports on back versions of Windows is that they use a custom crash report system that predates WER.

How WER Protects Your Privacy

The WER system is part of the Windows Quality Online Services (Winqual) program, whose primary purpose is to evaluate hardware and software for the "Designed for Windows" and ".NET Connected" logos.

There is no setup fee for a Winqual account and downloading WER data is free, but companies are required to purchase a VeriSign Code Signing ID, which costs $400 for a standard account and $695 for the Pro version. This allows Microsoft to verify the identity of companies submitting software or accessing the WER database.

Microsoft does not use WER data for any kind of marketing or trend analysis whatsoever. End-user identities are strictly protected, and Microsoft doesn't do any comparative analysis of the frequency or types of bugs reported for different products. Participating companies are required to sign an agreement promising to handle the data in a similar manner.

End-user privacy must be strictly protected, and the data can only be used for debugging purposes. If a court order compels disclosure of WER data, companies are required to oppose the request and notify Microsoft so they can handle it. Every person accessing the WER database must sign this agreement.

By default, error reports don't contain any identifying information except for what inadvertently is found on the stack (an area of memory that is heavily used by executing programs and included in the crash reports). However, the stack can contain personal information, depending on what you were doing at the time of the crash. It could reveal your most recent menu selections, the web site you were viewing, the credit card number you just entered, the business proposal you just composed, or the passionate email you just wrote to your darling.

WER participants are legally bound to keep WER information private, but if you were working on something at the time of the crash that you really don't want anyone else to see, you shouldn't send the report to Microsoft. You can look at the reports to check for private data before sending them, but sifting through 50-100k of technical data can be tedious.

Vendors can request additional information beyond the basic data provided, but they can't just ask for anything. Developers at Microsoft monitor the requests and will deny any that aren't needed for debugging. For example, Microsoft will not allow vendors to obtain the user name or machine name from the registry. (This information could be captured through the Corporate Error Reporting tool, however, which will be discussed in another article.)

Submission and Response

Error reports are encrypted before transmission to Microsoft through HTTPS, and are stored on a set of secure servers that are used for no other purpose. Access requires a username and password, and companies can only see reports on their own products.

After you send the basic report to Microsoft, the database is queried for any responses that the vendor has requested to be displayed for the reported error. You may see a message from the vendor thanking you for submitting the report. Or you may be given a link to the vendor's support site or a hotfix on the Windows Update site.

Sometimes vendors ask end-users to submit additional information that will help diagnose the problem. These can take the form of WMI queries, lists of registry keys, or questionnaires that ask you for information directly. Submission of additional information is voluntary, just as submission of the basic report is voluntary. Note that if you give identifying information in a questionnaire or sign up to be notified when a bug is fixed, your error report will no longer be anonymous.

Figure 3. Request for additional data for crash report.

Microsoft strongly encourages vendors to provide end-users with some sort of response to their error reports, even if it's only a simple "Thank you, we'll look into it." Otherwise end-users can feel they are whistling in the wind, and may not send future error reports. If you don't receive a vendor response after submitting a report, it's the developer of the software that crashed who's ignoring you, not Microsoft (unless, of course, it was a Microsoft program that crashed).

The WER request for a system snapshot may be disconcerting in this age of widespread privacy violations, but there are no hidden motives here. Microsoft goes to great lengths to protect user privacy. The main purpose and main result of sending in crash reports is hardware and software with fewer bugs, and that benefits everyone.

Sheryl Canter has been a Contributing Editor to PC Magazine since 1993, a software developer since the early 1980's, and was the editor of PC Magazine's Utilities column from 1993-2002.

Extra Info for Developers: How to Get Crash Reports on Non-XP Systems

HomeWritingSpeakingWeb DesignGraphic DesignBioBlogContact