An informal analysis of vendor acknowledgement of vulnerabilities

["Steven M. Christey" <coley () LINUS MITRE ORG>]

Title: An informal analysis of vendor acknowledgement of vulnerabilities
Authors: Steve Christey (coley () mitre org)
         Barbara Pease (bjp () mitre org)
Date: March 11, 2001


Many disclosure debates focus on researchers who discover
vulnerabilities.  Little attention is given to the impact on busy
security analysts who must determine which vulnerabilities exist, and
if they can be patched.  There is little or no emphasis on the role of
vendors of the vulnerable software.  Given continued discussions of
vulnerability disclosure practices, most recently regarding vendor
contacts on the PEN-TEST list, we decided to offer some results of an
informal analysis we performed in October 2000.  We also make some
recommendations for improvements.

In anticipation of the Guardent/eWeek vulnerability summit in early
November 2000, we conducted an informal experiment in which we tried
to obtain the following information:

1) Whether the vendor publicly showed awareness of an announced
   vulnerability, and admitted that the problem was real ("vendor
   acknowledgement," also referred to as confirmation).

2) Whether the vendor provided contact information for security
   problems.

Although our results are not quantitative, we do consider them to be
useful in understanding some of the problems in assessing which
vulnerabilities can be patched.


Disclaimer
----------

This is an informal analysis.  It is the sole product of the
individual authors, and does not represent an official position of The
MITRE Corporation.

This analysis does not attempt to identify or address the various
reasons for why vulnerability information may be incomplete or
unavailable, as that topic has often been discussed in the past.

The analysis evolved as it was being conducted, and as such was not
subjected to normal scientific rigor.  Thus, it is not appropriate to
list specific vulnerabilities, vendors, or researchers.

This analysis focuses on a test set of "worst case" vulnerabilities.
It is not necessarily typical of all vulnerabilities or all vendors.


Basic Conclusions
-----------------

1) Without a standard location for security vulnerability information
   on vendor web sites, it is often extremely difficult to even find
   the appropriate page where vulnerability and patch information
   might be discussed.

2) Without a standard contact name for asking questions about
   vulnerabilities, it is difficult for a security analyst to find out
   which individuals or groups at a vendor web site have the
   information needed.  This problem also makes it difficult for the
   vulnerability researcher to reach the right people.

3) The customer-focused nature of vendor web sites often makes it
   difficult for security analysts to access vulnerability
   information, even if the site has that information.  Security
   analysts normally are not the vendor's customers.

4) Frequently, there is no apparent public acknowledgement of the
   vulnerability, or the web site is too difficult to navigate.

5) In some cases, the vendor may or may not have acknowledged the
   vulnerability, but the vendor's information is too vague to be
   certain.

6) When the researcher's original report did not include detailed
   vendor and product information (such as URLs and version numbers),
   it made it more difficult for the security analyst to find the
   proper vendor web page to examine for acknowledgement.


Focus of the Analysis
---------------------

We examined a test set of approximately 150 announced vulnerabilities.
This test set EXCLUDED any vulnerability with a well-established
advisory, e.g. from CERT, well-known software vendors, or well-known
security companies with vulnerability analysis teams.  Most were
announced between 1997 and 2000.  The most recent vulnerabilities had
been publicized at least 2 months before the analysis took place.

Vendor acknowledgement of vulnerabilities was examined from the point
of view of a "busy security analyst."  Assumptions were:

1) The analyst is responsible for identifying and mitigating IT
   security vulnerabilities in a relatively large enterprise with
   diverse platforms, applications, and security requirements.  Thus,
   every announced vulnerability may need to be addressed in one way
   or another.

2) The analyst is busy, and so only has about 20 minutes to research
   each vulnerability.  At a rate of approximately 100 new
   vulnerabilities per month - which is not unusual these days - that
   could consume almost one staff week per month, just seeing if
   vulnerabilities are "real" according to the vendor.

3) The analyst does not necessarily know if the researcher's report is
   accurate, e.g. whether the vendor was properly contacted.

4) The analyst only wants to concentrate on vulnerabilities for which
   there is a known fix or workaround that is approved by the vendor.

The basic task was to see if the analyst could find sufficient vendor
acknowledgement of a vulnerability within a maximum of 20 minutes.


Process for Determining Acknowledgement
---------------------------------------

The six steps below comprise the basic method we used.  Although
carried out in a disciplined way, there were variations in each
author's approach.  As we continued the work, the process itself
evolved.

1) Only look for the vendor's direct public acknowledgement.  We did
   not consider a researcher's claim that "the vendor has fixed the
   problem" to be sufficient support for vendor acknowledgement; a
   security analyst may not know how reliable the researcher is.  A
   link to a patch was also considered weak evidence.  A security
   analyst might not be a consumer of the software, so he or she might
   not be able to verify that a suggested patch actually fixes the
   problem, assuming there is enough time to test the patch.

2) Consult the original disclosure (typically a *Bugtraq post) to see
   if the researcher provided any supporting information (vendor
   links, patches, etc.)  See if there are any follow-ups by the
   affected vendor.

3) Consult the vendor web site.  Look for web sections named
   "security," "support," "changes," product-specific pages, etc.
   When information can not be obtained in this fashion, try a keyword
   search, including keywords such as "security," the type of
   vulnerability (e.g. "buffer overflow"), software name, and version
   number.

4) Consult supporting vulnerability databases for additional pointers
   if information can not be found on the vendor web site.  Two
   well-known online vulnerability databases were used, to see if they
   had conclusive fix information that satisfied the requirements in
   (1) above, or pointers to vendor web sites.

5) If acknowledgement can not be determined, try to obtain a contact
   name from the vendor web site.  (The "less busy" security analyst
   might want to ask the vendors for acknowledgement.)

6) Record the discovered information, including URL's visited.

The information we collected did not lend itself easily to
quantitative results.  Therefore, we report our results as
observations based on the information recorded.


High-Level Observations
-----------------------

The following high-level observations were made.  Although many
vendors faithfully report vulnerabilities, the focus of our work was
on finding vendor confirmation within the 20 minute time frame of the
"busy security analyst."  Therefore, these observations are not
necessarily indicative of actual vendor reporting levels.  The results
reflect more the ease of finding the information, especially for
vendors that do not have a well-established security advisory
capability.

1) Approximately 15% of the vulnerabilities in the test set had clear
   vendor acknowledgement.

2) Approximately 15% of the vulnerabilities in the test set had
   possible vendor acknowledgement.  This included vague vendor
   statements such as "it's really important for customers to update
   their software," or a "fixed a security bug" statement in a change
   log.  It was not necessarily clear whether the problem being fixed
   was the same as the problem that was disclosed by the researcher.

3) For approximately 20% of the vulnerabilities, it was too difficult
   to find a page on the vendor web site that might include
   vulnerability information.

4) Unless a software vendor posted a follow-up to an email, it would
   normally take between 15 and 30 minutes for the security analyst to
   determine if there was any acknowledgement.

5) For approximately 1/3 of the vulnerabilities, the researcher said
   that they had contacted the vendor.  For at least 1/3 of the
   vulnerabilities, the researcher did not say anything about
   contacting the vendor.  Data is unavailable for the remainder of
   the vulnerabilities, but either (a) the researcher explicitly said
   that they did *not* contact the vendor, or (b) the researcher did
   not say anything about contacting the vendor.

6) Sometimes, the researcher provided little vendor information.  This
   increased our workload by forcing us to search for information via
   common search engines.  For example, if the researcher did not
   provide a URL to the vendor site, then we would would have to
   search the WWW for the vendor and product.  The search would be
   more difficult when the researcher didn't spell the software or
   vendor names correctly.  If the researcher didn't provide a product
   link, then we would have to search the vendor site for product
   pages.  If the researcher didn't provide the version number of the
   affected software, then we would often have to do some additional
   searching; or, we wouldn't be able to know if a vendor-reported fix
   was addressing the vulnerability found by the researcher.

7) Most vendor web sites were focused exclusively on the customer,
   instead of the security analyst.  This could make it difficult or
   impossible for the analyst to obtain the appropriate security
   information.

   Often, if a "security" web page was available on a vendor web site,
   the page would point to security features of the product - not
   security announcements.  (This was also encountered when performing
   keyword searches for "security".)  Often, the "support" web page
   did not include pointers to security contacts, or security
   advisories.  Sometimes, the "support" web page was inaccessible
   because we were not a customer, or the web site required
   registration.  A busy security analyst would not necessarily want
   to register with every vendor whose software is used in the
   analyst's enterprise.

8) Some vendor web sites did not provide any email address or phone
   number for support, instead relying on a web-based form.  The form
   would often require information such as the version and platform of
   the product being used.  This type of interface would be highly
   inconvenient for a security analyst to ask the vendor for
   acknowledgement, if it is not already available on the web site.

9) Some freeware/open source vendors would be very clear in their
   acknowledgement of vulnerabilities, whether in a "recent news" page
   or in a change log.

   If the freeware vendor was "small," then normally the web site
   would be simple, and contact information would be easily
   accessible.

10) Sometimes, a change report would include a brief entry that was
   too vague to be certain whether the vendor fixed the vulnerability
   that had been reported by the researcher.  If the researcher left
   out critical information such as the version number, this would be
   more difficult to determine if the vulnerability was fixed.

   Sometimes, the vendor would describe a "serious bug" but not say
   that it was a security problem, so it could not be certain that the
   fix had security implications - or, which vulnerability was being
   fixed.

   Sometimes, we were only able to determine that the vulnerability
   had been fixed by manual source code inspection.  Often, however,
   there were no surrounding comments to indicate for certain that the
   source code had been patched.

   Sometimes, several vulnerabilities in the same product and version
   would be disclosed, but we would only see acknowledgement of a
   single vulnerability.  It's possible that the vendor fixed all the
   problems in the same patch, but it could not be certain.

11) Often, when the vendor included cross references, this made it
   much easier to ensure that the proper vulnerability was being
   addressed.  This was especially useful in cases in which multiple
   vulnerabilities were discovered in the same software product.

12) Many large vendor web sites were difficult to navigate, especially
   if the vendor site offered information on a large number of
   products.  Just finding the product page could take a long time.
   We often found ourselves relying on the site's search engine, where
   a keyword search for "security" would rarely produce useful results
   from an analyst's perspective.

   While a time limit for the analysis was set for each vulnerability,
   we sometimes found ourselves "lost" in these sites, following many
   false leads, to the point where we would lose track of time and
   exceed the 20 minute maximum.  Finding the appropriate location in
   a web site was often frustrating, and it was often tempting to
   quickly give up.

13) Sometimes, on very large web sites, we visited more than 15
   different URL's before giving up, especially if we had to rely on
   knowledge base or site-wide keyword searches.

14) If the vendor web site did not have any apparent acknowledgement,
   sometimes the vulnerability databases that we consulted did not
   have any additional fix or vendor information.  This was probably a
   reflection of the lack of information on the vendor web sites.
   Sometimes, however, the vulnerability databases had pointers to
   vendor web site pages that we had not been able to find on our own
   before our time limit expired.  This is perhaps a reflection of the
   complexity of the vendor web site.

15) We rarely encountered a case in which a vendor publicly
   acknowledged a vulnerability before a fix was available.  This made
   it difficult to determine if the vendor was even aware of the
   problem.  (Note that some vendors have said that many of their
   customers prefer this approach [1]).

16) Even in cases in which a vendor had a well-established security
   advisory and contact mechanism in place, if a reported security
   vulnerability did not have an associated advisory, it would be
   difficult to determine if the vendor was even aware of the report.
   This was often due to the same site navigation problems that we
   encountered with other vendors that did not have advisory
   capabilities.

17) Some vendor web sites relied on bulletin boards for technical
   support.  These boards were often difficult to navigate or search,
   but sometimes they were the only potential location in which
   acknowledgement might have appeared.

18) For vulnerabilities that were more than a year old, it was much
   more difficult to obtain acknowledgement, due to factors such as:
   (a) web sites were deleted or relocated, (b) the software was
   discontinued or no longer supported, or (c) the software was
   otherwise renamed, e.g. as part of a corporate acquisition.

19) Sometimes, the vendor would post an acknowledgement to an internal
   customer list, and that post would be forwarded to a vulnerability
   disclosure source such as Bugtraq.  However, there would not be any
   apparent public acknowledgement of the vulnerability.

20) Often, the researcher would not include vendor contact
   information, or a record of their communications with the vendor,
   which could help a security analyst to determine if the vendor is
   aware of the problem and developing a patch.  (The practice of
   including "communications logs" seems to be growing more prevalent
   since our analysis in October 2000.)

21) Many vulnerabilities in the test set had a high risk level, e.g. a
   buffer overflow that allowed a remote attacker to execute arbitrary
   code.  Unfortunately, the most serious vulnerabilities in our test
   set did not necessarily produce an increase in the visibility of
   the vendor's acknowledgement.  In some cases, this might be due to
   an incorrect assessment of the actual risk level.  For example,
   buffer overflows are often described as having only a denial of
   service impact.  It is likely that many of those overflows could be
   forced to execute code by a sufficiently skilled programmer, and it
   is possible that the affected vendor did not realize the potential
   seriousness of the problem.


Some Suggestions for Improvement
--------------------------------

Following are some suggestions for improvement.  These suggestions are
intended to make it easier for security analysts to obtain information
from vendors.  They build on suggestions that others have made in the
past.  Note that vendors may have specific reasons for not providing
easy access to vulnerability information.

1) A vendor could make its security acknowledgements more easily
   accessible by placing them in a standard section of the web site,
   either (a) a security page, or (b) the support page.  If the vendor
   offers multiple products, then it could provide a link to a
   security page from each product page.

2) If a vendor restricts access to some support pages, or requires
   registration, then it could provide their acknowledgement in a
   place that has full access to the public, which would make it much
   more convenient for security analysts.  Note: this is only a
   recommendation for the acknowledgement of the problem; we recognize
   that the vendor may wish to restrict access to customer-specific
   resources, such as patches.

3) A vendor could feature a security contact more prominently on its
   web site.  The contact could be listed under the security, support,
   and product pages.  The vendor could provide a standardized alias
   for reporting security problems or asking for clarification, such
   as security () example com.  RFPolicy 2.0 [2] suggests additional
   aliases.

4) A vendor could annotate vulnerability-related web pages with
   standard keywords such as "security" and "vulnerability" to
   facilitate web searches.

5) If a vendor uses a web-based form for support, then it could
   provide options that allow non-customers to make requests or
   comments about security without requiring product license
   information, platform information, or other information that
   non-customers would not have.

6) A vendor could precisely identify which vulnerabilities are being
   fixed by using commonly accepted cross references [3].

7) If a vendor offers products that have security features, then the
   vendor could include a pointer to their security page.

8) A researcher could make it easier for security analysts by
   including the following information in the vulnerability report:
   (a) URL to vendor web site; (b) URL to product page; (c) URL to
   vendor acknowledgement, if available; (d) product version and
   platform information; and (e) record of communications with the
   vendor, especially who was contacted and how.

9) Finally, a Request For Comments (RFC) could be drafted and adopted
   so that there is established guidance for all participants in the
   vulnerability disclosure process, especially the vendor and the
   researcher.


Footnotes
---------

[1] eWeek, "Security Core: Best practices."  November 13, 2000.
    http://www.zdnet.com/eweek/stories/main/0,10228,2652346-3,00.html

[2] RFPolicy 2.0.  http://www.wiretrip.net/rfp/policy.html

    While this policy is focused on how researchers should participate
    in the disclosure process, it assumes certain lines of
    communication with vendors that often do not exist, such as the
    standard security email aliases.

[3] It should be noted that both authors are involved in a common
    cross-referencing effort.