Re: [RFC v2] Vulnerability library proposal

[Djalal Harouni <tixxdz () opendz org>]

On Sun, Aug 07, 2011 at 12:40:45AM +0100, Djalal Harouni wrote:
> Hi list,
>
> This is a proposal for a new NSE vulnerability library. The library is
> designed to help managing discovered vulnerabilities and to make the
> output more consistent.
This is a new version of the proposal that includes corrections and
suggestions. I've also attached a diff file to not wast your time.

Thanks for the help.



vulns library version 2
-----------------------


Table:
------
1) Functionality.
2) portrule and hostrule scripts.
3) prerule and postrule scripts (vulnerability management scripts).


Note: when we say post-processing scripts we are referring to postrule
scripts.


1) Functionality:
-----------------
* This library is only for managing reported vulnerabilities.

* The library must produce consistent output for all vulnerabilities.

* The library should be able to silently store the vulnerabilities
  information in the registry for post-processing.

* The library should also store and report vulnerabilities that have been
  checked but aren't present (not vulnerable, patched, etc).

* The library must handle different states:
  UNKNOWN:  We don't know the state of the vulnerability.
  LIKELY_VULN: The program seems vulnerable (results of version
               comparisons, this state can cover possible false positive
               situations).
  NOT_VULN: The program was confirmed not to be vulnerable.
  VULN: The program was confirmed to be vulnerable.
  EXPLOIT: The program was confirmed to be vulnerable and was exploited
           successfully (the VULN state will automatically be set).
  DoS: The program was confirmed to be vulnerable and a Denial of Service
       was caused (the VULN state will automatically be set).

* The library will accept different IDs ('CVE', 'OSVDB', 'YOUR_DB', etc),
  which are used to reference and to map vulnerability entries. Duplicate
  entries that are reported by several scripts must also be handled, and
  in this case the vulnerabilities state can also be updated.
  (i.e. from 'LIKELY_VULN' to 'NOT_VULN' or from 'LIKELY_VULN' to 'VULN')


Vulnerabilities will be saved in the registry (global database) only
when a post-processing script is selected by the user. The script will
set a special variable or call a function that will activate the global
vulnerability database, this way we do not wast memory and extra
processing, and the user will have the feature without specifying any
script argument.


2) portrule and hostrule scripts:
---------------------------------

Vulnerability table info:

  - Vulnerability title (mandatory field).
  - State: UNKNOWN, NOT_VULN, LIKELY_VULN, VULN, EXPLOIT, DoS
           (mandatory field).

  - IDS: CVE, OSVDB, BID ...  The IDs entries will help the library
                              to reference vulnerabilities and to
                              track duplicate ones (optional).

  - "Risk factor": This optional field will take one of these string
                   values: "High", "Medium" or "Low".

  - Vulnerability Scores: a table of vulnerability scores (optional).
     SCORES = {CVSS = '', CVSSv2 = '', Exploitability = '',  ...}
                  
  - "References": reference links (optional).
                  If the IDs of the popular vulnerabilities references
                  are specified then their links will be constructed
                  automatically and pushed to this "References" list.
                  Popular IDs: CVE, OSVDB, BID ...

  - "Date": disclosure date table (optional).

  - "Description": vulnerability description (optional).
  
  - "Check results" (optional). This field can be the result of the
                  vulnerable check, did the server return anything ?
                  This can help specialists to investigate the results
                  and decide if the program is vulnerable or not.
  
  - "Exploitation results" (optional), if present then show it.
  
  - More information will be shown according to the debug level.


Vulnerability table example:

vuln_info = {

  -- Mandatory fields
  title = 'Exim string_format Function Remote Overflow', -- string
  state = vulns.State.VULN, -- number

  -- The following fields are all optional
  IDS = {CVE = 'CVE-2010-4344', OSVDB = '69685'}, -- IDs map
  risk_factor = 'High', -- string
  scores = {CVSSv2 = '9.3 (AV:N/AC:M/Au:N/C:C/I:C/A:C)'}, -- Scores map
  description = 'vulnerability description ...', -- string
  references = {
    -- list of references
    -- popular IDs references will be constructed automatically
  },
  date = {y = 2010, m = 12, d = 14} -- date table

  -- These fields are all optional and they will not be saved in the
  -- global database (registry).
  check_results = '...', -- string
  exploit_results = '...', -- string
                           -- will probably be the result of table.concat()
  extra_info = '...', -- string (result of table.concat() or whatever)
  ...
}


Sample output:
Portrule/Hostrule output:
--@output
-- PORT   STATE SERVICE
-- 25/tcp open  smtp
-- | smtp-vuln-cve2010-4344:
-- | Vulnerabilities:
-- |   Exim string_format Function Remote Overflow:
-- |     State: VULNERABLE
-- |     IDs:  CVE-2010-4344; OSVDB 69685
-- |     Risk factor: High   CVSSv2: 9.3 (AV:N/AC:M/Au:N/C:C/I:C/A:C)
-- |     References:
-- |       http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4344
-- |       http://osvdb.org/show/osvdb/69685
-- |  
-- |   Exim exim User Account Configuration File Directive Local privileges escalation:
-- |     State: VULNERABLE
-- |     IDs:  CVE-2010-4345; OSVDB 69860
-- |     Risk factor: Medium   CVSSv2: 6.9 (AV:L/AC:M/Au:N/C:C/I:C/A:C)
-- |     References:
-- |       http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4345
-- |_      http://osvdb.org/69860


API for portrule and hostrule scripts:
--------------------------------------

* Create and return a new local report object.
  local report = vulns.Report:new(SCRIPT_NAME, host, port)

  Note: SCRIPT_NAME, host and port information can be used by
  post-processing scripts.

* Report vulnerabilities.
  Arguments are: valid vulnerabilities tables.
  return report:make_output(vuln_table, ...)

* Add vulnerabilities.
  Arguments are: valid vulnerabilities tables.
  report:add(vuln_table, ...)
  return report:make_output()


General usage:
--------------
local report = vulns.Report:new(SCRIPT_NAME, host, port)
return report:make_output(vuln_table)



3) prerule and postrule scripts (vulnerability management scripts):
-------------------------------------------------------------------

* Scripts can make the vulns.lua store silently the vulnerability reports.
  
  The library can store silently all reported vulnerabilities, this is
  very useful for post-processing scripts (postrule) that produce final
  reports. To enable this feature these post-processing scripts must first
  call vulns.save_reports(), then any call to report:add(...) will
  automatically save vulnerabilities in the global database (registry).


* Vulnerability information for post-processing scripts:

  vuln_info = {

    script_name = "the name of the script that pushed me",

    -- If the 'host' table was specified when calling
    -- vulns.Report:new()
    -- e.g. local report = vulns.Report:new(SCRIPT_NAME, host)
    host = {
      ip = host.ip
      targetname = host.targetname,
      bin_ip = host.bin_ip,
    },

    -- If the 'host' and 'port' tables were specified when calling
    -- vulns.Report:new()
    -- e.g. local report = vulns.Report:new(SCRIPT_NAME, host, port)
    port = {
      number = port.number,
      protocol = port.protocol,
      service = port.service,
      version = port.version,
    }


    -- mandatory fields
    title = 'Exim string_format Function Remote Overflow', -- string
    state = vulns.State.VULN, -- number

    -- the following fields are all optional
    IDS = {CVE = 'CVE-2010-4344', OSVDB = '69685'}, -- IDs map
    risk_factor = 'High', -- string
    description = 'vulnerability description ...', -- string
    references = { -- list of references
      'http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4344',
      'http://osvdb.org/show/osvdb/69685',
    },
    date = {y = 2010, m = 12, d = 14} -- date table
  }


Functions list:
---------------
* Instruct the vulns library to store vulnerabilities reported by all the
  scripts in the database (registry). This function can take a callback
  function as an argument to filter vulnerabilities before they are saved
  in the registry, this way scripts can store only the interested discovered
  vulnerabilities. The returned value is a uniq ID that other functions
  can use to reference only the saved vulnerabilities that satisfy the
  callback filter. The callback function is an optional arugment.

  You should call this function in the prerule of the post-processins scripts.
  function vulns.save_reports()

  Examples:
  1)
  local FID -- The filter ID of the current script
  prerule = function()
    FID = vulns.save_reports() -- no callback filters, all vulnerabilities
                               -- will be saved in the registry.
    return true
  end

  postrule = function()
    if vulns.get_ids(FID) then
      return true
    end
  end


  2)
  local FID = {} -- The filters ID table of the current script

  local callback_filter_vuln_high = function(vuln_info)
    -- process vuln_info and return true only if the vulnerability was
    -- confirmed and if the risk factor is high
  end

  local callback_filter_not_vuln = function(vuln_info)
    -- process vuln table and return true if the program is not vulnerable
  end

  prerule = function()
    FID['vuln_high'] = vulns.save_reports(callback_filter_vuln_high)
    FID['not_vuln'] = vulns.save_reports(callback_filter_not_vuln)
    return true
  end

  postrule = function()
    for _, f in pairs(FID) do
      if vulns.get_ids(f) then
        return true
      end
    end
  end


The following functions have effects only if the vulns.save_reports() was
called.

* Add vulnerability IDs to the database.
  Note: the library will handle and add the appropriate ids that are found
  in the vuln_table, so you don't need to call this function.
  function vulns.add_ids(filter_id)


* Get the IDS table.
  function vulns.get_ids(filter_id)
  
  Argument filter_id The filter ID returned by the vulns.save_reports().


* Add vulnerability entries to the database, however it's better for scripts
  to use the Report class (i.e. call report:add(vuln_table, ...) ).

  function vulns.add(script_name, vuln_table, ...)


* Search for the specified vulnerability ID and return true if there are
  hosts that are affected or were tested against this vulnerability.
  (All vulnerability states are counted).
  
  function vulns.lookup_id(filter_id, vuln_id_type, entry_id)
  
  Arguments:
  o filter_id The filter ID returned by the vulns.save_reports()
  o vuln_id_type The vulnerability ID type: 'CVE', 'OSVDB' or 'BID' etc
  o entry_id The vulnerability ID.


* Search and return a list of vulnerability entries according to the
  specified selection filter.

  function vulns.find(filter_id, selection_filter)

  Arguments:
  o filter_id The filter ID returned by the vulns.save_reports()
  o selection_filter is a table with the current fields:
    state: The state mask (optional field).
    hosts_filter: A function that will inspect the host table and returns
                  true or false. (Optional field).
    ports_filter: A function that will inspect the host and port tables
                  and returns true or false. (Optional field).
    risk_factor: The risk factor (optional).
    id_type: The ID type ('CVE', 'OSVDB' or whatever) (optional).
    id: The vulnerability ID (optional).


  -- Return all vulnerabilities
  local list = vulns.find(filter_id)

  -- Select the returned vulnerabilities
  local list = vulns.find(filter_id,
                          {state = bit.bor(vulns.State.VULN,
                                           vulns.State.LIKELY_VULN),

                          -- The argument is the vuln host table if it
                          -- was specified.
                          hosts_filter = function(host)
                                            -- parse and match data
                                            -- return true or false
                                         end,

                          -- The argument is the vuln host and port tables
                          -- if they were specified.
                          ports_filter = function(host, port)
                                            -- parse and match data
                                            -- return true or false
                                         end,

                          risk_factor = "High",
                          id_type = 'CVE', id ='CVE-XXXX-XXXX'})


* Return a list of vulnerability entries (hosts that are affected or were
  tested against the specified vulnerability ID) according to the state
  mask. 
  function vulns.find_id_entries(filter_id, vuln_id_type,
                                 entry_id, state_mask)

  e.g.
  -- all states
  local list = vulns.find_id_entries(filter_id, 'CVE', 'CVE-XXXX-XXXX')

  -- Hosts and entries that are "NOT VULNERABLE"
  local list = vulns.find_id_entries(filter_id, 'CVE', 'CVE-XXXX-XXXX',
                                     vulns.State.NOT_VULN)

  -- Hosts and entries that are: "LIKELY VULNERABLE" and "VULNERABLE"
  local list = vulns.find_id_entries(filter_id, 'CVE', 'CVE-XXXX-XXXX',
                                     bit.bor(vulns.State.LIKELY_VULN,
                                     vulns.State.VULN))

  With this function you can construct a list of major vulnerability
  issues and see if your network is vulnerable or not.


* Return a vulnerability iterator. This function will take a selection
  filter like the vulns.find() function.
  function vulns.get_vulnerabilities(filter_id, selection_filter)


* Return formatted output to be displayed to the user. This function will
  take a selection filter like the vulns.find() function.
  function vulns.make_output(filter_id, selection_filter)

  -- Post-processing scripts can do:
  return vulns.make_output(filter_id,
                           {state = vulns.State.VULN, risk_factor = "High"})


* Return formatted output to be displayed to the user. This function will
  take a vulnerability entry that was returned by one of the previous
  functions: vulns.find(), vulns.find_id_entries() and
             vulns.get_vulnerabilities()

  function vulns.make_output_vuln(filter_id, vuln_table)


Usage:
------

  -- Post-processing script:
  local FID

  prerule = function()
    FID = vulns.save_reports()
    return true
  end
 
  postrule = function()
    if vulns.get_ids(FID) then
      return true
    end
  end

  prerule_action = function()
    return nil
  end

  postrule_action = function()
    -- parse, manipulate and report saved vulnerabilities
    -- ...

    -- Return all the stored vulnerabilities that have a state:
    -- 'VULNERABLE' or 'LIKELY_VULNERABLE'
    return vulns.make_output(FID,
                             {state = bit.bor(vulns.State.VULN,
                                              vulns.State.LIKELY_VULN)})
  end

  tactions = {prerule = prerule_action, postrule = postrule_action}
  action = function(...) return tactions[SCRIPT_TYPE](...) end



Vulnerability database Internal data representation:
----------------------------------------------------

VULNS = {

  -- Vulnerability entries
  ENTRIES = {

    HOSTS = {
      [host_a] = {

        vuln_1 = {
          title = 'Program X vulnerability',
          state = vulns.State.VULN,
          IDS = {CVE = 'CVE-XXXX-XXXX', OSVDB = 'XXXXX'},

          -- the following fields are all optional
          risk_factor = 'High',
          description = 'vulnerability description ...',

          references = VULNS.SHARED.REFERENCES[x],
        },

        vuln_2 = {
          ...
        },
        ...
      },

      [host_b] = {
        ...
      },
    },
    
    NETWORK = {

    },
  },


  -- Store shared data between vulnerabilities here (type of data: tables)
  SHARED = {
    -- List of references, members will be referenced by the previous
    -- vulnerability entries.
    REFERENCES = {
      {
        ["http://...";] = true,
        ["http://...";] = true,
      },
      {
      },
    },
  },


  -- These are tables that are associated with the different filters.
  -- This will help the vulnerabilities lookup mechanism.
  --
  -- Just caches to reference all the vulnerabilities information:
  -- tables, maps etc. Only memory addresses are stored here.
  FILTER_ID_1_IDS = {

    'CVE' = {
    
      'CVE-XXXX-XXXX' = {
        references = VULNS.SHARED.REFERENCES[x],

        entries = {
          HOSTS = {
            -- References to the hosts affected by this vulnerability.
            [host_x] = VULNS.ENTRIES.HOSTS[host_x][vuln_x],
            [host_y] = VULNS.ENTRIES.HOSTS[host_y][vuln_a],
            ...
          },
      },

      'CVE-YYYY-YYYY' = {

      },

    },

    'OSVDB' = {
      'XXXXX' = {
        references = VULNS.SHARED.REFERENCES[x],

        entries = {
          ...
        },
      },
      'YYYYY' = {
        references = VULNS.SHARED.REFERENCES[y],
        entries = {
          ...
        },
      },
    },

    'YOUR_FAVORITE_ID' = {
      'XXXXX' = {
        ...
      },
    },

    -- Entries whithout the vulnerability ID are stored here.
    'NMAP_IDS' = {
      'XXXXX' = {
        ...
      },
    },
  },

  
  FILTER_ID_2_IDS = {
    ...
  },

  FILTER_ID_3_IDS = {
    ...
  },

}

Thanks.

-- 
tixxdz
http://opendz.org

Attachment: vuln_library_proposal.diff
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/