Barnyard2 fatal error duplicate references, but there are no duplicates

[elof () sentor se]

I just upgraded to barnyard2.1.10, and it complains...


I dropped and created a brand new database so that there are no old 
garbage data.
I pre-populated the reference-system with all the metadata like I've done 
for years.
I ran barnyard2 in testmode, but it bails:



barnyard2 in self-test mode:
barnyard2 -T -v -c barnyard2.conf -d /log -f snort.unified2 --pid-path /var/run
Found pid path directive (/var/run)
Running in Test mode

         --== Initializing Barnyard2 ==--
Initializing Input Plugins!
Initializing Output Plugins!
Parsing config file "barnyard2.conf"
Found pid path directive (/var/run)
WARNING database: Defaulting Reconnect/Transaction Error limit to 10
WARNING database: Defaulting Reconnect sleep time to 5 second
Checking PID path...
PID path stat checked out ok, PID path set to /var/run
Writing PID "45577" to file "/var/run/barnyard2_mon0.pid"
Chroot directory = /var/log/snort
ERROR database: Query [SELECT ref_id FROM reference WHERE ref_system_id = '7' AND ref_tag = 
'blog.fireeye.com/research/2010/10/feodosoff-a-new-botnet-on-the-rise.html#more';] 
returned more than one result
[SystemCacheSynchronize()], Call to ReferencePopulateDatabase() failed
[CacheSynchronize()]:, SystemCacheSyncronize() call failed.
ERROR: database [DatabaseInitFinalize()]: CacheSynchronize() call failed 
...
Fatal Error, Quitting..
Done. Cleaning up.



So I guess the "returned more than one result" is the actual problem, and 
the following failed cache errors are just a result of it.


Now, the strange thing is that there are no multiple results for this 
query!
> From the same machine, I run 'psql', using the same configuration as 
barnyard2 use, and run the exact command as listed in the errorlog:

psql -h 10.10.10.10 foo foo
Password for user foo:

foo=> SELECT ref_id FROM reference WHERE ref_system_id = '7' AND ref_tag = 
'blog.fireeye.com/research/2010/10/feodosoff-a-new-botnet-on-the-rise.html#more';
  ref_id
--------
    2872
(1 row)



There's just one result, so why do barnyard2 complain about "returned more 
than one result" ?

I guess the error message is related to some other query, not the one 
logged on screen.



So, apparently this specific url-based reference was _not_ duplicated.

However, my old system that pre-populates the reference-system populates 
things just like they did a long time ago, before barnyard even existed 
and the database plugin was builtin in snort. I.e. it populates the same 
reference multiple times if several rules use the same reference:
Example:
The ref_id 7 = 'url'. This table unique.
However, the following three rules would generate three rows in the 
reference table:
alert udp $HOME_NET any -> any 53 (msg:"blah1"; ...
reference:url,www.f-secure.com/weblog/archives/00002227.html; sid:2013481; 
rev:1;)
alert udp $HOME_NET any -> any 53 (msg:"blah2"; ...
reference:url,www.f-secure.com/weblog/archives/00002227.html; sid:2013482; 
rev:2;)
alert udp $HOME_NET any -> any 53 (msg:"blah3"; ...
reference:url,www.f-secure.com/weblog/archives/00002227.html; sid:2013483; 
rev:1;)

ref_id   ref_system_id   ref_tag
17       7               www.f-secure.com/weblog/archives/00002227.html
18       7               www.f-secure.com/weblog/archives/00002227.html
19       7               www.f-secure.com/weblog/archives/00002227.html

I'm not sure, but I can guess that the new barnyard2, in some other query 
that is not logged, complain about these kind of duplicates.




1)
Should there only be only ONE 'www.f-secure.com/weblog/archives/00002227.html' 
in the database, and the signatures 2013481, 2013482 and 2013483 should 
all reuse it, not having their own ref_id instance?

2)
In the new barnyard2.1.10, you seem to have added some kind of 
pre-population of the reference system.
Does this mean that there's no longer any need to pre-populate it using a 
separate system?
If so that would be great - one system less to deal with.
Also, if I'm correct in my assumptions above, the problem should not 
appear at all.


/Elof




my barnyard2.conf:

var SENSOR_NAME foo
var DB_USER foo
var DB_PASSWORD foo
var INTERFACE mon0
var DB_HOST 10.10.10.10
var DB_NAME foo

config reference_file: /usr/local/etc/snort/reference.config
config classification_file: /usr/local/etc/snort/classification.config
config gen_file: /usr/local/etc/snort/gen-msg.map
config sid_file: /usr/local/etc/snort/barnyard2.sid-msg.map
config logdir: /var/log/snort
config hostname: $SENSOR_NAME
config interface: $INTERFACE
config alert_with_interface_name
config chroot: /var/log/snort
config set_gid: snort
config set_uid: snort
config show_year
config umask: 022
input unified2
output log_tcpdump: $SENSOR_NAME-barnyard2.tcpdump
output database: log, postgresql, user=$DB_USER password=$DB_PASSWORD 
dbname=$DB_NAME host=$DB_HOST sensor_name=$SENSOR_NAME


Version:

   ______   -*> Barnyard2 <*-
  / ,,_  \  Version 2.1.10 (Build 310)
  |o"  )~|  By Ian Firns (SecurixLive): http://www.securixlive.com/
  + '''' +  (C) Copyright 2008-2012 Ian Firns <firnsy () securixlive com>



------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!