Re: Converting IPFIX flows to columnar format with all IEs - tool recommendations?

[Mel Beckman via NANOG <nanog () lists nanog org>]

Here's a StackOverflow thread that got me started on the Python code. These days you could probably get Grok to write 
it for you 🙂

https://stackoverflow.com/questions/65010451/how-can-i-parse-all-the-packets-in-my-pcap-file-instead-of-one
________________________________
From: Mel Beckman via NANOG <nanog () lists nanog org>
Sent: Thursday, January 15, 2026 10:24 AM
To: nanog () lists nanog org <nanog () lists nanog org>
Cc: Jonas Muecke <jonas.muecke () tu-dresden de>; Mel Beckman <mel () beckman org>
Subject: Re: Converting IPFIX flows to columnar format with all IEs - tool recommendations?

I have used MapForce Pro edition, a generic commeercial tool for any-to-any data conversion, for similar tasks in the 
past. Specifically I was converting pcap logs of SMTP sessions to create a database of source and destination email for 
a custom security project. It worked pretty well, and the tool is reasonably priced. I did end up writing a Python pcap 
preprocessor but that was only to filter out stuff I didn't want.

  -mel
________________________________
From: Jonas Muecke via NANOG <nanog () lists nanog org>
Sent: Thursday, January 15, 2026 9:57 AM
To: nanog () lists nanog org <nanog () lists nanog org>
Cc: Jonas Muecke <jonas.muecke () tu-dresden de>
Subject: Converting IPFIX flows to columnar format with all IEs - tool recommendations?

Hi,

I'm looking for recommendations on conversion and long-term storage of
raw IPFIX flow data. Specifically, I need to convert IPFIX flows stored
in pcap files into a columnar format like Parquet to enable easy and
detailed historic analysis that isn't possible with aggregated data.

Requirements:
- Parse IPFIX from pcap files (including templates)
- Preserve ALL information elements, including custom IEs with
enterprise PENs
- Output enterprise number + IE ID + data (detailed interpretation of
the data not needed)
- Handle IP fragment reassembly or large IP packets

nfdump [1] gets close, but it skips custom IEs. Other tools require
replaying the pcaps which risks overflowing buffers, so reading directly
from pcap files would be preferred.

Has anyone had similar requirements and found a solution? I'm open to
multi-step conversions (e.g., via JSON). Long-term I'll capture directly
to a better format, but need to process existing pcap archives first.

Thanks,
Jonas

[1] https://github.com/phaag/nfdump

--
Jonas Muecke
Phd Student, TU Dresden

_______________________________________________
NANOG mailing list
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/U7ZR5BJFNCBWI4EBLRDUVPVEV45GHID6/
_______________________________________________
NANOG mailing list
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/473NHRQNYUDTJ7BXBCWWNBOKCCWWJPAR/
_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/FBMRTUQ3O3DUUU7SYHHAYENYYKIGRU6T/