NetFlow: Pre-requisites and configuration

This service template has been designed to meet specific needs. It requires various fields to be completed to target bandwidth consumption

Ideally, each deployed service should meet a particular need, such as measuring throughput generated by the e-mail service. In this case, the user will complete the various fields required for this measurement (destination IP of the mail server, SMTP port 25...).

Introduction

Netflow service models allow you to monitor the average flow generated over a defined period of time by an application, source IP or destination IP and generate alerts if defined thresholds are exceeded. They also report data and performance graphs in the same way as other service models.

Netflow is a network protocol used to assess IP network traffic. It was developed by Cisco Systems. Today, NetFlow has become an industry standard supported by many devices. There are several versions of the protocol, but the most common versions are versions 5 and 9.

Network flows

NetFlow uses the concept of a flow to capture data about network behavior, such as the source and destination of network traffic, the applications using the network, and the amount of bandwidth allocated to those applications.

A stream is a unidirectional sequence of packets between a given source and a given destination, defined by a 7-tuple key comprising the following fields:

• Source IP address
• Destination IP address
• Source Port
• Destination port
• IP Protocol
• Input interface
• Type of IP service

NetFlow records

The NetFlow information collected by a Flow Publisher is managed by creating records for each flow. Each record is managed in the NetFlow cache. When packets are captured, the statistics for active flows are updated. Once a stream has been created and placed in the NetFlow cache, it remains active until it expires. Once the flow expires, the flow record is added to a NetFlow export datagram for transmission to the NetFlow collector.

NetFlow support

In addition to Cisco, many other network equipment manufacturers offer NetFlow support on their devices. The list includes Juniper, Alcatel-Lucent and Nortel, among others. With respect to software platforms, there is support on VMware and Linux servers .

Some manufacturers use another names for this technology:

• Jflow or cflowd at Juniper Networks
• NetStream at 3Com/HP
• NetStream at Huawei Technologies
• Cflowd at Alcatel-Lucent
• Rflow at Ericsson
• AppFlow at Citrix

Architecture

Network elements (switches and routers) compile statistics on the network flow data they export to collectors. These detailed statistics can include the number of packets and bytes, application ports, IP addresses, QoS fields, the interfaces through which they pass, etc.

The architecture for collecting IP network traffic information is as follows:

• NetFlow exporter: Observes data in packets, creates monitored network traffic records and transmits this data to the NetFlow Collector.
• NetFlow collector: Collects the records sent by the exporter and stores them in a local database.
• ServiceNav Box: Retrieves information collected by the NetFlow Collector
• SNP (Monitoring Platform) allows you to configure the NetFlow template to use the data reported by the ServiceNav Box

Configuration of network equipment

The network equipment must be configured to export the flows to the Netflow Collector Storage.

Our procedure Configuring NetFlow devices provides you with a large number of Netflow activation procedures depending on your device.

Setting up NetFlow Collector Storage

A collector can act as a central collector for both NetFlow and sFlow exports.

sFlow procedure: https://servicenav.coservit.com/en/documentations/how-to-use-our-service-model-networkanalysis-sflow/ 

Sizing NetFlow Collector Storage

How much disk space should an average NetFlow deployment consume? One of the biggest considerations is the impact exporting NetFlow data will have on available bandwidth, CPU overhead on devices and the hard drives that store it..

It is important to note that a network flow data export can contain records containing up to 30 conversations or streams . This is important because the average volume of NetFlow is directly proportional to the number of unique TCP/UDP sockets created by clients and servers on the network.

This aggregated nature of NetFlow, and the fact that NetFlow packets are composed solely of IP header information (i.e., not the packet payload itself), explains why the export consumes only 1-2% of the interface rate. Since 2004, Cisco's NetFlow experts have maintained a rule of thumb that NetFlow will create only 1-1.5% of throughput on the interface to which it is exported.

What is the typical stream volume per PC? The answer is: it depends, the trend seems to be about 100 flows / minute per computer, with a peak of 350

For example, a company has 1000 nodes and that each node generates 200 feeds per minute. This is equivalent to about 200,000 feeds in one minute, which is about 3300 flows per second.  Why so much flow?

Applications generate a lot of unique feeds, especially web browsers and most applications. Here are some typically very talkative applications:

• Java, Adobe, Anti-virus, web browsers
• Skype is very talkative and generates traffic to DNS
• Web page feeds generating images, ads, etc.
• Email constantly checking inbox
• NetBios

A flow stored on the NetFlow Colletor Storage occupies 150 bytes of disk space, it is therefore recommended to provision 2 GB per day, per 100 nodes:

NetFlow Collector Storage Deployment

The NetFlow Collector Storage will be created from a ServiceNav Box master image.

It is therefore necessary to follow fully documentation on implementing a ServiceNav Box.

Configuring Collector Storage

Connecting to Collector Storage

Once the Collector Storage operating system has booted it asks for login information, this information is the following:

Login : Ask support
Password: Ask support

 Caution: Never update your monitoring box. Updates are sent from the central platform.

Downloading the installation script

You need to download the installation script, to do so execute the following commands:

sudo su -

cd /home/coadmin

ftp -p software.servicenav.io

# Enter login & password

 Login : Ask support

Password: Ask support

 cd TOOLS

get xflow_installation.tar

exit

tar xvf xflow_installation.tar

Launching the installer

To launch the installer, execute these three commands:

sudo su -

cd /home/coadmin/xflow_installation

./xflow_installation.pl

Then follow the instructions on the screen:

###### xFLOW COLLECTOR INSTALLATION ##########
Copyright CoServIT 2012-2013. All rights reserved.
## Configure your flow Collector
## - Configure the rules
## - Configure the directory
## - Start and save process n(s)fcapd
## Data retention configuration
## Delete a configuration 
## Note: You can exit this program with Ctrl+C 
Press any key to continue

Press any key

##### xFLOW COLLECTOR CONFIGURATION ############

Do you want to configure the Flow Collector (y/n) [default: n] :

Type "y" and then press Enter to validate your choice

NetFlow (n) or sFlow (s) [default = n] :

We configure Netflow, type "n" and then press Enter to validate your choice

Listening Port : 9995

Fill in the listening port that you have defined and press Enter to validate your choice:

Directory : WAN

Fill in the name of the export destination directory you have defined and press Enter to validate your choice 

Do you want to save this configuration ?  (y/n) [default: n] :

Type "y" and then press Enter to validate your choice 

Do you want to configure the data retention (y/n) [default : n] :

To set up data retention on the Collector, type "y" and then press Enter to validate your choice 

NetFlow (n) or sFlow (s) [default = n] :

We configure Netflow, type "n" and then press Enter to validate your choice

Data retention in days (example 2 for two days): 2

Fill in the number of days of data retention and press Enter to validate your choice 

Do you want to save this configuration ?  (y/n) [default: n] :

Type "y" and then press Enter to validate your choice

Configuring the NetworkAnalysis-NetFlow Service Template

Once the prerequisites have been configured via the installer, you can use the service template NetworkAnalysis-NetFlow by referring to the following procedure How to use our NetworkAnalysis-NetFlow service template

If an error occurs, contact support or refer to the following topics: "Checking" "Debug"

Checking Netflow Collector Storage

After launching the installer, wait about twenty minutes and check that the exported files arrive in the destination directory by executing the following command:

ll /home/coadmin/network_analysis/

Monitoring of NetFlow Collector Storage

In addition to the Netflow monitoring undertaken following installation of the Netflow Collector Storage, add self-monitoring services based on the following service template:

• LIN-DirectorySize to monitor the size of your destination directories.
• Lin-ProcessName  to monitor the proper execution of nfcapd processes

Debug

Tasks performed by the script :

• Creation of an ACL authorizing listening on the specified port
• Creation of the destination directory
• Launching the NetFlow process
• Creating a cron task to monitor the netflow process
• Creation of a startup file to restart process in case of Collector reboot
• Creating a Data Deletion and Retention Task

Creation of an ACL authorizing listening on the specified port

Check that the rule has been created by executing the following command:

Expected result: Presence of the rule by light authorizing the port filled in during installation.

iptables -L 

If the rule is not present, execute the following commands:

sudo su -

iptables -A INPUT -p udp -dport -j ACCEPT

/etc/init.d/iptables.sh restart

Check again if there is a problem, contact support.

Creation of the destination directory

By default the installer creates the destination directories under /home/coadmin/network_analysis . Check that the directory has been created by executing the following command: "Expected result: Presence of the directory specified during installation.

sudo su -

ll /home/coadmin/network_analysis/

If the directory does not exist, execute the following commands:

sudo su -

mkdir//home/coadmin/network_analysis/

chmod -R 777 /home/coadmin/network_analysis/

chown coadmin:coadmin /home/coadmin/network_analysis/

Check again, if there is a problem, contact support.

Launching the NetFlow process

The installer launches the netflow process based on the listening port and destination directory entered.

Check that the process is running by executing the following command:

ps -aux | grep nfcapd

and also check that the listening port and destination directory match those entered during installation.

sudo su -

nfcapd -w -D -l /home/coadmin/network_analysis/ -p

Check again, if there is a problem, contact support.

Creation of a netflow process startup file for when the Collector is rebooted

(with no impact on the initialization and operation of the Collector)

The installer creates a netflow process reset file when the Collector is restarted.

Check the correct configuration by executing the following command:

sudo su -

ll /etc/rc0.d/ 

In case of problems, contact support.

Creation of a cron task to monitor the netflow process (without impacting the running of the Collector)

The installer creates a cron task that runs a command every minute to check the process and restart it if necessary.

Check the creation of the task by executing the following command:

crontab -l

In case of problems, contact support.

Creating a Data Deletion Task

If your data is not deleted or the data retention does not match what was entered during installation, execute the following command:

more /usr/local/nagios/libexec/nfcapd_deleteCache.sh

If the information returned is not correct or the number of retention days does not match what was set during installation, contact support.

Checking exports.

As a reminder, the files are generated by and are fed continuously by your NetFlow exporterThis means that the presence of these files does not necessarily mean that the exports have been successfully consumed by the Collector.

To check, execute the following command:

Sudo su -

ll /home/coadmin/network_analysis/

If there are no files, repeat the verification steps or contact support.

If the configurations are correct and the files are 276 bytesin size: which means that the files do not contain data, there are two possible causes:

• Exports do not reach the Collector Storage
• Exports look good in the Collector but do not contain any data.

In this case, contact support or run the following check commands:

tcpdump -i src

Ex:

Sudo su -
tcpdump -i ens160 src 192.168.238.156

If exports occur, you should have connection information from your NetFlow Exporter to the Collector on the defined port. (for info, the above example is related to sFlow exports)

In case of problem check the configuration on your NetFlow Exporter or contact support.