FortiAnalyzer FortiView not showing data

Had an interesting one recently, I had a customer get in touch with me regarding their FortiAnalyzer. The FortiView pane was showing 'No Data', and the Log View section was gone.

All data was gone in FortiView

Further investigation via CLI revealed the drive was apparently unmounted, or at least unused. To see this I ran this command:

execute lvm info

We observed the following output:

Disk being unused is a bad thing - usually means its unmounted

Warning: Use the following steps at your own risk, if you don't feel comfortable with a Linux/Unix style shell, call Fortinet support. And more importantly backup your config/data before making any changes.

Seeing as the FortiAnalyzer is *nix based, it's common behavior for a full disk to be unmounted/put in read only mode. After a bit of troubleshooting I resolved to contact Fortinet's Support line for some more advanced troubleshooting. We enabled the system shell on the FortiAnalyzer and noted the disks were indeed full.

This is bad

This confirmed my suspicion, the disk was full and it was full, and FortiAnalyzer couldn't use it, so data wasn't able to be processed and displayed. Now we need to understand what happened and why it filled up so quickly, this is a 200GB disk and shouldn't fill up this quickly. According to support, it's a known issue with this version of FortiAnalyzer (v6.0.3) that enabling packet logging in an IPS profile could cause the disk to fill up. The packet logging setting essentially is a no limit packet capture intended for analysis and tuning, and not meant to be running all the time. So we went to Security Profiles > Intrusion Prevention and checked the IPS profile:

If this is enabled, you're storing a copy of every packet that traverses the IPS engine

Bingo! We now know that the packet logging was causing the drive to fill up. Now all we have to do is clear out the packet logs from the /var/private/ips_files/<FG_SerialNo> and get the disk to remount. But first, we will need to disable packet logging on the FortiGate's IPS profile.

Once we cleared the packet logs, we ran execute lvm info again:

Much better! It says used.

When we logged back into the FortiAnalyzer's web interface, FortiView still had no data, for this we will need to rebuild the database hcache by issuing the following command:

diagnose sql hcache rebuild-fortiview

Once that runs you'll be in great shape ready to do more FortiAnalyzing!

The key takeaway here is that Packet Logging wasn't meant to be run all the time, only used as a tool for troubleshooting. Being the overly zealous person that I am, I likely enabled it thinking it was an additional layer of logging, not a full blown packet capture.