Splunk Forwarder Troubleshoot

1. check if splunk process is running on splunk forwarder

For windows check in services

And for linux use below command

ps -ef |grep splunkd  Or

cd $SPLUNK HOME/bin

./splunk status

2.Check on indexer if receiving is enabled on port 9997 and port 9997 is open on indexer

check if receiving is configured :On indexer go to setting>>forwarding and receiving >> check if receiving is enabled on port 9997.If not enable it

4. check if you are able to telnet  indexer  from forwarder host

telnet  indexer name 9997

If you are not able to telnet to the server ,then check network and firewall issue

5.Confirm on indexer if your file is already indexed or not by using below search query

In the Splunk UI, run the following search - index=_internal "FileInputTracker" *<path_to_file>*

6.Check if forwarder has completed processing log file i.e. tailing process by using below url

https://splunk forwarder server name:8089/services/admin/inputstatus/TailingProcessor:FileStatus

in tailing process output you can check if forwarder is having any issue for processing file

7. Check out log file permissions which you are sending to splunk.verify if splunk user has access to log file

8.Checkout filestamp for last modification and verify if forwarder is monitoring it

9.Verify inputs.conf and outputs.conf gor proper configuration

--------------------------------------------------------------------------

10. Checkout disk space availability on indexer

11. check splunkd.log on forwarder at location $SPLUNK_HOME/var/log/splunk for any errors.like for messages that are from 'TcpOutputProc', they should give you an indication as to what is occurring when the forwarder tries to connect to the indexer

12. tcpdump port 997 data for any errors

tcpdump -i etho port 997

13. check out ulimit if you have installed forwarder on linux. and set it to unlimites or max

ulimit is limit set by default in linux is limit for number files opened by a process

check ulimit command:

ulimit -n

set ulimit command:

ulimit -n expected size

14. check metrics.log if any queue is blocked , if it is blocked then resolve the issue.

15. Restart the forwarder