1) Where Splunk stores the data ?
Ans: Splunk has its own database
and stores data in flat files.
2) How much disc space is
required to store data in splunk ?
Ans: Splunk stores data in 2 type
of files/directories 1) actual data in zip files takes ~15% of file
size 2) index files takes ~35% of file size So around 50% of files
size require to store that file and other than this space is required to
store search results.
3)How the indexer stores indexes
?
Ans: As the indexer indexes your
data, it creates a number of files. These files contain two types of
data: The raw data in compressed form (rawdata) Indexes that point
to the raw data, plus some metadata files (index files) Together, these files
constitute the Splunk Enterprise index.
4) What is bucket ? How data ages
in Splunk ?
Ans: An index directory is called
a bucket. A bucket moves through several stages as it ages:
hot warm cold frozen thawed
5) What are components of
splunk/splunk architecture?
Ans: Splunk Components are: 1)
Search head - Front end and provides GUI for searching 2) Indexer - indexes
logs/machine data 3) Forwarder -Forwards logs/data to Indexer 4) Deployment
server -Manages splunk components in distributed environment
6) What is Splunk?
Ans: Splunk is a software or tool
used to monitor, report, analyze machine data
7) What is default index in
Splunk ?
Ans: main, If you do not assign
any index to data/log then this log/data goes to default index main.
8) What are common port numbers
used by Splunk?
Ans: Below are common port
numbers used by splunk,however you can change them if required 1) Service Port
number Used
2) Splunk Web Port: 8000 3)
Splunk Management Port: 8089 4) Splunk Indexing Port: 9997 5) Splunk Index Replication
Port 8080 6) Splunk network port: 514 (Used to get data in from netwok port
i.e. UDP data) 7) KV store 8191
9) Which is latest splunk version
in use?
Ans: splunk 7.1.1
10) What is splunk indexer?What
are stages of splunk indexing?
Ans: The indexer is the Splunk
Enterprise component that creates and manages indexes. The primary functions of
an indexer are: 1) Indexing incoming data. 2) Searching the indexed data.
11) What are
types of splunk forwarder?
Ans: There are two types of
splunk forwarders: a) universal forwarder(UF) -Splunk agent installed on
non-Splunk system to gather data locally, can’t parse or index data b)
Heavy weight forwarder(HWF) - full instance of splunk with advance
functionality. - Generally works as a remote collector, intermediate forwarder,
and possible data filter because they parse data, they are not
recommended for production systems
12) what are most important configuration files of splunk?
Ans: 1) inputs.conf 2)
outputs.conf 3) props.conf 4) indexes.conf 5) transforms.conf 6) server.conf
13)What is the function of props.conf and transforms.conf files?
Ans: props.conf: you can create
fields, event break,line break for multiline events, assign timestamp, event
segmentation transforms.conf: you can create fields,Anonymizing data,
index-time field extractions, Configuring regex-based host and source
type overrides, Routing specific events to a particular index
14) What are types of splunk licenses?
Ans: 1) Enterprise license 2)
Free license, expires in 60 days and 500MB per day 3) Forwarder license 4) Beta
license 5) Licenses for search heads (for distributed search) 6) Licenses for
cluster members (for index replication)
15) What are diffrence between splunk app and splunk add-on?
Ans: splunk app is container/directory
of configurations,searches,dashboards etc. in splunk where you can create
dashboard, report etc. splunk add-on is used to define the data, field
extractions and does not have gui.
16) How you can assign index to a log file ?
Ans: In inputs.conf under file
monitor stanza write this: index= your index name
17) what happens if the license master is unreachable?
Ans: license slave will start a
24-hour timer, after which search will be blocked on the license slave (though
indexing continues). users will not be able to search data in that slave until
it can reach license master again
18) what is summary index in splunk?
Ans: Summary index is used to
give fast result of report/dashboard. You can store any cron/save search
result in summary index so that you can reduce the data in summary index.
19) What is splunk DB connect?
Ans: Splunk DB Connect is an
splunk app (generic SQL database plugin that allows you to
easily integrate database information with Splunk queries and reports.
20) Can you write down a general regular expression for extracting ip
address from logs?
Ans: There are multiple ways we
can extract ip address from logs.Below are few examples. Regular Expression for
extarcting ip address: rex field=_raw
"(?\d+\.\d+\.\d+\.\d+)" OR rex field=_raw
"(?([0-9]{1,3}[\.]){3}[0-9]{1,3})"
21) What is difference between stats vs transaction command?
Ans: The transaction command is
most useful in two specific cases: Unique id (from one or more fields) alone is
not sufficient to discriminate between two transactions. This is the case
when the identifier is reused, for example web sessions identified by
cookie/client IP. In this case, time span or pauses are also used to
segment the data into transactions. In other cases when an identifier is
reused, say in DHCP logs, a particular message may identify the beginning or
end of a transaction. When it is desirable to see the raw text of the events
combined rather than analysis on the constituent fields of the
events. In other cases, it's usually better to use stats as the
performance is higher, especially in a distributed search environment.
Often there is a unique id and stats can be used.
22) How to troubleshoot splunk performance issues?
Ans: Answer to this question
would be very wide but basically interviewer would be looking for
following keywords in intreview: -Check splunkd.log for any errors -Check
server performance issues i.e. cpu/memory usag,disk i/o etc -Install SOS
(Splunk on splunk) app and check for warning and errors in dashboard -check
number of saved searches currently running and their system resources
consumption - install Firebug, which is a firefox extension. After it's
installed and enabled, log into splunk (using firefox), open firebug's
panels, switch to the 'Net' panel (you will have to enable it). The Net panel
will show you the HTTP requests and responses along with the time spent in
each. This will give you a lot of information quickly over which requests are
hanging splunk for a few seconds, and which are blameless. etc..
23) What are buckets? explain splunk bucket lifecycle?
Ans: Splunk places indexed data
in directories, called as "buckets". It is physically a
directory containing events of a certain period.A bucket moves through several
stages as it ages: Hot - Contains newly indexed data. Open for writing. One or
more hot buckets for each index. Warm - Data rolled from hot. There are many
warm buckets. Colld - Data rolled from warm. There are many cold buckets.
Frozen - Data rolled from cold. The indexer deletes frozen data by default, but
you can also archive it. Archived data can later be thawed (Data in
frozenbuckets is not searchable) By default, your buckets are located in
$SPLUNK_HOME/var/lib/splunk/defaultdb/db. You should see the hot-db there, and
any warm buckets you have.By default, Splunk sets the bucket size to 10GB
for 64bit systems and 750MB on 32bit systems.
24) What is the different between stats and eventstats commands?
Ans: Stats command generate
summary statistics of all existing fields in your search results and save
them as values in new fields. Eventstats is similar to the stats command,
except that aggregation results are added inline to each event and only if the
aggregation is pertinent to that event. eventstats computes the requested
statistics like stats, but aggregates them to the original raw data.
25) Who are the biggest direct competitors to Splunk?
Ans:
ELK(Elasticsearch,logstash,Kibana), Loggly, Loglogic,sumo logic etc..
26) splunk licenses specify what ?
Ans: how much data you can index
per calendar day
27) how does splunk determine 1 day, from a licensing perspective ?
Ans: midnight to midnight on the
clock of the license master
28) how are forwarder licenses purchased ?
Ans: They are included with
splunk, no need to purchase separately
29) What is command for restarting just the splunk webserver?
Ans: splunk start splunkweb
30) What is command for restarting just the splunk daemon?
Ans: splunk start splunkd
31) What is command to check for running splunk processes on unix/Linux
?
Ans: ps aux | grep splunk
32) What is Command to enable splunk to boot start?
Ans: $SPLUNK_HOME/bin/splunk
enable boot-start
33) How to disable splunk boot start?
Ans: $SPLUNK_HOME/bin/splunk
disable boot-start
34) What is sourcetype in splunk?
Ans: Sourcetype is splunk way of
identifying data
35) How to reset splunk admin password?
Ans: To reset your password log
in to server on which splunk is installed and rename passwd file at below
location and then restart splunk.After restart you can login using default
username:admin password:changeme $splunk-home\etc\passwd
35) How to reset splunk admin password?
Ans: To reset your password log
in to server on which splunk is installed and rename passwd file at below
location and then restart splunk.After restart you can login using default
username:admin password:changeme $splunk-home\etc\passwd
36) How to disable splunk launch message?
Ans: Set value OFFENSIVE=Less in
splunk_launch.conf
37) How to clear splunk search history?
Ans: Delete following file on
splunk server $splunk_home/var/log/splunk/searches.log
38) What is btool or how will you troubleshoot splunk configuration
files?
Ans: splunk btool is a command
line tool that helps us to troubleshoot configuration file issues or just
see what values are being used by your Splunk Enterprise installation in
existing environment
39) What is difference between splunk app and splunk add on?
Ans: Basiclly both contains
preconfigured configuration and reports etc but splunk add on do not have
visual app. Splunk apps have preconfigured visual app
40) What is .conf files precedence in splunk?
Ans: File precedence is as
follows: System local directory -- highest priority App local directories App
default directories System default directory -- lowest priority
41) what is fishbucket or what is fishbucket index?
Ans: Its a directory or index at
default location /opt/splunk/var/lib/splunk . It contains seek pointers and
CRCs for the files you are indexing, so splunkd can tell if it has read them
already.We can access it through GUI by seraching for “index=_thefishbucket”
42) How do i exclude some events from being indexed by Splunk?
Ans : This can be done by
defining a regex to match the necessary event(s) and send everything else to
nullqueue. Here is a basic example that will drop everything except events that
contain the string error.
In props.conf:
[source::/var/log/foo]
TRANSFORMS-set =
setnull,setparsing
In transforms.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = error
DEST_KEY = queue
FORMAT = indexQueue
43) How can i tell when splunk is finished indexing a log file?
Ans: By watching data from
splunk's metrics log in real-time. index="_internal"
source="*metrics.log" group="per_sourcetype_thruput"
series="<your_sourcetype_here>" | eval MB=kb/1024 | chart
sum(MB) or to watch everything happening split by sourcetype....
index="_internal" source="*metrics.log"
group="per_sourcetype_thruput" | eval MB=kb/1024 | chart sum(MB)
avg(eps) over series And if you're having trouble with a data input and
you want a way to troubleshoot it, particularly if your whitelist/blacklist
rules arent working the way you expect, go to this URL:
https://yoursplunkhost:8089/services/admin/inputstatus
44) How to set the default search time in Splunk 6?
Ans: To do this in Splunk
Enterprise 6.0, use ui-prefs.conf. If you set the value in
$SPLUNK_HOME/etc/system/local, all your users should see it as the default
setting. For example, if your $SPLUNK_HOME/etc/system/local/ui-prefs.conf file
includes: 1. [search] 2. dispatch.earliest_time = @d 3. dispatch.latest_time =
now The default time range that all users will see in the search app will be
today.
45) What is dispatch directory?
Ans:
$SPLUNK_HOME/var/run/splunk/dispatch contains a directory for each search that
is running or has completed. For example, a directory named 1434308943.358 will
contain a CSV file of its search results, a search.log with details about
the search execution, and other stuff. Using the defaults (which you can override
in limits.conf), these directories will be deleted 10 minutes after the search
completes - unless the user saves the search results, in which case the results
will be deleted after 7 days.
46) What is difference between search head pooling and search head
clustering?
Ans: Both are features provided
splunk for high availability of splunk search head in case any one search
head goes down.Search head cluster is newly introduced and search head pooling
will be removed in next upcoming versions.Search head cluster is managed
by captain and captain controls its slaves. Search head cluster is more
reliable and efficient than search head pooling.
47) If I want to add/onboard folder access logs from a windows machine
to splunk how can I add same?
Ans: Below are steps to add
folder access logs to splunk 1.Enable Object Access Audit through group policy
on windows machine on which folder is located 2. Enable auditing on specific
folder for which you want to monitor logs 3.Install splunk universal forwarder on
windows machine 4.Configure universal forwarder to send security logs to splunk
indexer
48) How would you handle/troubleshoot splunk license violation warning
error?
Ans: License violation warning
means splunk has indexed more data than our purchased license quota. We have to
identify which index/sourcetype has received more data recently than usual
daily data volumr. We can check on splunk license master pool wise available
quota and identify the pool for which violation is occuring. Once we know the pool
for which we are receiving more data then we have to identify top
sourcetype for which we are receiving more data than usual data.Once sourcetype
is identified then we have to find out source machine which is sending
huge number of logs and root cause for the same and troubleshoot
accordingly.
49) What is mapreduce algorithm?
Ans: Maprduce algorithm is secret
behind splunk fast data searching speed. It's an algorithm typically used for
batch based large scale parallelization. It's inspired by functional
programming's map() and redce () functions.
50) How splunk avoids duplicate indexing of logs ?
Ans: At indexer splunk keeps
track of indexed events in a directory called fish buckets (default location
/opt/splunk/var/lib/splunk). It contains seek pointers and CRCs for the files
you are indexing, so splunkd can tell if it has read them already.
51) Where Hunk stores the data ?
Ans: Hunk does not have its own
data , its uses Hadoop or NoSQL data to analyze.
52) What is Virtual index ?
Ans: Its data container, and
contain information about files or directories in Hadoop.
53)What is ERP ?
Ans: in ERP you can provide
hadoop enviornment information. ERP takes the request from search head and
execute map-reduce job in Hadoop.
54) Can I use splunk and hunk in same instance ?
Ans: yes, with different License
(hunk and splunk enterprise).
55) Can I read Hunk archived data exported in Hadoop by using pig or
hive?
Ans: Yes, you can read but you
need to use bucket reader.
56) Splunk can be replaced by Hunk ?
Ans: No, Hunk is used for Hadoop,
NoSQL data analysis.
57) How Hunk search head cluster is different than Enterprise Splunk
search head cluster?
Ans: It is same as Enterprise
search head cluster.
58) What customers are using Hunk ?
Ans: Yahoo, Comcast, Ericsson and
many more...
59) Can we archive all Splunk buckets in Hadoop by using Hunk?
Ans: No, We can archive only warm
and cold buckets.
60) Can we archive some search result/data from splunk in Hadoop by
using Hunk?
Ans: No, you can not archive
splunk search output/result in Hadoop by using Hunk. You have to archive
all index data(cold,warm buckets) in Hadoop.
61) What is datamodel ? Have you
created any datamodel? Explain?
62) What is pivot
63) Dashboard vs form
64) chart vs timechart
65) What are the methods to
improve dashboard performance?
Optimize search, datamodel
acceleration, post processing, summary indexeing, report acceleration
66) What is postprocessing?
Postprocess is a mechanism where
one main/base search runs and it shows different panel chart , It improves
the performance of the dashboard.
67)What is lookup table ?
68) What is timebased lookup ?
69) tag vs eventtype
70) Summary indexing vs data
model acceleration
71) ELK vs Splunk
72) how you can assign index to a
log file?
In inputs.conf file under file
monitor stanza
index=<index name>
73) stats vs eventstats vs
streamstats
74) How you can drop INFO, DEBUG
events before indexing.
[source::/var/log/foo]
TRANSFORMS-set = setnull
In transforms.conf
[setnull]
REGEX = . (INFO|DEBUG)
DEST_KEY = queue
FORMAT = nullQueue
75) How you can perform load
balancing in forwarder?
use multiple indexer ip/server
and port in outputs.conf in forwarder
1. [tcpout:<target_group>]
2.
3. server
= [<indexer ip >]:<port>, [<indexer ip>]:<port>
76) How you can send same data to
different indexer for High Availability?
use multiple stazas for indexer
ip/server and port in outputs.conf in forwarder
1. [tcpout]
2.
3. defaultGroup
= <target_group1>, <target_group2>
1. [tcpout:<target_group1>]
2.
3. server
= [<indexer ip >]:<port>
4.
1. [tcpout:<target_group2>]
2.
3. server
= [<indexer ip >]:<port>
4.
77) How you can send data to
selective indexes?
78) How you can filter out files
from directory to monitor?
79) How you can create index time
field?
80) Index vs search time field
extraction
81) fast mode vs verbose mode vs
smart mode for search
82) How LDAP authentication works
?
83) How you can assign indexes to
users so that they only can search those indexes data?
84) What are the config files to
implement authentication?
authentication.conf
85) What is the deployment server
?
86) Gui changes vs config files
changes
87) How can I create an alert
that should not run again in next 1 hour?
88) What are common log files to
debug the splunk issues?
89) What is indexer cluster ?
90) What is search head cluster ?
91) What is replication factor?
92) What is search factor?
93) search head cluster vs search
head pool
94) What is the role of master
node in indexer cluster?
95) What is the role of deployer
in search head cluster?
96) Why it is not recommended to
use deployment manager to manage indexer cluster nodes?
97) What is the role of captain
in search head cluster ?
98) How to change timezone in
events coming into splunk?
In props.conf use TZ
=<timezone>
99) I am getting search reached
maximum error/warning, How can I fix this issue?
100) My data size reached beyond
my splunk license limit. Can I search the data or data is indexed in splunk ?
101) what are the steps required
for data onboarding in splunk ?
