Tag Archives: splunk

Apache Custom LogFormats

Take the leg work out of reading your apache logs by converting them to a structured format that's easy to read. Enter JSON :)

Create the file /etc/apache2/vhosts.d/00_default_logging.conf
You will also need to ensure the "logio_module" is enabled.

For the access log, I use the format: Apache doc.

LogFormat "{\"time\":\"%{msec}t\", \"bytes_in\":\"%I\", \"bytes_out\":\"%O\", \"cookie\":\"%{Cookie}i\", \"server\":\"%v\", \"dest_port\":\"%p\", \"http_content_type\":\"%{Content-type}i\", \"http_method\":\"%m\", \"http_referrer\":\"%{Referer}i\", \"http_user_agent\":\"%{User-agent}i\", \"ident\":\"%l\", \"response_time_microseconds\":\"%D\", \"client\":\"%h\", \"remoteAddr\":\"%a\", \"status\":\"%>s\", \"uri_path\":\"%U\", \"uri_query\":\"%q\", \"user\":\"%u\"}" apache_json

And for the error log, I use: Apache doc.

ErrorLogFormat "{\"time\":\"%{msec}t\", \"client\":\"%a\", \"log_level\":\"%l\", \"pid\":\"%P\", \"srcln\":\"%F\", \"error_code\":\"%E\", \"message\":\"%M\"}"

In the vhost.conf I have the following at the bottom of my VirtualHost block:

<VirtualHost>
    ...
    CustomLog "/var/log/apache2/ssl_access.log" apache_json
    ErrorLog /var/log/apache2/ssl_error.log
</VirtualHost>

Give the command to test the config is sane:

apachectl configtest

If nothing is returned, go ahead and restart apache.
This also makes importing the logs much easier into tools like ELK or Splunk.

Pesky Control Characters

For when plain text goes mad. Handy for printf, sed, vim

Control Characters

CTRL   (^D means to hold the CTRL key and hit d)

Oct  Dec Char  Hex  Key    Comments
\000   0  NUL  \x00  ^@ \0 (Null byte)
\001   1  SOH  \x01  ^A    (Start of heading)
\002   2  STX  \x02  ^B    (Start of text)
\003   3  ETX  \x03  ^C    (End of text)
\004   4  EOT  \x04  ^D    (End of transmission)
\005   5  ENQ  \x05  ^E    (Enquiry)
\006   6  ACK  \x06  ^F    (Acknowledge)
\007   7  BEL  \x07  ^G    (Ring terminal bell)
\010   8   BS  \x08  ^H \b (Backspace)  (\b matches backspace inside [] only)
\011   9   HT  \x09  ^I \t (Horizontal tab)
\012  10   LF  \x0A  ^J \n (Line feed)
\013  11   VT  \x0B  ^K    (Vertical tab)
\014  12   FF  \x0C  ^L \f (Form feed)
\015  13   CR  \x0D  ^M \r (Carriage return)
\016  14   SO  \x0E  ^N    (Shift out)
\017  15   SI  \x0F  ^O    (Shift in)
\020  16  DLE  \x10  ^P    (Data link escape)
\021  17  DC1  \x11  ^Q    (Device control 1) (XON) (Default UNIX START char.)
\022  18  DC2  \x12  ^R    (Device control 2)
\023  19  DC3  \x13  ^S    (Device control 3) (XOFF)  (Default UNIX STOP char.)
\024  20  DC4  \x14  ^T    (Device control 4)
\025  21  NAK  \x15  ^U    (Negative acknowledge)
\026  22  SYN  \x16  ^V    (Synchronous idle)
\027  23  ETB  \x17  ^W    (End of transmission block)
\030  24  CAN  \x18  ^X    (Cancel)
\031  25  EM   \x19  ^Y    (End of medium)
\032  26  SUB  \x1A  ^Z    (Substitute character)
\033  27  ESC  \x1B  ^[    (Escape)
\034  28  FS   \x1C  ^\    (File separator, Information separator four)
\035  29  GS   \x1D  ^]    (Group separator, Information separator three)
\036  30  RS   \x1E  ^^    (Record separator, Information separator two)
\037  31  US   \x1F  ^_    (Unit separator, Information separator one)
\177 127  DEL  \x7F  ^?    (Delete)

Printable Characters

Specials (32-47)

Oct  Dec Char Hex    Comments
\040  32 " " \x20    (space)
\041  33  !  \x21    EXCLAMATION POINT(bang)
\042  34  "  \x22    QUOTATION MARK, DIAERESIS
\043  35  #  \x23:   NUMBER SIGN (Pound sign)
\044  36  $  \x24    DOLLAR SIGN
\045  37  %  \x25    PERCENT SIGN
\046  38  &  \x26    AMPERSAND
\047  39  '  \x27    APOSTROPHE, RIGHT SINGLE QUOTATION MARK, ACUTE ACCENT (single quote)
\050  40  (  \x28    LEFT PARENTHESIS  (open parenthesis)
\051  41  )  \x29    RIGHT PARENTHESIS (close parenthesis)
\052  42  *  \x2A    ASTERISK
\053  43  +  \x2B    PLUS SIGN
\054  44  ,  \x2C    COMMA, CEDILLA
\055  45  -  \x2D    HYPHEN, MINUS SIGN
\056  46  .  \x2E    PERIOD, DECIMAL POINT, (Full Stop)
\057  47  /  \x2F    SLANT (SOLIDUS), slash

Digits

Oct  Dec Char Hex
\060  48  0  \x30
\061  49  1  \x31
\062  50  2  \x32
\063  51  3  \x33
\064  52  4  \x34
\065  53  5  \x35
\066  54  6  \x36
\067  55  7  \x37
\070  56  8  \x38
\071  57  9  \x39

Specials (58-64)

Oct  Dec Char Hex    Comments
\072  58  :  \x3A    COLON
\073  59  ;  \x3B    SEMICOLON
\074  60  <  \x3C    LESS-THAN SIGN  (left angle bracket)
\075  61  =  \x3D    EQUALS SIGN
\076  62  >  \x3E    GREATER-THAN SIGN  (right angle bracket)
\077  63  ?  \x3F    QUESTION MARK
\100  64  @  \x40    COMMERCIAL AT

Latin Capital Letters

Oct  Dec Char Hex       Oct  Dec Char Hex       Oct  Dec Char Hex
\101  65  A  \x41	\112  74  J  \x4A	\123  83  S  \x53
\102  66  B  \x42	\113  75  K  \x4B	\124  84  T  \x54
\103  67  C  \x43	\114  76  L  \x4C	\125  85  U  \x55
\104  68  D  \x44	\115  77  M  \x4D	\126  86  V  \x56
\105  69  E  \x45	\116  78  N  \x4E	\127  87  W  \x57
\106  70  F  \x46	\117  79  O  \x4F	\130  88  X  \x58
\107  71  G  \x47	\120  80  P  \x50	\131  89  Y  \x59
\110  72  H  \x48	\121  81  Q  \x51	\132  90  Z  \x5A
\111  73  I  \x49	\122  82  R  \x52

Specials (91-96)

Oct  Dec Char Hex    Comments
\133  91  [  \x5B    LEFT (SQUARE) BRACKET (open bracket) 
\134  92  \  \x5C    REVERSE SLANT (REVERSE SOLIDUS) (backslash, backslant) 
\135  93  ]  \x5D    RIGHT (SQUARE) BRACKET (closing bracket) 
\136  94  ^  \x5E    CIRCUMFLEX ACCENT 
\137  95  _  \x5F    UNDERLINE (LOW LINE)
\140  96  `  \x60    LEFT SINGLE QUOTATION MARK, GRAVE ACCENT 

Latin Small Letters

Oct  Dec Char Hex       Oct  Dec Char Hex       Oct  Dec Char Hex
\141  97  a  \x61	\152 106  j  \x6A	\163 115  s  \x73
\142  98  b  \x62	\153 107  k  \x6B	\164 116  t  \x74
\143  99  c  \x63	\154 108  l  \x6C	\165 117  u  \x75
\144 100  d  \x64	\155 109  m  \x6D	\166 118  v  \x76
\145 101  e  \x65	\156 110  n  \x6E	\167 119  w  \x77
\146 102  f  \x66	\157 111  o  \x6F	\170 120  x  \x78
\147 103  g  \x67	\160 112  p  \x70	\171 121  y  \x79
\150 104  h  \x68	\161 113  q  \x71	\172 122  z  \x7A
\151 105  i  \x69	\162 114  r  \x72

Specials (123-126)

Oct  Dec Char Hex    Comments
\173 123  {  \x7B    LEFT BRACE (LEFT CURLY BRACKET) (open brace)
\174 124  |  \x7C    VERTICAL LINE (pipe)
\175 125  }  \x7D    RIGHT BRACE (RIGHT CURLY BRACKET) (closing brace)
\176 126  ~  \x7E    TILDE (OVERLINE) (squiggle)

Control (127)

Oct  Dec Char Hex Key   Comments
\177 127 DEL \x7F ^?    (Delete)

Useful Splunk Tips

Disable the "open in search" function in a dashboard.

<option name="link.visible">false</option>

One shot log ingestion: (super useful!)

$SPLUNK_HOME/bin/splunk add oneshot "/path/to/file" -index <index> -sourcetype <sourcetype> -auth admin:changeme

Remove indexed logs

splunk cmd btprobe -d $SPLUNK_DB/fishbucket/splunk_private_db --file --reset

track files being read

https://<server>:8089/services/admin/inputstatus/TailingProcessor%3AFileStatus

Splunk Dropdown Menu

You can create drop down menus in most apps eg search or a custom one.  This can be achieved creating the following path:

/opt/splunk/etc/apps/<app_name>/local/data/ui/nav

Then created the file default.xml.  An example of the content is:

<nav search_view="search" color="#002B48">
 <view name="search" default='true' />
 <collection label="Dashboards">
 <collection label="Business Intelligence">
 <view name="betting_stats" />
 </collection>
 <collection label="Native Mobile Intelligence">
 <view name="crittercism_overview" />
 <view name="crittercism_trends" />
 </collection>
 <collection label="Operational Intelligence">
 <view name="operational_intelligence" />
 </collection>
 <collection label="Social Media Posture">
 <view name="social_media_posture" />
 </collection>
 </collection>
</nav>

The second line is the view that is loaded.  This can be a dashboard or as above, I've just used the default search view.  Below that are each option in the menu separated by the "<collection>" tags.
Each view name is the name of the dashboard to open.

Example

Useful Splunk searches.

List all agents and versions :)

index=_internal source=*metrics.log group=tcpin_connections | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | dedup sourceHost | where sourceIp != sourceHost | table sourceHost sourceIp os version | sort 0 version

List all sourcetypes and event dates

| metadata type=sourcetypes index=* | convert ctime(firstTime) ctime(lastTime) | table firstTime lastTime sourcetype | sort +firstTime

List sources and size

index="_internal" source="*metrics.log" group="per_source_thruput"| eval GB=kb/1024/1024 | chart sum(GB) by series | sort - sum(GB)

List License days remaining

| rest /services/licenser/licenses | search group_id="enterprise" status="valid" OR status="expired" | rename group_id as Type | rename splunk_server as "Splunk Server" | eval "Quota (GB)"=(quota/1024/1024/1024) | eval now=now() | eval "Days Remaining"=(expiration_time-now)/86400 | eval "Expiration Time"=strftime(expiration_time, "%Y-%m-%d  %H:%M:%S") | eval "Days Remaining"=round('Days Remaining', 0) | table "Type" "Quota (GB)" "Expiration Time" "Days Remaining" "Splunk Server" status | where 'Days Remaining' < 10

List index size per day

index=_internal source="*license_usage.log"
| eval indexname = if(len(idx)=0 OR isnull(idx),"(UNKNOWN)",idx)
| eval sourcetypename = st
| bin _time span=1d
| stats values(poolsz) as poolsz sum(b) as b by _time, pool, indexname, sourcetypename
| eval GB=(b/1024/1024/1024)
| eval pool=(poolsz/1024/1024/1024)
| fields _time, indexname, sourcetypename, GB, pool
| search indexname=* sourcetypename=*
| stats sum(GB) as GB by _time indexname
| eval GB = round(GB,4)

List index size per indexer

| rest /services/data/indexes
| where disabled = 0
| search title= "wh_aws"
| eval currentDBSizeGB = round( currentDBSizeMB / 1024)
| where currentDBSizeGB > 0
| eval maxTotalDataSizeMB = round( maxTotalDataSizeMB / 1024)
| eval frozenTimePeriodInSecs= round( frozenTimePeriodInSecs / 60 /60/ 24)
| eval first_event=strptime(minTime,"%Y-%m-%dT%H:%M")
| eval last_event=strptime(maxTime,"%Y-%m-%dT%H:%M")
| eval TimeDiff=last_event-first_event | eval TimeDiff = round(TimeDiff/60/60/24,2)
| table splunk_server title summaryHomePath_expanded minTime maxTime currentDBSizeGB totalEventCount frozenTimePeriodInSecs maxTotalDataSizeMB TimeDiff
| eval diff=(latest-earliest)
| rename minTime AS earliest maxTime AS latest summaryHomePath_expanded AS index_path currentDBSizeGB AS Size(GB) totalEventCount AS Count frozenTimePeriodInSecs AS Days maxTotalDataSizeMB AS "Max(GB)" title AS index TimeDiff as "Duration"

List users

|rest /services/authentication/users splunk_server=local |fields title type realname email roles | nomv roles| rename title as Username type as Authentication realname as "Full Name" email as Email roles as Roles

List sourcetypes where data is older than X

| tstats latest(_time) as latest where index=* by sourcetype host index | where latest < relative_time(now(), "-4h") | convert ctime(latest)

List indexes on remote search head

| rest /services/data/indexes | search NOT (title="_*" OR title="history" OR title="summary" OR title="example" OR title="main" OR title="splunklogger") | dedup title | table title currentDBSizeMB maxTotalDataSizeMB totalEventCount minTime maxTime

# Find top hosts

| tstats count by host | eventstats sum(count) as total | eval percentage = round(count/total*100,0) | fields - total | sort - count | head 40

Splunk & Systemd

No longer required from Splunk version 7.2.2 onwards

Universal Forwarder

Copy this text into a file named "/usr/lib/systemd/system/splunkforwarder.service".

[Unit]
Description=Splunk Universal Forwarder
Wants=network.target
After=network.target

[Service]
Type=forking
RemainAfterExit=yes
ExecStart=/opt/splunkforwarder/bin/splunk start
ExecStop=/opt/splunkforwarder/bin/splunk stop
ExecReload=/opt/splunkforwarder/bin/splunk restart
StandardOutput=syslog
Restart=always

[Install]
WantedBy=multi-user.target

Next, create a symlink that systemd will use to fetch this control file.

systemctl enable splunkforwarder

That's it.  You can either reboot or run

systemctl start splunkforwarder

SearchHead, Indexer, Heavy Forwarder

Copy this text into a file named "/usr/lib/systemd/system/splunkforwarder.service".

[Unit]
Description=Splunk
Wants=network.target
After=network.target

[Service]
User=splunk
Group=splunk
Type=forking
RemainAfterExit=yes
ExecStart=/opt/splunk/bin/splunk start --answer-yes --no-prompt --accept-license
ExecStop=/opt/splunk/bin/splunk stop
ExecReload=/opt/splunk/bin/splunk restart
StandardOutput=syslog
#ExecStart=/bin/sh -c "echo never >/sys/kernel/mm/transparent_hugepage/enabled"
#ExecStart=/bin/sh -c "echo never >/sys/kernel/mm/transparent_hugepage/defrag"
#ulimit -Sn 65535
#ulimit -Hn 65535
LimitNOFILE=65535
#ulimit -Su 20480
#ulimit -Hu 20480
LimitNPROC=20480
#ulimit -Hf unlimited
#ulimit -Sf unlimited
LimitFSIZE=infinity
LimitCORE=infinity
#Restart=always

[Install]
WantedBy=multi-user.target
If you have THP compiled into your kernel (hint: it's enabled by default), then you'll need to uncomment the two ExecStart lines