SumoLogic Concepts: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(71 intermediate revisions by the same user not shown)
Line 7: Line 7:
The search syntax is based on the "funnel" or the "pipeline" concept. The pipeline input receives all SumoLogic data, and data is filtered b entering [[#Keyword|keywords]] and [[#Operator|operators]], separated by pipes ("|"). Each operator acts on the results produced by previous operators, so data is being progressively filtered out. The typical search query syntax is similar to:
The search syntax is based on the "funnel" or the "pipeline" concept. The pipeline input receives all SumoLogic data, and data is filtered b entering [[#Keyword|keywords]] and [[#Operator|operators]], separated by pipes ("|"). Each operator acts on the results produced by previous operators, so data is being progressively filtered out. The typical search query syntax is similar to:


  [[#Keyword_Search|keyword search]] ''or'' [[#String_Search|string search]] | parse | where | group-by | sort | limit
  [[#Keyword_Search|keyword search]] ''or'' [[#String_Search|string search]] | [[#parse|parse]] | where | group-by | [[#sort|sort]] | limit


All queries start with a [[#Keyword_Search|keyword search]] or a [[#String_Search|string search]].  
All queries start with a [[#Keyword_Search|keyword search]] or a [[#String_Search|string search]].
 
As queries get longer and more complex, it is a best practice to format your queries by using a soft return before the pipes, such as:
 
_sourcecategory=apache
| parse "* --" as src_ip
| count by src_ip
| sort _count
 
Searches are not instantaneous, and some of them can take a long time. Thus, they can be [https://help.sumologic.com/05Search/Get-Started-with-Search/Search-Basics/Pause_or_Cancel_a_Search paused or canceled].
 
Searches can be [https://help.sumologic.com/05Search/Get-Started-with-Search/Search-Basics/Save-a-Search saved] in the [[#Library|Library]]. Once a search is saved, it can be further [https://help.sumologic.com/Dashboards-and-Alerts/Alerts/02-Schedule-a-Search scheduled]. To save a search, compose the search in an "Unnamed" tab, then "Save As". By default, the search is saved in the [[#Personal_Folder|Personal folder]].


==Keyword Search==
==Keyword Search==
  _sourceCategory=CloudWatch


==String Search==
==String Search==
==Search Comments==
// comments on a single line
/*
  multi-line comments
*/
==Export Search Results==
{{External|[https://help.sumologic.com/05Search/Get-Started-with-Search/Search-Basics/Export-Search-Results Export Search Results]}}
Up to 100,000 rows of CSV can be downloaded from the browser.
==Search Job API==
{{External|[https://help.sumologic.com/APIs/Search-Job-API/About-the-Search-Job-API Search Job API]}}
==Scheduled Search==
{{External|[https://help.sumologic.com/Dashboards-and-Alerts/Alerts/02-Schedule-a-Search Scheduled Search]}}


=Keyword=
=Keyword=
Line 25: Line 59:


</font>
</font>
Keyword expressions. Keyword expressions include [[#Metadata_Fields|metadata field]] expressions.
=Field=
The search language allows alphanumeric characters, hyphens and underscores for valid [[#Field|field]] names. If the field names contain special characters, a [https://help.sumologic.com/05Search/Get-Started-with-Search/Search-Basics/Reference_a_Field_with_Special_Characters special syntax] is required.


=Metadata=
=Metadata=
Line 30: Line 69:
{{External|[https://help.sumologic.com/05Search/Get-Started-with-Search/Search-Basics/Search-Metadata Search Metadata]}}
{{External|[https://help.sumologic.com/05Search/Get-Started-with-Search/Search-Basics/Search-Metadata Search Metadata]}}


Metadata fields:
Sumo Logic attaches [[#Metadata_Fields|metadata fields (or tags)]] to the log messages when the data is collected; the tags provide information about [[#Collector|Collectors]] and [[#Source|Sources]] data was ingested from, the type of logs, etc. Usually, filtering by metadata tags is the first step in building a [[#Keyword_Search|keyword search]] query. An overview of available metadata tags is presented below:
 
==<span id='Metadata_Fields'></span><span id='Metadata_Tags'></span>Metadata Fields (or Tags)==


====_collector====
====_collector====


The name of the [[#Collector|Collector]], as set when the Collector was installed, that received the log message.
The name of the [[#A_Collector|Collector]], as set when the Collector was installed, that received the log message.


====_source====
====_source====
Line 47: Line 88:


The category of the [[#Source|Source]] that collected the message. The source category can be a maximum of 1,024 characters.
The category of the [[#Source|Source]] that collected the message. The source category can be a maximum of 1,024 characters.
_sourceCategory=CloudWatch


====_sourceHost====
====_sourceHost====


The host name of the [[#Source|Source]]. For local Sources the name of the Source as set when the Source is configured. For remote [[#Collector|Collectors]], this field uses the remote host's name. The _sourceHost metadata field is populated using a reverse DNS lookup. If the name cannot be resolved, _sourceHost is displayed as "localhost". This can be a maximum of 128 characters.
The host name of the [[#Source|Source]]. For local Sources the name of the Source as set when the Source is configured. For remote [[#Collector|Collectors]], this field uses the remote host's name. The _sourceHost metadata field is populated using a reverse DNS lookup. If the name cannot be resolved, _sourceHost is displayed as "localhost". This can be a maximum of 128 characters.
_sourceHost=/up/test/up-plat-svc-blah


====_messageCount====
====_messageCount====
Line 62: Line 107:
====_receiptTime====
====_receiptTime====


The time the [[#Collector|Collector]] received the message in milliseconds.
The time the [[#Collector|Collector]] received the message in milliseconds. Also see [https://help.sumologic.com/05Search/Get-Started-with-Search/How-to-Build-a-Search/Use-Receipt-Time User Receipt Time].


==== _size====
==== _size====
Line 77: Line 122:


=Operator=
=Operator=
==Individual Operators==
===parse===
<font color=darkgray>Strings can be parsed based on start and stop anchor points in messages, and then aliased as user-created fields:</font>
... | parse "* --" as src_ip | ...
The above parses out the IP address into a [[#Field|field]] named "src_ip", using an <font color=darkgray>endpoint anchor.</font>
===sort===
{{External|[https://help.sumologic.com/05Search/Search-Query-Language/Search-Operators/sort sort]}}
Sorting by the timestamp in natural order:
... | sort by +_messageTime
Also see [[#Message_Sorting|Message Sorting]] below.
===count===
count by
===keyvalue===
... | keyvalue regex " ([A-Z_-]+?)='([^']+?)'" keys "TYPE", "MESSAGES", "CHANNEL", "DOCUMENT-URI" | count by %"docment-uri"
===outlier===
Tracks the moving average of the standard deviation of a value.
==Group Operators==
{{External|[https://help.sumologic.com/05Search/Search-Query-Language/aaGroup Group Operators]}}
===group by===
<font color=darkgray>example, did not work.</font>


=Pipe=
=Pipe=
Line 82: Line 167:
=Wildcards=
=Wildcards=


* means zero or more characters.
'*' means zero or more characters.


? means a single character.
? means a single character.


=Collector=
=<span id='A_Collector'></span>Collector=
 
To access a list of all available Collectors: Sumo Logic -> Hamburger -> Manage Data -> Collection.


=Source=
=Source=


{{External|[https://help.sumologic.com/03Send-Data/Sources Sources]}}
{{External|[https://help.sumologic.com/03Send-Data/Sources Sources]}}
A Source is an environment that produces data. Each Source is configured to collect files in a specific way, depending on the type of [[#A_Collector|Collector]] is using.
=<span id='View'></span>Views=
==Message Table View==
===Message Sorting===
By default, message sorting is configured to show the newest messages first. That can be changed as follows: the "Messages" tab -> top-right gear icon -> Display Message Preferences -> Sort By: "Oldest Message First".
==Aggregates View==
The Aggregates view seems to become available if a [[#Group_Operators|group operator]] is used. The view (tab) should be a peer of "Messages" tab. <font color=darkgray>TO TEST. TO PROCESS https://help.sumologic.com/05Search/Get-Started-with-Search/Search-Basics/Chart-Search-Results</font>
=Metrics vs. Logs=
==Metrics==
{{External|[https://help.sumologic.com/Metrics/Metric-Queries-and-Alerts/Metrics_Monitors_and_Alerts Metric Monitors and Alerts]}}
==Logs==
=<span id='Chart'></span>Charts=
{{External|[https://help.sumologic.com/Dashboards-and-Alerts/Dashboards/Chart-Panel-Types Charts]}}
=<span id='Alert'></span>Alerts=
{{External|[https://help.sumologic.com/Dashboards-and-Alerts/Alerts Alerts]}}
==Log Alerts==
A log alert is a notification or action that is triggered when a pre-defined condition is detected in the log data: a particular string appears in the log, or a certain type of error passes a certain threshold. When an alert is triggered, Sumo can: send an e-mail, run a script, post on slack. To create an alert, the sequence is:
* Build a log query
* Save it as a [[#Scheduled_Search|Scheduled Search]]. The "Scheduled Search" detects the problem.
* Schedule: how frequently.
==Metrics Alerts==
=<span id='Library'></span>The Library=
{{External|[https://help.sumologic.com/05Search/Library The Library]}}
==Personal Folder==
The "Personal" folder is accessible as Sumo Logic -> Hamburger -> The "Personal" folder icon or Sumo Logic -> Hamburger -> Library -> User's Name -> Personal
=Large Messages=
{{External|[https://help.sumologic.com/05Search/Get-Started-with-Search/Search-Basics/Search-Large-Messages Search Large Messages]}}
=To Process=
* Statistically significant alerts using outliers youtu.be/uBsyCyQ6v3Q

Latest revision as of 23:48, 30 January 2019

Internal

Search

The search syntax is based on the "funnel" or the "pipeline" concept. The pipeline input receives all SumoLogic data, and data is filtered b entering keywords and operators, separated by pipes ("|"). Each operator acts on the results produced by previous operators, so data is being progressively filtered out. The typical search query syntax is similar to:

keyword search or string search | parse | where | group-by | sort | limit

All queries start with a keyword search or a string search.

As queries get longer and more complex, it is a best practice to format your queries by using a soft return before the pipes, such as:

_sourcecategory=apache
| parse "* --" as src_ip
| count by src_ip
| sort _count

Searches are not instantaneous, and some of them can take a long time. Thus, they can be paused or canceled.

Searches can be saved in the Library. Once a search is saved, it can be further scheduled. To save a search, compose the search in an "Unnamed" tab, then "Save As". By default, the search is saved in the Personal folder.

Keyword Search

 _sourceCategory=CloudWatch

String Search

Search Comments

// comments on a single line
/*
  multi-line comments
*/

Export Search Results

Export Search Results

Up to 100,000 rows of CSV can be downloaded from the browser.

Search Job API

Search Job API

Scheduled Search

Scheduled Search

Keyword

Keywords are case insensitive.

How to figure out the complete list of valid keywords.

Most used keywords:

  • _sourceCategory

Keyword expressions. Keyword expressions include metadata field expressions.

Field

The search language allows alphanumeric characters, hyphens and underscores for valid field names. If the field names contain special characters, a special syntax is required.

Metadata

Search Metadata

Sumo Logic attaches metadata fields (or tags) to the log messages when the data is collected; the tags provide information about Collectors and Sources data was ingested from, the type of logs, etc. Usually, filtering by metadata tags is the first step in building a keyword search query. An overview of available metadata tags is presented below:

Metadata Fields (or Tags)

_collector

The name of the Collector, as set when the Collector was installed, that received the log message.

_source

The name of the Source, as set when the Source was configured.

_sourceName

The name of the log file, as the path that was uses when the Source was configured.

_sourceCategory

The category of the Source that collected the message. The source category can be a maximum of 1,024 characters.

_sourceCategory=CloudWatch 

_sourceHost

The host name of the Source. For local Sources the name of the Source as set when the Source is configured. For remote Collectors, this field uses the remote host's name. The _sourceHost metadata field is populated using a reverse DNS lookup. If the name cannot be resolved, _sourceHost is displayed as "localhost". This can be a maximum of 128 characters.

_sourceHost=/up/test/up-plat-svc-blah

_messageCount

A sequence number, maintained by Source, added by the Collector when the message was received.

_messageTime

The timestamp of the message in milliseconds. If the message doesn't have a timestamp, _messageTime uses the _receiptTime.

_receiptTime

The time the Collector received the message in milliseconds. Also see User Receipt Time.

_size

The size of the log message in bytes.

_format

The pattern used for parsing the timestamp. See here for more details.

_raw

The raw log message.

Operator

Individual Operators

parse

Strings can be parsed based on start and stop anchor points in messages, and then aliased as user-created fields:

... | parse "* --" as src_ip | ...

The above parses out the IP address into a field named "src_ip", using an endpoint anchor.

sort

sort

Sorting by the timestamp in natural order:

... | sort by +_messageTime

Also see Message Sorting below.

count

count by

keyvalue

... | keyvalue regex " ([A-Z_-]+?)='([^']+?)'" keys "TYPE", "MESSAGES", "CHANNEL", "DOCUMENT-URI" | count by %"docment-uri"

outlier

Tracks the moving average of the standard deviation of a value.

Group Operators

Group Operators

group by

example, did not work.

Pipe

Wildcards

'*' means zero or more characters.

? means a single character.

Collector

To access a list of all available Collectors: Sumo Logic -> Hamburger -> Manage Data -> Collection.

Source

Sources

A Source is an environment that produces data. Each Source is configured to collect files in a specific way, depending on the type of Collector is using.

Views

Message Table View

Message Sorting

By default, message sorting is configured to show the newest messages first. That can be changed as follows: the "Messages" tab -> top-right gear icon -> Display Message Preferences -> Sort By: "Oldest Message First".

Aggregates View

The Aggregates view seems to become available if a group operator is used. The view (tab) should be a peer of "Messages" tab. TO TEST. TO PROCESS https://help.sumologic.com/05Search/Get-Started-with-Search/Search-Basics/Chart-Search-Results

Metrics vs. Logs

Metrics

Metric Monitors and Alerts

Logs

Charts

Charts

Alerts

Alerts

Log Alerts

A log alert is a notification or action that is triggered when a pre-defined condition is detected in the log data: a particular string appears in the log, or a certain type of error passes a certain threshold. When an alert is triggered, Sumo can: send an e-mail, run a script, post on slack. To create an alert, the sequence is:

  • Build a log query
  • Save it as a Scheduled Search. The "Scheduled Search" detects the problem.
  • Schedule: how frequently.

Metrics Alerts

The Library

The Library

Personal Folder

The "Personal" folder is accessible as Sumo Logic -> Hamburger -> The "Personal" folder icon or Sumo Logic -> Hamburger -> Library -> User's Name -> Personal

Large Messages

Search Large Messages

To Process

  • Statistically significant alerts using outliers youtu.be/uBsyCyQ6v3Q