Application logs are the source of truth for production behavior. The five levels (DEBUG, INFO, WARN, ERROR, FATAL), the three patterns (structured, unstructured, sampled), and the one rule: log with purpose, not volume. The team that logs everything is the team that finds nothing in the noise.
Table of contents
- The five levels
- The three patterns
- The structured pattern
- The unstructured pattern
- The sampled pattern
- The one rule
- FAQ
The five levels
The five log levels, in order of severity:
- DEBUG. Detailed information for debugging. The team that sets the log level to DEBUG in production is going to drown in the noise.
- INFO. General information about the application’s behavior. The default for production.
- WARN. Something unexpected, but the application can recover. The team that has a lot of WARNs is the team that has a lot of potential incidents.
- ERROR. Something failed. The team should investigate.
- FATAL. The application is unable to continue. The team should page.
The three patterns
The three patterns for application logs:
- Structured. The log message is a structured object (JSON, key-value). The team can query, filter, and aggregate the logs.
- Unstructured. The log message is a free-form text. The team can grep, but cannot easily query.
- Sampled. The team logs a sample of the events (e.g., 1% of the requests). The right answer for the workload that has too much volume to log everything.
The structured pattern
The structured pattern:
- The log message is a JSON object.
- The fields are well-defined: timestamp, level, message, request_id, user_id, and the application-specific fields.
- The team can query the logs with the log provider’s query language (Lucene, Splunk SPL, etc.).
- The team can aggregate the logs: count by level, count by request_id, count by user_id.
The right answer for most production workloads. The cost: the team has to define the schema and the team has to use a logger that supports structured logging.
The unstructured pattern
The unstructured pattern:
- The log message is a free-form text.
- The team can grep the logs for keywords, but cannot easily query or aggregate.
- The team that has a lot of unstructured logs is the team that has a hard time debugging.
The right answer for the team that is starting out or the team that has a small workload. The cost: the team cannot easily scale the log analysis.
The sampled pattern
The sampled pattern:
- The team logs a sample of the events (e.g., 1% of the requests).
- The right answer for the workload that has too much volume to log everything.
- The trade-off: the team does not have the full picture; the team has to be careful about what is sampled and what is not.
The team that samples 1% of the requests still has 99% of the data lost. The team that samples 1% of the errors still has the errors. The team that samples 1% of the successes is going to miss the silent failures.
The one rule
The one rule: log with purpose, not volume.
The team that logs everything is the team that finds nothing in the noise. The team that logs with purpose is the team that finds the bug, the failure, the performance issue, the security incident.
The purpose: the team should be able to answer, in under 10 seconds, the question “what is happening in production right now?” The team that can answer that question is the team that has the right logs.
FAQ
What are the five log levels?
DEBUG (detailed information for debugging), INFO (general information), WARN (something unexpected but recoverable), ERROR (something failed), FATAL (the application is unable to continue).
What is structured logging?
The log message is a structured object (JSON, key-value). The team can query, filter, and aggregate. The right answer for most production workloads.
What is sampled logging?
The team logs a sample of the events (e.g., 1% of the requests). The right answer for the workload that has too much volume to log everything.
What is the right log level for production?
INFO for the default. DEBUG for the team that is debugging a specific issue. WARN, ERROR, FATAL for the events that the team is alerting on.
What should I include in every log message?
Timestamp, level, message, request_id (if applicable), user_id (if applicable), and the application-specific fields. The team should be able to answer, in under 10 seconds, the question “what is happening in production right now?”
How do I reduce log volume?
Sample the logs, use a higher log level, or filter the logs at the source. The team that logs everything is the team that finds nothing in the noise.
If you are sizing the observability tier for a new project, the RunxBuild hosting calculator is the place to model the line items. The logs, the metrics, the traces - each one is a separate number, and the team’s mental model for the platform is the sum of those numbers. The RunxBuild dashboard is where the team sees the actual application logs in one place.