JSON Lines (JSONL): The Format Built for Logs, Streams, and Big Data

By Deepak Kumar Developer & builder of JSON Formatter Hub · About the author

Published June 22, 2025 · 8 min read · JSON Formatter Hub

A standard JSON array requires the whole document to be valid before you can safely read any of it — one malformed record at the end, or a process that dies mid-write, and the entire file becomes unparseable. JSON Lines (also written JSONL, or newline-delimited JSON / NDJSON) fixes this by dropping the wrapping array entirely: each line is its own complete, independently parseable JSON value.

The Format

A JSONL file is plain text where every line is a self-contained JSON value, almost always an object, separated by a single newline character (\n). There is no wrapping [ ], and no commas between records:

{"timestamp": "2026-07-12T09:00:00Z", "level": "info", "message": "Server started"}
{"timestamp": "2026-07-12T09:00:03Z", "level": "warn", "message": "High memory usage", "pct": 87}
{"timestamp": "2026-07-12T09:00:07Z", "level": "error", "message": "Connection timeout", "host": "db-2"}

Each line above is independently valid JSON. That single property — line-level independence — is what standard JSON arrays lack and what makes JSONL suited to a different class of problem.

Why Wrap-Array JSON Breaks Down at Scale

Compare the same data as a standard JSON array:

[
  {"timestamp": "2026-07-12T09:00:00Z", "level": "info", "message": "Server started"},
  {"timestamp": "2026-07-12T09:00:03Z", "level": "warn", "message": "High memory usage", "pct": 87},
  {"timestamp": "2026-07-12T09:00:07Z", "level": "error", "message": "Connection timeout", "host": "db-2"}
]

Three practical problems emerge once this file grows to gigabytes or is written incrementally:

• You can't parse it incrementally without a streaming parser. A naive JSON.parse() needs the entire file in memory and requires the trailing ] to exist — a process that crashes mid-write leaves an unparseable file, because the array was never closed.
• You can't append safely. Adding a record means seeking back past the closing ], inserting a comma, writing the new object, and re-closing the array — not a simple append. With JSONL, appending a new record is just writing one more line to the end of the file.
• You can't process it line-by-line with standard Unix tools. grep, wc -l, tail -f, and `split` all assume line-oriented data. A pretty-printed JSON array spans arbitrary numbers of lines per record with no reliable boundary; JSONL's boundary is always exactly one line.

JSONL solves all three by making every record independently addressable and appendable.

Where JSONL Is the Right Tool

• Log files — each log entry is appended as it happens; a crash mid-write corrupts at most the last partial line, not the entire file
• Streaming APIs — OpenAI's and Anthropic's streaming chat completion APIs, and many webhook delivery systems, use newline-delimited JSON so the client can process each chunk as it arrives without waiting for the full response
• ML training data and datasets — Hugging Face datasets, OpenAI fine-tuning files, and most large-scale ML pipelines use JSONL as the canonical training-data format, since a single 50GB dataset never needs to fully fit in memory to be processed
• Data pipeline handoffs — tools like jq, Spark, and BigQuery load JSONL natively because each line maps cleanly to one row

Reading and Writing JSONL in Python

No special library is required for basic use — plain line iteration plus json.loads/json.dumps is often enough:

import json

# Reading — stream line by line, never load the whole file into memory
def read_jsonl(path):
    with open(path, 'r') as f:
        for line in f:
            line = line.strip()
            if line:
                yield json.loads(line)

for record in read_jsonl('logs.jsonl'):
    print(record['level'], record['message'])

# Writing — append one record at a time
def append_jsonl(path, record):
    with open(path, 'a') as f:
        f.write(json.dumps(record) + '\n')

append_jsonl('logs.jsonl', {"timestamp": "2026-07-12T09:00:10Z", "level": "info", "message": "Request handled"})

For larger workloads, the jsonlines package wraps this pattern with a slightly cleaner API and built-in error handling for malformed lines:

pip install jsonlines

import jsonlines

with jsonlines.open('logs.jsonl') as reader:
    for obj in reader:
        print(obj)

with jsonlines.open('logs.jsonl', mode='a') as writer:
    writer.write({"timestamp": "2026-07-12T09:00:12Z", "level": "info", "message": "Done"})

Reading and Writing JSONL in Node.js

For small-to-medium files, splitting on newlines is sufficient:

const fs = require('fs');

// Reading
const lines = fs.readFileSync('logs.jsonl', 'utf8').split('\n').filter(Boolean);
const records = lines.map(line => JSON.parse(line));

// Writing / appending
function appendJsonl(path, record) {
  fs.appendFileSync(path, JSON.stringify(record) + '\n');
}
appendJsonl('logs.jsonl', { timestamp: new Date().toISOString(), level: 'info', message: 'Request handled' });

For large files where you cannot hold the whole thing in memory, stream it line-by-line with Node's readline module:

const fs = require('fs');
const readline = require('readline');

async function processLargeJsonl(path) {
  const rl = readline.createInterface({
    input: fs.createReadStream(path),
    crlfDelay: Infinity,
  });

  for await (const line of rl) {
    if (!line.trim()) continue;
    const record = JSON.parse(line);
    // process record here, one at a time — constant memory usage
  }
}

Command-Line Tools That Understand JSONL

jq processes JSONL natively — pass one record through the filter for each line:

# Filter to only error-level log lines
cat logs.jsonl | jq 'select(.level == "error")'

# Extract just the message field from every line
cat logs.jsonl | jq -r '.message'

# Convert a JSON array file into JSONL
jq -c '.[]' data.json > data.jsonl

# Convert JSONL back into a JSON array
jq -s '.' data.jsonl > data.json

Standard Unix tools also work directly because the format is line-oriented: wc -l logs.jsonl counts records, tail -f logs.jsonl tails a live-growing log, and split -l 100000 logs.jsonl chunk_ splits a huge file into fixed-size chunks without ever parsing a single record.

JSONL vs a JSON Array: When to Use Each

Use a standard JSON array when the data is a single cohesive document that's always read and written as a whole — an API response, a config file, a small dataset that comfortably fits in memory. Use JSONL when records are generated incrementally over time, need to be processed one at a time without loading everything into memory, or need to survive partial writes gracefully — logs, event streams, and large training datasets are the classic cases. If you're unsure, ask whether the file is ever appended to after creation; if yes, JSONL is almost always the better fit.

Summary

JSON Lines trades the single-document guarantees of a JSON array for line-level independence: constant-memory streaming reads, safe appends, and compatibility with line-oriented Unix tooling. It's not a replacement for JSON arrays in general — it's the right specialized format specifically for logs, streams, and large sequential datasets where records arrive or need to be processed one at a time.