from_reader 读文件很慢
主线位置:反序列化主线的最底层。格式每要一个字节就调一次
Read::next,from_reader的IoRead把这一次next变成一次io::Read::read。
flowchart TB
de["Deserializer 要一个字节"] --> nx["IoRead::next"]
nx --> by["io::Bytes 取一个字节"]
by --> q{reader 是什么}
q -->|File| sys["一次 read 系统调用"]
q -->|BufReader| mem["从 8 KiB 缓冲里拿"]
de2["from_slice"] --> sl["SliceRead 直接索引"]
IoRead 怎么实现、from_slice 为什么比 BufReader 还快,在 反序列化一侧 的「from_reader:一个字节一次 read」里讲过。这一页只讲怎么认出这个坑、怎么改。
现象
serde_json::from_reader(File::open(path)?) 看起来是最直接的写法,结果比先读进内存再解析慢一百多倍。src/bin/reader_slow.rs 写一个 20000 篇 Post 的 JSON 文件,用一个数 read 调用次数的包装器读四遍:
use std::fs::File;
use std::io::{self, BufReader, Read};
use std::time::Instant;
use serde::{Deserialize, Serialize};
#[derive(Serialize, Deserialize)]
struct Post {
id: u32,
title: String,
tags: Vec<String>,
}
// Same shape, but borrows the title from the input buffer.
#[derive(Deserialize)]
struct PostRef<'a> {
#[allow(dead_code)]
id: u32,
title: &'a str,
#[allow(dead_code)]
tags: Vec<&'a str>,
}
// Counts how many times read() is called and how many bytes it returns.
struct Counting<R> {
inner: R,
calls: usize,
bytes: usize,
}
impl<R: Read> Read for Counting<R> {
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize> {
self.calls += 1;
let n = self.inner.read(buf)?;
self.bytes += n;
Ok(n)
}
}
fn counting(path: &str) -> Counting<File> {
Counting { inner: File::open(path).unwrap(), calls: 0, bytes: 0 }
}
fn main() {
let path = std::env::args().nth(1).unwrap();
let posts: Vec<Post> = (0..20_000)
.map(|id| Post { id, title: format!("post number {id}"), tags: vec!["rust".into(), "serde".into()] })
.collect();
std::fs::write(&path, serde_json::to_vec(&posts).unwrap()).unwrap();
let t = Instant::now();
let mut r = counting(&path);
let v: Vec<Post> = serde_json::from_reader(&mut r).unwrap();
println!("from_reader(File): {:>4} ms, read calls = {:>7}, bytes = {}, posts = {}", t.elapsed().as_millis(), r.calls, r.bytes, v.len());
let t = Instant::now();
let mut r = counting(&path);
let v: Vec<Post> = serde_json::from_reader(BufReader::new(&mut r)).unwrap();
println!("from_reader(BufReader<File>): {:>4} ms, read calls = {:>7}, bytes = {}, posts = {}", t.elapsed().as_millis(), r.calls, r.bytes, v.len());
let t = Instant::now();
let mut r = counting(&path);
let mut buf = Vec::new();
r.read_to_end(&mut buf).unwrap();
let v: Vec<Post> = serde_json::from_slice(&buf).unwrap();
println!("read_to_end + from_slice: {:>4} ms, read calls = {:>7}, bytes = {}, posts = {}", t.elapsed().as_millis(), r.calls, r.bytes, v.len());
let t = Instant::now();
let v: Vec<PostRef> = serde_json::from_slice(&buf).unwrap();
println!("from_slice, borrowed &str: {:>4} ms, posts = {}, last title = {:?}", t.elapsed().as_millis(), v.len(), v[v.len() - 1].title);
}
M2 Max、macOS、rustc 1.99.0,--release,连跑两次,read 次数相同,毫秒数第一次如下(第二次 485 / 4 / 3 / 2):
from_reader(File): 483 ms, read calls = 1277782, bytes = 1277781, posts = 20000
from_reader(BufReader<File>): 4 ms, read calls = 157, bytes = 1277781, posts = 20000
read_to_end + from_slice: 3 ms, read calls = 18, bytes = 1277781, posts = 20000
from_slice, borrowed &str: 2 ms, posts = 20000, last title = "post number 19999"
文件 1277781 字节,直接读 File 时 read 被调了 1277782 次:每个字节一次,最后一次返回 0 表示文件结束。每一次都是一个系统调用,这就是那 483 ms。
从哪里来
from_reader 只是把 reader 包成 IoRead,而 IoRead::new 用的是标准库的 bytes() 迭代器:
pub fn new(reader: R) -> Self {
IoRead {
iter: LineColIterator::new(reader.bytes()),
ch: None,
// ...
}
}
io::Bytes::next 每次向底层 reader 要一个字节,IoRead::next 每次调它一次。serde_json 自己不加缓冲,from_reader 的文档写着:「When reading from a source against which short reads are not efficient, such as a [File], you will want to apply your own buffering because serde_json will not buffer the input. See [std::io::BufReader].」
issue #160「Parsing 20MB file using from_reader is slow」开于 2016-10-12,到 1.0.154 仍未关闭。文档里那句提醒是三年后才加的:PR #544(首次发布于 1.0.41)的说明是「Today I stumbled across the fact that from_reader is slow. There seems to be a suggestion from back in 2016 to add a comment to the docs ... but it seems it was never done. This PR finally adds that note as apparently people like me still stumble across the bug.」它加的就是 from_reader 文档里现在这句:「Note that counter to intuition, this function is usually slower than reading a file completely into memory and then applying [from_str] or [from_slice] on it. See [issue #160].」
修法
二选一,看文件多大、要不要借用。
| 写法 | read 次数 | 时间 | 什么时候用 |
|---|---|---|---|
from_reader(File) | 1277782 | 483 ms | 不要用 |
from_reader(BufReader::new(file)) | 157 | 4 ms | 输入很大、不想整个放进内存,或者本来就是流 |
read_to_end / fs::read + from_slice | 18 | 3 ms | 文件能放进内存;最快 |
同上,字段用 &str | — | 2 ms | 数据只在这块缓冲活着的时候用 |
BufReader默认 8 KiB,1277781 字节要 156 次加一次读到 0。serde_json 那边依然逐字节next,只是落在内存里,不再是系统调用。from_slice走SliceRead,能一次扫多个字节、能借用。最后一行的PostRef里title: &'a str直接指向buf,不分配String。字符串里有转义时借不成,见 expected a borrowed string。from_reader只收DeserializeOwned,PostRef这样的类型根本传不进去(生命周期 'de 与借用 的「DeserializeOwned 与 from_reader」)。要借用就只能先读进内存。
已经包过 BufReader 的 reader(比如 TcpStream 外面的 BufReader,或者 hyper / axum 给的已经在内存里的 body)不需要再包。axum 的 Json 提取器本来就是先收齐 body,再用 Deserializer::from_slice 解析(body 提取器)。
怎么确认没中招
- 搜代码里的
from_reader(,看传进去的是不是裸的File、TcpStream、ChildStdout这类每次read都是系统调用的东西。&mut file、&file也一样不缓冲。 - 拿不准时,用上面的
Counting包一下传进去:calls接近bytes,就是没有缓冲;接近bytes / 8192,就是BufReader在起作用。
小结
- serde_json 不缓冲输入:
IoRead每个字节调一次底层read,给File就是每字节一次系统调用。 - 包
BufReader:read从 1277782 次降到 157 次,483 ms 到 4 ms。 - 文件能放进内存就
read_to_end/fs::read加from_slice:3 ms,而且字段可以借用输入。 - 检查办法:搜
from_reader(,或者用计数包装器看read次数是不是接近字节数。