Rust in Everything
基于 sea-orm 2.0.4

Loader 报 too many arguments for query: 65536

主线位置:查询主线上 Loader 的第二条语句。posts.load_one(user::Entity, &db) 把父记录的外键去重,拼成一条 WHERE ... IN (...),每个键一个 $N,最后照常走 sqlx::query_with。机制见 Loader 与 N+1 的「很大的输入:不分批」;参数上限本身是 sqlx 和 Postgres 协议的事,见 sqlx 卷的 too many arguments for query: 65536。

flowchart TB
  parents["n 个父记录"] --> keys["取外键,去重"]
  keys --> cond["一条 IN,每键一个参数"]
  cond --> check{"参数数超过 65535"}
  check -->|否| ok["发出,按键装回"]
  check -->|是| err["sqlx 拒发 Protocol 错误"]

本页实验在 M2 Max、macOS、rustc 1.99.0、Postgres 16.14 上运行,sea-orm 2.0.4(sqlx 0.9.0),以及本站用的 1.1.20(sqlx 0.8.6),同一份代码。SQL 取自 sqlx 的语句日志(sqlx=info)。

现象

一个后台任务把一张大表的记录读出来,再 load_one 它们的关联记录,数据量小时正常,某天开始报:

Query Error: encountered unexpected or invalid data: PgConnection::run(): too many arguments for query: 65536 (sqlx_postgres::connection::executor:222)

这是 DbErr 的 Display;Debug 是 Query(SqlxError(Protocol("PgConnection::run(): too many arguments for query: 65536 (sqlx_postgres::connection::executor:222)")))。

最小复现

在内存里造 n 篇文章,每篇指向一个不同的作者(库里只有 3 个作者,不影响结果),再 load_one(author)。src/bin/loader_big.rs:

// n posts in memory, each pointing at a different author (only 3 exist)
fn posts(n: i32, distinct: i32) -> Vec<post::Model> {
  (1..=n).map(|i| post::Model { id: i, title: String::new(), likes: 0, author_id: (i - 1) % distinct + 1 }).collect()
}
// ...
  for n in [65_535, 65_536] {
    match posts(n, n).load_one(author::Entity, &db).await {
      Ok(v) => println!("n={n}: Ok, {} slots, {} found", v.len(), v.iter().flatten().count()),
      Err(e) => println!("n={n}: Err({e:?})\n  Display: {e}"),
    }
  }
  let v = posts(70_000, 3).load_one(author::Entity, &db).await?;

2.0.4 的输出(db.statement 有六万多个占位符,从 $17 往后截掉,标为 ...):

summary="SELECT \"author\".\"id\", \"author\".\"name\" FROM …" db.statement="\n\nSELECT \"author\".\"id\", \"author\".\"name\" FROM \"author\" WHERE (\"author\".\"id\") IN (($1), ($2), ($3), ($4), ($5), ($6), ($7), ($8), ($9), ($10), ($11), ($12), ($13), ($14), ($15), ($16), ($17 ... rows_affected=3 rows_returned=3 elapsed=63.560833ms elapsed_secs=0.063560833
n=65535: Ok, 65535 slots, 3 found
summary="SELECT \"author\".\"id\", \"author\".\"name\" FROM …" db.statement="\n\nSELECT \"author\".\"id\", \"author\".\"name\" FROM \"author\" WHERE (\"author\".\"id\") IN (($1), ($2), ($3), ($4), ($5), ($6), ($7), ($8), ($9), ($10), ($11), ($12), ($13), ($14), ($15), ($16), ($17 ... rows_affected=0 rows_returned=0 elapsed=143.792µs elapsed_secs=0.000143792
n=65536: Err(Query(SqlxError(Protocol("PgConnection::run(): too many arguments for query: 65536 (sqlx_postgres::connection::executor:222)"))))
  Display: Query Error: encountered unexpected or invalid data: PgConnection::run(): too many arguments for query: 65536 (sqlx_postgres::connection::executor:222)
summary="SELECT \"author\".\"id\", \"author\".\"name\" FROM …" db.statement="\n\nSELECT \"author\".\"id\", \"author\".\"name\" FROM \"author\" WHERE (\"author\".\"id\") IN (($1), ($2), ($3)) ORDER BY \"author\".\"id\" ASC\n" rows_affected=3 rows_returned=3 elapsed=1.494583ms elapsed_secs=0.001494583
n=70000, 3 distinct keys: Ok, 70000 slots, 70000 found

65535 个键正好能发,65536 个失败;失败的那一行日志是 0.14 ms、rows_returned=0,语句没有到服务端。七万篇文章只有 3 个不同作者时一切正常:上限按去重后的键数算。

1.1.20 上结果相同,只是条件写成 "author"."id" IN ($1, $2, ...)、不加 ORDER BY,报错的位置是 sqlx 0.8.6 的 executor:220:

n=65536: Err(Query(SqlxError(Protocol("PgConnection::run(): too many arguments for query: 65536 (sqlx_postgres::connection::executor:220)"))))
  Display: Query Error: encountered unexpected or invalid data: PgConnection::run(): too many arguments for query: 65536 (sqlx_postgres::connection::executor:220)

出处

loader_impl_impl 把所有父记录的键收进一个 Vec,交给 prepare_condition 拼成一个条件,再执行一次。Postgres 上走 prepare_condition_with_save_as,它先 keys.iter().unique() 去重,然后每个键变成一个元组 (($n)),复合键占 arity 个参数。中间没有任何按大小切分的逻辑。load_one、load_many 都经过这个函数,load_many_to_many 自己拼语句,但条件用的也是这个 prepare_condition;2.0 的 Entity Loader 里一对多的部分也交给 Loader(见 Loader 与 N+1)。

拒绝发生在 sqlx:PgConnection::run 在写任何消息之前做 u16::try_from(arguments.len()),Bind 消息里参数个数是 16 位。sea-orm 把这个 sqlx::Error::Protocol 原样包成 DbErr::Query,不重试、不降级。

在 SeaQL/sea-orm 里搜 too many arguments、loader too many arguments、load_many too many、loader parameters limit、IN clause limit,没有找到相关 issue。

修法

把父记录切片,每片分别加载,再按顺序拼起来。load_one 的返回值和输入一一对应(第 i 个父记录对应第 i 个 Option),切片拼接不会错位。同一个程序的后半段:

let parents = posts(70_000, 70_000);
let mut authors = Vec::with_capacity(parents.len());
for chunk in parents.chunks(20_000) {
  authors.extend(chunk.load_one(author::Entity, &db).await?);
}
println!("chunks: Ok, {} slots, {} found, slot of post 3 = {:?}", authors.len(), authors.iter().flatten().count(), authors[2]);
--- fix: 70000 distinct keys in chunks of 20000
summary="SELECT \"author\".\"id\", \"author\".\"name\" FROM …" db.statement="\n\nSELECT \"author\".\"id\", \"author\".\"name\" FROM \"author\" WHERE (\"author\".\"id\") IN (($1), ($2), ($3), ($4), ($5), ($6), ($7), ($8), ($9), ($10), ($11), ($12), ($13), ($14), ($15), ($16), ($17 ... rows_affected=3 rows_returned=3 elapsed=28.874ms elapsed_secs=0.028874
summary="SELECT \"author\".\"id\", \"author\".\"name\" FROM …" db.statement="\n\nSELECT \"author\".\"id\", \"author\".\"name\" FROM \"author\" WHERE (\"author\".\"id\") IN (($1), ($2), ($3), ($4), ($5), ($6), ($7), ($8), ($9), ($10), ($11), ($12), ($13), ($14), ($15), ($16), ($17 ... rows_affected=0 rows_returned=0 elapsed=12.348917ms elapsed_secs=0.012348917
summary="SELECT \"author\".\"id\", \"author\".\"name\" FROM …" db.statement="\n\nSELECT \"author\".\"id\", \"author\".\"name\" FROM \"author\" WHERE (\"author\".\"id\") IN (($1), ($2), ($3), ($4), ($5), ($6), ($7), ($8), ($9), ($10), ($11), ($12), ($13), ($14), ($15), ($16), ($17 ... rows_affected=0 rows_returned=0 elapsed=12.789792ms elapsed_secs=0.012789792
summary="SELECT \"author\".\"id\", \"author\".\"name\" FROM …" db.statement="\n\nSELECT \"author\".\"id\", \"author\".\"name\" FROM \"author\" WHERE (\"author\".\"id\") IN (($1), ($2), ($3), ($4), ($5), ($6), ($7), ($8), ($9), ($10), ($11), ($12), ($13), ($14), ($15), ($16), ($17 ... rows_affected=0 rows_returned=0 elapsed=14.20625ms elapsed_secs=0.01420625
chunks: Ok, 70000 slots, 3 found, slot of post 3 = Some(Model { id: 3, name: "cid" })

四条语句(20000 + 20000 + 20000 + 10000),第 3 篇文章拿到的仍是 3 号作者。1.1.20 上同样是四条、同样的结果。片的大小按「片内去重后的键数 × 主键列数 ≤ 65535」算。

不想多发语句时,可以绕过 Loader,把所有键绑成一个数组参数,写 = ANY($1),参数永远只有一个,做法见 sqlx 卷的 too many arguments for query: 65536;在 sea-orm 里要开 postgres-array feature,结果也要自己按键分组。

怎么确认没中招

  • 搜代码里的 load_one / load_many / load_many_to_many(2.0 还有 load().with(..)),看传进去的父记录集合有没有上界:来自分页查询的(每页几十条)没问题;来自「全表」「全部待处理」的要切片。本站目前不用 LoaderTrait。
  • 关联键很少重复时(每个父记录指向不同的子记录),父记录数就是参数数;外键高度重复时参数数远小于父记录数,测试数据要按真实分布造,否则测不出来。

小结

  • Loader 把去重后的键全放进一条 IN,不分批;超过 65535 个参数时 sqlx 在发送前报 Protocol("... too many arguments for query: N ..."),sea-orm 原样返回。
  • 上限按去重后的键数乘主键列数算;2.0.4 和 1.1.20 行为相同,只是 IN 的写法不同。
  • 修法:父记录 chunks(n) 后逐片 load_*,结果按顺序拼接;或者自己用数组参数 = ANY($1) 查。

💬 关联讨论

发起讨论
加载中...