fix(formatter_markdown): preserve line breaks around Chinese/Japanese characters in all proseWrap (#27331)

Align with Prettier 3.10 (not released) and also dprint and biome.
This commit is contained in:
leaysgur committed 2026-10-05 05:13:53 +00:00
1 parent e2c68b17fa
commit 7d0e54dc20
16 files changed
+514 -486

No files matched your search

@@ -245,7 +245,7 @@
- [externals/prettier/markdown/thematicBreak/simple.md](diffs/markdown/externals__prettier__markdown__thematicBreak__simple.md.md)
- crates/oxc_formatter_markdown/DIVERGENCES.md#leading-thematic-break
### Option 2: 270/282 (95.74%)
### Option 2: 269/282 (95.39%)
```json
{"printWidth":100,"proseWrap":"always"}
@@ -273,6 +273,8 @@
- crates/oxc_formatter/DIVERGENCES.md#suppressed-terminator-per-semi
- [externals/prettier/markdown/paragraph/cjk.md](diffs/markdown/externals__prettier__markdown__paragraph__cjk.md.md)
- crates/oxc_formatter_markdown/DIVERGENCES.md#container-directive
- [externals/prettier/markdown/splitCjkText/symbolSpaceNewLine.md](diffs/markdown/externals__prettier__markdown__splitCjkText__symbolSpaceNewLine.md.md)
- crates/oxc_formatter_markdown/DIVERGENCES.md#cj-line-break
- [externals/prettier/markdown/thematicBreak/simple.md](diffs/markdown/externals__prettier__markdown__thematicBreak__simple.md.md)
- crates/oxc_formatter_markdown/DIVERGENCES.md#leading-thematic-break
@@ -0,0 +1,179 @@
# externals/prettier/markdown/splitCjkText/symbolSpaceNewLine.md
## Option 2
`````json
{"printWidth":100,"proseWrap":"always"}
`````
### Diff
`````diff
===================================================================
--- prettier
+++ oxfmt
@@ -1,14 +1,65 @@
-日本語、にほんご。汉语, 中文. 日本語,にほんご.English
-words!? 漢字!汉字?「セリフ」(括弧) 文字(括弧)文字【括弧】日本語English
+日本語
+、
+にほんご
+。
+汉语,
+中文.
+日
+本
+語
+,
+に
+ほ
+ん
+ご
+.
+English words!?
+漢字
+!
+汉字
+?
+「セリフ」
+(括弧)
+文字
+(括弧)
+文字
+【括弧】
+日本語
+English
-「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)」「禁則(きんそく)処理(しょり)」「禁則(きんそく)処理(しょり)!!!!」「禁則(きんそく)処理(しょり)!!!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」
+「禁則(きんそく)処理(しょり)!」
+「禁則(きんそく)処理(しょり)!」
+「禁則(きんそく)処理(しょり)!」
+「禁則(きんそく)処理(しょり)」
+「禁則(きんそく)処理(しょり)!」
+「禁則(きんそく)処理(しょり)!」
+「禁則(きんそく)処理(しょり)」
+「禁則(きんそく)処理(しょり)」
+「禁則(きんそく)処理(しょり)!!!!」
+「禁則(きんそく)処理(しょり)!!!」
+「禁則(きんそく)処理(しょり)!」
+「禁則(きんそく)処理(しょり)!」
+「禁則(きんそく)処理(しょり)!」
+「禁則(きんそく)処理(しょり)!」
+「禁則(きんそく)処理(しょり)!」
+「禁則(きんそく)処理(しょり)!」
-中点・中点
+中点
+・
+中点
-禁則処理にわざと違反した文章のテストを今から行います。準備はいいでしょうか?レディ、ゴー!
+禁則処理にわざと違反した文章のテストを今から行います。準備はいいでしょうか?レデ
+ィ、ゴー!
[ウ ィキペディア]
[ウ ィキペディア]: https://ja.wikipedia.org/
-C言語・C++・Go・Rust
+C言
+語
+・
+C++
+・
+Go
+・
+Rust
`````
### Actual (oxfmt)
`````md
日本語
、
にほんご
。
汉语,
中文.
日
本
語
,
に
ほ
ん
ご
.
English words!?
漢字
!
汉字
?
「セリフ」
(括弧)
文字
(括弧)
文字
【括弧】
日本語
English
「禁則(きんそく)処理(しょり)!」
「禁則(きんそく)処理(しょり)!」
「禁則(きんそく)処理(しょり)!」
「禁則(きんそく)処理(しょり)」
「禁則(きんそく)処理(しょり)!」
「禁則(きんそく)処理(しょり)!」
「禁則(きんそく)処理(しょり)」
「禁則(きんそく)処理(しょり)」
「禁則(きんそく)処理(しょり)!!!!」
「禁則(きんそく)処理(しょり)!!!」
「禁則(きんそく)処理(しょり)!」
「禁則(きんそく)処理(しょり)!」
「禁則(きんそく)処理(しょり)!」
「禁則(きんそく)処理(しょり)!」
「禁則(きんそく)処理(しょり)!」
「禁則(きんそく)処理(しょり)!」
中点
・
中点
禁則処理にわざと違反した文章のテストを今から行います。準備はいいでしょうか?レデ
ィ、ゴー!
[ウ ィキペディア]
[ウ ィキペディア]: https://ja.wikipedia.org/
C言
語
・
C++
・
Go
・
Rust
`````
### Expected (prettier)
`````md
日本語、にほんご。汉语, 中文. 日本語,にほんご.English
words!? 漢字!汉字?「セリフ」(括弧) 文字(括弧)文字【括弧】日本語English
「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)」「禁則(きんそく)処理(しょり)」「禁則(きんそく)処理(しょり)!!!!」「禁則(きんそく)処理(しょり)!!!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」「禁則(きんそく)処理(しょり)!」
中点・中点
禁則処理にわざと違反した文章のテストを今から行います。準備はいいでしょうか?レディ、ゴー!
[ウ ィキペディア]
[ウ ィキペディア]: https://ja.wikipedia.org/
C言語・C++・Go・Rust
`````
+1 -1
View File
@@ -87,7 +87,7 @@ Pin fixtures are named after their entry's slug (a comment would be an HTML bloc
### Fixture fingerprint
`tests/fixtures/fingerprint.rs` serializes the AST's meaning (structure, decoded text, destinations, labels) and ignores what formatting may change
(spans, markers, fence style, text splitting, whitespace runs, tightness, whitespace next to Chinese / Japanese characters, trailing whitespace of verbatim lines, info string whitespace).
(spans, markers, fence style, text splitting, whitespace runs, tightness, trailing whitespace of verbatim lines, info string whitespace).
The harness asserts it is identical for input and output: every fixture is a `parse(format(x)) ≅ parse(x)` check, which idempotency alone cannot give (a corrupted output is often a fixpoint).
Extend the ignore set only with a reason written next to it.
+23 -8
View File
@@ -505,27 +505,42 @@ delimiter escaped when a later line could close it as front matter (a line start
or a thematic break, which prints `---`): the next parse (Prettier's and ours) would read the block as front matter.
Prettier prints it as written (the leading blank line that kept it out of front matter is dropped).
## autolink-cjk-space
## cj-line-break
- Why: semantics
- Pin: `tests/fixtures/markdown/prose-wrap/autolink-cjk-space.md`
- Why: semantics (prettier/prettier#20143)
- Pin: `tests/fixtures/markdown/prose-wrap/cj-line-break.md`
- Conformance: `markdown/splitCjkText/symbolSpaceNewLine.md`
- Oxfmt: `externals/prettier/markdown/splitCjkText/symbolSpaceNewLine.md`
```markdown
<!-- input, proseWrap always -->
日本語の文章は
改行しても
そのままです。
見て http://x.y2.
。次
<!-- ours -->
見て http://x.y2. 。次
日本語の文章は
改行しても
そのままです。
見て http://x.y2.
。次
<!-- prettier -->
日本語の文章は改行してもそのままです。
見て http://x.y2.。次
```
A line break right after the trailing punctuation of an autolink literal stays a space, whatever the
Chinese / Japanese rules would make of it: the literal runs to the next whitespace, so joined to `。次`
it becomes `http://x.y2.。次` on the next parse (`autolink_stretch`).
Prettier drops the break between the two punctuation characters and the link changes.
A line break next to Chinese / Japanese text is kept under every `proseWrap`,
except between a Korean and a CJ letter, where it is a space.
Browsers disagree on such a break (Firefox drops it, Chrome and Safari render a space),
so removing it or making it a space changes the rendered text in some of them.
Joined, an autolink literal also takes the CJ punctuation after it (`http://x.y2.。次`).
Prettier `main` keeps the break since #20143; the pin (3.9.9) removes it or makes it a space.
## wiki-link-risk-link-text
+28 -67
View File
@@ -1,20 +1,18 @@
//! Word kinds for the CJK line-breaking rules (Prettier's `splitText` / `printWhitespace`).
//!
//! A whitespace-delimited word is split into runs of non-CJK characters
//! and single CJK characters (a variation selector stays with its base);
//! only the edge runs matter to the printer,
//! and the split as a whole feeds the "does this sentence put spaces around CJ text" statistic.
//! Only the characters at the ends of a whitespace-delimited word matter to the printer
//! (a variation selector stays with its base).
//!
//! The tables are Prettier's own character classes
//! (`constants.evaluate.js`: `cjk-regex` + Script_Extensions / General_Category for CJK,
//! ASCII punctuation + `\p{P}` (+ U+3000, U+FF5E) for punctuation, `\p{Script_Extensions=Hangul}` for Korean),
//! `\p{P}` (+ U+3000, U+FF5E) for punctuation, `\p{Script_Extensions=Hangul}` for Korean),
//! enumerated with Node 26 (Unicode 17).
//! Regenerate by testing every code point against those regexes and collapsing the hits into ranges.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Kind {
NonCjk,
/// Chinese or Japanese letter: no spaces between words, a line break is not a space.
/// Chinese or Japanese letter: no spaces between words, a line break next to it is kept.
CjLetter,
/// Korean letter: words are space-separated, as in Latin script.
KLetter,
@@ -27,67 +25,33 @@ impl Kind {
}
}
/// The run at one end of a word, as the whitespace next to it sees it.
#[derive(Clone, Copy, Debug)]
pub struct Edge {
pub kind: Kind,
/// The run's character on the whitespace side.
pub ch: char,
/// That character is punctuation (a CJK punctuation run always is).
pub punctuation: bool,
/// The kind of the word's first character.
pub fn first_kind(word: &str) -> Option<Kind> {
word.chars().next().map(kind)
}
/// The kinds of a word's runs, in order.
pub fn kinds(word: &str) -> impl Iterator<Item = Kind> + '_ {
runs(word).map(|(_, kind)| kind)
}
/// The word's first and last run, as edges.
pub fn edges(word: &str) -> Option<(Edge, Edge)> {
let mut it = runs(word);
let (first_run, first_kind) = it.next()?;
let (last_run, last_kind) = it.last().unwrap_or((first_run, first_kind));
let first_ch = first_run.chars().next()?;
let last_ch = last_run.chars().next_back()?;
let edge = |kind: Kind, ch: char| Edge {
kind,
ch,
punctuation: kind == Kind::CjkPunctuation || is_punctuation(ch),
};
Some((edge(first_kind, first_ch), edge(last_kind, last_ch)))
}
/// Splits a word into runs: each CJK character (plus a following variation selector) on its own,
/// non-CJK characters together.
fn runs(word: &str) -> impl Iterator<Item = (&str, Kind)> {
let mut rest = word;
std::iter::from_fn(move || {
let c = rest.chars().next()?;
let (len, kind) = if is_cjk(c) {
let mut len = c.len_utf8();
if let Some(vs) = rest[len..].chars().next()
&& is_variation_selector(vs)
{
len += vs.len_utf8();
}
let kind = if is_punctuation(c) {
Kind::CjkPunctuation
} else if is_hangul(c) {
Kind::KLetter
} else {
Kind::CjLetter
};
(len, kind)
} else {
let len = rest.find(is_cjk).unwrap_or(rest.len());
(len, Kind::NonCjk)
};
let (run, tail) = rest.split_at(len);
rest = tail;
Some((run, kind))
/// The kind of the word's last character; a trailing variation selector stays with its base.
pub fn last_kind(word: &str) -> Option<Kind> {
let mut chars = word.chars();
let last = chars.next_back()?;
Some(match chars.next_back() {
Some(base) if is_variation_selector(last) && is_cjk(base) => kind(base),
_ => kind(last),
})
}
fn kind(c: char) -> Kind {
if !is_cjk(c) {
Kind::NonCjk
} else if is_punctuation(c) {
Kind::CjkPunctuation
} else if is_hangul(c) {
Kind::KLetter
} else {
Kind::CjLetter
}
}
fn is_variation_selector(c: char) -> bool {
matches!(u32::from(c), 0xFE00..=0xFE0F | 0xE0100..=0xE01EF)
}
@@ -100,11 +64,8 @@ fn is_hangul(c: char) -> bool {
in_ranges(HANGUL, c)
}
/// ASCII punctuation, U+3000, U+FF5E, or Unicode general category P.
pub fn is_punctuation(c: char) -> bool {
if c.is_ascii() {
return c.is_ascii_punctuation();
}
/// U+3000, U+FF5E, or Unicode general category P (only asked about CJK characters).
fn is_punctuation(c: char) -> bool {
matches!(c, '\u{3000}' | '\u{FF5E}') || in_ranges(PUNCTUATION, c)
}
@@ -272,13 +272,13 @@ pub fn collect_inlines<'a>(
let break_kept = |j: usize, f: &MarkdownFormatter<'_, 'a>| -> bool {
f.context().raw_text().get() != Raw::No || raw_line(j + 1, f) != Raw::No
};
// Prettier's sentence is a run of texts joined by soft breaks; its CJ spacing style is one statistic.
let mut sentence_cj_spaces: Option<Option<bool>> = None;
// Whether the sentence (a run of texts joined by soft breaks) has CJK text, scanned once per sentence.
let mut sentence_has_cjk: Option<bool> = None;
// The previous sibling was a line break that stays in the output
let mut last_break_kept = false;
for (i, child) in children.iter().enumerate() {
if !matches!(child, Inline::Text(_) | Inline::SoftBreak(_)) {
sentence_cj_spaces = None;
sentence_has_cjk = None;
}
let after_kept_break = last_break_kept;
if !matches!(child, Inline::SoftBreak(_) | Inline::HardBreak(_)) {
@@ -332,8 +332,8 @@ pub fn collect_inlines<'a>(
} else {
(None, None)
};
let cj_spaces = *sentence_cj_spaces
.get_or_insert_with(|| sentence_cj_spaces_at(children, i, f));
let has_cjk =
*sentence_has_cjk.get_or_insert_with(|| sentence_has_cjk_at(children, i));
// A paragraph's first line that opens a block
// (the rest of a paragraph a definition was split from) is escaped: `\- x`, `1\. x`
if i == 0
@@ -365,7 +365,7 @@ pub fn collect_inlines<'a>(
before_soft_break: matches!(children.get(i + 1), Some(Inline::SoftBreak(_))),
next_word: next_word_of(children, i, parent.delimiter, f),
glued_last_word: glued_last_word(children, i, raw, f),
cj_spaces,
has_cjk,
..words::TextContext::default()
};
words::push_text(raw, cx, parts, f);
@@ -381,19 +381,19 @@ pub fn collect_inlines<'a>(
// A separator, not content: the next line is measured on its own
parts.push_sep(Sep::HardLine);
} else {
let cj_spaces = *sentence_cj_spaces
.get_or_insert_with(|| sentence_cj_spaces_at(children, i, f));
let has_cjk =
*sentence_has_cjk.get_or_insert_with(|| sentence_has_cjk_at(children, i));
let cx = words::TextContext {
next_word: next_word_of(children, i, parent.delimiter, f),
after_liquid: follows_liquid(children, i),
// Only the CJK rules look back (a trailing `\` was escaped by the text);
// a word glued to an autolink literal is part of it, the break after it stays a space
prev_word: if cj_spaces.is_some() && !glued_to_autolink(children, i, f) {
// Only the CJK rules look back (a trailing `\` was escaped by the text),
// and `preserve` keeps the break anyway
prev_word: if has_cjk && f.options().prose_wrap != ProseWrap::Preserve {
prev_word_of(children, i, f)
} else {
None
},
cj_spaces,
has_cjk,
..words::TextContext::default()
};
words::push_whitespace(true, &cx, parts, f);
@@ -707,40 +707,17 @@ fn autolink_stretch(children: &[Inline<'_>], i: usize, raw: &str) -> usize {
}
}
/// The text before node `i` is one word in an autolink literal's stretch (`http://x.y2` + `.`):
/// what follows the break must not glue to it (`push_text` makes the same call for its first word).
fn glued_to_autolink<'a>(
children: &'a [Inline<'a>],
i: usize,
f: &MarkdownFormatter<'_, 'a>,
) -> bool {
let Some(Inline::Text(t)) = children.get(i.wrapping_sub(1)) else { return false };
let raw = f.context().slice(t.span);
!raw.contains(is_split_whitespace) && autolink_stretch(children, i - 1, raw) > 0
}
fn follows_liquid(children: &[Inline<'_>], i: usize) -> bool {
matches!(children.get(i.wrapping_sub(1)), Some(Inline::Liquid(_)))
}
/// `usesCJSpaces` of the sentence (run of texts and soft breaks) containing node `i`;
/// `None` without CJK text, which is the common case and skips the scan and the CJK rules.
fn sentence_cj_spaces_at<'a>(
children: &'a [Inline<'a>],
i: usize,
f: &MarkdownFormatter<'_, 'a>,
) -> Option<bool> {
let in_sentence = |c: &Inline<'a>| matches!(c, Inline::Text(_) | Inline::SoftBreak(_));
/// The sentence (run of texts and soft breaks) containing node `i` has CJK text;
/// without it, the common case, the CJK rules are skipped.
fn sentence_has_cjk_at(children: &[Inline<'_>], i: usize) -> bool {
let in_sentence = |c: &Inline<'_>| matches!(c, Inline::Text(_) | Inline::SoftBreak(_));
let start = (0..i).rev().take_while(|&j| in_sentence(&children[j])).last().unwrap_or(i);
let end = (i..children.len()).take_while(|&j| in_sentence(&children[j])).last().unwrap_or(i);
let texts = children[start..=end].iter().filter_map(|c| match c {
Inline::Text(t) => Some(t),
_ => None,
});
if !texts.clone().any(|t| t.contains_cjk) {
return None;
}
Some(words::uses_cj_spaces(texts.map(|t| f.context().slice(t.span))))
children[start..=end].iter().any(|c| matches!(c, Inline::Text(t) if t.contains_cjk))
}
/// A word character directly before or after node `i`.
@@ -5,7 +5,7 @@ use oxc_allocator::{Allocator, ArenaVec, StringBuilder};
use oxc_formatter_core::{
Buffer, Format,
builders::{
dedent_to_root, hard_line_break, literal_line_break, mark_as_root, soft_line_break,
dedent_to_root, hard_line_break, literal_line_break, mark_as_root,
soft_line_break_or_space, text,
},
write,
@@ -46,8 +46,6 @@ impl<'a> Format<'a, MarkdownFormatContext<'a>> for Atom<'a> {
/// A fill separator.
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub enum Sep {
/// A break or nothing (a line break between CJ characters).
SoftLine,
Line,
HardLine,
}
@@ -55,7 +53,6 @@ pub enum Sep {
impl<'a> Format<'a, MarkdownFormatContext<'a>> for Sep {
fn fmt(&self, f: &mut MarkdownFormatter<'_, 'a>) {
match self {
Sep::SoftLine => write!(f, soft_line_break()),
Sep::Line => write!(f, soft_line_break_or_space()),
Sep::HardLine => write!(f, hard_line_break()),
}
@@ -79,13 +76,12 @@ pub struct Parts<'a> {
}
impl<'a> Item<'a> {
/// The item as the parser will read it: a break is a newline, a soft line between CJ letters nothing.
/// The item as the parser will read it: a break is a newline, a line a space.
fn flat(&self) -> &'a str {
match self {
Item::Atom(Atom::Str(s)) => s,
Item::Atom(_) | Item::Sep(Sep::HardLine) => "\n",
Item::Sep(Sep::Line) => " ",
Item::Sep(Sep::SoftLine) => "",
}
}
}
+31 -127
View File
@@ -7,7 +7,7 @@
//! - a line break before a word that would start a block (`-`, `1.`, `#`, `>`) never breaks
//!
//! Around Chinese / Japanese text a whitespace follows Prettier's `printWhitespace`:
//! a line break between CJ characters is not a space, and a space next to one never breaks
//! a line break next to CJ is kept, and a space next to one never breaks
//! (`cjk` classifies the word edges).
use std::borrow::Cow;
@@ -19,7 +19,7 @@ use crate::options::ProseWrap;
use super::{
MarkdownFormatter,
cjk::{self, Edge, Kind},
cjk::{self, Kind},
escape::{
ends_with_unescaped_backslash, is_fake_setext_underline, leading_run_escaped,
print_delimited_word,
@@ -76,9 +76,8 @@ pub struct TextContext<'a> {
/// The last word of the previous sibling text (a soft break's other side), for the CJK rules only;
/// a trailing `\` there was escaped by the text (see `push_text`).
pub prev_word: Option<&'a str>,
/// The sentence puts spaces between CJ and non-CJK words (Prettier's `usesCJSpaces`);
/// `None` when it has no CJK text at all.
pub cj_spaces: Option<bool>,
/// The sentence has CJK text (the parser's `contains_cjk`); without it the word edges are not classified.
pub has_cjk: bool,
/// Inside a reference link's raw content, where a line break never prevents a break.
pub is_link: bool,
}
@@ -99,8 +98,6 @@ pub fn push_text<'a>(
let mut is_first = true;
let leading_ws = raw.starts_with(is_split_whitespace);
let mut prev_word: Option<&'a str> = None;
// `prev_word` is the text's first word
let mut prev_is_first = false;
loop {
// Whitespace before the next word (leading, or the run after the previous word).
let after_ws = rest.trim_start_matches(is_split_whitespace);
@@ -126,9 +123,7 @@ pub fn push_text<'a>(
};
let cx = TextContext {
next_word: next,
// The first word inside an autolink literal's stretch is part of the link:
// the whitespace after it stays a space (a CJ neighbor must not glue to the link)
prev_word: prev_word.filter(|_| !(cx.autolink_stretch > 0 && prev_is_first)),
prev_word,
after_liquid: is_first && cx.after_liquid,
..cx
};
@@ -144,7 +139,6 @@ pub fn push_text<'a>(
break;
}
prev_word = Some(word);
prev_is_first = is_first;
let printed: Cow<'a, str> = if in_delimiter {
let prev = if is_first { cx.edge_prev } else { Some(' ') };
let next = if is_last { cx.edge_next } else { Some(' ') };
@@ -184,7 +178,7 @@ pub fn push_text<'a>(
/// The whitespace (a space run, or one holding a `newline`) between `cx.prev_word`
/// and `cx.next_word` (`None`: the whitespace touches a node edge):
/// a separator that may break, a space, or nothing.
/// a kept line break, a separator that may break, or a space.
pub fn push_whitespace<'a>(
newline: bool,
cx: &TextContext<'a>,
@@ -208,134 +202,44 @@ pub fn push_whitespace<'a>(
options.prose_wrap
};
if prose_wrap == ProseWrap::Preserve && newline {
if newline && prose_wrap == ProseWrap::Preserve {
parts.push_sep(Sep::HardLine);
return;
}
// The CJK rules only apply to a sentence with CJK text
let (prev, next) = if cx.cj_spaces.is_some() {
// A space breaks only under `always`; otherwise it stays a space whatever its neighbors
let (prev, next) = if cx.has_cjk && (newline || prose_wrap == ProseWrap::Always) {
(
cx.prev_word.and_then(|w| cjk::edges(w).map(|(_, last)| last)),
cx.next_word
.filter(|w| w.in_sentence)
.and_then(|w| cjk::edges(w.word).map(|(first, _)| first)),
cx.prev_word.and_then(cjk::last_kind),
cx.next_word.filter(|w| w.in_sentence).and_then(|w| cjk::first_kind(w.word)),
)
} else {
(None, None)
};
let can_be_space = !newline || line_break_can_be_space(prev, next, cx);
// Next to CJ, except Korean next to a CJ letter (a space everywhere)
let cj_bound = (prev.is_some_and(Kind::is_cj) || next.is_some_and(Kind::is_cj))
&& !matches!(
(prev, next),
(Some(Kind::KLetter), Some(Kind::CjLetter))
| (Some(Kind::CjLetter), Some(Kind::KLetter))
);
// Prettier's `isLineBreakAmbiguous`: browsers disagree on whether the break is a space or nothing
if newline && cj_bound {
parts.push_sep(Sep::HardLine);
return;
}
// Prettier's `isBreakable`: never a break between CJ and a neighbor word,
// a node edge (`None`) may break
let breakable = prose_wrap == ProseWrap::Always
&& f.context().no_wrap_depth().get() == 0
&& is_breakable(prev, next, cx);
match (breakable, can_be_space) {
(true, true) => parts.push_sep(Sep::Line),
(true, false) => parts.push_sep(Sep::SoftLine),
(false, true) => parts.push_str(" "),
(false, false) => {}
&& !(cj_bound && prev.is_some() && next.is_some());
// Any line break left here is a space (Prettier's `lineBreakCanBeConvertedToSpace`)
if breakable {
parts.push_sep(Sep::Line);
} else {
parts.push_str(" ");
}
}
/// Prettier's `lineBreakCanBeConvertedToSpace`.
fn line_break_can_be_space(prev: Option<Edge>, next: Option<Edge>, cx: &TextContext<'_>) -> bool {
if cx.is_link {
return true;
}
let (Some(prev), Some(next)) = (prev, next) else { return true };
let latin_like = |kind: Kind| matches!(kind, Kind::NonCjk | Kind::KLetter);
// Between non-CJK / Korean words, or Korean and CJ, a line break is a space
if (latin_like(prev.kind) && latin_like(next.kind))
|| matches!(
(prev.kind, next.kind),
(Kind::KLetter, Kind::CjLetter) | (Kind::CjLetter, Kind::KLetter)
)
{
return true;
}
// A delimiter run glued to a CJ character gains flanking it did not have (`」\n**` → `」**` closes);
// a space keeps it inert, as the line break did
if is_delimiter_char(prev.ch) || is_delimiter_char(next.ch) {
return true;
}
// Around CJK punctuation or between CJ letters it is nothing
if prev.kind == Kind::CjkPunctuation
|| next.kind == Kind::CjkPunctuation
|| (prev.kind == Kind::CjLetter && next.kind == Kind::CjLetter)
{
return false;
}
// Between CJ and non-CJK: a space next to ASCII punctuation (`:::` fences),
// nothing next to other punctuation (`〜`, `…`), else the sentence's own style.
if prev.ch.is_ascii_punctuation() || next.ch.is_ascii_punctuation() {
return true;
}
if prev.punctuation || next.punctuation {
return false;
}
cx.cj_spaces.unwrap_or(false)
}
/// `*` `_` `~` `` ` ``: runs of these pair up by flanking or by length, so their neighbors matter.
fn is_delimiter_char(c: char) -> bool {
matches!(c, '*' | '_' | '~' | '`')
}
/// Prettier's `isBreakable` for a `" "` / `"\n"` whitespace under `always`.
fn is_breakable(prev: Option<Edge>, next: Option<Edge>, cx: &TextContext<'_>) -> bool {
if cx.is_link {
return true;
}
let (Some(prev), Some(next)) = (prev, next) else { return true };
// Korean next to CJ breaks;
// anything else next to CJ never does (browsers turn the break into a space)
matches!(
(prev.kind, next.kind),
(Kind::KLetter, Kind::CjLetter) | (Kind::CjLetter, Kind::KLetter)
) || !(prev.kind.is_cj() || next.kind.is_cj())
}
/// Prettier's `usesCJSpaces`, over the words of one sentence (the texts of a soft-break run):
/// whether spaces outnumber nothing between CJ and non-CJK words.
/// `words` yields each text's words with the whitespace before them (`None` for the first).
pub fn uses_cj_spaces<'a>(texts: impl Iterator<Item = &'a str>) -> bool {
let (mut spaces, mut nothing) = (0u32, 0u32);
let counts = |a: Kind, b: Kind| {
matches!((a, b), (Kind::CjLetter, Kind::NonCjk) | (Kind::NonCjk, Kind::CjLetter))
};
for text in texts {
let mut prev_last: Option<Kind> = None;
let mut rest = text;
loop {
let after_ws = rest.trim_start_matches(is_split_whitespace);
let ws = &rest[..rest.len() - after_ws.len()];
rest = after_ws;
let word_end = rest.find(is_split_whitespace).unwrap_or(rest.len());
let (word, tail) = rest.split_at(word_end);
if word.is_empty() {
break;
}
let mut kinds = cjk::kinds(word);
let first = kinds.next().unwrap_or(Kind::NonCjk);
if let Some(prev) = prev_last
&& !ws.is_empty()
&& !ws.contains('\n')
&& counts(prev, first)
{
spaces += 1;
}
let mut last = first;
for kind in kinds {
if counts(last, kind) {
nothing += 1;
}
last = kind;
}
prev_last = Some(last);
rest = tail;
}
}
spaces > nothing
}
/// Never break before a word that would start a block.
pub fn prevents_break(newline: bool, next: Option<NextWord<'_>>, prose_wrap: ProseWrap) -> bool {
let Some(next) = next else { return false };
+1 -28
View File
@@ -312,33 +312,6 @@ impl Walker<'_> {
}
/// Whitespace runs are a formatter freedom (`proseWrap`, indentation), their content is not.
/// A run touching a Chinese / Japanese character is nothing: the printer treats a space
/// and a line break next to CJ text as interchangeable with no whitespace
/// (browsers drop a segment break between CJ characters), so the oracle equates all three.
fn collapse_whitespace(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut pending_space = false;
for word in s.split_whitespace() {
if pending_space && !out.ends_with(is_cj) && !word.starts_with(is_cj) {
out.push(' ');
}
out.push_str(word);
pending_space = true;
}
out
}
/// Chinese / Japanese (not Korean) by Unicode block; deliberately not the printer's table.
fn is_cj(c: char) -> bool {
matches!(u32::from(c),
0x2E80..=0x303F // CJK radicals, Kangxi, ideographic description, CJK symbols and punctuation
| 0x3040..=0x30FF // Hiragana, Katakana
| 0x3100..=0x312F // Bopomofo
| 0x31C0..=0x31EF // CJK strokes
| 0x3200..=0x9FFF // enclosed CJK, compatibility, ext A, unified ideographs
| 0xF900..=0xFAFF // compatibility ideographs
| 0xFE30..=0xFE4F // compatibility forms
| 0xFF00..=0xFF9F // fullwidth forms, halfwidth katakana
| 0x20000..=0x3FFFF // ext B and beyond
)
s.split_whitespace().collect::<Vec<_>>().join(" ")
}
@@ -1,2 +0,0 @@
見て http://x.y2.
。次
@@ -1,41 +0,0 @@
---
source: crates/oxc_formatter_markdown/tests/fixtures/mod.rs
---
==================== Input ====================
見て http://x.y2.
。次
==================== Output ====================
---------------------------------------
{ printWidth: 80, proseWrap: "always" }
---------------------------------------
見て http://x.y2. 。次
----------------------------------------
{ printWidth: 100, proseWrap: "always" }
----------------------------------------
見て http://x.y2. 。次
--------------------------------------
{ printWidth: 80, proseWrap: "never" }
--------------------------------------
見て http://x.y2. 。次
---------------------------------------
{ printWidth: 100, proseWrap: "never" }
---------------------------------------
見て http://x.y2. 。次
-----------------------------------------
{ printWidth: 80, proseWrap: "preserve" }
-----------------------------------------
見て http://x.y2.
。次
------------------------------------------
{ printWidth: 100, proseWrap: "preserve" }
------------------------------------------
見て http://x.y2.
。次
===================== End =====================
@@ -0,0 +1,21 @@
日本語の文章は
改行しても
そのままです。
漢字と English の間の改行も
English の前で残ります。
営業時間は月〜
金の9時〜18時です。
中文!
한국어
한국어와
漢字の間の改行は空白になります。
更多细节,请参考文档:
www.example.com 和 `code`。
見て http://x.y2.
。次
@@ -0,0 +1,174 @@
---
source: crates/oxc_formatter_markdown/tests/fixtures/mod.rs
---
==================== Input ====================
日本語の文章は
改行しても
そのままです。
漢字と English の間の改行も
English の前で残ります。
営業時間は月〜
金の9時〜18時です。
中文!
한국어
한국어와
漢字の間の改行は空白になります。
更多细节,请参考文档:
www.example.com 和 `code`。
見て http://x.y2.
。次
==================== Output ====================
---------------------------------------
{ printWidth: 80, proseWrap: "always" }
---------------------------------------
日本語の文章は
改行しても
そのままです。
漢字と English の間の改行も
English の前で残ります。
営業時間は月〜
金の9時〜18時です。
中文!
한국어
한국어와 漢字の間の改行は空白になります。
更多细节,请参考文档:
www.example.com 和 `code`。
見て http://x.y2.
。次
----------------------------------------
{ printWidth: 100, proseWrap: "always" }
----------------------------------------
日本語の文章は
改行しても
そのままです。
漢字と English の間の改行も
English の前で残ります。
営業時間は月〜
金の9時〜18時です。
中文!
한국어
한국어와 漢字の間の改行は空白になります。
更多细节,请参考文档:
www.example.com 和 `code`。
見て http://x.y2.
。次
--------------------------------------
{ printWidth: 80, proseWrap: "never" }
--------------------------------------
日本語の文章は
改行しても
そのままです。
漢字と English の間の改行も
English の前で残ります。
営業時間は月〜
金の9時〜18時です。
中文!
한국어
한국어와 漢字の間の改行は空白になります。
更多细节,请参考文档:
www.example.com 和 `code`。
見て http://x.y2.
。次
---------------------------------------
{ printWidth: 100, proseWrap: "never" }
---------------------------------------
日本語の文章は
改行しても
そのままです。
漢字と English の間の改行も
English の前で残ります。
営業時間は月〜
金の9時〜18時です。
中文!
한국어
한국어와 漢字の間の改行は空白になります。
更多细节,请参考文档:
www.example.com 和 `code`。
見て http://x.y2.
。次
-----------------------------------------
{ printWidth: 80, proseWrap: "preserve" }
-----------------------------------------
日本語の文章は
改行しても
そのままです。
漢字と English の間の改行も
English の前で残ります。
営業時間は月〜
金の9時〜18時です。
中文!
한국어
한국어와
漢字の間の改行は空白になります。
更多细节,请参考文档:
www.example.com 和 `code`。
見て http://x.y2.
。次
------------------------------------------
{ printWidth: 100, proseWrap: "preserve" }
------------------------------------------
日本語の文章は
改行しても
そのままです。
漢字と English の間の改行も
English の前で残ります。
営業時間は月〜
金の9時〜18時です。
中文!
한국어
한국어와
漢字の間の改行は空白になります。
更多细节,请参考文档:
www.example.com 和 `code`。
見て http://x.y2.
。次
===================== End =====================
@@ -1,32 +1,11 @@
<!-- Whitespace around Chinese / Japanese: a line break between CJ characters is nothing,
a space next to CJ text never breaks, Korean behaves like Latin -->
<!-- Whitespace around Chinese / Japanese: a space next to CJ text never breaks, Korean behaves like Latin
(a line break next to CJ text: DIVERGENCES.md#cj-line-break) -->
日本語の文章は
改行しても
空白になりません。
漢字と English の間に空白を置く文書では、改行も
English の前で空白になります。
漢字とEnglishの間に空白を置かない文書では、改行は
Englishの前でも消えます。
漢字と English の間に空白を置く文書でも、その空白では折り返さずに、段落の終わりまで一行のまま続きます。
한국어는 라틴 문자처럼
띄어쓰기로 단어를 나눕니다.
営業時間は月〜
金の9時〜
18時です。詳しくは……
後日お知らせします……
括弧の前 (paren) と後ろ、ASCII 記号の
:: 前後は空白になります。
<!-- The rules stop at another node's edge, as Prettier's sentence does:
the whitespace there may break and a line break stays a space
(joined, `:www.` would no longer be an autolink literal) -->
<!-- The rules stop at another node's edge, as Prettier's sentence does: the whitespace there may break -->
使用方法请参考文档中的说明,如果需要了解更多信息,请参考 [应用部署](./deployment.md)。
更多细节,请参考文档:
www.example.com 和 `code`。
@@ -2,215 +2,104 @@
source: crates/oxc_formatter_markdown/tests/fixtures/mod.rs
---
==================== Input ====================
<!-- Whitespace around Chinese / Japanese: a line break between CJ characters is nothing,
a space next to CJ text never breaks, Korean behaves like Latin -->
<!-- Whitespace around Chinese / Japanese: a space next to CJ text never breaks, Korean behaves like Latin
(a line break next to CJ text: DIVERGENCES.md#cj-line-break) -->
日本語の文章は
改行しても
空白になりません。
漢字と English の間に空白を置く文書では、改行も
English の前で空白になります。
漢字とEnglishの間に空白を置かない文書では、改行は
Englishの前でも消えます。
漢字と English の間に空白を置く文書でも、その空白では折り返さずに、段落の終わりまで一行のまま続きます。
한국어는 라틴 문자처럼
띄어쓰기로 단어를 나눕니다.
営業時間は月〜
金の9時〜
18時です。詳しくは……
後日お知らせします……
括弧の前 (paren) と後ろ、ASCII 記号の
:: 前後は空白になります。
<!-- The rules stop at another node's edge, as Prettier's sentence does:
the whitespace there may break and a line break stays a space
(joined, `:www.` would no longer be an autolink literal) -->
<!-- The rules stop at another node's edge, as Prettier's sentence does: the whitespace there may break -->
使用方法请参考文档中的说明,如果需要了解更多信息,请参考 [应用部署](./deployment.md)。
更多细节,请参考文档:
www.example.com 和 `code`。
==================== Output ====================
---------------------------------------
{ printWidth: 80, proseWrap: "always" }
---------------------------------------
<!-- Whitespace around Chinese / Japanese: a line break between CJ characters is nothing,
a space next to CJ text never breaks, Korean behaves like Latin -->
<!-- Whitespace around Chinese / Japanese: a space next to CJ text never breaks, Korean behaves like Latin
(a line break next to CJ text: DIVERGENCES.md#cj-line-break) -->
日本語の文章は改行しても空白になりません。
漢字と English の間に空白を置く文書では、改行も English の前で空白になります。
漢字とEnglishの間に空白を置かない文書では、改行はEnglishの前でも消えます。
漢字と English の間に空白を置く文書でも、その空白では折り返さずに、段落の終わりまで一行のまま続きます。
한국어는 라틴 문자처럼 띄어쓰기로 단어를 나눕니다.
営業時間は月〜金の9時〜18時です。詳しくは……後日お知らせします……
括弧の前 (paren) と後ろ、ASCII 記号の :: 前後は空白になります。
<!-- The rules stop at another node's edge, as Prettier's sentence does:
the whitespace there may break and a line break stays a space
(joined, `:www.` would no longer be an autolink literal) -->
<!-- The rules stop at another node's edge, as Prettier's sentence does: the whitespace there may break -->
使用方法请参考文档中的说明,如果需要了解更多信息,请参考
[应用部署](./deployment.md)。
更多细节,请参考文档: www.example.com 和 `code`。
----------------------------------------
{ printWidth: 100, proseWrap: "always" }
----------------------------------------
<!-- Whitespace around Chinese / Japanese: a line break between CJ characters is nothing,
a space next to CJ text never breaks, Korean behaves like Latin -->
<!-- Whitespace around Chinese / Japanese: a space next to CJ text never breaks, Korean behaves like Latin
(a line break next to CJ text: DIVERGENCES.md#cj-line-break) -->
日本語の文章は改行しても空白になりません。
漢字と English の間に空白を置く文書では、改行も English の前で空白になります。
漢字とEnglishの間に空白を置かない文書では、改行はEnglishの前でも消えます。
漢字と English の間に空白を置く文書でも、その空白では折り返さずに、段落の終わりまで一行のまま続きます。
한국어는 라틴 문자처럼 띄어쓰기로 단어를 나눕니다.
営業時間は月〜金の9時〜18時です。詳しくは……後日お知らせします……
括弧の前 (paren) と後ろ、ASCII 記号の :: 前後は空白になります。
<!-- The rules stop at another node's edge, as Prettier's sentence does:
the whitespace there may break and a line break stays a space
(joined, `:www.` would no longer be an autolink literal) -->
<!-- The rules stop at another node's edge, as Prettier's sentence does: the whitespace there may break -->
使用方法请参考文档中的说明,如果需要了解更多信息,请参考 [应用部署](./deployment.md)。
更多细节,请参考文档: www.example.com 和 `code`。
--------------------------------------
{ printWidth: 80, proseWrap: "never" }
--------------------------------------
<!-- Whitespace around Chinese / Japanese: a line break between CJ characters is nothing,
a space next to CJ text never breaks, Korean behaves like Latin -->
<!-- Whitespace around Chinese / Japanese: a space next to CJ text never breaks, Korean behaves like Latin
(a line break next to CJ text: DIVERGENCES.md#cj-line-break) -->
日本語の文章は改行しても空白になりません。
漢字と English の間に空白を置く文書では、改行も English の前で空白になります。
漢字とEnglishの間に空白を置かない文書では、改行はEnglishの前でも消えます。
漢字と English の間に空白を置く文書でも、その空白では折り返さずに、段落の終わりまで一行のまま続きます。
한국어는 라틴 문자처럼 띄어쓰기로 단어를 나눕니다.
営業時間は月〜金の9時〜18時です。詳しくは……後日お知らせします……
括弧の前 (paren) と後ろ、ASCII 記号の :: 前後は空白になります。
<!-- The rules stop at another node's edge, as Prettier's sentence does:
the whitespace there may break and a line break stays a space
(joined, `:www.` would no longer be an autolink literal) -->
<!-- The rules stop at another node's edge, as Prettier's sentence does: the whitespace there may break -->
使用方法请参考文档中的说明,如果需要了解更多信息,请参考 [应用部署](./deployment.md)。
更多细节,请参考文档: www.example.com 和 `code`。
---------------------------------------
{ printWidth: 100, proseWrap: "never" }
---------------------------------------
<!-- Whitespace around Chinese / Japanese: a line break between CJ characters is nothing,
a space next to CJ text never breaks, Korean behaves like Latin -->
<!-- Whitespace around Chinese / Japanese: a space next to CJ text never breaks, Korean behaves like Latin
(a line break next to CJ text: DIVERGENCES.md#cj-line-break) -->
日本語の文章は改行しても空白になりません。
漢字と English の間に空白を置く文書では、改行も English の前で空白になります。
漢字とEnglishの間に空白を置かない文書では、改行はEnglishの前でも消えます。
漢字と English の間に空白を置く文書でも、その空白では折り返さずに、段落の終わりまで一行のまま続きます。
한국어는 라틴 문자처럼 띄어쓰기로 단어를 나눕니다.
営業時間は月〜金の9時〜18時です。詳しくは……後日お知らせします……
括弧の前 (paren) と後ろ、ASCII 記号の :: 前後は空白になります。
<!-- The rules stop at another node's edge, as Prettier's sentence does:
the whitespace there may break and a line break stays a space
(joined, `:www.` would no longer be an autolink literal) -->
<!-- The rules stop at another node's edge, as Prettier's sentence does: the whitespace there may break -->
使用方法请参考文档中的说明,如果需要了解更多信息,请参考 [应用部署](./deployment.md)。
更多细节,请参考文档: www.example.com 和 `code`。
-----------------------------------------
{ printWidth: 80, proseWrap: "preserve" }
-----------------------------------------
<!-- Whitespace around Chinese / Japanese: a line break between CJ characters is nothing,
a space next to CJ text never breaks, Korean behaves like Latin -->
<!-- Whitespace around Chinese / Japanese: a space next to CJ text never breaks, Korean behaves like Latin
(a line break next to CJ text: DIVERGENCES.md#cj-line-break) -->
日本語の文章は
改行しても
空白になりません。
漢字と English の間に空白を置く文書では、改行も
English の前で空白になります。
漢字とEnglishの間に空白を置かない文書では、改行は
Englishの前でも消えます。
漢字と English の間に空白を置く文書でも、その空白では折り返さずに、段落の終わりまで一行のまま続きます。
한국어는 라틴 문자처럼
띄어쓰기로 단어를 나눕니다.
営業時間は月〜
金の9時〜
18時です。詳しくは……
後日お知らせします……
括弧の前 (paren) と後ろ、ASCII 記号の
:: 前後は空白になります。
<!-- The rules stop at another node's edge, as Prettier's sentence does:
the whitespace there may break and a line break stays a space
(joined, `:www.` would no longer be an autolink literal) -->
<!-- The rules stop at another node's edge, as Prettier's sentence does: the whitespace there may break -->
使用方法请参考文档中的说明,如果需要了解更多信息,请参考 [应用部署](./deployment.md)。
更多细节,请参考文档:
www.example.com 和 `code`。
------------------------------------------
{ printWidth: 100, proseWrap: "preserve" }
------------------------------------------
<!-- Whitespace around Chinese / Japanese: a line break between CJ characters is nothing,
a space next to CJ text never breaks, Korean behaves like Latin -->
<!-- Whitespace around Chinese / Japanese: a space next to CJ text never breaks, Korean behaves like Latin
(a line break next to CJ text: DIVERGENCES.md#cj-line-break) -->
日本語の文章は
改行しても
空白になりません。
漢字と English の間に空白を置く文書では、改行も
English の前で空白になります。
漢字とEnglishの間に空白を置かない文書では、改行は
Englishの前でも消えます。
漢字と English の間に空白を置く文書でも、その空白では折り返さずに、段落の終わりまで一行のまま続きます。
한국어는 라틴 문자처럼
띄어쓰기로 단어를 나눕니다.
営業時間は月〜
金の9時〜
18時です。詳しくは……
後日お知らせします……
括弧の前 (paren) と後ろ、ASCII 記号の
:: 前後は空白になります。
<!-- The rules stop at another node's edge, as Prettier's sentence does:
the whitespace there may break and a line break stays a space
(joined, `:www.` would no longer be an autolink literal) -->
<!-- The rules stop at another node's edge, as Prettier's sentence does: the whitespace there may break -->
使用方法请参考文档中的说明,如果需要了解更多信息,请参考 [应用部署](./deployment.md)。
更多细节,请参考文档:
www.example.com 和 `code`。
===================== End =====================
@@ -2,7 +2,7 @@
source: crates/oxc_formatter_markdown/tests/conformance.rs
expression: report
---
markdown compatibility: 1263/1275 (99.06%), 0 files skipped
markdown compatibility: 1262/1275 (98.98%), 0 files skipped
# Failed
@@ -19,4 +19,5 @@ markdown compatibility: 1263/1275 (99.06%), 0 files skipped
| markdown/commonmark-test-suite/snippet: example-54.md (Thematic breaks) | 💥 | 0.00% |
| markdown/link/encodedLink.md | 💥 | 86.67% |
| markdown/paragraph/cjk.md | 💥💥✨ | 60.00% |
| markdown/splitCjkText/symbolSpaceNewLine.md | 💥 | 20.25% |
| markdown/thematicBreak/simple.md | 💥 | 0.00% |