groff stopped distributing pdfmark/pdfroff

[jarnocek]

According to an announcement last november, groff has stopped distributing pdfroff as part of the removing of pdfmark; you may find some discussion in this thread. The new repository seems to be broken, at least from my domains. I don't know if this should affect the use/maintenance of the ms, man or newest mdoc formats for generating PDFs.

[jgm]

This shouldn't make a difference for us as long as pdfroff is still available. Hopefully the problem with the repository is just temporary.

[jgm]

It's possible, but we'd have to make sure this is fully featured enough. I suspect there were good reasons we went with pdfroff in the first place, but I really don't recall them.

[jgm]

I just tried

pandoc MANUAL.txt -t ms -s -Vfontfamily=T | groff -ms -mpdfmark -mspdf -Tpdf -e -t -KUTF-8 -i  > m.pdf

and the results were pretty good. I needed to specify the fontfamily because the default P wasn't found. I got these errors/warnings:

pdf.tmac:/opt/homebrew/Cellar/groff/1.23.0/share/groff/1.23.0/tmac/spdf.tmac:251: error: pdfhref:init: unknown feature '-PT'
troff:<standard input>:5555: warning [p 43, 6.3i]: cannot adjust line
troff:<standard input>:5816: warning [p 46, 2.0i]: cannot adjust line
troff:<standard input>:5831: warning [p 46, 3.0i]: cannot adjust line
troff:<standard input>:6862: warning: special character 'ty' not defined
Use of uninitialized value $c in substr at /opt/homebrew/bin/gropdf line 1456, <> line 47345.
troff:<standard input>:6880: warning: special character 'do' not defined
troff:<standard input>:11404: warning: special character 'be' not defined
troff:<standard input>:11422: warning: special character 're' not defined

I haven't done a thorough comparison with pdfroff's results, but this is fairly encouraging so far.
(Update: the "cannot adjust line" warnings also appear with pdfroff, but not the others.)
(Update 2: even -P-pa4 seems to work; so all the options we used with pdfroff seem to work.)
(Update 3: math seems to work too.)

The warning about ty comes from

.pdfhref O 3 "Extension: citations \(typst\)"

Here groff is evidently interpreting \(ty as a standard roff special character. I'm not sure why this didn't happen with pdfroff, but it could be because of something special in spdf.tmac, which it seems isn't being loaded here? Also in the PDF's contents we see

(Update: this happens with pdfroff too, so pdfroff isn't better in this respect; it just doesn't emit a warning!)

I think we can omit -mspdf -mpdfmark; at least we still get contents and internal links. This removes the error about spdf.tmac, but we still get the errors about ty etc.

[jgm]

What I've learned so far:

• we can probably use groff instead of pdfroff
• but in either case we need to improve what we do with .pdfhref O; things aren't being properly escaped, as you can see by looking at the contents in the PDF.

[jgm]

OK, I've made some improvements in how the .pdfhref macros are generated. This works better now with pdfroff but the output doesn't work with groff -Tpdf. There seems to be a difference here. E.g.

.pdfhref O 1 "\376\377\000T\000e\000s\000t\000 \003\261\000 \000\050\000p\000a\000r\000e\000n\000s\000\051\000 \040\034\000h\000i\040\035"

works fine with pdfroff but gives you just a string of numbers in the PDF TOC with groff.

[jarnocek]

OK, I've made some improvements in how the .pdfhref macros are generated. This works better now with pdfroff but the output doesn't work with groff -Tpdf. There seems to be a difference here. E.g.

.pdfhref O 1 "\376\377\000T\000e\000s\000t\000 \003\261\000 \000\050\000p\000a\000r\000e\000n\000s\000\051\000 \040\034\000h\000i\040\035"

works fine with pdfroff but gives you just a string of numbers in the PDF TOC with groff.

Just got back to my desk... [updated] I've tested with pandoc 3.6.3 (nightly; sliglhtly behind 3.6.4) and found that the full section on Pandoc's markdown is typeset, but disappears from the PDF outline with pdfroff, whereas with the -Tpdf option it is typeset and reported in the outlne, but with the bookmark in numeric codes,

Another difference is the file sizes of the PDFs: the pdfroff version is far smaller than the -Tpdf option, apparently because of filtering the output through ghostscript (pdfroff does it; in fact it requeres it, while groff -Tpdf does not).

[update] The pdfmark macro package is also loaded by the Perl script called by the -Tpdf option, that's why it is redundant in the command line: gropdf.pl is in fact, another postprocessor, so the pdfmark macro code, the one removed from the groff main distribution, is also removed...

It seems to me that they've just shot their foot. I'll check that anomaly in the Groff development list, to see if this bug/misfeature is already there.

[jarnocek]

After some research, trial and error, I figured out the following:

1. The gropdf perl postprocessor does not use Keith Marshall's pdfmark groff macro package for the bash pdfroff script; instead, it uses the -mpdf macro set; so removing pdfmark/pdfroff should not affect the -Tpdf option for groff;
2. The gropdf postprocessor produces larger PDFs, which might be reduced by filtering them through ghostscript, just as pdfroff does;
3. The offending string

.pdfhref O 1 "\376\377\000P\000a\000n\000d\000o\000c\040\031\000s\000 \000M\000a\000r\000k\000d\000o\000w\000n"

in our sample seems to be produced by a faulty conversion of the original # Pandoc's markdown string in the source, from UTF-8 to UTF-16, which is required by the PDF standard: the string begins with the standard BOM sequence þÿ (octal: \376\377), and the rest of the string prepends \000 to the characters intended to be in the bookmark.

So far, this is clear to me.

[update] The faulty conversion seems to occur whenever there is some non-alphabetic character in the original string, such as rightparen ( or quote '. This may explain the other (ty, do, be, re, which follow parenright in the faulty headers) errors:

$ pandoc --eol lf manual.txt -st ms | sed -e 's/\\376\\377//g' | groff -Kutf8 -et -ms -Tpdf > man2pdf.pdf
troff:<standard input>:5553: warning [p 35, 5.1i]: cannot adjust line
troff:<standard input>:5829: warning [p 37, 8.9i]: cannot adjust line
troff:<standard input>:6860: warning: special character 'ty' not defined
troff:<standard input>:6878: warning: special character 'do' not defined
Use of uninitialized value $c in substr at /usr/bin/gropdf line 1456, <> line 44218.
troff:<standard input>:11402: warning: special character 'be' not defined
troff:<standard input>:11420: warning: special character 're' not defined

[jgm]

Yes, I agree that things like this work

.pdfhref O 1 "Pandoc's Markdown"

But try one with non-ASCII characters:

.pdfhref O 1 "Pandoc's Markdown with Greek αβΓ letters and «quotes»"

[jarnocek]

Overall, it seems to me that the unescaped strings do their job with the gropdf script/engine, acknowledging some limitations.

I prepared a test file with pangrams.

# Queda gazpacho, fibra, látex, jamón, kiwi y viñas.

Benjamín pidió una bebida de kiwi y fresa; Noé, sin vergüenza, la más exquisita champaña del menú.

## Eble ĉiu kvazaŭ-deca fuŝĥoraĵo ĝojigos homtipon.

Laŭ Ludoviko Zamenhof bongustas freŝa ĉeĥa manĝaĵo kun spicoj.

# Voix ambiguë d’un cœur qui au zéphyr préfère les jattes de kiwi

Voyez le brick géant que j’examine près du wharf.

## »Fix, Schwyz!« quäkt Jürgen blöd vom Paß

Falsches Üben von Xylophonmusik quält jeden größeren Zwerg.

# Ζαφείρι δέξου πάγκαλο, βαθῶν ψυχῆς τὸ σῆμα.

Ταχίστη αλώπηξ βαφής ψημένη γη, δρασκελίζει υπέρ νωθρού κυνός.

Indeed, Greek is poorly if supported at all by either script (pdfroff/gropdf): it is faked via the symbol font; and anything beyond Latin 1 is faulty. However, that should be expected, given the limitations of the Base35 Adobe fonts, which are the default on groff.

Here is the test.ms file, with UTF8 encoded text, and the result afterwards.

.\" **** Custom macro definitions *********************************
.\" * Super/subscript
.\" (https://lists.gnu.org/archive/html/groff/2012-07/msg00046.html)
.ds { \v'-0.3m'\\s[\\n[.s]*9u/12u]
.ds } \s0\v'0.3m'
.ds < \v'0.3m'\s[\\n[.s]*9u/12u]
.ds > \s0\v'-0.3m'
.\" * Horizontal line
.de HLINE
.LP
.ce
\l'20'
..
.\" **** Settings *************************************************
.\" text width
.nr LL 6.5i
.\" left margin
.nr PO 1i
.\" top margin
.nr HM 1i
.\" bottom margin
.nr FM 1.25i
.\" header/footer width
.nr LT \n[LL]
.\" point size
.nr PS 10p
.\" line height
.nr VS 12p
.\" font family: A, BM, H, HN, N, P, T, ZCM
.fam P
.\" paragraph indent
.nr PI 0m
.\" interparagraph space
.nr PD 0.4v
.\" footnote width
.nr FL \n[LL]
.\" footnote point size
.nr FPS (\n[PS] - 2000)
.\" color used for strikeout
.defcolor strikecolor rgb 0.7 0.7 0.7
.\" color for links (rgb)
.ds PDFHREF.COLOUR   0.35 0.00 0.60
.\" border for links (default none)
.ds PDFHREF.BORDER   0 0 0
.\" point size difference between heading levels
.nr PSINCR 1p
.\" heading level above which point size no longer changes
.nr GROWPS 2
.\" comment these out if you want a dot after section numbers:
.als SN SN-NO-DOT
.als SN-STYLE SN-NO-DOT
.\" page numbers in footer, centered
.rm CH
.ds CF %
.\" pdf outline fold level
.nr PDFOUTLINE.FOLDLEVEL 3
.\" start out in outline view
.pdfview /PageMode /UseOutlines
.\" ***************************************************************
.\" PDF metadata
.pdfinfo /Title ""
.pdfinfo /Author ""
.hy
.\" 1 column (use .2C for two column)
.1C
.SH 1
Queda gazpacho, fibra, látex, jamón, kiwi y viñas.
.pdfhref O 1 "Queda gazpacho, fibra, látex, jamón, kiwi y viñas."
.pdfhref M "queda-gazpacho-fibra-latex-jamon-kiwi-y-vinas."
.XS
.pdfhref L -D "queda-gazpacho-fibra-latex-jamon-kiwi-y-vinas." \
 -- "	Queda gazpacho, fibra, látex, jamón, kiwi y viñas."
.XE
.LP
Benjamín pidió una bebida de kiwi y fresa; Noé, sin vergüenza, la más
exquisita champaña del menú.
.SH 2
Eble ĉiu kvazaŭ-deca fuŝĥoraĵo ĝojigos homtipon.
.pdfhref O 2 "Eble ĉiu kvazaŭ-deca fuŝĥoraĵo ĝojigos homtipon."
.pdfhref M "eble-ciu-kvazau-deca-fushorajo-gojigos-homtipon."
.XS
.pdfhref L -D "eble-ciu-kvazau-deca-fushorajo-gojigos-homtipon." \
 -- "		Eble ĉiu kvazaŭ-deca fuŝĥoraĵo ĝojigos homtipon."
.XE
.LP
Laŭ Ludoviko Zamenhof bongustas freŝa ĉeĥa manĝaĵo kun spicoj.
.SH 1
Voix ambiguë d\(cqun cœur qui au zéphyr préfère les jattes de kiwi
.pdfhref O 1 "Voix ambiguë d’un cœur qui au zéphyr préfère les jattes de kiwi"
.pdfhref M "voix-ambigue-dun-c_u339_ur-qui-au-zephyr-prefere-les-jattes-de-kiwi"
.XS
.pdfhref L -D "voix-ambigue-dun-c_u339_ur-qui-au-zephyr-prefere-les-jattes-de-kiwi" \
 -- "	Voix ambiguë d\(cqun cœur qui au zéphyr préfère les jattes de kiwi"
.XE
.LP
Voyez le brick géant que j\(cqexamine près du wharf.
.SH 2
»Fix, Schwyz!« quäkt Jürgen blöd vom Paß
.pdfhref O 2 "»Fix, Schwyz!« quäkt Jürgen blöd vom Paß"
.pdfhref M "fix-schwyz-quakt-jurgen-blod-vom-pa_u223_"
.XS
.pdfhref L -D "fix-schwyz-quakt-jurgen-blod-vom-pa_u223_" \
 -- "		»Fix, Schwyz!« quäkt Jürgen blöd vom Paß"
.XE
.LP
Falsches Üben von Xylophonmusik quält jeden größeren Zwerg.
.SH 1
Ζαφείρι δέξου πάγκαλο, βαθῶν ψυχῆς τὸ σῆμα.
.pdfhref O 1 "Ζαφείρι δέξου πάγκαλο, βαθῶν ψυχῆς τὸ σῆμα."
.pdfhref M "_u950__u945__u966__u949__u943__u961__u953_-_u948__u941__u958__u959__u965_-_u960__u940__u947__u954__u945__u955__u959_-_u946__u945__u952__u8182__u957_-_u968__u965__u967__u8134__u962_-_u964__u8056_-_u963__u8134__u956__u945_."
.XS
.pdfhref L -D "_u950__u945__u966__u949__u943__u961__u953_-_u948__u941__u958__u959__u965_-_u960__u940__u947__u954__u945__u955__u959_-_u946__u945__u952__u8182__u957_-_u968__u965__u967__u8134__u962_-_u964__u8056_-_u963__u8134__u956__u945_." \
 -- "	Ζαφείρι δέξου πάγκαλο, βαθῶν ψυχῆς τὸ σῆμα."
.XE
.LP
Ταχίστη αλώπηξ βαφής ψημένη γη, δρασκελίζει υπέρ νωθρού κυνός.
.TC
.pdfsync

And the result from gropdf is somewhat expected, given the above limitations...

compared to the ghostscript postprocessed pdfroff

Notice that the Greek Outline/Bookmark produced by pdfroff is rendered correctly by the PDF reader, and not the document itself, because of the limitation of the fonts, and not of the encoding of the string once processed through ghostscript. The outline produced the gropdf script is incorrect because of the limitation of the perl code itself in supporting anything beyond Latin 1 (perhaps Win1252, but I need further testing).

Perhaps these limitations deserve a warning in the documentation.

[jgm]

OK, that's good to know that many non-ASCII characters work, and that Greek is a special case.

So I think the bottom line is this:

1. We can provide a groff option for --pdf-engine, using groff -Kutf8 -et -ms -Tpdf.

2. When groff is being used, we need to avoid escaping the strings in .pdfref O macros. This could be done in two ways:

   a. Add a groff extension to ms (-t ms+groff) that triggers the non-encoded behavior. Or:
   b. Just "unescape" the encoded strings in Text.Pandoc.PDF before passing to groff.

   Option a would allow you to produce a .ms file that you process manually with groff, so it's probably better. The groff extension could be added automatically when groff is selected as pdf-engine.

3. MAYBE: change default font family in the template to T, since this works better on macOS with the broken homebrew groff install.

[jgm]

Created issue #10738.

[jarnocek]

given the issue open after this general discussion, I think it's about time to move the discussion there...