GitLab

On branch pipeline-input-segment-size, commit df1c96af, the following command gives incorrect output.

cameron@cs-osl-08:~/parabix-devel/build16$ bin/nfd ~/Wikibooks/wiki-books-all.xml --SeparatedPipelineStages=2 --ByteMerging=1 --thread-num=1 >winfd4

The difference is illustrated compared with correct output produced when nfd is run without the separated pipeline stages.

bin/nfd ~/Wikibooks/wiki-books-all.xml --ByteMerging=1 --thread-num=1 >winfd2
cameron@cs-osl-08:~/parabix-devel/build16$ diff winfd2 winfd4
6926652c6926652
< File:Russie - Moscou - place rouge saint basile 3.jpg
---
> File:Russie - Moscou - uge saint basile 3.jpg

The hexdump shows that the missing part of the output is actually a run of nulls that has not been filled in.

cameron@cs-osl-08:~/parabix-devel/build16$ head -6926652 winfd4 | tail -1 | hexdump -C
00000000  46 69 6c 65 3a 52 75 73  73 69 65 20 2d 20 4d 6f  |File:Russie - Mo|
00000010  73 63 6f 75 20 2d 20 00  00 00 00 00 00 00 00 75  |scou - ........u|
00000020  67 65 20 73 61 69 6e 74  20 62 61 73 69 6c 65 20  |ge saint basile |
00000030  33 2e 6a 70 67 0a                                 |3.jpg.|
00000036