how to convert doctags code generated by granite docling via vLLM

[saikanov]

Question

as the title, i tried using doctagsdocument but it fail, somehow it cant turn doctags to markdown

[dosubot]

To convert doctags generated by Granite Docling via vLLM to Markdown, you need to follow this workflow: load your doctags into a DocTagsDocument, then use DoclingDocument.load_from_doctags() (as a static method) to create a DoclingDocument, and finally call export_to_markdown() on that DoclingDocument. Calling export_to_markdown directly on a DocTagsDocument will not work and will fail as you described.

Here’s a minimal example:

from docling_core.types.doc.document import DocTagsDocument, DoclingDocument

# Suppose doctags_raw is your doctags string (from Granite Docling/vLLM)
# If you have images, provide them as a list; if not, create dummy images for each page
doctags_doc = DocTagsDocument.from_doctags_and_image_pairs([doctags_raw], [dummy_image])

# Correct usage: static method!
doc = DoclingDocument.load_from_doctags(doctags_doc, document_name="MyDoc")

# Export to markdown
markdown = doc.export_to_markdown()
print(markdown)

Common pitfalls:

• If you only have doctags and no images, create dummy images (e.g., blank PIL images) for each page, as the loader expects them [source].
• Make sure to use DoclingDocument.load_from_doctags as a static method, not as an instance method [source].
• If your markdown output is empty or incomplete, check that your doctags are well-formed and match the expected tag structure. Malformed or incomplete doctags can cause parsing failures [source].
• Layout (like columns) may be lost unless the content is detected as a table. For lossless structure, consider using docling-json instead of doctags, or post-process the output to group elements by layout [source].

If you share a sample of your doctags or the specific error, I can help debug further.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[saikanov]

To convert doctags generated by Granite Docling via vLLM to Markdown, you need to follow this workflow: load your doctags into a DocTagsDocument, then use DoclingDocument.load_from_doctags() (as a static method) to create a DoclingDocument, and finally call export_to_markdown() on that DoclingDocument. Calling export_to_markdown directly on a DocTagsDocument will not work and will fail as you described.

Here’s a minimal example:

from docling_core.types.doc.document import DocTagsDocument, DoclingDocument

Suppose doctags_raw is your doctags string (from Granite Docling/vLLM)

If you have images, provide them as a list; if not, create dummy images for each page

doctags_doc = DocTagsDocument.from_doctags_and_image_pairs([doctags_raw], [dummy_image])

Correct usage: static method!

doc = DoclingDocument.load_from_doctags(doctags_doc, document_name="MyDoc")

Export to markdown

markdown = doc.export_to_markdown()
print(markdown)
Common pitfalls:

• If you only have doctags and no images, create dummy images (e.g., blank PIL images) for each page, as the loader expects them [source].
• Make sure to use DoclingDocument.load_from_doctags as a static method, not as an instance method [source].
• If your markdown output is empty or incomplete, check that your doctags are well-formed and match the expected tag structure. Malformed or incomplete doctags can cause parsing failures [source].
• Layout (like columns) may be lost unless the content is detected as a table. For lossless structure, consider using docling-json instead of doctags, or post-process the output to group elements by layout [source].

If you share a sample of your doctags or the specific error, I can help debug further.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other    

i got empty output with this code, or even when i remove

from docling_core.types.doc.document import DocTagsDocument, DoclingDocument
from pathlib import Path

mains = """
<doctags>
<heading level="1">Introduction</heading>
<paragraph>SmolDocling is a compact vision-language model for document understanding.</paragraph>
<table>
  <row><cell>Model</cell><cell>Params</cell></row>
  <row><cell>SmolDocling</cell><cell>256M</cell></row>
</table>
</doctags>
"""
# Suppose doctags_raw is your doctags string (from Granite Docling/vLLM)
# If you have images, provide them as a list; if not, create dummy images for each page
doctags_doc = DocTagsDocument.from_doctags_and_image_pairs([mains], [Path("img/test-ocr-10.png")])

# Correct usage: static method!
doc = DoclingDocument.load_from_doctags(doctags_doc, document_name="MyDoc")

# Export to markdown
markdown = doc.export_to_markdown()
print(markdown)

[dosubot]

The empty markdown output is likely because your doctags string uses tags like <heading> and <paragraph>, but the parser expects specific tag names such as <section_header_level_1> for headings and <text> for paragraphs. The recognized tags are: title, section_header_level_1 to _level_6, text, otsl (for tables), picture, list_item, code, formula, and a few others. If your doctags don't use these exact tags, nothing will be parsed and your markdown will be empty (source).

Try changing your doctags string to:

<doctags>
  <section_header_level_1>Introduction</section_header_level_1>
  <text>SmolDocling is a compact vision-language model for document understanding.</text>
  <otsl>
    <row><cell>Model</cell><cell>Params</cell></row>
    <row><cell>SmolDocling</cell><cell>256M</cell></row>
  </otsl>
</doctags>

Also, your image argument is fine—Path objects are accepted and converted to PIL images internally (source).

If you update your doctags to use the expected tag names, your markdown export should work. Recent updates also improved markdown output for nested and grouped items, so make sure you're using an up-to-date version if you want the best results (source, source).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[saikanov]

The empty markdown output is likely because your doctags string uses tags like <heading> and <paragraph>, but the parser expects specific tag names such as <section_header_level_1> for headings and <text> for paragraphs. The recognized tags are: title, section_header_level_1 to _level_6, text, otsl (for tables), picture, list_item, code, formula, and a few others. If your doctags don't use these exact tags, nothing will be parsed and your markdown will be empty (source).

Try changing your doctags string to:

Introduction SmolDocling is a compact vision-language model for document understanding. ModelParams SmolDocling256M Also, your image argument is fine—`Path` objects are accepted and converted to PIL images internally ([source](https://github.com/docling-project/docling-core/blob/abfa81f0651e70c1b3bc479f5621f706d66f0cbd/docling_core/types/doc/document.py)).

If you update your doctags to use the expected tag names, your markdown export should work. Recent updates also improved markdown output for nested and grouped items, so make sure you're using an up-to-date version if you want the best results (source, source).

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other    

i got something like this from vllmserve it is different than the one you provide

<loc_47><loc_11><loc_211><loc_20>LLM Security Verification Standard 0.0.1
<loc_433><loc_11><loc_452><loc_20>2024
<loc_46><loc_33><loc_110><loc_43>Frontispiece
<loc_46><loc_56><loc_130><loc_64>About the Standard
<loc_46><loc_74><loc_453><loc_102>The Large Language Model Security Verification Standard is a list of specific AI and LLM security requirements or tests that can be used by architects, developers, testers, security professionals, tool vendors, and consumers to define, build, test and verify secure LLM driven applications.
<loc_46><loc_118><loc_141><loc_127>Copyright and License
<loc_46><loc_136><loc_221><loc_144>Version 0.0.1 (Bleeding Edge version), 2024
<loc_212><loc_150><loc_287><loc_170><loc_46><loc_178><loc_115><loc_186>Figure 1: license
<loc_46><loc_196><loc_454><loc_224>Copyright © 2008-2024 The OWASP Foundation. This document is released under the Creative Commons Attribution-ShareAlike 4.0 International License. For any reuse or distribution, you must make clear to others the license terms of this work.
<loc_46><loc_240><loc_104><loc_249>Project Leads
<loc_185><loc_264><loc_305><loc_273>Vandana Sehgal Elliot Ward
<loc_46><loc_295><loc_192><loc_303>Other Contributors and Reviewers
<loc_42><loc_312><loc_454><loc_373>Eric AllenFrawa VetterliRory McNamaraRaulMoshe(Lakera)(Lakera)(Snyk)Onitza-KlugmanBen-NehemiaSam Watts(Snyk)(Lakera)
<loc_46><loc_389><loc_452><loc_407>If a credit is missing from the 0.0.1 credit list above, please log a ticket at GitHub to be recognized in future 0.x updates.
<loc_46><loc_412><loc_453><loc_450>The Large Language Model Security Verification Standard is built upon the initial research performed into LLM security by the Snyk Security labs team in 2023. Much of the concept, structure, boilerplate and tooling for the LLMSVS has been adapted from the OWASP ASVS project. Thank you to all those previously involved in the OWASP ASVS.
<loc_447><loc_472><loc_452><loc_480>3

i think this is just a loc, does i miss something that i need to add to get the docstring?

[dosubot]

Your vllmserve output uses <loc_> tags, which are just location markers for bounding boxes—they aren't recognized as semantic doctags by Docling, so nothing gets parsed and your markdown output is empty. Docling's parser expects tags like <section_header_level_1>, , (for tables), etc. Only these will be converted to markdown; <loc_> tags are ignored except for provenance info, not content.

To get markdown output, you need your pipeline (model or prompt) to emit valid doctags with those recognized tags. There’s no built-in tool or official docling-core or docling-serve feature to convert <loc_> tags to semantic doctags automatically—you’d need to add a custom post-processing step that maps your <loc_>-tagged text to the right doctags structure, or adjust your model/prompt to output the correct tags directly. For reference, here’s a minimal valid doctags example:

<doctags>
  <section_header_level_1>Introduction</section_header_level_1>
  <text>SmolDocling is a compact vision-language model for document understanding.</text>
  <otsl>
    <row><cell>Model</cell><cell>Params</cell></row>
    <row><cell>SmolDocling</cell><cell>256M</cell></row>
  </otsl>
</doctags>

If you want to preserve layout or location info for LLMs, consider working with docling-json or doctags and post-processing them to group elements by layout before converting to markdown, since markdown export flattens structure unless content is detected as a table (source). If you share more about your vllmserve pipeline or prompts, I can help brainstorm how to generate valid doctags.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[saikanov]

Your vllmserve output uses <loc__> tags, which are just location markers for bounding boxes—they aren't recognized as semantic doctags by Docling, so nothing gets parsed and your markdown output is empty. Docling's parser expects tags like <section_header_level_1>, , (for tables), etc. Only these will be converted to markdown; <loc__> tags are ignored except for provenance info, not content.

To get markdown output, you need your pipeline (model or prompt) to emit valid doctags with those recognized tags. There’s no built-in tool or official docling-core or docling-serve feature to convert <loc__> tags to semantic doctags automatically—you’d need to add a custom post-processing step that maps your <loc__>-tagged text to the right doctags structure, or adjust your model/prompt to output the correct tags directly. For reference, here’s a minimal valid doctags example:

Introduction SmolDocling is a compact vision-language model for document understanding. ModelParams SmolDocling256M If you want to preserve layout or location info for LLMs, consider working with docling-json or doctags and post-processing them to group elements by layout before converting to markdown, since markdown export flattens structure unless content is detected as a table ([source](https://github.com/docling-project/docling-serve/issues/374)). If you share more about your vllmserve pipeline or prompts, I can help brainstorm how to generate valid doctags.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other    

how do i do that? with prompting?

i do the exact same with vLLM native code and it worked, but the serve just give me the loc

[dosubot]

Prompting can work—but only if the model and pipeline actually support emitting semantic doctags (like <section_header_level_1>, <text>, <otsl>, etc.) in the output. vllmserve does not convert <loc_*> tags to semantic doctags automatically, and there’s no config or middleware in vllmserve to do this for you. If vLLM native code gives you semantic doctags but vllmserve only gives <loc_*> tags, the difference is almost always in the prompt template or pre/post-processing: vllmserve just passes through what the model outputs, so you need to make sure your prompt (and any system prompt or pipeline config) explicitly instructs the model to emit the correct tags, not just location markers.

Try running a minimal test where you prompt vllmserve with a clear instruction and example output using the semantic doctags you want. If that still doesn’t work, compare the exact prompt and pipeline config between your working vLLM native setup and vllmserve—sometimes the serve layer adds or strips context, or uses a different default prompt.

If you’re stuck with <loc_*> tags only, you’ll need a custom post-processing script to map those to semantic doctags before passing them to Docling for markdown export—there’s no built-in tool for this in docling-core or docling-serve (source).

If you want, share your prompt or pipeline config for both vLLM native and vllmserve and I can help debug the difference.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}