Repository navigation
Proposal: a new general purpose Block extension #1175
Description
Activity
My initial proposal suggests that these blocks are different from fenced code blocks, but another option would be to use the fenced code block delimitators and also provide for a 'code' type. In our implementation, a user could provide their own 'code' template which might allow them to provide their own custom syntax highlighting solution. If we went this way, the user would simply use this extension instead of the
fenced_codeextension, avoiding any conflicts.I am undecided which way to go on this. Do we overload fenced code blocks, or create a completely new and different fenced block? Similar approaches (overloading fenced code blocks) have been taken by other implementations. For example r-markdown uses fenced code blocks to define the
rcode which is executed with its output being included in the rendered document. I have also seen at least one Markdown-to-reStructuredText bridge which used overloaded fenced code blocks to replicate directives (sorry, I don't recall which one). To my mind, the primary benefit of overloading fenced code blocks is that when a document using the blocks are parsed by other Markdown implementations, the block is simply parsed as a plain code block. In contrast, introducing a completely new syntax could result in some weird output from other implementations. The only reason the existing admonitions extension mostly avoids this is because the content is indented and becomes a code block. But I would prefer to not indent the content of the new block.In general, when I went through the pain of getting fenced blocks to work under lists and such via SuperFences, I just made it so that users could create custom fences to generate whatever kind of block they wanted. With that said, it still doesn't run inline with normal block processors, so it has some limitations as the content is stored under placeholders like code.
I'd love an approach that uses the normal block processors, and if it is possible, I'd love to see how that would work.
- The way Python Markdown currently normalizes line breaks and such between blocks is frustrating for code blocks as any code blocks with multiple empty lines always get normalized to one.
- Even in Admontions there is still a degree of complexity involved with appending additional blocks as children and accounting for lists and such. I think this is partly due to how Python Markdown consumes blocks.
Anyways, I'm at least interested to see how this may potentially work.
@facelessuser you make some valid points. I have not looked too closely at how you implemented superfences, and was curious how you worked around the problems with the existing block parser. Sounds like you didn't. It may not make sense to even attempt this until after we refactor how blocks are parsed. However, I'm not really interested in tackling that problem right now. Mostly I am simply trying to get my ideas recorded at this point. I haven't given much thought to how it might be implemented.
See also Pandoc Markdown's Divs and Spans https://pandoc.org/MANUAL.html#divs-and-spans which have a bit different but similar idea.
@facelessuser you make some valid points. I have not looked too closely at how you implemented superfences, and was curious how you worked around the problems with the existing block parser. Sounds like you didn't. It may not make sense to even attempt this until after we refactor how blocks are parsed. However, I'm not really interested in tackling that problem right now. Mostly I am simply trying to get my ideas recorded at this point. I haven't given much thought to how it might be implemented.
Yeah, I accomplish everything through the preprocessor still. It's a bit hacky, but it was the only way to avoid the above issues. Anyways, I think this is a good idea, and generally, if/when block parsers are able to handle blocks better, this will be a good addition on top of it.
As a user, I think this is a wonderful idea!
I'm currently migrating a mediawiki site to git/markdown. For admonitions, the pandoc mediawiki→markdown converter produced::: warning, which is the same that markdown-it, docusaurus and some others use. It'll take time to update hundreds of wiki pages to!!! warningand indented content that mkdocs uses. Also, if I ever need to switch away from mkdocs in the future, I'll have to convert the content once more.
The::: class(and the::: tag) ideas are so powerful, that will probably make!!! admonitionsobsolete if they're implemented! I hope we can see this in production soon! :)Reacted by Konstantinos GidarakosJust stumbled upon MyST, a Commonmark parser which is intended as a reStructuredText replacement. I've seen many before, but this is the first one where I like the directives syntax. It is very similar to what I was trying to accomplish here. See also, their optional colon_fence extension, which looks similar to (but not the same as) Pandoc's syntax.
Also of interest is their discussion of various existing proposals and implementations out in the wild and why they chose what they did.
Just stumbled upon MyST, a Commonmark parser which is intended as a reStructuredText replacement. I've seen many before, but this is the first one where I like the directives syntax. It is very similar to what I was trying to accomplish here. See also, their optional colon_fence extension, which looks similar to (but not the same as) Pandoc's syntax.
Unfortunately, us doing code this way, without a huge overhaul of the parser to get block handling where we want, may not be possible. And I definitely wouldn't try to use the
```{directive}format unless the block handling was refactored as the only real way to properly preserve code is using the PreProcessor, and that syntax will conflict with fenced code (at least as things are currently).But, if you don't need to preserve empty lines and such for code blocks and you just want to handle normal Markdown content, it may be possible to cook something up now. I've been considering possibly tinkering with a way to pull this off now that I've had some time to think about it, at least the logic in regards to the handling of the "fences". Assuming some sort of prototype could be pulled off, we'd at least have something to work with now.
So, I actually started to play around with this, and it is in a very early prototype stage. I wanted to make sure I could get it to work in lists and such. I kind of played with the directive syntax some. I'm not sure if anything below, syntax or behavior-wise, will be the final implementation, but this is currently just exploratory.
The thing I like is that you don't have to do indentations for admonitions and such, and it doesn't break my editor's syntax highlighting :). I guess some common admonitions could be created like
warningand if people want something different, they could use the genericadmonitiontype. Anyways, this was a simple test:- ::::{admonition} This is really important! --- type: warning --- Don't do that, for these reasons: - :::{details} Here is a summary This is nested! ::: :::: :::{html} div --- attributes: {id: some-id, class: these are classes} --- Some other content :::
Results
<ul> <li> <div class="admonition warning"> <div class="admonition-title">This is really important!</div> <p>Don't do that, for these reasons</p> <ul> <li> <details> <summary>Here is a summary</summary> <p>This is nested!</p> </details> </li> </ul> </div> <div class="these are classes" id="some-id"> <p>Some other content</p> </div> </li> </ul>
So far the directives are pretty simple objects. They have
on_createevents andon_endevents. You can store the content as you accumulate it and then process it once you hiton_end, or not store it and let the Markdown parser just parse it as Markdown content. That's really it.class DirectiveTemplate: """Directive template.""" # Set to something if argument should be split. # Arguments will be split and white space stripped. ARG_DELIM = '' NAME = '' STORE = False def __init__(self, length): """Intitialize.""" self.store = [] self.length = length def config(self, args, **options): """Parse configuration.""" self.args = [a.strip() for a in args.split(self.ARG_DELIM)] if args and self.ARG_DELIM else [args] self.options = options def on_create(self, el): """On create event.""" def on_end(self, el): """Perform action on end."""
Anyways, I figured I share it and see if people had any thoughts.
Yeah, it is pretty easy to just derive to create shortcut for Note admonitions and such:
class Admonition(DirectiveTemplate): """Admonition.""" NAME = 'admonition' def on_create(self, parent): """Create the element.""" el = etree.SubElement(parent, 'div') t = self.options.get('type', '').lower() title = self.args[0] if self.args and self.args[0] else t.title() classes = [c for c in self.options.get('classes', '').split(' ') if c] if t != 'admonition': classes.insert(0, t) classes.insert(0, 'admonition') ad_title = etree.SubElement(el, 'div', {'class': 'admonition-title'}) ad_title.text = title el.set('class', ' '.join(classes)) return el class Note(Admonition): """Note.""" NAME = 'note' def config(self, args, **options): """Parse configuration.""" super().config(args, **options) self.options['type'] = 'note'
And then this:
:::{admonition} This is really important! --- type: warning --- Don't do that! ::: :::{note} Just a note ::: :::{note} With a title And some words. :::Becomes this:
<div class="admonition warning"> <div class="admonition-title">This is really important!</div> <p>Don't do that!</p> </div> <div class="admonition note"> <div class="admonition-title">Note</div> <p>Just a note</p> </div> <div class="admonition note"> <div class="admonition-title">With a title</div> <p>And some words.</p> </div>
Very cool. Although, when I initially proposed a template block, I meant to actually use a template. Something like the following, which would allow the end user to define any layout they want.
<div class="admonition {{ type }}> <div class="admonition-title">{{ title }}</div> {{ body }} </div>Of course, that would ideally end up as part of the etree, which adds additional complications, so I understand why you haven't taken that approach. It's just that requiring users to use the etree API to define their own custom blocks narrows the target audience. Although, I suppose for specific predefined block types like admonitions, using the etree API is probably more performant. However, a wrapper around what you have so far could provide a more general purpose system which uses actual templates. I just wouldn't name what you have "template."
However, a wrapper around what you have so far could provide a more general purpose system that uses actual templates. I just wouldn't name what you have "template."
Yeah, the idea of templates wasn't really laid out, and I do realize what I have isn't a template. The naming is quite wrong in that regard. I haven't even thought about true templates yet. I'm still working through getting the blocks to not get messed up when passing through lists. There are always list corner cases...always 😢 .
I feel there is an advantage to more advanced, non-template type variants, but I can also see the attraction for actual tempaltes.
Reacted by Waylan Limberg58 remaining items
Keep in mind that mandatory YAML fences must be enabled currently, they aren't default.
Just in case you don't know it, there is an extension, Custom Blocks, quite similar to what you are proposing here but in my opinion (skewed as i am the author) it uses a more simpler syntax (no need for a closing tag, no need for a separator between the block name and the parameters... It also has a more flexible mapping of parameters to python function parameters which makes defining custom blocks quite easy.
::: blocktype value1 "long value2" param3=value3 param4="long value4" Indented content which may contain any other blocks as long as they are **indented**.By default it generates:
<div class="blocktype value1 long-value2" param3="value3" param4="long value4"> <p>Indented content which may contain any other blocks as long as they are <b>indented</b>.</p> </div>
That is:
- the type of the block as class
- keyless values as classes
- keyworded values as attributes
- indented content reparsed as markdown
But you can customize it with a simple python function like this one:
def blocktype(ctx, param1, param2, param3, param4='default'): return E('.blocktype', dict(param1=param1), E('span.title', param2), ctx.parse(ctx.content))
To generate:
<div class="blocktype" param1="value1"> <div class="title">long value2</div> <p>Indented content which may contain any other blocks as long as they are <b>indented</b>.</p> </div>
- keyworded attributes are assigned first to parameters by name
- keyless values are then assigned by position to the rest of the parameters
- the indented content is passed as ctx.content, and ctx.parser generates html with markdown. Depending on your block type you might not do that (source code, dot diagrams...)
Reacted by villeodellOne of the things we are trying to eliminate is the indentation. This requires an end. I've had complaints from others (and myself) about how many syntax highlighters style indentations in Markdown as code blocks ( if they aren't lists. Avoiding indentations silvers this and was mentioned earlier in the thread.
I'll go a little further to say, the intent is not to invalidate work others have done, and if an existing generic block satisfies your needs, I'd say that maybe what we are doing here won't make a difference to you.
With that said, I've tried to put a lot of thought into the generic blocks we are currently implementing.
- They won't require indentation which many have complained about before.
- They should be flexible allowing users to specify if they should treat the content as a block, span, or even raw text which they should preserve through the markdown process.
- They should allow you to tap into various events to make things a bit easier. Do you need to process the content after the entire content under a block has been parsed? There's an
on_endevent for that. Does your generic block need to parse the content but then run a treeproessor on it later, maybe after some other extension? You should be able to use theon_registerevent to add a tree processor in conjunction.
While there are other solutions that may be sufficient for others, I am hoping that some will find what we are trying to do here a welcome addition. I'm trying to involve users of Python Markdown to give their input to try and make this a useful solution moving forward. I am hoping to have a pre-release soon to start allowing people to try it out. It has definitely been more involved to write this extension than what I initial thought, so work near the end has slowed as other obligations in life have crept in, but I think we are very soon reaching an alpha release. Hopefully, with feedback, we can polish any rough edges and get something useful out.
Reacted by Ofek LevIt was just a heads up. Long live to the opensource... and dupped efforts :-) At least i hope you could take a look at it, to learn from its hits and failures. How easy is to add a new kind of block (or span or blockquote or whatever). How indentation or the lack of it may affect readability on nesting. Also the unsolved problem of integrating fenced code because of its pre and post processing hacks. Let me also suggest taking a look on how it uses introspection to map block parameters to user's generator functions, if you take a similar approach you may even reuse the builtin generators CustomBlocks already has with your unindented syntax.
Keep on.
How easy is to add a new kind of block (or span or blockquote or whatever).
I guess I should give some background. I am not new to implementing plugins in the Python Markdown ecosystem. I support an extremely popular set of extensions. You are even using them in your own documentation 🙂 . I am also a member of the Python Markdown team and have helped restructure large parts of the code and implement many features. I have a pretty intimate knowledge of the inner workings of Python Markdown.
How indentation or the lack of it may affect readability on nesting
I am familiar with the tradeoff that would be happening hear. I support a number of plugins that employ this nesting technique already: Details and Tabbed for instance.
Indented blocks can be more readable, but also pose a frustration for standard Markdown highlighters in editors, etc. Additionally, I do believe there are some issues with
md_in_htmland indented extensions, or at least this existed in the past. Additionally, some people also don't like using indented code blocks except in lists. I don't think there is a right or wrong answer here, but this is a well-understood tradeoff.Could we allow some flexibility and allow users to choose whether they'd like to still use indentation, that is a possibility, but not currently in the plans.
Also the unsolved problem of integrating fenced code because of its pre and post processing hacks.
I'm not quite sure what you are referring to here. I've been down the road and have implemented complex plugins using fenced code and am all too familiar with the process. I am the author of SuperFences. Yes, SuperFences used pre and post-processing to implement fenced code and is quite complicated because of it, but there were specific reasons as to why we had to use pre and post-processors. I've also spent a lot of time thinking about how to implement fenced code without using pre and post-processing and have successfully pulled it off with this new generic block extension.
I should be clear and say I already have a working prototype for generic blocks. This isn't theoretical. It successfully allows for fenced code blocks using the standard block processor and no pre and post-processors. It can handle implementing the content as blocks, spans, or even preserved raw data. The only weakness or drawback is that Python Markdown will not retain more than one new line between blocks which doesn't make this approach useful for code blocks where such a requirement is important, but is fine for standard Markdown or preserving raw content where the number of new lines is not really an issue.
I am open to dialogue about any concerns or ideas for the new plugin. I will be looking for feedback when I release this as an alpha. I am interested in whether there are people who truly do prefer a more indented style. It would be interesting to see if those people are a majority or a minority.
I do understand that you may not be interested as you have your own plugin, and that is fine. Our goals and perspectives when approaching this may be different, and there is nothing wrong with that. With that said, if you have specific concerns about the current approach we are heading in and would like to have a broader discussion or suggestions to build upon what we are trying, I'd be happy to delve further into such a conversation.
I know i am not talking to a kiddie. ;-) Apologizes if i said something that could be interpreted in the opposite sense.
Forget code blocks. It was just an example of a fail with CustomBlocks. Maybe you could help but OT, nevermind. Sorry.
The final comment referred to the following: CustomBlocks decouples syntax parsing from html generation. The parser and all the markdown_py extension framework details are dealt by the extension. User extends custom blocks dealing just with html generation, defining a function which receives the parameters (in a headline, in a yaml block, whatever syntax), the content (marked, indented, whatever) and spits html. My point is that if you are getting to a similar abstraction, even if we are using different markdown syntax, maybe the generators could be shared somehow. That was the idea i wanted to share if you want to get it.
I know i am not talking to a kiddie. ;-) Apologizes if i said something that could be interpreted in the opposite sense.
No worries. I have no ideas how much you are aware of what I'm familiar with.
Forget code blocks. It was just an example of a vokimon/markdown-customblocks#1 with CustomBlocks. Maybe you could help but OT, nevermind. Sorry.
If you help me understand what the issue is, I may potentially be able to help 🤷🏻. Though, you'd probably have to ping me over there to keep things on topic here.
My point is that if you are getting to a similar abstraction, even if we are using different markdown syntax, maybe the generators could be shared somehow. That was the idea i wanted to share if you want to get it.
I don't know much of the approach that CustomBlocks uses as I have not looked yet. I can explain what I am doing in the current Block extension.
The idea is to have a single generic format for the block themselves. The base extension handles finding the generic block syntax and processing it and feeding the content of the block through the Markdown parser in the appropriate way. If the block declares that the content should be parsed as block content then all content under it is processed as a block, if the content is declared as a span then it will be parsed as if the content is inline. If declared as raw content, it will be preserved under the wrapper element(s).
Blocks are specified by registering a "Block" object with the main Blocks extension. These objects declare any allowed global options, block-specific options, etc. They provide the parent wrapper element via the
on_createevent and specify how the content should be handled via theon_markdownevent. If desired, after the block has been parsed, anon_endevent can be used to do any further processing, rearranging of elements, etc. There are a few more events for handling some specific scenarios, but you get the idea.Syntax parsing is separate in this sense, the block syntax is generic and handled by the base extension, but content you could handle however you wish. You could use an
on_registerevent to register traditional Markdown extensions that target your special blocks alongside the generic block. For instance, I have a tabbed container created with generic blocks, but we generate ID slugs for each tab. To make sure we do not conflict with Header slugs, we register a Treeprocessor alongside the Tab block so that we can slugify tab IDs after Header slugs are handled via the TOC extension. This way we can avoid conflicting IDs. Additionally, you could preserve the content as raw text and have a special parser to handle the content during theon_endevent. It's flexible to try and allow for some less straightforward block ideas. I'm sure there will be some limitations, but I'm trying to be flexible as possible.Once I get the alpha out, I'm more than happy to have people give criticism and/or suggestions.
I have released an alpha containing the current changes: https://pypi.org/project/pymdown-extensions/#history.
Please provide feedback in this discussion: facelessuser/pymdown-extensions#1868
The main feedback I am looking for now is on syntax.
Maybe I have overlooked it, but is there a simple "hello world" example that we can try which allows me to define a custom block? I'd be primarily interested to test this from an extension author perspective.
@squidfunk I am purposely not focusing on the API right now as I want to nail down syntax first. If you really want to jump into the API, I think the Admonition block is a good template to get started. I would just ignore the
on_registerandon_parseevents as those are used to handle dynamic type generation for the block and are more advanced.At the very least you need an
on_createevent.NAMEmust be a unique name to avoid conflicts with other registered blocks.ARGUMENTSis used to specify what some may think of as the "title" input:/// block | argument.OPTIONSspecifies block-specific inputs that can be given via a YAML block in the header. Regardless of the block, theattributesoption is always special and allows adding extra classes, ID, or whatever else is desired, so that option name is kind of reserved.CONFIGis used to specify any global config settings for the block. These are done when registering your block and only then.
If you have general feedback, please provide it at the alpha's discussion topic here: facelessuser/pymdown-extensions#1868
If you have additional questions related to the API, please create a separate discussion thread on the repo. I'm happy to answer any additional questions, but since API is not the primary focus of the first alpha, and the API could change some before I document the API for a future alpha, that is not my focus, and I'd like to keep the main focus, at least in the alpha release topic, on nailing down the syntax.
Reacted by Martin DonathWell, it may be time to update all of this.
GitHub has implemented their Admonition style, and it's a Blockquote fallback.
Maybe implement the new style for specific admonitions (presumed), and then leave the
!!!for an unstructured block as suggested, that continues to render as it does.>[!NOTE]
!!!- follow the Material for MkDocs style for custom blocks
???- use as a "Spoiler" indicator, with the same unstructured block hidden by default.Here's the current supported array:
Note
Highlights information that users should take into account, even when skimming.
Tip
Optional information to help a user be more successful.
Important
Crucial information necessary for users to succeed.
Warning
Critical content demanding immediate user attention due to potential risks.
Caution
Negative potential consequences of an action.
> [!NOTE] > Highlights information that users should take into account, even when skimming. > [!TIP] > Optional information to help a user be more successful. > [!IMPORTANT] > Crucial information necessary for users to succeed. > [!WARNING] > Critical content demanding immediate user attention due to potential risks. > [!CAUTION] > Negative potential consequences of an action.Reacted by Benoit de Chezelles@matchavez I think that syntax has zero ideological overlap with this particular thread. The GitHub syntax lets you achieve only these 5 exact blocks with absolutely zero leeway for extensibility (any deviation makes the syntax unrecognized by GitHub).
That said, I will advertise a thread where there is interest in making this work despite these limitations, not as a general syntax
Reacted by Timothée MazzucotelliI'm closing this issue as the proposed extension was implemented some time ago as the Blocks extension of pymdown-extensions starting with version 9.10 (see also facelessuser/pymdown-extensions/discussions/1973).
I realize others might want something different, but the Bocks extension meets the needs I was trying to address here in my proposal.
I haven't solidified the proposal yet, but I am envisioning something like a fenced code block (except its not for code and would use different delimitators) which might support two different block types:
Therefore, if the user provided
div(for an element block), the content is parsed as Markdown and then wrapped in adivelement with any additional attributes being set on thediv. However, if the user provides a 'type' ofadmonition, then theadmonitiontemplate is used. Presumably the template would expect a set of attributes (including the admonition type), a title and a body and insert those values into the HTML template. Other templates might accept other options.This would also allow users to use the CSS provided by whichever CSS framework they are using. For example, the MkDocs default theme is using Bootstrap, which provides it own set of alerts. However, MkDocs doesn't use them, but instead also provides CSS for Rst style admonitions because that is what the admonition extension expects. With a new block extension, a Bootstrap alert template could allow Bootstrap's CSS to be used along with all of Bootstrap's alert features (icons, dismissal, etc) removing the need for the MkDocs theme to also include the Rst based CSS.
A few additional things to note:
I would prefer to not add any new extensions to the core library. So I would expect this to be developed in a separate repo. However, I mention it here because it could effect how we proceed with #1174. Also, I would appreciate any feedback on the idea and/or input on a possible syntax proposal.