support replacing [refid][] syntax link content with alternate content

Add programmable reference overrides

If a user provides a ReferenceOverride function, then reference ids
will be passed to the given ReferenceOverride function first, before
consulting the generated reference table.

The goal here is to enable programmable support for
"WikiWords"-style identifiers or other application-specific
user-generated keywords.

Example, writing documentation:

 The [Frobnosticator][] is a very important class in our codebase.
 While it is used to frobnosticate widgets in general, it can also
 be passed to the [WeeDoodler][] to interesting effect.

This might be solveable with the HTML Renderer relative prefix, but
I didn't see a good way of making a short link to 'Frobnosticator'
relatively without having to write it twice. Maybe
'<Frobnosticator>' should work? Should Autolinks work for relative
links?

In addition, I wanted a little more richness. I plan to support
Godoc links by prefixing references with a '!', like so:

  Check out the [Frobnosticator][] helper function
  [!util.Frobnosticate()][]

The first link links to the Frobnosticator architectural overview
documentation, whereas the second links to Godoc.

Better advice on how to implement this sort of think with
Blackfriday is highly desired.

Merge pull request #131 from russross/minor-cleanup

Minor cleanup.

Merge pull request #128 from bjornerik/angled-quotes

Add support for angled, double quotes

Merge pull request #130 from shurcooL/master

Fix test case.

Merge pull request #129 from halostatue/header-prefixes-and-suffixes

Allow configurable header ID prefix/suffixes.

Merge pull request #126 from halostatue/generate-unique-header-ids

Prevent header collisions on generation.

Merge pull request #125 from halostatue/auto-header-id

Add a flag to turn on header ID generation.

Merge pull request #124 from halostatue/fix-header-id-toc-rendering

Use supplied header ID for TOC rendering.

Merge pull request #118 from rtfb/test-relative-links

Make sure relative URLs are preserved

Merge pull request #117 from rtfb/remove-sanitization

Remove sanitization

Merge pull request #111 from rtfb/master

Add Travis configuration and a badge

Fixup tests after 67002b0

Merge pull request #109 from tummychow/html5-code-block

Use HTML5 recommended style of language on code blocks

Merge pull request #107 from shurcooL/master

Improve parser to detect LineBreak independently of renderer.

Fix misleading wording in README

Fixes #103.

Merge pull request #98 from cpuguy83/master

Add titleblock support

Merge pull request #95 from anonx/master

Whitelisted th and td to fix #79

Merge pull request #93 from shurcooL/patch-1

Add a reference to github_flavored_markdown in README.

Merge pull request #88 from dimfeld/master

Remove unnecessary HTML_ABSOLUTE_LINKS flag

Merge pull request #86 from dimfeld/master

Add absolute link transformation and footnote enhancements

Merge pull request #82 from dimfeld/master

Sanitize shouldn't filter out URLs without protocol.

Update README.md

link to markdownfmt, add note about LaTeX output

Merge pull request #78 from JeffPaine/patch-1

Add Usage heading to README

Merge pull request #39 from athom/tilde-escape

allow \~ to escape as ~

Merge pull request #77 from mprobst/cleanup_sanitize_test

Move sanitization tests into their own file.

Merge pull request #73 from johnsto/bugfix/fenced-code-append

Fix fenced code extension modifying data beyond slice

Merge pull request #76 from mprobst/self-closing

feat: Write self-closing tags with a />

Merge pull request #75 from mprobst/sanitize_test

Avoid raw mode parsing so that tags like <script> don't cause escaping

Merge pull request #74 from mprobst/sanitize_test

Add a test for the correct handling of escaped entities in HTML.

Merge pull request #71 from mprobst/master

Add support for a bunch more safe HTML element tags, and bring them into...

Merge pull request #70 from mprobst/master

fix: Handle all different token types that the parser can emit (d'oh).

Merge branch 'gihnius-master'

Allow rel attribute in sanitizer

Fixes issue #68.

Merge pull request #69 from mprobst/master

Use go.net/html's parser to sanitize HTML.

Merge pull request #64 from willnix/master

Add table tags to the whitelist.

Merge pull request #61 from shurcooL/feature/dont-expand-tabs-inside-fenced-code-blocks

Don't expand tabs inside fenced code blocks.

Merge pull request #60 from shurcooL/fix/fenced-code-block-extra-newline

Fix for potential extra newline added inside fenced code blocks.

Merge pull request #59 from johnsto/master

Header ID specifiers

Merge pull request #56 from muhqu/issue/45

Fix for Fenced Code Blocks without a blank line before

Merge pull request #58 from aspic/master

Explicit return byte array at end of function.

Merge pull request #55 from rtfb/master

Autolink fixes

Merge pull request #50 from rtfb/master

Better protection against JavaScript injection

Merge pull request #52 from laslowh/master

add HTML_NOFOLLOW_LINKS

Merge pull request #44 from FreakyDazio/safe-relatives

Relative URIs are considered safe

Merge pull request #43 from microcosm-cc/master

Cells in THEAD > TR are now TH.

Change GOROOT references to GOPATH in README

fix smartypants to pass single backticks through, issue #38

panic fix (issue #33) with test case

Merge pull request #32 from bertzzie/master

Enable Parsing Inside a Link

Merge pull request #31 from aybabtme/patch-1

Fix typo.

update license language to match OSI

Merge pull request #29 from athom/master

add EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK flag to make it closer to GFM

Merge pull request #27 from moshee/master

Footnotes (addresses #14)

Merge pull request #22 from rtfb/master

Add some protection against script injection

Merge pull request #24 from subosito/sample-fixes

Fix table syntax example on README

Merge pull request #16 from cespare/blockcodegithub_doc_fix

Fix up method documentation formatting for the BlockCodeGithub method.

Merge pull request #15 from moshee/master

HTML5

recognize fraction slash as well as regular slash to make fractions

link directly to blackfriday-tool

readme updates for go 1

updates for go 1

version bump to v1.1

permit backslash-escaped vertical bars in tables

fixed bug with blank line handling within list items

table unit tests and fix for a crash uncovered by them

bug in tables when a row has too few columns

tag as version 1.0

additional doc comments

doc improvements, commenting

readme tweak

readme updates

simplify naming of parsing functions

finished removing redundant end-of-buffer checks in block parsing; code cleanup

bounds checking stress tests

missing bounds check

removing more redundant checks, additional cleanup of block parsing

more consistent spacing of block-level elements

remove redundant tests for tab characters in parsing

remove NO_EXPAND_TABS options

removing redundant end-of-buffer checks in block parsing

move whitespace stripping to parser, not renderers

corner case spacing issue with table of contents

simplify inline callback interface

version bump to 0.6

inline helpers put parser arg first

Renderer is now an interface

preparing for switch to rendering interface

table of contents support beefed up

complete page rendering is now an option in the library

refactoring: inline renderers return bools, preparing rendering struct to become an interface

render -> Parser, made parsing functions methods of *Parser

camel case

added simplified interface for common usage

version number, few more options for command-line tool

example markdown binary: try to guess a title

options to supress tab expansion or to expand tabs to 8 spaces instead of 4

fenced code: ending marker must match beginning marker, tests for fenced code blocks

readme tweak

README tweak

simplified BSD license

preformatted html block tests

tests for ordered lists

horizontal rule and list testing

more robust whitespace stripping and matching corrections to tests

fixed minor bugs uncovered by more testing

unit tests for underlined headers, improved whitespace handling for the same

fixed headers nested in lists, added prefix header unit tests

eliminate a buffering level for paragraphs

clean up main markdown function: split out first and second passes

refactoring: newlines as hard breaks changed from HTML option to global markdown option

refactoring paragraph rendering

reduce copying for lists

experiment: render headers directly to output buffer to avoid a copy; minor speed boost

dumb tweak that gives a little speed bump

rewrite of attrEscape: cleaner and faster

enable profiling from command-line tool

fraction example in readme

more inline unit tests

output validates, command-line tool has useful options

improved (hopefully) smart quote handling

convert test files to unix format, fix a few broken ones

unit test for linebreaks

tab expansion bug

fix test name conflicts

Merge pull request #2 from kjk/markdown-tests

integrate tests for markdown 1.0.3 test files by comparing them with refe

starting inline unit tests, fix a few minor bugs they exposed

readme updates

comments, minor cleanups

rewrote the stinking pile of code that was blockParagraph

tab expansion fixed to handle multibyte unicode characters

gofmt

allocate new buffers on stack; mild speed improvement

export all names from Renderer struct
This enables new back-ends that are not part of the package
Basically a big search-and-replace for this commit

performance fix: with autolinking on, it is almost twice as fast now

remove dependency on less function

readme tweak

rudimentary latex backend, additional cleanup

split parser into multiple files, clean up naming

cleanup in markdown: better naming, misc fixes

comments on performance

fix smartypants and html entity escaping

escape entities when using smartypants

return result instead of taking buffer as input

features list

mdash

readme file

refactored into a proper package

smartypants

cleanup

compatibility fixes

output matches upskirt for markdown test suite

fixing link parsing

basics working, still a few renderers to write

parsing done but untested

working on inline parsing

emph parsing

reference extraction

setup, starting reference handling

fixed ordered lists

all block-level parsers

working on listitem

initial commit

integrate tests for markdown 1.0.3 test files by comparing them with reference files rendered with upskirt (no extensions)

Add some HTML5

HTML5 doctype, Wrap TOC with <nav>

<nav> makes the TOC more easily identifiable and workable with CSS.

Fix html tag ordering in doc string.

Fix up method documentation formatting.

fix table syntax example

Improve html element stripping code

Fix typo

Add HTML_SKIP_SCRIPT to MarkdownCommon

Couple more tests

More <script> stripping

Partially addresses issue #11.

Add an option to strip <script> elements

Partially addresses issue #11.

Make isHtmlTag() case insensitive

Extract repetitive code to a func

Fix bug in isHtmlTag()

Fix what seems to be a typo. j should iterate through all tagname, so it
should be initialized to zero. The test exposes this bug.

Make a way to parameterize inline tests

Expose extensions and html flags parameters so that tests could specify
what code paths they want to exercise.

parser no longer returns prematurely from empty footnote ref

leftover debug stuff

added slice bounds check

Implementation and some tests for inline footnotes. Also I noticed the list items had the wrong ids, that was silly of me.

Referenced footnotes appear to be functional. Inline still unimplemented.

new tests pass but old tests now fail...

First attempt at supporting Pandoc-style footnotes. The existing tests have not broken but the new functionality does not work yet.

add testcase for GFM autolink

make autolink peforms like GFM

add EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK flag to make it closer to GFM(Github flavor Markdown)

Fix typo.

Image inside a link now works.

Added test for link inside image

Updated tests to check for th tags instead of td tags within thead blocks

Added th to table headers so that styling with things like Twitter Bootstrap and typeset.css work as expected. Cells in headers should always be TH unless they are advisory cells within headers in which case TD is acceptable (but being Markdown a user with such needs could just enter HTML for this)

Tests for links when using  HTML_SAFELINK

  Relative URIs are considered safe

add HTML_NOFOLLOW_LINKS

Add a note about JavaScript injections to README

Rename HTML_SKIP_SCRIPT to HTML_SANITIZE_OUTPUT

Rewrite protection against JavaScript injection

This drops the naive approach at <script> tag stripping and resorts to
full sanitization of html. The general idea (and the regexps) is grabbed
from Stack Exchange's PageDown JavaScript Markdown processor[1]. Like in
PageDown, it's implemented as a separate pass over resulting html.

Includes a metric ton (but not all) of test cases from here[2]. Several
are commented out since they don't pass yet.

Stronger (but still incomplete) fix for #11.

[1] http://code.google.com/p/pagedown/wiki/PageDown
[2] https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet

Extract useful code to separate func

Fix bug in autolink with trailing semicolon

In case the link ends with escaped html entity, the semicolon is a part
of the link and should not be interpreted as punctuation.

Fix bug in autolink overescaping html entities

If autolink encounters a link which already has an escaped html entity,
it would escape the ampersand again, producing things like these:
    &  --> &
    " --> "
This commit solves that by first looking for all entity-looking things
in the link and copying those ranges verbatim, only considering the rest
of the string for escaping.
Doesn't seem to have considerable performance impact.
The mailto: links are processed the old way.

Extract a chain of ifs into separate func

This gives a ~10% slowdown of a full test run, which is tolerable.
Switch statement is still slightly slower (~5%). Using map turned out to
be unacceptably slow (~3x slowdown).

go fmt

Fix bug in autolink termination

Detect the end of link when it is immediately followed by an element.

Fix bug with overzealous autolink processing

When the source Markdown contains an anchor tag with URL as link text
(i.e. <a href=...>http://foo.bar</a>), autolink converts that link text
into another anchor tag, which is nonsense. Detect this situation with
regexp and early exit autolink processing.

Explicit return byte array at end of function.

Revert "add an infinity-loop detection to block-level parsing"

This reverts commit 0c62e28e900533ff5d0376fac2e5b0c4894e1fa3.

optimisation: only fix fenced code blocks if the extensions parser flag is set... ;-)

out-comment stderr debug output

add some stderr output to reference stress tests

fix issue #45: 'Fenced Code Blocks without a blank line before'

Add missing newline between paragraph and fenced code block within `firstPass()`.

add error message when panic has been raised within `doTestsBlock()`

add an infinity-loop detection to block-level parsing

add test cases for issue #45

Also support header IDs in ## headers ##

Correctly emit trailing header ID brace

Add Header IDs to default extensions

Add header ID support and tests: # Header {#myid}

Fix bug where newlines were inserted inside fenced code blocks.

Change firstPass() code that checks for fenced code blocks to check all
of them and properly keep track of lastFencedCodeBlockEnd.
This way, it won't misinterpret the end of a fenced code block as a
beginning of a new one.

Add failing test for an issue introduced by PR #56.

The issue is that when there are more than 1 fenced code blocks with a
blank line before and after, the parser introduces a single extra new
line to all the fenced code blocks except the last one.

Don't expand tabs inside fenced code blocks.

Still do normalize newlines inside fenced code blocks.

tagWhitelist allows alignment attribute now

This is the closest I could get to removing everything "unsave" without introducing an additional regex.

Add table tags to the whitelist.

Fixing:
https://github.com/russross/blackfriday/commit/55cd82008e9b35b9a03a80e06d5a4c4601320211

This commit introduced a html tag whitelist which does not include any table tags (<td>,<tr>,<thead>...). Therefore even tables the markdown parser itself generated will be removed.

Use go.net/html's parser to sanitize HTML.

Use an HTML5 compliant parser that interprets HTML as a browser would to parse
the Markdown result and then sanitize based on the result.
Escape unrecognized and disallowed HTML in the result.
Currently works with a hard coded whitelist of safe HTML tags and attributes.

Merge branch 'master' of github.com:gihnius/blackfriday into gihnius-master

Conflicts:
	html.go
	inline_test.go

test: add nofollow ref for non internal links only

add nofollow ref for non internal links only

add target blank attr

fix: Handle all different token types that the parser can emit (d'oh).

Add support for a bunch more safe HTML element tags, and bring them into some order.

Add a test for the correct handling of escaped entities in HTML.

The sanitization code does not retain any particular escaped entities - it
parses the HTML and thus loses the information on what entities were in the
original. The result is correct UTF-8 HTML though.

Avoid raw mode parsing so that raw mode tags like <script> don't cause issues.

Certain tags like <script> but also <title> and others switch an HTML5 parser
into raw mode, which causes the rest of the HTML string to be always parsed as
text, including any elements or entities that we do want to support (e.g. <p>).

As we're going to escape any of the raw text elements anyway (it's e.g. script,
style, title, xmp, noframes, and a couple of others) we can just switch of raw
text parsing by disabling it after each starting tag.

feat: Write self-closing tags with a />

Adds tests for self-closing tags both for correct writing and for correct
sanitization, i.e. stripping attributes on them.

Avoid double alloc

Fix fenced code extn modifying data beyond slice

Move sanitization tests into their own file.

Also adds an explicit test for [link](...) syntax to be sanitized.

allow \~ to escape as ~

Add Usage heading

Sanitize shouldn't filter out URLs without protocol.

Fix spelling error

Add tests for absolute prefix

Add tests for new footnote functionality

Move footnote prefix to a better place

Use parameters object for extra options. Enhance footnote support.

Option to add return links.
Option to make footnote prefixes unique, for rendering multiple
documents per page.

Add footnote prefix option. Needs testing

Add ability to convert relative links to absolute

Remove unnecessary HTML_ABSOLUTE_LINKS flag

Add a reference to github_flavored_markdown in README.

Whitelisted th and td to fix #79

Add titleblock support

Improve parser to detect LineBreak independently of renderer.

When checking if it's a newline preceeded by two spaces, look at the input data rather than the output, since the output depends on the renderer implementation.

Use HTML5 recommended style of language on code blocks

For code blocks that contain a certain language of code, the recommended
attribute structure is <pre><code class="language-foo">. This also
corresponds to the behavior expected by various JS syntax highlighters.

The GitHub code block implementation was obsolete, and identical to the
normal implementation except for its attribute structure, so it was
removed.

Closes #108.

Remove go tip from Travis build matrix

More Travis stuff: better go get, plus build badge

Add a simple Travis config

Change Sanitize() to SanitizeBytes() in example

Document usage of blackfriday along w/ bluemonday

Rip off all blackfriday's html sanitization effort

As per discussion in issue #90.

Convert silly funcs to consts

Make sure relative URLs are preserved

Add tests to make sure we don't break relative URLs again.
Extracted common html flags and common extensions for easy access from
tests.

Closes issue #104, which was fixed as a side effect of cf6bfc9.

Use supplied header ID for TOC rendering.

- Fixes #112 so that `#header {#header-id}` renders the TOC with
  `#header-id` instead of `#toc_1`.

Add a flag to turn on header ID generation.

- Fixes #51, #101, and #102.
- Uses the [code][gfm] mentioned by @shurcooL from his Github
  Flavored Markdown parser extension in a [comment on #102][comment].
  Since this was mentioned, I assumed that @shurcooL would be OK with
  this being included under the licence provided by blackfriday (there
  is no licence comment on his code).
- I’ve added it behind another flag, EXTENSION_AUTO_HEADER_IDS, that
  would need to be turned on for it to work. It works with both prefix
  and underline headers.

[gfm]: https://github.com/shurcooL/go/blob/3bec0366a85101d116bf88b477e92ea6cdd75e4e/github_flavored_markdown/main.go#L90-L102
[comment]: https://github.com/russross/blackfriday/issues/102#issuecomment-51272260

Prevent generated header collisions, less naively.

> This is a rework of an earlier version of this code.

The automatic header ID generation code submitted in #125 has a subtle
bug where it will use the same ID for multiple headers with identical
text. In the case below, all the headers are rendered a `<h1
id="header">Header</h1>`.

  ```markdown
  # Header
  # Header
  # Header
  # Header
  ```

This change is a simple but robust approach that uses an incrementing
counter and pre-checking to prevent header collision. (The above would
be rendered as `header`, `header-1`, `header-2`, and `header-3`.) In
more complex cases, it will append a new counter suffix (`-1`), like so:

  ```markdown
  # Header
  # Header 1
  # Header
  # Header
  ```

This will generate `header`, `header-1`, `header-1-1`, and `header-1-2`.

This code has two additional changes over the prior version:

1.  Rather than reimplementing @shurcooL’s anchor sanitization code, I
    have imported it as from
    `github.com/shurcooL/go/github_flavored_markdown/sanitized_anchor_name`.

2.  The markdown block parser is now only interested in *generating* a
    sanitized anchor name, not with ensuring its uniqueness. That code
    has been moved to the HTML renderer. This means that if the HTML
    renderer is modified to identify all unique headers prior to
    rendering, the hackish nature of the collision detection can be
    eliminated.

Allow configurable header ID prefix/suffixes.

This is specifically driven by the Hugo usecase where multiple documents
are often rendered into the same ultimate HTML page.

When a header ID is written to the output HTML format (either through
`HTML_TOC`, `EXTENSION_HEADER_IDS`, or `EXTENSION_AUTO_HEADER_IDS`), it
is possible that multiple documents will hvae identical header IDs. To
permit validation to pass, it is useful to have a per-document prefix or
suffix (in our case, an MD5 of the content filename, and we will be
using it as a suffix).

That is, two documents (`A` and `B`) that have the same header ID (`#
Reason {#reason}`), will end up having an actual header ID of the form
`#reason-DOCID` (e.g., `#reason-A`, `#reason-B`) with these HTML
parameters.

This is built on top of #126 (more intelligent collision detection for
`EXTENSION_AUTO_HEADER_IDS`).

Fix test case.

Fix a minor issue in expected anchor after recent PR. The tests were written before the improvement that squashes non-alphanumeric characters into a single dash, and does not include dashes at the beginning and end. This updates the test case to match that behavior so that tests pass and Travis is green.

Add support for angled, double quotes

The flag `HTML_SMARTYPANTS_ANGLED_QUOTES` combined with `HTML_USE_SMARTYPANTS` configures rendering of double quotes as angled left and right quotes (« »).

The SmartyPants documentation mentions a special syntax for these, `<<>>`, a syntax neither pretty nor user friendly.

Typical use cases would be either or, or combined, but never in the same document. As an example would be a person from Norway; he has a blog in both English and Norwegian (his native tounge); he would then configure Blackfriday to use angled quotes for the Norwegian section, but keep them as reqular double quotes for the English.

If the flag `HTML_SMARTYPANTS_ANGLED_QUOTES` is not provided, everything works as before this commit.

Minor cleanup.

Apply gofmt on html.go.
Apply goimports-compatible formatting on block.go (space between standard and third party imports).
Move Travis build status image in a more pleasing, common location.
Remove "Markdown pretty-printer output engine" from TODO steps; this is already done in markdownfmt.
Remove unneeded trailing whitespace in README.