avatar

Writing a markov-chain IRC bot in Go

Note : The original idea was not mine. While looking for tutorials on markov chains, I stumbled across this article which is great and helped me a lot understanding what was actually going on.

For the past few years, I've been aggregating a quite a lot of IRC logs and I was wondering what to do with those. Markov chains looked like a pretty cool principle so I decided to work on that for a few days and see if I could use those logs to create random sentences from them.

A basic Go bot

Configuration file handling and writing the base of the bot

First of all, let's start by creating the conf package (conf being short for configuration obviously). It's quite a handy little snippet of code that I usually include in all my projects that needs configuration files. The configuration file format is yaml.

Now create a new folder called conf, and edit conf.go.

// project/conf/conf.go
package conf

import (
    "io/ioutil"
    "log"

    "gopkg.in/yaml.v2"
)

// Configuration is the main struct that represents a configuration.
type Configuration struct {
    Server      string
    Channel     string
    BotName     string
    TLS         bool
    InsecureTLS bool
}

// C is the Configuration instance that will be exposed to the other packages.
var C = new(Configuration)

// Load parses the yml file passed as argument and fills the Config.
func Load(cp string) error {
    conf, err := ioutil.ReadFile(cp)
    if err != nil {
        return fmt.Errorf("Conf : Could not read configuration : %v", err)
    }
    if err = yaml.Unmarshal(conf, &C); err != nil {
        return fmt.Errorf("Conf : Error while parsing yaml : %v", err)
    }
    return nil
}

This package is intented to work that way : Load a configuration file, and fill a C instance (which is exposed outside the package) of the Configuration struct. Now create a new file at the root of your project and name it conf.yml. The configuration file must contain the following information :

botname: my-bot-nickname
server: irc.freenode.net:6667
channel: #my-awesome-channel
tls: false
insecuretls: false

I'd advise you not to version that file as it may contain information about the channels you go to and/or the servers you're connected on.

Now, back to the project/main.go file, we will load the configuration file named conf.yml placed at the root of your project. We will also initialise the IRC connection to suit your configuration file. Note that in the conf.yml file, you'll also define whether or not to use TLS. If you decide to use TLS, don't forget to change the port of the server in the server variable. Also if the server you're connecting to doesn't have a valid certificate or whatever, set the insecuretls variable to true.

// project/main.go
package main

import (
    "crypto/tls"
    "log"

    "github.com/yourusername/project/conf"
    "github.com/thoj/go-ircevent"
)

func main() {
    var err error

    // Load the configuration.
    if err = conf.Load("conf.yml"); err != nil {
        log.Fatal(err)
    }

    // Initialize the bot and setup the TLS parameters if needed
    ib := irc.IRC(conf.C.BotName, conf.C.BotName)
    if conf.C.TLS {
        ib.UseTLS = true
        if conf.C.InsecureTLS {
            ib.TLSConfig = &tls.Config{InsecureSkipVerify: true}
        }
    }

    // Connect to the server
    if err = ib.Connect(conf.C.Server); err != nil {
        log.Fatal(err)
    }

    // On connection to the server, automatically join the configured channel
    ib.AddCallback("001", func(e *irc.Event) {
        ib.Join(conf.C.Channel)
    })

    // Callback to execute when a message is received either on the channel or
    // directly to the bot (query/msg for example)
    ib.AddCallback("PRIVMSG", func(e *irc.Event) {
        m := e.Message()
        log.Printf("I just received : '%v'", m)
    })
    ib.Loop()
}

The code is commented and should allow you to understand what's going on. First we load the conf.yaml file and exit the program if there is an error. Then we initialize the irc bot by calling successive functions. You can already compile your bot and start it. Once it receives a message it will just print it out in the console it was launched in. Nothing special.

Your project should now look like this :

project
├── conf
│   └── conf.go
├── conf.yml
└── main.go

Markov chains !

Finally ! The base of our bot is ready so now we can switch to the more interesting part which is the ellaboration of the markov chain. Once again, I did not write all the code by myself and took a large part of the code from here which is an example of how to implement markov chains in Go. So let's get started. Let's create a new package named markov and edit the file markov.go :

// project/markov/markov.go
package markov

import (
    "math/rand"
    "strings"
    "time"
)

// PrefixLen is the number of words per Prefix defined as the key for the map.
const PrefixLen = 2

// MainChain is the chain that will be available outside the package.
var MainChain *Chain

// Prefix is a Markov chain prefix of one or more words.
type Prefix []string

// String returns the Prefix as a string (for use as a map key).
func (p Prefix) String() string {
    return strings.Join(p, " ")
}

// Shift removes the first word from the Prefix and appends the given word.
func (p Prefix) Shift(word string) {
    copy(p, p[1:])
    p[len(p)-1] = word
}

// Chain contains a map ("chain") of prefixes to a list of suffixes.
// A prefix is a string of prefixLen words joined with spaces.
// A suffix is a single word. A prefix can have multiple suffixes.
type Chain struct {
    Chain map[string][]string
}

// Build builds the chain using the given string parameter
func (c *Chain) Build(s string) {
    p := make(Prefix, PrefixLen)
    for _, v := range strings.Split(s, " ") {
        key := p.String()
        c.Chain[key] = append(c.Chain[key], v)
        p.Shift(v)
    }
}

// Generate returns a string of at most n words generated from Chain.
func (c *Chain) Generate() string {
    p := make(Prefix, PrefixLen)
    var words []string
    for {
        choices := c.Chain[p.String()]
        if len(choices) == 0 {
            break
        }
        next := choices[rand.Intn(len(choices))]
        words = append(words, next)
        p.Shift(next)
    }
    return strings.Join(words, " ")
}

// NewChain returns a new Chain with prefixes of prefixLen words.
func NewChain() *Chain {
    return &Chain{make(map[string][]string)}
}

// Init initializes the markov chain
func Init() {
    rand.Seed(time.Now().UnixNano())
    MainChain = NewChain()
}

Here things are getting a bit complicated. As usual the code is already commented but let's explain a bit more the concept here. What will actually happen when we build the markov chain ?

> "Hello."
{" ": ["Hello."]}
> "Hello !"
{
    " ": ["Hello.", "Hello"], 
    "Hello": ["!"]
}
> "Hello World !"
{
    " ": ["Hello.", "Hello", "Hello"],  
    "Hello": ["!", "World"], 
    "Hello World": ["!"]
}

As you can see, each time a word or a sentence is typed in, it will modify the chain to include all the possible changes and paths. Then, when we will as the chain to generate a new sentence from the previous ones, it will walk through the chain, performing a random operation on each node. Of course things are starting to get interesting once the chain gets a bit more complicated than these three "Hello"s. As you can see, each node is naturally weighted by repetition. There is a higher chance that the first word will be "Hello" than "Hello.". Also there is a 50% chance that the said "Hello" will turn into a "Hello World !" or "Hello !".

Integrating the markov chain

Let's edit the project/main.go file, and more specifically the PRIVMSG callback so that it doesn't just log the message it has received, but also update the markov chain.

// project/main.go

    // Place this right after loading the configuration :
    markov.Init()

    // Callback to execute when a message is received either on the channel or
    // directly to the bot (query/msg for example)
    ib.AddCallback("PRIVMSG", func(e *irc.Event) {
        m := e.Message()
        if strings.HasPrefix(m, "!") {
            if strings.HasPrefix(m, "!mk") {
                ib.Privmsg(conf.C.Channel, markov.MainChain.Generate())
            }
        } else if strings.HasPrefix(m, conf.C.BotName) {
            ib.Privmsg(conf.C.Channel, markov.MainChain.Generate())
        } else {
            markov.MainChain.Build(m)
        }
    })

When the bot receives a message, it will check if there is a specific command or its own name as the first word. If so, then it will generate a random sentence using the markov chain. If neither of these conditions are met, then it will append the received message to the chain.

Coupon on Udemy

A small note before you start the article. I was recently contact by the staff of udemy. They're offering a course about creating GUI using Python and Qt to the 50 first persons that register to the course with the code "MARKDOWNBLOG". This course normally costs $79 so I personnaly think it's a good deal.

Link : Python GUI Programming on Udemy
Coupon : MARKDOWNBLOG
Cost : Free for the first 50 persons (otherwise $79)

Mistune Parser

As you may (or may not) know, I recently switched from Misaka (GitHub) to Mistune (GitHub) mainly because Mistune is a pure python markdown parser. It means that it is easier to declare new grammars (and new behaviors) than modifying Misaka itself (which, just as a reminder, is a binding for Sundown, markdown engine written in C). I needed to modify the parser's behavior because carado asked me for a MathJax support (see my previous update for mor information). It took me a really long time to figure out how I could do that using Mistune, because the documentation about adding additionnal behaviors isn't as clear as I expected.

MathJax Support

The Javascript library

First of all, I started by including the library in the base template for every blog. For now, the lib is loaded whenever you're reading a blog even if it doesn't use MathJax. Although that behavior may change as I intend to add article-specific settings. For example if you want to enable or disable some parsing options or modify the way some items are rendered. But that will come later.

<script type="text/x-mathjax-config">
    MathJax.Hub.Config({
      tex2jax: {inlineMath: [['$','$']]}
    });
</script>
<script type="text/javascript" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>

As you can see the delimiters for inline math is $. It's obviously easier to parse if the inline blocks are defined using $...$ and blocks are defined using $$...$$.

The mistune renderer

class MathBlockGrammar(mistune.BlockGrammar):
    block_math = re.compile(r"^\$\$(.*?)\$\$", re.DOTALL)
    latex_environment = re.compile(r"^\\begin\{([a-z]*\*?)\}(.*?)\\end\{\1\}", re.DOTALL)


class MathBlockLexer(mistune.BlockLexer):
    default_rules = ['block_math', 'latex_environment'] + mistune.BlockLexer.default_rules

    def __init__(self, rules=None, **kwargs):
        if rules is None:
            rules = MathBlockGrammar()
        super(MathBlockLexer, self).__init__(rules, **kwargs)

    def parse_block_math(self, m):
        """Parse a $$math$$ block"""
        self.tokens.append({
            'type': 'block_math',
            'text': m.group(1)
        })

    def parse_latex_environment(self, m):
        self.tokens.append({
            'type': 'latex_environment',
            'name': m.group(1),
            'text': m.group(2)
        })


class MathInlineGrammar(mistune.InlineGrammar):
    math = re.compile(r"^\$(.+?)\$", re.DOTALL)
    block_math = re.compile(r"^\$\$(.+?)\$\$", re.DOTALL)
    text = re.compile(r'^[\s\S]+?(?=[\\<!\[_*`~$]|https?://| {2,}\n|$)')


class MathInlineLexer(mistune.InlineLexer):
    default_rules = ['block_math', 'math'] + mistune.InlineLexer.default_rules

    def __init__(self, renderer, rules=None, **kwargs):
        if rules is None:
            rules = MathInlineGrammar()
        super(MathInlineLexer, self).__init__(renderer, rules, **kwargs)

    def output_math(self, m):
        return self.renderer.inline_math(m.group(1))

    def output_block_math(self, m):
        return self.renderer.block_math(m.group(1))


class MarkdownWithMath(mistune.Markdown):
    def __init__(self, renderer, **kwargs):
        if 'inline' not in kwargs:
            kwargs['inline'] = MathInlineLexer
        if 'block' not in kwargs:
            kwargs['block'] = MathBlockLexer
        super(MarkdownWithMath, self).__init__(renderer, **kwargs)

    def output_block_math(self):
        return self.renderer.block_math(self.token['text'])

    def output_latex_environment(self):
        return self.renderer.latex_environment(self.token['name'], self.token['text'])

Let's start by wondering what we're trying to achieve here. MathJax is a javascript library that will read the content of the page it is included in and replace everything that is between $ or $$ delimiters with custom html/css (or even svg) to display beautiful math. This means that the only thing we're trying to achieve there is just to not parse what's between those delimiters.

Now this is getting a bit complicated here. Mainly due to how Mistune works, you first need to define a grammar class (MathInlineGrammar and MathBlockGrammar) which mainly consist of regular expressions for the things to match in the markdown text. We then declare the lexers that will use the grammar defined earlier (MathInlineLexer and MathBlockLexer)

We'll then define a new markdown parser that will be able to understand and use the two lexers defined above (MarkdownWithMath).

Syntax Highlighter

This part is quite similar to the article I wrote about how to do a markdown syntax highlighter with Misaka. The code doesn't change much.

class HighlighterRenderer(mistune.Renderer):

    def block_code(self, code, lang=None):
        if not lang:
            lang = 'text'
        try:
            lexer = get_lexer_by_name(lang, stripall=True)
        except:
            lexer = get_lexer_by_name('text', stripall=True)
        formatter = HtmlFormatter()
        return "{open_block}{formatted}{close_block}".format(
            open_block="<div class='code-highlight'>" if lang != 'text' else '',
            formatted=highlight(code, lexer, formatter),
            close_block="</div>" if lang != 'text' else ''
        )

Now let's add the support for MathJax and centered images in our renderer :

class HighlighterRenderer(mistune.Renderer):

    def block_code(self, code, lang=None):
        if not lang:
            lang = 'text'
        try:
            lexer = get_lexer_by_name(lang, stripall=True)
        except:
            lexer = get_lexer_by_name('text', stripall=True)
        formatter = HtmlFormatter()
        return "{open_block}{formatted}{close_block}".format(
            open_block="<div class='code-highlight'>" if lang != 'text' else '',
            formatted=highlight(code, lexer, formatter),
            close_block="</div>" if lang != 'text' else ''
        )

    def table(self, header, body):
        return "<table class='table table-bordered table-hover'>" + header + body + "</table>"

    def image(self, src, title, text):
        if src.startswith('javascript:'):
            src = ''
        text = mistune.escape(text, quote=True)
        if title:
            title = mistune.escape(title, quote=True)
            html = '<img class="img-responsive center-block" src="%s" alt="%s" title="%s"' % (src, text, title)
        else:
            html = '<img class="img-responsive center-block" src="%s" alt="%s"' % (src, text)
        if self.options.get('use_xhtml'):
            return '%s />' % html
        return '%s>' % html

    # Pass math through unaltered - mathjax does the rendering in the browser
    def block_math(self, text):
        return '$$%s$$' % text

    def latex_environment(self, name, text):
        return r'\begin{%s}%s\end{%s}' % (name, text, name)

    def inline_math(self, text):
        return '$%s$' % text

Now that we are all set, we can initialize the renderer like this :

markdown_renderer = MarkdownWithMath(renderer=HighlighterRenderer(escape=False))

Centered images and MathJax support !

Baby

MarkDownBlog now officially supports MathJax. It also works when you preview your mardkown on the new/edit page.
Oh also, the images are now alway centered.
Oh and I changed the markdown engine from Misaka (GitHub) to Mistune (GitHub) as it is easier to declare a new grammar and/or new blocks in Mistune. I'll write an article later about how to achieve the MathJax parsing + syntax highlighter and custom image renderer using Mistune.


MathJax Block

In order to create a MathJax block, the syntax is $$ your mathjax code here $$. You can see an example below :

$$ état(u) = \sum_{p \in POP(u)} (\int_{REL(u)} DDL(p,u,x)^{r} dx)^{i} $$

MathJax Inline Block

In order to create an inline MathJax block, the syntax is $ your mathjax code here $. Using that syntax you can put math expressions in the middle of a sentence. You can see an example below :

When $a \ne 0$, there are two solutions to $ax^2 + bx + c = 0$ and they are : $$x = {-b \pm \sqrt{b^2-4ac} \over 2a}.$$

(Don't hesitate to click on the upper-right eye icon to see the raw markdown)