Rosio Pavoris a blog

#!/usr/bin/python2

import sys
import bz2

def classify(text, langs=('english', 'german', 'french')):
    results = {}
    for lang in langs:
        with open(lang + '.txt') as f:
            corpus = f.read()

        compressed = len(bz2.compress(corpus))
        results[lang] = len(bz2.compress(corpus + text)) - compressed

    return sorted(results, key=results.__getitem__)

if __name__ == '__main__':
    print "Most likely %s." % classify(sys.stdin.read())[0].capitalize()

$ wget -qO - http://www.gutenberg.org/ebooks/31469.txt.utf8 | ./classific.py
Most likely English.
$ wget -qO - http://www.gutenberg.org/ebooks/22367.txt.utf8 | ./classific.py
Most likely German.
$ wget -qO - http://www.gutenberg.org/ebooks/4968.txt.utf8 | ./classific.py
Most likely French.

Alright, we’ve covered search trees in some detail, and they work great for problems where we have clear states and rules of production to move from one state to the next. Sometimes that’s not a very convenient way to state a problem, though, and a more natural way to think about things is as a bunch of variables which can take values in a certain domain, and a number of constraints which describe the relationships of these variables to each other.
The canonical example here is Dijkstra’s eight queens problem. However, that’s been done to death, so let’s instead have two queens and seven knights, and instead of the usual 8×8 chess board, let’s have a 6×6 one.

Read the rest of this entry »

I’ve written this, so I might as well share it.

In my post on the Mandelbrot set earlier, I mentioned the Julia sets of the quadratic polynomial f_c(z) = z² + c where c is a given (constant) complex number and z are the points of the complex plane. Because I wanted to visualise how those Julia sets changed as c varied, I’ve written a short program to do that for me.
You can find it here. As usual, you’ll need Allegro, and the compilation instruction is on the first line.

What it does is take two complex numbers as parameters, plus the number of steps it should take to go from the first to the second. At each step, it will calculate and display the Julia set of the quadratic polynomial with that complex number as c, and hopefully your computer is fast enough that the successive Julia sets look like an animation.
For example, if you invoke it as:

./julia -0.8 -1 -0.8 1 200

you’ll see the following:

Though probably not at the same speed. I’ve made no effort to maintain a certain frame rate; the whole thing moves as quickly as your CPU can keep up, because I just wanted a visualisation of how Julia sets change, not a screensaver. If it’s moving too slowly for you, you can try reducing the number of steps, or lowering the numbers in the ZOOM or ITERS #defines, though the first one will make the image smaller and the second will make it darker. If you aren’t interested in the window title, you can also remove the snprintf and set_window_title steps for a significant speed-up.
If it’s too fast, you can do the reverse, or you can build in a delay with Allegro’s install_timer and rest, or POSIX’s usleep or nanosleep.

(Once it’s done, it will just pause at the last Julia set. Press any key to close it. If you want to close it before it’s done, you’ll have to kill it manually.)

The interesting points to explore are the ones inside the Mandelbrot set, as anything else will just be Fatou dust (though you’ll probably still be able to see it because of the grey). For those points, the salient area is the one within 2 unit lengths of the origin, which is why the field displayed ranges from (-2, 2) in the top left corner to (2, -2) in the bottom right (or probably (-2, -2) to (2, 2), I don’t remember). If you need a bigger plane, replace all instances of ZOOM * 4 with ZOOM * (bigger number), and all instances of ZOOM * 2 with ZOOM * (half of bigger number) (if you want to keep the origin in the center of the window).
If you actually want to save the animations, man 3alleg save_bitmap and assemble the images yourself in something like the GIMP. I initially started out doing it this way, but animated GIFs get really big really quickly, so I went with this instead.

Enjoy.

People who take an active interest in AI are quite unlikely to have very many friends, so it should come as no surprise that trying to get computers to play games has always been a popular subfield of AI. Traditionally that game has mostly been chess, but I feel chess has a grinding tedium to it, so we’re going to look at tic-tac-toe instead, because that at least has the benefit of being over quickly.

Read the rest of this entry »

Last time we looked at how to solve the eight puzzle using the hill climbing algorithm, which gave us a result much more quickly than a blind depth-first search did, but we wondered if the solution we found was the best we could do, and we asked if there was a way to use heuristics to find not just a solution, but the best solution. Today, we’ll see that there is, and it’s actually really straightforward.

Read the rest of this entry »

(This post assumes you read the previous one.)

Today we’ll be looking at the hill climbing algorithm, which is just a plain old depth-first search with heuristics added.
“Heuristics” is a fancy word (from the Greek εὑρίσκω, “I discover”) for a very simple concept. In the context of search trees, it simply means that at a every node, you’re going to look at each possible branch, and take the one that looks the most promising first, instead of just one at random. “Most promising” can be a tricky concept, though.⁰
Our river-crossing example isn’t necessarily the best one to demonstrate the concept, so let’s go with another classic: the 8 puzzle.

Read the rest of this entry »

A decent proportion of my readers are noobie programmers or people who aren’t in a position to receive a formal CS education, so I thought I’d cover the basics of a fundamental concept most people cover in their first semester of algorithms or AI today: search trees. The fact that my college considers this to be third-year material so advanced they cannot in good faith make the class compulsory is neither here nor there.

Consider the famous problem of the farmer who wants to cross a river with his fox, goose, and grain, though the only boat can only carry himself and one of these three possesions. Ignore for a moment why a farmer would own a fox, and let’s stretch credibility a bit more by assuming that while the fox and goose are well-trained enough not to wander off in the absence of the farmer, they are not trained not to eat the goose or the grain, respectively, in said absence. How can he safely get to the other side without losing his goose or grain?

Read the rest of this entry »

I’m in a class called Netwerkbeheer (Network Management), which spans two semesters and is a transparent excuse to peddle CCNA certifications. As a result, I spend a lot of time playing with Cisco routers and switches, and one of the many, many things that annoy me about Cisco’s IOS is their cavalier attitude towards security and cryptosystems. A particularly egregious example of this is Cisco’s type 7 encryption.
If you’ve ever configured a Cisco router, you’ve probably encountered it. When the misleadingly named service password-encryption is running, setting a password with the enable password command “encrypts” the password, so that when you issue the show running-config command, you’ll see a line like

enable password 7 08314940000A

instead of the plaintext password, which you’d see if the so-called “password-encryption” was turned off.
Type 7 “encryption” manifests itself in a few other places, including in FTP passwords and various routing protocol authentication passwords.

Type 7 has been known to be broken for a decade and a half now,⁰ but people continue to use it, almost always for bad reasons.^1,2 To drive home just how broken type 7 is, let’s look at it in detail.

The general form of the type 7 “ciphertext” is (0[0-9]|1[0-5])([0-9A-F]{2})+. Some experimenting finds that the length of the “ciphertext” is always twice the length of the plaintext, plus two. Can you guess why?

The “encryption” key is always a number in the range 0-15, which would be easy enough to bruteforce, but that turns out to be unnecessary, since it’s provided (in decimal form) as the first two characters of the “ciphertext”.
That key determines the starting point in a table of twenty-six secondary keys (which, incidentally, is dsfd;kfoA,.iyewrkldJKDHSUB; I don’t know why the table has 26 entries instead of 16), which are XORed in turn with the characters in the plaintext. If the key is, say, 7, the first character in the plaintext is XORed with the seventh character in the table, the second character in the plaintext is XORed with the eighth character in the table, the third with the ninth, &c.
Each resulting character is then converted to two hexadecimal digits (the input can only be ASCII, of course) and appended to the ciphertext.

And that’s seriously all there’s to it. The result is a “cipher” that’s either slightly less or slightly more secure than writing out your passwords in permanent marker on the outside of the door of the server room, depending on how you manage your configuration files.
Because I know this is going to be an issue at some point, I’ve written a simple utility that encrypts and decrypts passwords using type 7, which you can find here.

You’d think this would be a moot point because people should realise their configuration files are sensitive information, but people are, of course, idiots. In that sense, type 7 isn’t just worthless, but actively harmful, because it gives people a false sense of security.

⁰ http://insecure.org/sploits/cisco.passwords.html

¹ The original intent of type 7 was apparently to foil shoulder-surfers, who might see your configuration file as it scrolls by on your screen. Cisco’s official stance (now) is that if security is an issue, the router configuration file itself should be treated as vulnerable data, not just the passwords that may or may not be displayed in it. That would be fair enough, if it wasn’t at odds with Cisco’s default way of saving and loading configuration files, which is through plain TFTP over the regular network, with no options for encryption of either the config or the passwords themselves. But, you know.
(The claim that type 7 is so weak because the router has to be able to reverse it is bullshit, of course. At most it’s true for PAP authentication, but anyone who considers PAP passwords secret information has no business being anywhere near a router.)

² Cisco themselves now advise against using it, instead suggesting people use type 5, which isn’t encryption, but just hashing with MD5. Which is also broken, of course. The CCNA materials also state that at least type 7 is “better than no encryption”, but I’d argue that it’s worse, because its security is equivalent to plaintext, while also giving idiot network admins the impression that it’s not.
I’m told a type 6 exists now, which is based on AES and supposed to be better. AFAIK our routers don’t support it, and I’m not holding my breath either way.

There’s been some interest in tripcode crackers lately, so I thought I’d write on in Haskell. I mentioned this before, but I’ve improved it a bit since.
I’ll be discussing the code step by step in this post. By the end, we should have a working application that takes a POSIX regex as an argument, and then outputs tripcodes that match it.

If everything goes right, this post should be literate Haskell, but I can’t promise that it’ll actually work, what with WordPress being what it is.
Let’s get started.

The tripcode algorithm is relatively straightforward: the input is converted to SJIS, there are some XML character entity substitutions, then a salt is calculated, and the whole thing is passed to Unix crypt.
We won’t be dealing with SJIS conversions, since our input will be ASCII only, which (with one exception) is a subset of SJIS anyway, and our target board will probably be Shiichan or Futallaby anyway, neither of which even does it. We also won’t be writing our own crypt implementation in Haskell, so we’ll have to get it from a C library. To do this, we use the Foreign Function Interface language extension, like so:

> {-# LANGUAGE ForeignFunctionInterface #-}

The usual boilerplate:

> module Main (main) where

> import Char (chr, ord)
> import Data.List (inits)
> import Foreign.C
> import System (getArgs, exitFailure)
> import System.IO.Unsafe (unsafePerformIO)
> import Text.Regex.Posix ((=~))

It’ll become clear why we need all of these as we go along.
Let’s import our C crypt:

> foreign import ccall unsafe "DES_crypt" crypt :: CString -> CString -> CString

We’re using OpenSSL’s implementation, because GNU crypt is slow as fuck, and this thing is going to be slow enough as it is. That bit of code is just saying to take a C function named DES_crypt from a linked library, and expose it as a Haskell function named crypt, with the listed type signature.

We said the tripcode algorithm involves some XML character entity substitutions, so let’s write a function to do that. The canonical algorithm just escapes ", <, and >. If yours escapes more (or fewer), just add (or remove) them here.

> xmlescape :: String -> String
> xmlescape [] = []
> xmlescape (x:xs) = case x of
>     '"' -> (++) """ $ xmlescape xs
>     '<' -> (++) "&lt;"   $ xmlescape xs
>     '>' -> (++) "&gt;"   $ xmlescape xs
>     otherwise -> (:) x   $ xmlescape xs

Straightforward enough.
Next, we’ll need a function to generate the salt. crypt‘s salt is string of length 2 whose characters are in the range [a-zA-Z0-9.]. The tripcode function obtains this by appending H.. to the input and taking the second and third characters, performing some transformations to ensure they’re in the allowed range:

> salt :: String -> String
> salt t  =
>     map f . take 2 . tail $ t ++ "H.."
>     where
>         f c
>             | notElem c ['.'..'z'] = '.'
>             | elem c [':'..'@'] = chr $ ord c + 7
>             | elem c ['['..'`'] = chr $ ord c + 6
>             | otherwise = c

Now we’re ready for the actual tripcode.
This will happen in the IO monad, because we’re dealing with the FFI. We don’t have to use unsafePerformIO to escape from it, but to keep our algorithm conceptually cleaner, we will anyway.¹

> tripcode :: String -> String
> tripcode tr = unsafePerformIO $ do
>     trip <- newCString t
>     salt <- newCString $ salt t
>     peekCString (crypt trip salt) >>= return . drop 3
>     where t = xmlescape tr

Great. Now that we have everything we need to calculate the tripcode for a given input, we need a way to generate inputs.
Let’s first start by specifying the characters we want to allow. We’re leaving out # and ! because they’re usually separator characters and as such illegal in tripcodes (though most boards will allow !), and \ because that’s the aforementioned ASCII/SJIS exception.

If you’re targetting a specific algorithm and you know which characters are fine and which aren’t, you can edit this. For Shiichan, for instance, the only forbidden character is #.

> allowedChars :: [Char]
> allowedChars = filter (\c -> c `notElem` "#!\\") [' '..'~']

You can avoid the call to xmlescape altogether by disallowing characters that would be escaped, of course; that’s what I did for my C implementation, too. That will obviously reduce your search space, though.

Now we can use these characters to generate our (infinite) list of inputs:

> ins :: [String]
> ins = (inits . repeat) allowedChars >>= sequence

This will generate all possible combinations, from "" to strings of infinite length. Since crypt disregards input over eight characters wide, it will generate far more combinations than you’ll ever need. Since it will take almost forever to even get up to that point, though, the issue is kind of moot.²

Now that we have our infinite input list, we can turn this into the input/output combinations we need; first we’ll team up each input with its corresponding output with zip, and then we’ll filter it with our regular expression. Obviously, this is where Haskell’s laziness is really handy:

> tripPairs :: String -> [(String, String)]
> tripPairs regex = filter (\(a, b) -> b =~ regex) $ zip ins $ map tripcode ins

It’s cute that (=~) is used for regular expressions. It’s also handy that it takes a string as its second argument and we don’t have to dick around with compiling regular expressions and what have you.
(=~) actually has a pretty complicated type signature, but its return value is a polymorphic value that converts into a boolean without complaints, so we don’t have to worry about that.

So what do we have now? We have our tripcode function, our list of inputs, and a function that filters this list based on a regex argument. We’re pretty much done. Now we only need the main function:

> main :: IO ()
> main = getArgs >>= f
>     where
>         f []       = putStrLn " USAGE: tripcode [regex]" >> exitFailure
>         f (arg:xs) = mapM_ (putStrLn . show) $ tripPairs arg

When no argument is passed on the command line, we display a short usage note and return failure, otherwise we generate our list of matching pairs, turn them into displayable strings, and then print them to stdout. Try it, it works!³

It’ll be slow as fuck, though,⁴ for a number of reasons. The first is that OpenSSL’s crypt, while much faster than GNU crypt, still isn’t very fast. The second is that Haskell’s Text.Regex.Posix is very slow (if you know how, I suggest you use another regex library; I went with Text.Regex.Posix because most casual Haskellers (which I am too) aren’t likely to have any of the others). The third is that it can only use one processor at a time, whereas most others are multithreaded or multiprocess affairs. The fourth is that a high-level language like Haskell is obviously going to be slower than tripcode crackers written in C and Sepples by moon people. The fifth is that I suck at Haskell.

Still, at least it’s written in the world’s leading fictional programming language.

Edit: Just to make it absolutely clear, because this post gets a lot of hits: you’d have to be an idiot to actually use this to look for tripcodes. Write your own cracker in C. It’s easy and will be much faster. (Or if you don’t want to, I wrote one that was posted elsewhere in the thread you got this link from.)

¹ If we didn’t, the type signature would be String -> IO String, of course.

² If you only want to check eight-character inputs, though, you can do something like ins = sequence . take 8 $ repeat allowedChars instead.

³ Copy/paste this post into a file named Tripfind.lhs, then compile it using ghc -lssl Tripfind.lhs

⁴ Slower than my C one almost by an order of magnitude, even after adjusting for the fact that my C one can use both of my processors. And since my C one is already slower than, say, Tripcode Explorer by an order of magnitude…

Inspired by rmuser’s Youtube videos on information theory (and specifically the one about Huffman encoding), I wrote a Python script to calculate a Huffman encoding for text.

It reads input from stdin (preferably in ASCII), calculates a Huffman mapping, and shows it to you. It also calculates how long the text would be if encoded with that mapping, and how many bytes you’ve saved compared to ASCII¹, which just uses a byte for each character, regardless of how frequently it’s used.

Here’s the result of running it on itself:

$ python huffman.py < huffman.py
Symbol	Freq	Encoding
' '	560	10
'e'	262	111
't'	138	1100
's'	134	1101
'r'	125	00001
'\n'	110	00011
'f'	105	00100
'n'	98	00110
'l'	96	00111
'a'	75	01100
'i'	72	01110
'o'	67	000000
'd'	61	000100
'.'	54	001010
'('	48	010000
')'	48	010001
'c'	46	010011
','	46	010100
'q'	45	010101
'm'	36	011011
'b'	35	011110
'u'	32	0000010
':'	30	0001010
'_'	26	0001011
'='	26	0010110
'y'	24	0010111
'p'	24	0100100
'h'	22	0101100
'g'	20	0101110
'['	11	01001011
'%'	11	01011010
'#'	11	01011011
'1'	10	01101000
'"'	10	01101001
'0'	9	01101010
"'"	9	01101011
']'	9	01111100
'F'	9	01111101
'\\'	8	000001110
'T'	5	010111100
'+'	5	010111101
'v'	5	010111110
'w'	5	010111111
'/'	4	0000011010
'R'	4	011111100
'3'	4	011111101
'x'	4	011111110
'k'	4	011111111
'I'	3	0100101000
'2'	3	0100101001
'j'	3	0100101010
'S'	3	0100101011
'>'	2	00000110000
'C'	2	00000110010
'A'	2	00000110011
'E'	2	00000111100
'B'	2	00000111101
'P'	2	00000111110
'H'	2	00000111111
'*'	1	000001100010
'!'	1	000001100011
'8'	1	000001101100
'-'	1	000001101101
'W'	1	000001101110
'L'	1	000001101111

Encoded message length: 12237 bits (1529.62 bytes)
This message contained 2634 characters. Huffman encoding saved 1104 bytes
compared to ASCII.

As you can see, the most frequently-used characters have the shortest encoding, while the rarest have the longest. I’m assuming that means it’s working the way it should.

Simple toy, but it beats paying attention in class.

¹ If the input isn’t in ASCII, it should still come up with a correct mapping, but that last bit will be off by a bit.

Prime numbers are ridiculously important in cryptography, and I recently found myself needing a way to generate them (for a toy implementation of the Diffie-Hellman key exchange algorithm). It keeps amazing me how few languages actually have libraries for generating prime numbers, or even just for primality testing.
Because I didn’t feel like thinking, and I didn’t need incredibly massive primes, I just wrote a quick implementation of the sieve of Eratosthenes, which is an interesting enough algorithm that I’d thought I’d share it (this is first-year stuff even for us, but I guess I have non-programmers in my audience).

The basic idea is that you start with a list of integers from 2 up to the number you want to find the largest prime smaller than. You then take the smallest number in this list (2), and cross off all multiples of this number (except for 2 itself). You repeat this for every other number, in order, and you’ll be left with just a list of primes.

Wikipedia has a visualisation which could be clearer but is still pretty good:

The implementation (in Python) is dead simple:

def sieve1(n):
    l = range(n)
    l[0], l[1] = None, None
    for i in xrange(int(__import__('math').sqrt(n)) + 1):
        if l[i] is not None:
            j = i * 2
            while j < n:
                l[j] = None
                j += i
    return filter(None, l)

We could just use range(2, n) and then work with an offset, but the convenience of having the indices equal to the values at those indices is too useful to complain about the extra memory required for two additional integers. Note also the obvious optimisation of only going up to sqrt(n) + 1 (no apologies for the unorthodox import).
filter(None, l) gets rid of the Nones in our list. Passing None as the function argument just applies the identity function, which has the effect of getting rid of None, False, 0, and empty lists, tuples, and dictionaries (anything which would evaluate to False in an if statement).

This is nice, but it does have the disadvantage of having to generate the whole list every time you want more primes. You can’t just input a list of pregenerated primes and have it continue from there.
The fix is easy enough:

def sieve2(n, primes = [2, 3]):
    if n <= primes[-1]:
        return filter(lambda m: m <= n, primes)
    
    offset = primes[-1] + 1
    l = range(offset, n)
    
    max = int(__import__('math').sqrt(n))
    for p in primes:
        if p > max:
            break
        i = p * 2 - offset
        while i < 0:
            i += p
        while i < len(l):
            l[i] = None
            i += p
    
    for k in xrange(max - offset + 1):
        if l[k] is not None:
            i = k * 2 + offset
            while i < len(l):
                l[i] = None
                i += l[k]
    
    l = filter(None, l)
    primes.extend(l)
    
    return primes

This function uses memoisation, and takes advantage of a feature of Python’s function defaults which some believe to be a bug: if mutable default arguments are altered, they will remain altered for the next time the function is called.
This is a side-effect of the fact that functions have a single field that holds the default arguments (a tuple called func_defaults in the function’s namespace), which isn’t duplicated into a working copy when the function is called. Most default arguments will be things like integers and strings, which are immutable, so most people never even run into this. Being able to use it for memoisation is really handy, though. Alternatively, you could just use a global variable and access it from inside the function.

The algorithm is basically the same, except that if we’ve already generated the primes requested, we can just return them. Then we construct a list of all integers larger than our largest prime, and sieve it with our existing primes (now we really do have to work with an offset). Then we just sieve the rest classically. At the end, we add our newly sieved list to our old primes, and return that.
Next time we invoke the function, primes will still have the new primes. You can access them from outside the function at sieve2.func_defaults[0].

Evaluating sieve2(100) and then sieve2(1000) will be slower than just evaluating sieve1(1000), but it will be faster than evaluating sieve1(100) and then sieve1(1000). And if you happen to have a large number of pregenerated primes, you can save some more time.

The sieve of Eratosthenes is just one of a number of sieve-based prime number generators, but it’s the oldest one, and it compares favorably to modern improvements.

A classical cryptography library isn’t much of a toy to non-programmers, so in an effort to learn about Python’s Tkinter library, I wrote a quick-and-dirty GUI front-end to it:

I know it’s ugly, but that’s because it’s Tk. I think it’s meant to be ugly. Tell the Python dudes to use GTK+ bindings for Python’s standard toolset.
It works, though, even if you can’t paste text into the textboxes.

There are two ways to run this, and both require that you have Python installed (I still use 2.5.2, but I see no reason why it shouldn’t run under 2.6).
The first, and most obvious, is to copy all of that code into a file, and then copy all of this code into a different file called classicalcrypto.py. Put those into the same folder and just double-click on the first file to run it (if you aren’t using Windows, you’ll obviously need to chmod +x first).
The second and slightly more complicated way is to install the classical crypto module separately. Download and untar this file, and then install it by running the setup file like python setup.py install. Then just put this code into a file, and you won’t have to worry about keeping two files in the same folder.
If anyone has any problems getting it to work, let me know. I did try to package it using py2exe, but, unsurprisingly, py2exe is a piece of shit and that didn’t work.

Only most algorithms are included, because some have non-trivial key requirements; the four-square cipher, for example, requires two Polybius squares. I left the Solitaire cipher because it still works, for some reason, even though it shouldn’t. I wouldn’t trust it to be strong.
Atbash and ROT13 ignore the key you enter. Caesar and Rail Fence require it to be an integer (in Rail Fence’s case, higher than 1).
Keep in mind that Rail Fence is a permutation cipher, so trailing newlines are significant characters. For reasons I couldn’t be bothered to troubleshoot, Tk appends a newline when it updates the contents of the textbox, so if you just hit encrypt and decrypt in a row, you won’t get the original message back.

Specifically, a small classical cryptography library.
None of these things are very hard to implement (in fact, two of the hardest bits I haven’t even implemented myself), but it’s good practice for me and some people might hypothetically have a use for it.
I’ve taken a stab at documenting it, but I don’t know how Python documentation works, so forgive me if it’s useless.

Included functions are:

• atbash: The Atbash cipher. Optionally allows you to pass your own alphabet.
• bifid: Delastelle’s bifid cipher.
• caesar: The Caesar cipher. Optionally allows you to pass your own alphabet, preserves case and letters not in the alphabet.
• foursquare: The four-square cipher.
• hill: The Hill cipher, sans decryption function, because I honestly couldn’t figure out how to invert a matrix in a way that preserves integers (protip: it’s not the usual way; I wrote a Gauss-Jordan reduction function before realising that couldn’t work for most key matrices). If you know how to invert it yourself, you can do that and pass it as the key along with the ciphertext, and it’ll decrypt normally. Alternatively, you could tell me how to do it and I’ll fix it.
• keyword: The keyword cipher. Optionally allows you to pass your own alphabet, preserves case and letters not in the alphabet.
• playfair: The Playfair cipher. Note that the decryption doesn’t have a way to tell which characters were added during encryption and which were there in the plaintext (though it should be obvious unless your plaintext has a lot of Xs and Qs).
• railfence: The rail fence cipher. Encryption only, because it gave me a headache. Never mind, I added decryption. It’s not as elegant as I’d like, but it works.
• rot13: ROT13. Special case of the Caesar cipher.
• solitaire: Bruce Schneier’s solitaire cipher.
• solitaire_prng: The key generator for same.
• vigenere: The Vigenère cipher previously discussed.
• vigenere_autokey: Vigenère’s autokey cipher, also previously discussed.

Basically, everything in Wikipedia’s “classical ciphers” for which an algorithm was described, minus a few of the more boring ones.
There’s also a Polybius class, which allows for easy construction of Polybius squares, and methods for looking up things in them. It’s just a wrapper around a square matrix and a reverse look-up of the same.

The general principle behind these functions is that they take a message (which is a string) and a key appropriate to the cipher. To decrypt, you use the same function and pass an additional boolean decrypt, set to True.
Preservation of case and punctuation characters and the like is inconsistent, so if that’s important, you’re going to have to look at the code itself. This is because I am nothing if not easily bored.

Anyway, enjoy. I may add to this over time, so if there are ciphers you would like me to implement, let me know.

The Vigenère cipher is a more interesting cipher, in that it uses an actual key, and isn’t trivial to bruteforce. Central to the Vigenère cipher (and, in fact, most classical substitution ciphers) is the tabula recta:

Suppose that you have a message to encrypt starting with “Proletarians of Europe! The war has lasted more than a year.”, and a key “secret”. To encrypt your message, you line the plaintext and the ciphertext up like this:

And then you just look up vertical pairs in the table. The first letter to encrypt is P, so you go to the row marked P. The first letter to encrypt it with is S, so you look in the column S of the row P. The resulting ciphertext letter is H. You repeat this for the rest of the letters.
The resulting ciphertext will be:

Decrypting is the reverse of this: you go to the column corresponding to the key letter, find the letter H in that column, and then look at what row that letter is in.

Implementing this in Python turns out not to be very hard.
I actually constructed the tabula recta as a lookup table, mostly to see how hard it would be, but you don’t, of course, have to do that. All you need is a way to turn letters into numbers (A = 0, B = 1, &c.) and back, and encryption is as easy as:

ciphertext_letter =
    to_letter((to_number(plaintext_letter) + to_number(key_letter)) % 26)

Decryption just changes the + to a -.
This is also how I constructed the tabula recta. The to_letter function is easily accomplished by treating a string of the letters in alphabetical order as an array, and the for the to_number function I just used a reverse dictionary lookup of the same.
The tabula recta then takes literally three lines of code:

lookup = dict([(a, 0) for a in alpha])
for a in lookup:
    lookup[a] = dict([(b, alpha[(tonum[a] + tonum[b]) % 26]) for b in alpha])

You really don’t want to know how much longer that would be in Java.
A sample session:

$ ./vigenere.py
[E]ncrypt or [D]ecrypt? E
Phrase to encrypt? Proletarians of Europe! The war has lasted more than a year.
Strip whitespace and punctuation? (Y/n) Y
Key? secret
hvqcimsvkrrlgjglvhhivyipsvjrweswvvhfgvgkltfeavek
$ ./vigenere.py
[E]ncrypt or [D]ecrypt? D
Phrase to decrypt? hvqcimsvkrrlgjglvhhivyipsvjrweswvvhfgvg kltfeavek
Key? secret
proletariansofeuropethewarhaslastedmorethanayear

The Vigenère cipher is less easy to crack than the Caesar cipher, but not much.
If you know the length of the key, the ciphertext simply turns into so many Caesar-shifted ciphertexts, which may be subjected to frequency analysis. Though of course, we’ve already seen how great that turns out if you don’t have a massive amount of text to analyse.
If you don’t know the length of the key, but do have a huge ciphertext, you can look for repeating clusters of letters, which are likely to be bits of plaintext that happen to repeat at intervals equal to some multiple of the key length. If you have a bunch of different ones, you can find the largest common factor of the intervals, and this will likely be the key length. After this you can just use frequency analysis again. This method is known as the Kasiski examination, after the Prussian Friedrich Kasiski (though Charles Babbage also came up with it).
These things aren’t that hard to implement, but also not that interesting.

It should be clear, though, that the Vigenère cipher is a pretty weak one (it certainly doesn’t deserve the name «le chiffre indéchiffrable»), which is ironic, given that Blaise de Vigenère never actually came up with it, and the one he did come up with was actually quite a bit stronger.
Vigenère’s autokey cipher uses the same basic principle of the tabula recta, but instead of a repeating key, it’s going to use a short key to start with, and then for the rest of it use the plaintext.¹ Vigenère himself suggested using a single letter of the alphabet for the key, but that’s pretty stupid. Let’s instead use a full word, like so:

Again, not that hard to implement.
Rather than repeating, the key is now just the provided key plus the ciphertext, which saves a modulo operation in the lookup.
Decryption has an extra step, because every deciphered letter has to be added to the key while the decryption is in progress, but on the whole, it’s very similar to the other “Vigenère” cipher.

Breaking it is rather harder than breaking the other one, though. Frequency analysis isn’t really going to work, though if you use a single letter for the key, like Vigenère suggested, bruteforcing is trivial, and our own example is vulnerable to a dictionary attacks. Both of those are attacks against the key, though, not the cipher.
There are some relatively straightforward attacks possible, one of which is described in the Wikipedia article, and I’m working on those now.

More to come.

¹ You’ll remember that this basic principle also returns in a slightly different form in certain block cipher modes of operation in modern cryptography. It’s also used for some types of self-synchronising stream ciphers.

Alright, so it’s about programming again.
I’ve been meaning to write some tools to help me encrypt text using some classical ciphers, and also to break those ciphers. I’ve also been meaning to using Python more, because holy fuck Python is awesome.
Maybe it’s just the fact that I only really have any experience with seriously shit languages (Java, PHP, COBOL; I’ve used other languages, some of which didn’t suck, but never to the extent I’ve been forced to use those three), but after I wrote a Python script to deal with that ball algorithm in a third of the time it took me to write it in Java, which is probably the language I’ve used the most in my life, I redid some of the Project Euler problems, which I’d previously also mostly done in Java (some in Scheme), and I was amazed how much less painful it was. If this puts me on the same level as Randall Munroe I apologise.

Anyway.
The obvious place to start was with the Caesar cipher, which is the one where you take each letter and move it ahead X places in the alphabet. ROT13 is a Caesar cipher with the key set to 13.
There are two obvious attacks here: one is frequency analysis, where you count the number of letters and compare it to a chart of letter frequencies (the most common letter in the ciphertext is likely to be E if it’s an English (or Dutch, for that matter) text, for example); the other is bruteforcing, where you just go through all possible keys. This is possible because there are really only 25 possible keys, plus one (26, or 0) which produces a ciphertext identical to the plaintext.

Frequency analysis seemed overkill, since bruteforcing is so easy and much more fool-proof, especially for short ciphertexts, so I went with an algorithm that encrypts the ciphertexts with all 26 possible keys, and uses a dictionary file to count the number of dictionary words in the resulting texts. The one with the highest number of dictionary words is likely to be the plaintext, so that’s then returned.
You can find the code here.

A sample session (text in bold is entered by the user):

$ ./caesar.py
Phrase to encrypt? All that we see or seem is but a dream within a dream.
Shift by? 12
Encrypted: Mxx ftmf iq eqq ad eqqy ue ngf m pdqmy iuftuz m pdqmy.
Decrypting...
Decrypted: All that we see or seem is but a dream within a dream.
Success!

It’s slow as fuck because of the way it uses the dictionary file, of course (but that shouldn’t be too hard to optimise), and because the decrypting function doesn’t take punctuation into account it won’t count words that might actually be in the dictionary, which can be a problem for really short sentences with a lot of punctuation:

$ ./caesar.py
Phrase to encrypt? A ghost!
Shift by? 11
Encrypted: L rszde!
Decrypting...
Decrypted: L rszde!
Fail!

It also doesn’t work if punctuation and—more importantly—spaces have been stripped, though, as was usual for real-life applications of this cipher, and that’s more of a problem.
The easiest way around this is to let the user identify which the decrypted string is, of course, or you could look at random parts of the text to try to guess where words end and see if anything matches a dictionary word, but that’s a lot of work. It may be easier to solve this using frequency analysis.

Like I said, frequency analysis counts the number of letters in a word and guesses what they’re likely to map to based on how often they normally appear in a text written in the same language as the plaintext.
This sort of attack can break not just Caesar ciphers, but any simple substitution cipher, though for a polygraphic one (that is, one that replaces groups of letters rather than individual letters), you’ll need a polygraph frequency chart, of course. It doesn’t do too well with polyalphabetic substitution ciphers (where a plaintext glyph can map to multiple ciphertext ones), but given enough ciphertext, it can break Vigenère ciphers as well.

I wrote a function that does just this: it counts the number of times each letter appears in a ciphertext (ignoring punctuation and spacing, if any), and then naively replaces the letters in the ciphertext based on a letter frequency chart.
It works a lot faster than having to look up words in a dictionary, but the main problem seems to be finding a good letter frequency chart. ETAOIN SHRDLU is a famous sequence, of course, and if you look at the source you’ll see I used that first. My ciphertext was this speech by Karl Popper, ROT13′d (because it was the first large block of text that I could find that had been written in modern English).

With Linotype’s ETAOIN SHRDLU, the “decrypted” version started off like this:

Dr. Rujior, Dr. Mufg, Lfmtuz fgm Nugiludug

Io ipu 3rm Afjeliq oa Dumtjtgu oa ipu Jpfrluz Egtvurztiq fgm tiz Mufg, Hroauzzor Pouzjpl, fgm fll oa qoe kpo pfvu jodu puru iomfq, T ktzp io uxhruzz dq muuhlq auli ipfgwz aor ipu nrufi pogoer kptjp qoe fru jogaurrtgn og du.

Then I tried the one used by Samuel F. B. Morse to determine which letters should map to the least amount of dots and dashes in his code: E IT SAN HURDM WGVLFBK OPJXCZ YG.
This was the result:

Kr. Ruohlr, Kr. Dufv, Wfdsuc fvd Guvhwukuv

Hl heu 3rd Bfoawhz lb Kudsosvu lb heu Oefrwuc Avsnurcshz fvd shc Dufv, Mrlbucclr Elucoew, fvd fww lb zla iel efnu olku euru hldfz, S isce hl ujmrucc kz duumwz buwh hefvxc blr heu grufh elvlar iesoe zla fru olvburrsvg lv ku.

And finally, the one given by Robert Edward Lewand in Cryptographical Mathematics:

Dr. Rutlor, Dr. Wufg, Jfwiuc fgw Nugljudug

Lo lhu 3rw Aftejlq oa Duwitigu oa lhu Thfrjuc Egivurcilq fgw ilc Wufg, Proauccor Houcthj, fgw fjj oa qoe mho hfvu todu huru lowfq, I mich lo uxprucc dq wuupjq aujl lhfgkc aor lhu nrufl hogoer mhith qoe fru togaurrign og du.

In the first case, it got 47 letters right (not counting how many letters out of the 26 it matched up right, just counting how many letters in our “decrypted” paragraph match the original, ignoring spaces, punctuation, and numbers (out of 209)) and managed to turn “honour” into “pogoer”.
The second only got 25 right. For shame, Mr. Morse.
The third got a whopping 67 right, though “hogoer” isn’t really a word.

Unsurprisingly, the cryptography frequency chart came closest, but none of them did particularly well. Presumably they’d do better with more text, but it still surprised me.
It’s particularly surprising how few vowels they got right: both ETAOIN SHRDLU and the cryptography one got O, and the cryptography one also got I. And that’s it. Morse didn’t get a single one, and none of them got A, E, or U.
I suspect the “decrypted” text would be much easier to use as a starting point for a human code breaker if more vowels had been found. Maybe Popper is just really atypical, I don’t know.

Either way, this is just a taste of what I’ve been doing since my last exam on Friday. More to come, I’m sure.

I was browsing Reddit (with the quality of Slashdot’s editing going from utterly crap to considerably worse, and the discussion going from merely ignorantly libertarian to frothing-at-the-mouth ultra-conservatism, I need a new place to get my tech news from, so I’ve been trying a few other websites), and I came across this article on the “eight ball problem”.
If you’re averse to clicking links, the idea is that you’re given eight balls, one of which is slightly heavier than the others, and a set of scales. Your task is to find the heavier ball in as few weighings as possible.

I’ve been told it’s a popular job interview question, and the author of the article is probably right in saying that most computer science types would naturally try a binary search, which would get you the heavier ball in three weighings.
He then goes on to describe an algorithm that does it in two weighings: first you weigh balls 1 through 3 against 4 though 6. If both sets weigh the same, the heavier ball is either 7 or 8, and you just weigh those against each other. If not, take the set of three that was the heaviest and weigh the first ball against the second. If they weigh the same, the heavier ball is the third. If not, you have found the heavier ball.

This is indeed faster for eight balls, but is it faster in all situations, and how much?

The algorithm can be generalised as follows:

1. If you have one ball, you are done. It is the heavier. Otherwise:
2. Make two sets of n balls where n is the number of balls divided by three, rounded up. The third set, which may be empty, is the remainder of the balls.
3. Weigh the first set against the second.
4. If they weigh the same amount, your new set of balls is the third set. Otherwise, the new set is the heavier of the two sets. Discard the other balls.
5. Go back to step 1.

The astute among you may have noticed that while the binary search algorithm will always finish in the same time, this isn’t always true for this other algorithm; its best- and worst-case performance times differ for sets of balls that aren’t neatly divisible by 3 all the way down.
It’s not going to be a huge difference, especially for small amounts of balls, but it’s good to note.

The binary search will always take log₂(n) weighings to find the heavier ball, where n is the number of balls. It’s not immediately obvious how many the new algorithm will take, so let’s just write a program simulating it and run it a bunch of times.
While this seems an obvious place to use a functional language, I’m sorry to admit that I don’t yet feel comfortable with Haskell, and Scheme isn’t very legible, so I did it in Java.¹
That program outputs data in a gnuplot-friendly format, so let’s use it (gnuplot doesn’t like logarithms in base 2, but you’ll remember from your high school maths that log(n) / log(2) is equivalent):

We noted earlier that the second algorithm won’t always take the same amount of time, so the plotted line is the average time it will take. If you want to see the best- and worst-case performances plotted as well, you can do so yourself. The output data is here.

So yes, it does look like this algorithm is always faster, though binary search should really be fast enough for most purposes, and it’s somewhat easier to implement.
For 10,000 balls, this algorithm takes 8.84 weighings on average (and 9 in the worst case), while a binary search takes 14. For a million, it takes about 12.95 (13 in the worst case), and a binary search takes 20.
Unless someone is killing one of your loved ones for each weighing, I wouldn’t bother.

¹ Even as Java goes this is slower than it could be. For legibility’s sake I’m using Ball objects of my own making when I could be using ints or even booleans to represent the balls, for instance, and a getWeight() method call rather than a not-that-much-longer inline calculation. I invite you to write your own, better code in your favorite language.
Update: Like Python, for example. That was surprisingly painless, especially considering that it’s the first thing I’ve ever written in Python. It took maybe a third of the time it took me to write the Java version to write, and it’s not even that much slower, even though it’s an interpreted language.

After watching a few more videos by and about creationists on the YouTube, I realised that not only do I have more computing power at my fingertips than most scientists in history could ever have dreamed of and a greater-than-average ability to use said power, I also have a neat, educational, and easily implemented project to use it for.

Perhaps you remember the weasel program, but in case you don’t, here’s how my implementation worked:
The user enters a string consisting of characters from a certain pool (in our case, just capital letters and spaces). The program generates a random string of character from that pool, of the same length as the string the user entered. Every “generation”, the program takes one character in that string at random, and replaces it with another random character from the pool; it then compares the new (offspring) string and the old (parent) string to the target string the user entered, and discards the string that’s least like the target. The other one goes on to be the parent in the next generation.
This continues until the program’s own initially random string matches the user’s.

It’s a very simplistic example of Darwinian evolution, and while the point of it can be missed by people aiming to miss it, it demonstrates the power of evolution quite well.

Of course, casually running the program, seeing the number, and nodding absent-mindedly before forgetting all about it is no way to treat a nice algorithm, so I decided to drive the point home by modifying it a bit.

I decided to do a series of tests with successively longer strings (starting with one character, working up to fifty), and record how long it took on average to get from the random starting string to the target string (which is now just a series of As; I hope you realise why this doesn’t matter). The pool of characters was brought down from 27 possible characters to 10, to speed up execution times¹, and each test was then run five thousand times, to get rid of statistical artifacts.
The results were then plotted on a graph:

The X axis is the length of the string. The Y axis is the number of generations it took to get from a random starting string to the target. The blue line is the results my program found. The red line is how long you’d expect to take on average if you just rerolled the string entirely, which is how most creationists seem to think evolution works (the “tornado through a junkyard” fallacy).
For a string of length 50 this number is (10⁵⁰)/2, so forgive me for cutting it off the graph pretty early on.

There are some odd spikes the large sample size should have gotten rid of, which I blame on java.util.Random crapping out², but the trend is pretty clear all the same. (Edit: yeah, the problem was indeed that, and specifically how Java caches things in ways that breaks seeding random number generators. I fixed the issue as best I could and ran the program again; the results are here, and the spikes are indeed gone.)
It took about half an hour to run on my laptop, and the results are even clearer than I expected. Rather than the exponential growth in the number of generations needed so many creationists “predict”, there’s a linear one, which is much healthier.

You can debate how directly this applies to real-life evolution, since organisms tend to have genomes rather larger than fifty base pairs (though their pool of characters is only four, not ten), but they also tend to have more than one kid, and evolution is a trend over an entire population, not just one lineage, and they tend to have more than one mutation per generation (not to mention a bunch of other ways to stir things up, like chromosomal crossover), and sexual reproduction makes a whole new mess of everything.
The point, though, is to show the power of the Darwinian process; specifically, that it’s not just random chance, but something much, much more powerful.

Of course, people can show you all kinds of graphs and give you all kinds of programs to run, but it’s much more satisfying if you do it yourself and understand what you’re doing.
So I’m not going to post my code. Do it yourself. You have the algorithm (and if you think it sucked, improve on it (and post in the comments)), write your own code. It’s simple enough, it’s fun, and it’s quite gratifying.

¹ I also made a multi-threaded version of the program which works with a pool of fifty characters and goes up to string lengths of 500, and it’s been running on my reasonably pathetic cluster for the past two hours. I expect it to finish in a few weeks, unless the fans give out again and the entire cluster shuts down.

² ENTERPRISE TURKEY SOLUTIONS

In case you’re one of the three remaining peope who doesn’t know what Tor is, it’s basically an anonymising proxy on steroids.
Any request you make over a network (say, to retrieve a web page to display in your browser) is sent to a random node in the network, which then passes it on to the next node, which passes it on to the next node, and so on, until it finally reaches its destination. Each node only knows about the previous and the next node in the chain, so it becomes impossible to trace who made the original request.

Everything’s encrypted except for the final step between the last node and the webserver (for example), so some care should be taken when entering passwords and things, as a malicious exit node can intercept those if you don’t use things like TLS or other end-to-end encryption.
This is, of course, just as much of a risk on the internet in general (and one too many people aren’t aware of, too).

It’s pretty slow, since far more people are running clients than nodes (I’ll be setting up a node myself as soon as my ISP stops sucking; I’m giving it another week), but it’s not meant for general browsing (and certainly not filesharing) anyway; there’s a plug-in for Firefox that lets you turn it on briefly when you need it, and disable it when you don’t.

As with all privacy-preserving tools, genuinely undesirable activity is an issue (see picture), but the potential for good is considerable. While it may seem paranoid in (much of) the West (though maybe not even), much of China, for instance, depends on tools like these.
And you never know, you may need it yourself one day, and it’s better to become acquainted with it now than when it’s too late.

Get it here, if you don’t have it already. You don’t have to run a node (you can just set up the client (complete instructions for configuring Firefox to use it are there)), but if you can, please do. People depend on it.

Nearly all cultures have historically used numeral systems in base-10 (that is, the decimal system) or some multiple thereof (Mayans used base-20, Babylonians base-60), supposedly because a human hand has ten fingers.¹ If that’s the case, the ancients suffered from a severe lack of imagination.
If you count on your fingers in base-1 (that is, the normal way), you can count to ten. However, there is a way you can get up to 1023 using just both of your hands.

How? Use binary, of course.

It’s actually really easy once you get used to it. If your finger is up, that bit is set. If it’s down, it’s not.
For example, the following are the numbers 0, 24, 17, and 31 (only one hand is shown, because it’s easier; 31 is the highest you can go on one hand, obviously).²

Counting on your fingers in binary is a skill well worth picking up, especially if you intend to use computers more often than never, but also just because.
You can even count with negative numbers, if you use two’s complement or similar.

It might be harder to expand this to also use your toes, but every toe you add doubles your range of numbers (you can count up to 2ⁿ – 1, where n is the number of digits; including 0, that means you can represent 2ⁿ numbers), so you probably wouldn’t need all of them. If you’re counting up to 1,048,575 (or 2,097,151 if you’re a guy, hurr), you’re better off grabbing a calculator anyway.

¹ The Native American Yuki tribe actually used a base-4 system, because they counted the spaces between the fingers of one hand, which is interesting. Some Nigerian tribes use a duodecimal system (that is, base-12), because they are mutants.
(Actually, base-12 exists in a lot of places, mostly in the Imperial system of measurement (twelve rods to a hogshead, and all), and in various forms in time-keeping (twelve zodiac signs, twelve hours on the clock).)

² These hand pictures are actually repurposed from a chart detailing some variety of sign language.

So I probably should too.
This video describes a simple but effective attack again whole disk encryption and similar cryptosystems, based on the fact that, contrary to popular belief, modern DRAM retains its data for some time after power is cut, and the encryption key is stored in said DRAM.

There’s some more information in the full paper.

There are some obvious ways to defend against this as a user. One is to not leave your computer unattended while you’re logged in, or for several minutes after you’ve logged out and powered down.
The former should be obvious, but isn’t, and BitLocker’s thing that lets you boot straight to log-in makes it less than straightforward. Though if you use Vista, chances are you’re only doing the encryption thing because your boss made you anyway, and you don’t care about security or privacy.
Basically, if you aren’t typing the encryption password as well as your account password every time you boot, you have a problem.

It shouldn’t be too difficult to have the OS or some hardware device clear the entire RAM (or just a given area of it) as soon as a power failure is detected (it can keep running for a few milliseconds after the PSU sends the power failure signal, which is still a few thousand clock cycles), but that would be just as easy to get around, so it’s not worth the effort.

Another option is to just not store the key in RAM, but in a CPU register or the cache or something. I’m not sure how long their retain their information, but presumably it’s not nearly as long; possibly short enough to prevent this attack.
Of course, keeping something in the cache or the registers all the time isn’t something most OSes will play nice with, so that will require some OS-level retooling, and encryption keys tend to be comparatively large (128 to 256 bits may not seem like a lot, but a CPU register is 32 or 64 bits wide nowadays, and there aren’t that many of them), so that’s only going to increase the performance hit of whole disk encryption (which isn’t as big as people expect, but big enough that you don’t want to increase it).

The easiest thing is just to keep people away from your precious RAMs.