Last time we looked at how to solve the eight puzzle using the hill climbing algorithm, which gave us a result much more quickly than a blind depth-first search did, but we wondered if the solution we found was the best we could do, and we asked if there was a way to use heuristics to find not just a solution, but the best solution. Today, we’ll see that there is, and it’s actually really straightforward.
(This post assumes you read the previous one.)
Today we’ll be looking at the hill climbing algorithm, which is just a plain old depth-first search with heuristics added.
“Heuristics” is a fancy word (from the Greek εὑρίσκω, “I discover”) for a very simple concept. In the context of search trees, it simply means that at a every node, you’re going to look at each possible branch, and take the one that looks the most promising first, instead of just one at random. “Most promising” can be a tricky concept, though.0
Our river-crossing example isn’t necessarily the best one to demonstrate the concept, so let’s go with another classic: the 8 puzzle.
A decent proportion of my readers are noobie programmers or people who aren’t in a position to receive a formal CS education, so I thought I’d cover the basics of a fundamental concept most people cover in their first semester of algorithms or AI today: search trees. The fact that my college considers this to be third-year material so advanced they cannot in good faith make the class compulsory is neither here nor there.
Consider the famous problem of the farmer who wants to cross a river with his fox, goose, and grain, though the only boat can only carry himself and one of these three possesions. Ignore for a moment why a farmer would own a fox, and let’s stretch credibility a bit more by assuming that while the fox and goose are well-trained enough not to wander off in the absence of the farmer, they are not trained not to eat the goose or the grain, respectively, in said absence. How can he safely get to the other side without losing his goose or grain?
Having finished another popsci book on chaos theory recently (Ian Stewart’s Does God Play Dice?), I thought it’d be an interesting exercise to visualise the Lorenz attractor, and since it’s been a while since I’ve done anything new in programming, to take the opportunity to get into Xlib, the X Window System C library. Results aren’t very encouraging.
I mean, I got something to work easily enough, but any attempt at introducing color beyond black and white for clarity fails miserably and in non-deterministic ways. Eventually I gave up and redid it using something I know.
Compare:
(It’s prettier animated, so do compile the code yourself and see.)
In both cases, the screen represents the Cartesian plane (X-axis horizontal, Y-axis vertical, origin right in the center; one unit is ten pixels). In the Xlib version (left) the Z component is ignored entirely (so it’s really a projection of the attractor onto the Cartesian plane), in the Allegro version (right) some attempt at representing it using shades of gray has been made, with z=0 being black and z=55 being white (though because it is drawn with no real care, it will happily scribble dark lines over light ones if it has to).
You can mess with the variables and starting condition to see how it behaves, or swap around some Xs and Ys and Zs to get different angles, and at least in the Allegro version, messing with color is trivial enough.
Which brings me to my question: does anyone know of decent introductions to Xlib? The Internet is full of tutorials, and as usual, all of them seem to suck. I know Xlib isn’t really supposed to be used directly, but I want to.
I’m in a class called Netwerkbeheer (Network Management), which spans two semesters and is a transparent excuse to peddle CCNA certifications. As a result, I spend a lot of time playing with Cisco routers and switches, and one of the many, many things that annoy me about Cisco’s IOS is their cavalier attitude towards security and cryptosystems. A particularly egregious example of this is Cisco’s type 7 encryption.
If you’ve ever configured a Cisco router, you’ve probably encountered it. When the misleadingly named service password-encryption is running, setting a password with the enable password command “encrypts” the password, so that when you issue the show running-config command, you’ll see a line like
enable password 7 08314940000A
instead of the plaintext password, which you’d see if the so-called “password-encryption” was turned off.
Type 7 “encryption” manifests itself in a few other places, including in FTP passwords and various routing protocol authentication passwords.
Type 7 has been known to be broken for a decade and a half now,0 but people continue to use it, almost always for bad reasons.1,2 To drive home just how broken type 7 is, let’s look at it in detail.
The general form of the type 7 “ciphertext” is (0[0-9]|1[0-5])([0-9A-F]{2})+. Some experimenting finds that the length of the “ciphertext” is always twice the length of the plaintext, plus two. Can you guess why?
The “encryption” key is always a number in the range 0-15, which would be easy enough to bruteforce, but that turns out to be unnecessary, since it’s provided (in decimal form) as the first two characters of the “ciphertext”.
That key determines the starting point in a table of twenty-six secondary keys (which, incidentally, is dsfd;kfoA,.iyewrkldJKDHSUB; I don’t know why the table has 26 entries instead of 16), which are XORed in turn with the characters in the plaintext. If the key is, say, 7, the first character in the plaintext is XORed with the seventh character in the table, the second character in the plaintext is XORed with the eighth character in the table, the third with the ninth, &c.
Each resulting character is then converted to two hexadecimal digits (the input can only be ASCII, of course) and appended to the ciphertext.
And that’s seriously all there’s to it. The result is a “cipher” that’s either slightly less or slightly more secure than writing out your passwords in permanent marker on the outside of the door of the server room, depending on how you manage your configuration files.
Because I know this is going to be an issue at some point, I’ve written a simple utility that encrypts and decrypts passwords using type 7, which you can find here.
You’d think this would be a moot point because people should realise their configuration files are sensitive information, but people are, of course, idiots. In that sense, type 7 isn’t just worthless, but actively harmful, because it gives people a false sense of security.
0 http://insecure.org/sploits/cisco.passwords.html
1 The original intent of type 7 was apparently to foil shoulder-surfers, who might see your configuration file as it scrolls by on your screen. Cisco’s official stance (now) is that if security is an issue, the router configuration file itself should be treated as vulnerable data, not just the passwords that may or may not be displayed in it. That would be fair enough, if it wasn’t at odds with Cisco’s default way of saving and loading configuration files, which is through plain TFTP over the regular network, with no options for encryption of either the config or the passwords themselves. But, you know.
(The claim that type 7 is so weak because the router has to be able to reverse it is bullshit, of course. At most it’s true for PAP authentication, but anyone who considers PAP passwords secret information has no business being anywhere near a router.)
2 Cisco themselves now advise against using it, instead suggesting people use type 5, which isn’t encryption, but just hashing with MD5. Which is also broken, of course. The CCNA materials also state that at least type 7 is “better than no encryption”, but I’d argue that it’s worse, because its security is equivalent to plaintext, while also giving idiot network admins the impression that it’s not.
I’m told a type 6 exists now, which is based on AES and supposed to be better. AFAIK our routers don’t support it, and I’m not holding my breath either way.
I’m not even talking about apartheid or AIDS. I’m talking about language. Quite apart from the fact that they incomprehensibly keep pretending their dialect is a language in its own right, they keep applying the wrong words to things, and then passing them on to other languages.
Three examples.
This is probably the most famous example, and also an odd one, because mainstream Dutch is in the wrong here too. Examine the pictures above.
The one on the left is Suricata suricatta, a member of the mongoose family. In Afrikaans this has been called a meerkat, and this has been adopted into English. In regular Dutch it’s a stokstaartje (“little stick tail”).
The one on the right is a vervet monkey (Chlorocebus pygerythrus), one of thirty-five species of Old World monkey in the tribe Cercopithecini, which in Dutch are collectively called meerkatten.
This is, of course, an absurd name for either of those, as meerkat means “lake cat”. If anything should be called this, it should be Prionailurus viverrinus, which currently labors under the descriptive but utterly boring name fishing cat.
This medium-sized cat is semi-aquatic, and while it prefers streams and swamps to actual lakes, at least the name would be sort of appropriate. Being medium-sized, it’s also meer kat than the housecat.
Let’s just agree to call Suricata suricatta suricates, okay?
The one on the left is Raphicerus campestris, a small antelope native to southern and eastern Africa. For some reason, it was named “steenbok” after the one on the right, Capra ibex, the ibex, one of a few goat species called steenbok in Dutch. I have no idea what Afrikaners call the actual steenbok (the one on the right, that is; I know Germans call it Steinbock, and the other one Steinböckchen—that is to say, the diminutive), but the Dutch have taken to calling R. campestris “steenbokantilope”, which at least is fair enough.
Steenbok, of course, translates to stone buck (as in a male goat), which makes sense for the ibex because it lives in the Alps. It very much does not make sense for an antelope that spends its days in grassland.
Since it’s closely related to the two species of grysbok (which by rights should be spelled grijsbok), call it the fancy grysbok and stop confusing people.
Even though it’s not grey.
The last one is particularly ridiculous. Yes, that’s a moose. In Dutch, meese (elk, in European, though the North-American elk is something else; that’s a different discussion) are called eland. Afrikaners named two species of antelope eland because apparently they’re blind. Even the giant eland (Taurotragus derbianus, pictured) doesn’t even come close to Alces alces in size. The common eland, Taurotragus oryx, is even smaller.
The common eland (just eland in Afrikaans) is called the elandantilope in Dutch. The giant eland is reuzenelandantilope (the prefix reuzen- meaning “giant”). If you’re going to keep the dumb name, “eland antilope” and “giant eland antilope” seem like a good compromise.
I hate Afrikaans.
Having said that, there are some words that made it into English that they get right (boomslang, for example, means tree snake, and it’s exactly that), and a lot that, while dumb, aren’t confusing (aardvark (“earth pig”), aardwolf (“earth wolf”), wildebeest (“wild beast”), hartebeest (more correctly hertebeest, “deer-type animal”); all of these are at least vaguely misspelled by modern standards). Many non-animal words that made it into English are even fully accurate: spoor, veld, trek, and, of course, apartheid.
The three listed examples, though, are bunk. The animals are awesome enough to deserve decent names of their own.
And I still say Afrikaans is just a dialect of Dutch. It’s closer to standard Dutch than, say, Limburgs or West-Vlaams, and while there’s a movement to have those recognised as separate languages, that’s a tiny, tiny minority. The only real difference is that Afrikaans has a standardised spelling and good reasons to hate the Dutch.
(But then, so do we.)
You’ve had this problem before: you have a bunch of data points, and you want to interpolate between them.
For various reasons, higher order polynomial interpolation (where you try to find an nth-degree polynomial through n + 1 of your data points) can be a bad idea, so you decide that rather than using a simple equation, you’ll use a series of them to connect your data points. These equations are splines, and the simplest form of spline interpolation is just, well, connecting your data points directly:
That’s pretty ugly, though. Is there a way to achieve something like this:
instead?
Yes, obviously, and one of those ways is to use quadratic splines instead. Let’s use a simpler example, though. Suppose we only have four data points, (x0, y0) through (x3, y3):
The black dots are our actual data points, the red lines are our linear splines. What we’d actually like, though, is this:
Turns out that’s not that hard to do. As you can see, every spline is a quadratic equation, which obviously is of the form f(x) = ax² + bx + c. So each spline equation has three unknowns (a, b), and c, and there are three splines, for a total of nine unknowns (let’s call them a1 through a3 and so on).
Since two points are known for each spline equation, that gives us the following six equations:
To solve for nine unknowns, obviously we need nine equations. So what else do we know?
Well, the reason the linear spline interpolation looks like crap is because of the sharp breaks at the spline edges, so we would like our neighboring quadratic splines to have the same slope in the point that they share. In other words, if our spline equations are f, g, and h, we want the derivative of f to equal the derivative of g in the point (x1, y1), and we want the derivative of g to equal the derivative of h in the point (x2, y2).
The derivative is easy enough to find:
Filling in, this gives us two more equations:
Or equivalently:
Which brings our total to eight equations. We aren’t going to squeeze another legitimate equation out of this, so let’s just fill in one of the unknowns ourselves. If we make one of the as equal to 0, one of the quadratic splines becomes a linear spline, which is fine. Let’s take a1 for simplicity’s sake.
This enables us to construct the following matrix:
The first three columns are the as, the next three the bs, the next three the cs, and the final column will hold the solutions after reduction.
Filled in and solved for our particular dataset:
Which gives us the following equations for our splines:
Obviously this is a lot of work, but it’s mechanical work that doesn’t require a lot of judgement. Which is why I’ve written this Python script to do it for you. Feed it data points and it’ll produce gnuplot code to plot your splines:
$ python qsi.py < data.txt
plot 1.000000 <= x && x <= 3.000000 ? 0.000000 * x * x + 1.500000 * x + 1.500000 :\
3.000000 <= x && x <= 5.000000 ? -1.250000 * x * x + 9.000000 * x + -9.750000 :\
5.000000 <= x && x <= 9.000000 ? 1.125000 * x * x + -14.750000 * x + 49.625000 : 0/0 notitle
As you can tell, it’s not necessarily gorgeous, but it (probably) works, and it’s not like anyone has to see the code itself.
Format for the input file is as you’d expect: two numbers per line, first being x and second y, sorted. If gnuplot’s output is jagged, increase the sampling (set sample 1000).
And if it doesn’t work, fix it.
Edit: In light of overwhelming demand, this is a script that interpolates using a higher-order polynomial, as mentioned above. Here’s how the approaches compare for our sample dataset:
This script will fail if you only have one datapoint and its x value is 0, but everything else should work.
There’s been some interest in tripcode crackers lately, so I thought I’d write on in Haskell. I mentioned this before, but I’ve improved it a bit since.
I’ll be discussing the code step by step in this post. By the end, we should have a working application that takes a POSIX regex as an argument, and then outputs tripcodes that match it.
If everything goes right, this post should be literate Haskell, but I can’t promise that it’ll actually work, what with WordPress being what it is.
Let’s get started.
The tripcode algorithm is relatively straightforward: the input is converted to SJIS, there are some XML character entity substitutions, then a salt is calculated, and the whole thing is passed to Unix crypt.
We won’t be dealing with SJIS conversions, since our input will be ASCII only, which (with one exception) is a subset of SJIS anyway, and our target board will probably be Shiichan or Futallaby anyway, neither of which even does it. We also won’t be writing our own crypt implementation in Haskell, so we’ll have to get it from a C library. To do this, we use the Foreign Function Interface language extension, like so:
> {-# LANGUAGE ForeignFunctionInterface #-}
The usual boilerplate:
> module Main (main) where > import Char (chr, ord) > import Data.List (inits) > import Foreign.C > import System (getArgs, exitFailure) > import System.IO.Unsafe (unsafePerformIO) > import Text.Regex.Posix ((=~))
It’ll become clear why we need all of these as we go along.
Let’s import our C crypt:
> foreign import ccall unsafe "DES_crypt" crypt :: CString -> CString -> CString
We’re using OpenSSL’s implementation, because GNU crypt is slow as fuck, and this thing is going to be slow enough as it is. That bit of code is just saying to take a C function named DES_crypt from a linked library, and expose it as a Haskell function named crypt, with the listed type signature.
We said the tripcode algorithm involves some XML character entity substitutions, so let’s write a function to do that. The canonical algorithm just escapes ", <, and >. If yours escapes more (or fewer), just add (or remove) them here.
> xmlescape :: String -> String > xmlescape [] = [] > xmlescape (x:xs) = case x of > '"' -> (++) """ $ xmlescape xs > '<' -> (++) "<" $ xmlescape xs > '>' -> (++) ">" $ xmlescape xs > otherwise -> (:) x $ xmlescape xs
Straightforward enough.
Next, we’ll need a function to generate the salt. crypt’s salt is string of length 2 whose characters are in the range [a-zA-Z0-9.]. The tripcode function obtains this by appending H.. to the input and taking the second and third characters, performing some transformations to ensure they’re in the allowed range:
> salt :: String -> String > salt t = > map f . take 2 . tail $ t ++ "H.." > where > f c > | notElem c ['.'..'z'] = '.' > | elem c [':'..'@'] = chr $ ord c + 7 > | elem c ['['..'`'] = chr $ ord c + 6 > | otherwise = c
Now we’re ready for the actual tripcode.
This will happen in the IO monad, because we’re dealing with the FFI. We don’t have to use unsafePerformIO to escape from it, but to keep our algorithm conceptually cleaner, we will anyway.1
> tripcode :: String -> String > tripcode tr = unsafePerformIO $ do > trip <- newCString t > salt <- newCString $ salt t > peekCString (crypt trip salt) >>= return . drop 3 > where t = xmlescape tr
Great. Now that we have everything we need to calculate the tripcode for a given input, we need a way to generate inputs.
Let’s first start by specifying the characters we want to allow. We’re leaving out # and ! because they’re usually separator characters and as such illegal in tripcodes (though most boards will allow !), and \ because that’s the aforementioned ASCII/SJIS exception.
If you’re targetting a specific algorithm and you know which characters are fine and which aren’t, you can edit this. For Shiichan, for instance, the only forbidden character is #.
> allowedChars :: [Char] > allowedChars = filter (\c -> c `notElem` "#!\\") [' '..'~']
You can avoid the call to xmlescape altogether by disallowing characters that would be escaped, of course; that’s what I did for my C implementation, too. That will obviously reduce your search space, though.
Now we can use these characters to generate our (infinite) list of inputs:
> ins :: [String] > ins = (inits . repeat) allowedChars >>= sequence
This will generate all possible combinations, from "" to strings of infinite length. Since crypt disregards input over eight characters wide, it will generate far more combinations than you’ll ever need. Since it will take almost forever to even get up to that point, though, the issue is kind of moot.2
Now that we have our infinite input list, we can turn this into the input/output combinations we need; first we’ll team up each input with its corresponding output with zip, and then we’ll filter it with our regular expression. Obviously, this is where Haskell’s laziness is really handy:
> tripPairs :: String -> [(String, String)] > tripPairs regex = filter (\(a, b) -> b =~ regex) $ zip ins $ map tripcode ins
It’s cute that (=~) is used for regular expressions. It’s also handy that it takes a string as its second argument and we don’t have to dick around with compiling regular expressions and what have you.
(=~) actually has a pretty complicated type signature, but its return value is a polymorphic value that converts into a boolean without complaints, so we don’t have to worry about that.
So what do we have now? We have our tripcode function, our list of inputs, and a function that filters this list based on a regex argument. We’re pretty much done. Now we only need the main function:
> main :: IO () > main = getArgs >>= f > where > f [] = putStrLn " USAGE: tripcode [regex]" >> exitFailure > f (arg:xs) = mapM_ (putStrLn . show) $ tripPairs arg
When no argument is passed on the command line, we display a short usage note and return failure, otherwise we generate our list of matching pairs, turn them into displayable strings, and then print them to stdout. Try it, it works!3
It’ll be slow as fuck, though,4 for a number of reasons. The first is that OpenSSL’s crypt, while much faster than GNU crypt, still isn’t very fast. The second is that Haskell’s Text.Regex.Posix is very slow (if you know how, I suggest you use another regex library; I went with Text.Regex.Posix because most casual Haskellers (which I am too) aren’t likely to have any of the others). The third is that it can only use one processor at a time, whereas most others are multithreaded or multiprocess affairs. The fourth is that a high-level language like Haskell is obviously going to be slower than tripcode crackers written in C and Sepples by moon people. The fifth is that I suck at Haskell.
Still, at least it’s written in the world’s leading fictional programming language.
1 If we didn’t, the type signature would be String -> IO String, of course.
2 If you only want to check eight-character inputs, though, you can do something like ins = sequence . take 8 $ repeat allowedChars instead.
3 Copy/paste this post into a file named Tripfind.lhs, then compile it using ghc -lssl Tripfind.lhs
4 Slower than my C one almost by an order of magnitude, even after adjusting for the fact that my C one can use both of my processors. And since my C one is already slower than, say, Tripcode Explorer by an order of magnitude…
You know Sierpiński gaskets, right? I used one in my Christmas tree last December. They’re fractals created by taking a triangle, connecting the midpoints of the sides to divide it into four, removing the middle one, and then repeating that on the remaining triangles, ad infinitum (literally). They have an area of 0 and a Hausdorff dimension of log2(3).
There’s another, more interesting way of constructing them, though: take the three corner points of a triangle, and a random starting point x. Roll a three-sided die1 to select a random corner point, and mark the midpoint between that point and x. Then, this midpoint becomes x. Repeat forever.
It turns out the Sierpiński gasket is the attractor for this system. I’ve written a Python script to save you some paper and a large number of pencils.2 Here is the result I got after 10,000 iterations:
To run the script, you’ll need to have the Python Imaging Library installed. It takes three optional arguments: the side of the triangle in pixels (defaults to 1,000), the number of iterations (default 10,000), and the output filename (default out.png).
Strange attractors are fun. Coming up: more of the same.
Edit: If you’d prefer something you can see on the screen to something that dumps to a file, this may interest you. You can change the WIDTH and HEIGHT #defines to your actual resolution if you like (or basically any value, really; it should produce an equilateral gasket for any realistic resolution, though it might not for odd ones, including ones that are higher than they are wide).
You’ll need (besides a C compiler) the Allegro libraries. If you’re using a Debian-based distro, the package is liballegro-dev, IIRC, or you can get them here.
1 Or a six-sided one where you divide the result by two, rounding down, if you like.
2 It’s not exactly the same thing: raster images don’t have an infinite resolution, IEEE floating point numbers don’t have infinite precision, and you (probably) don’t have the patience to let your computer run forever.3
3 If you do, consider cracking tripcodes instead.
Inspired by rmuser’s Youtube videos on information theory (and specifically the one about Huffman encoding), I wrote a Python script to calculate a Huffman encoding for text.
It reads input from stdin (preferably in ASCII), calculates a Huffman mapping, and shows it to you. It also calculates how long the text would be if encoded with that mapping, and how many bytes you’ve saved compared to ASCII1, which just uses a byte for each character, regardless of how frequently it’s used.
Here’s the result of running it on itself:
$ python huffman.py < huffman.py
Symbol Freq Encoding
' ' 560 10
'e' 262 111
't' 138 1100
's' 134 1101
'r' 125 00001
'\n' 110 00011
'f' 105 00100
'n' 98 00110
'l' 96 00111
'a' 75 01100
'i' 72 01110
'o' 67 000000
'd' 61 000100
'.' 54 001010
'(' 48 010000
')' 48 010001
'c' 46 010011
',' 46 010100
'q' 45 010101
'm' 36 011011
'b' 35 011110
'u' 32 0000010
':' 30 0001010
'_' 26 0001011
'=' 26 0010110
'y' 24 0010111
'p' 24 0100100
'h' 22 0101100
'g' 20 0101110
'[' 11 01001011
'%' 11 01011010
'#' 11 01011011
'1' 10 01101000
'"' 10 01101001
'0' 9 01101010
"'" 9 01101011
']' 9 01111100
'F' 9 01111101
'\\' 8 000001110
'T' 5 010111100
'+' 5 010111101
'v' 5 010111110
'w' 5 010111111
'/' 4 0000011010
'R' 4 011111100
'3' 4 011111101
'x' 4 011111110
'k' 4 011111111
'I' 3 0100101000
'2' 3 0100101001
'j' 3 0100101010
'S' 3 0100101011
'>' 2 00000110000
'C' 2 00000110010
'A' 2 00000110011
'E' 2 00000111100
'B' 2 00000111101
'P' 2 00000111110
'H' 2 00000111111
'*' 1 000001100010
'!' 1 000001100011
'8' 1 000001101100
'-' 1 000001101101
'W' 1 000001101110
'L' 1 000001101111
Encoded message length: 12237 bits (1529.62 bytes)
This message contained 2634 characters. Huffman encoding saved 1104 bytes
compared to ASCII.
As you can see, the most frequently-used characters have the shortest encoding, while the rarest have the longest. I’m assuming that means it’s working the way it should.
Simple toy, but it beats paying attention in class.
1 If the input isn’t in ASCII, it should still come up with a correct mapping, but that last bit will be off by a bit.
Solve for n.
Edit: This is basically the reverse birthday problem, with the fixed probability being 50%. Just applying that approximating formula is a bit easier than working out the problem above:
Which is much lower than I expected and also crap. It means that the algorithm I’m using for my work-in-progress distributable tripcode searcher is broken. There are some obvious ways to fix it, but finding a way that’s both good and not likely to slow things down too much1 (though obviously the bottleneck is still going to be crypt itself) requires some thought.
1 Right now it’s running at about 330,000 tripcodes per second on my laptop, which is over four times the speed of tripper+ and over sixteen times the speed of my Haskell tripcode implementation, so I guess some slow-down wouldn’t kill it.
Though Asztal claimed 4 million tripcodes per second with Tripcode Explorer on his machine, so I suppose there’s still room for improvement, too.
One thing that continues to annoy me whenever my internets get into a discussion about race is that invariably, very nearly everyone gets it wrong. The most recent example of this is, of course, when some Stormfront morons declared war on Pharyngula.
On the one hand you have the common racists, which are wrong for obvious and uninteresting reasons, but on the other you have the “enlightened” people who claim race is entirely a social construct, or at least of no significance whatsoever. They’re wrong too.
The following is an excerpt from Richard Dawkins’ The Ancestor’s Tale, which I finished a few weeks ago. It may be the clearest explanation I’ve seen so far.
It is genuinely true that, if you measure the total variation in the human species and then partition it into a between-race component and a within-race component, the between-race component is a very small fraction of the total. Most of the variation among humans can be found within races as well as between them. Only a small admixture of extra variation distinguishes races from each other. That is all correct. What is not correct is the inference that race is therefore a meaningless concept. This point has been clearly made by the distinguished Cambridge geneticist A. W. F. Edwards in the recent paper called ‘Human genetic diversity: Lewontin’s fallacy’. R. C. Lewontin is an equally distinguished Cambridge (Mass.) geneticist, known for the strength of his political convictions and his weakness for dragging them into science at every possible opportunity. Lewontin’s view of race has become near-universal orthodoxy in scientific circles. He wrote, in a famous paper of 1972:
It is clear that our perception of relatively large differences between human races and subgroups, as compared to the variation within these groups, is indeed a biased perception and that, based on randomly chosen genetic differences, human races and populations are remarkably similar to each other, with the largest part by far of human variation being accounted for by the differences between individuals.
This is, of course, exactly the point I accepted above, not surprisingly since what I wrote was largely based on Lewontin. But see how Lewontin goes on:
Human racial classification is of no social value and is positively destructive of social and human relations. Since such racial classification is now seen to be of virtually no genetic or taxonomic significance either, no justification can be offered for its continuance.
We can happily agree that human racial classification is of no social value and is positively destructive of social and human relations. That is one reason why I object to ticking boxes in forms and why I object to positive discrimination in job selection. But that doesn’t mean that race is of ‘virtually no genetic or taxonomic significance’. This is Edward’s point, and he reasons as follows. However small the racial partition of the total variation may be, if such racial characteristics as there are are highly correlated with other racial characteristics, they are by definition informative, and therefore of taxonomic significance.
It’s not surprising that Lewontin’s1 views are most popular in the US, where casual racism is so common many smart people are so eager to dissociate themselves from it they swing too far in the other direction.
Dawkins then goes on to say that if we have a person and we are told about his sex, we immediately know more about the shape of his genitals, though not with absolute certainty. That is to say, our uncertainty about some of his attributes is reduced. Similarly, if we are told this person is black, our uncertainty about a number of his attributes, such as (but not exclusively) the color of his skin, is reduced as well, so it’s intuitively obvious that race cannot be exclusively a social construct.
The whole thing is worth reading, though the book as a whole is not his best. If you’re going to buy it, buy the hardcover version. It’s expensive, but the book relies on pictures too much for the paperback to be very useful.
Incidentally, contrary to what aforementioned Stormfront morons claim, there is no conclusive causative link between race and IQ. It’s true that blacks on average have a lower IQ than whites in the US, but that difference disappears once you adjust for class (the lower classes tend to have lower IQs than the upper classes, of course, given the strong correlation between IQ and education levels), and the fact that blacks on average tend to be lower class than whites seems to be more of a result of discrimination based on racism than it is of anything inherent in blacks.2
Either way, this whole discussion makes me tired. Talking to either side in it is like talking to a brick wall.
1 It should also not be surprising that Lewontin is an erstwhile compatriot of Gould’s, and a longtime opponent of the straw-man “genetic determinism” of evolutionary psychology.
2 And to the “other side”, before you try to dispute the validity of IQ testing, I suggest you at least read this article (Wikipedia has an article about that article).
Prime numbers are ridiculously important in cryptography, and I recently found myself needing a way to generate them (for a toy implementation of the Diffie-Hellman key exchange algorithm). It keeps amazing me how few languages actually have libraries for generating prime numbers, or even just for primality testing.
Because I didn’t feel like thinking, and I didn’t need incredibly massive primes, I just wrote a quick implementation of the sieve of Eratosthenes, which is an interesting enough algorithm that I’d thought I’d share it (this is first-year stuff even for us, but I guess I have non-programmers in my audience).
The basic idea is that you start with a list of integers from 2 up to the number you want to find the largest prime smaller than. You then take the smallest number in this list (2), and cross off all multiples of this number (except for 2 itself). You repeat this for every other number, in order, and you’ll be left with just a list of primes.
Wikipedia has a visualisation which could be clearer but is still pretty good:
The implementation (in Python) is dead simple:
def sieve1(n):
l = range(n)
l[0], l[1] = None, None
for i in xrange(int(__import__('math').sqrt(n)) + 1):
if l[i] is not None:
j = i * 2
while j < n:
l[j] = None
j += i
return filter(None, l)
We could just use range(2, n) and then work with an offset, but the convenience of having the indices equal to the values at those indices is too useful to complain about the extra memory required for two additional integers. Note also the obvious optimisation of only going up to sqrt(n) + 1 (no apologies for the unorthodox import).
filter(None, l) gets rid of the Nones in our list. Passing None as the function argument just applies the identity function, which has the effect of getting rid of None, False, 0, and empty lists, tuples, and dictionaries (anything which would evaluate to False in an if statement).
This is nice, but it does have the disadvantage of having to generate the whole list every time you want more primes. You can’t just input a list of pregenerated primes and have it continue from there.
The fix is easy enough:
def sieve2(n, primes = [2, 3]):
if n <= primes[-1]:
return filter(lambda m: m <= n, primes)
offset = primes[-1] + 1
l = range(offset, n)
max = int(__import__('math').sqrt(n))
for p in primes:
if p > max:
break
i = p * 2 - offset
while i < 0:
i += p
while i < len(l):
l[i] = None
i += p
for k in xrange(max - offset + 1):
if l[k] is not None:
i = k * 2 + offset
while i < len(l):
l[i] = None
i += l[k]
l = filter(None, l)
primes.extend(l)
return primes
This function uses memoisation, and takes advantage of a feature of Python’s function defaults which some believe to be a bug: if mutable default arguments are altered, they will remain altered for the next time the function is called.
This is a side-effect of the fact that functions have a single field that holds the default arguments (a tuple called func_defaults in the function’s namespace), which isn’t duplicated into a working copy when the function is called. Most default arguments will be things like integers and strings, which are immutable, so most people never even run into this. Being able to use it for memoisation is really handy, though. Alternatively, you could just use a global variable and access it from inside the function.
The algorithm is basically the same, except that if we’ve already generated the primes requested, we can just return them. Then we construct a list of all integers larger than our largest prime, and sieve it with our existing primes (now we really do have to work with an offset). Then we just sieve the rest classically. At the end, we add our newly sieved list to our old primes, and return that.
Next time we invoke the function, primes will still have the new primes. You can access them from outside the function at sieve2.func_defaults[0].
Evaluating sieve2(100) and then sieve2(1000) will be slower than just evaluating sieve1(1000), but it will be faster than evaluating sieve1(100) and then sieve1(1000). And if you happen to have a large number of pregenerated primes, you can save some more time.
The sieve of Eratosthenes is just one of a number of sieve-based prime number generators, but it’s the oldest one, and it compares favorably to modern improvements.
A classical cryptography library isn’t much of a toy to non-programmers, so in an effort to learn about Python’s Tkinter library, I wrote a quick-and-dirty GUI front-end to it:
I know it’s ugly, but that’s because it’s Tk. I think it’s meant to be ugly. Tell the Python dudes to use GTK+ bindings for Python’s standard toolset.
It works, though, even if you can’t paste text into the textboxes.
There are two ways to run this, and both require that you have Python installed (I still use 2.5.2, but I see no reason why it shouldn’t run under 2.6).
The first, and most obvious, is to copy all of that code into a file, and then copy all of this code into a different file called classicalcrypto.py. Put those into the same folder and just double-click on the first file to run it (if you aren’t using Windows, you’ll obviously need to chmod +x first).
The second and slightly more complicated way is to install the classical crypto module separately. Download and untar this file, and then install it by running the setup file like python setup.py install. Then just put this code into a file, and you won’t have to worry about keeping two files in the same folder.
If anyone has any problems getting it to work, let me know. I did try to package it using py2exe, but, unsurprisingly, py2exe is a piece of shit and that didn’t work.
Only most algorithms are included, because some have non-trivial key requirements; the four-square cipher, for example, requires two Polybius squares. I left the Solitaire cipher because it still works, for some reason, even though it shouldn’t. I wouldn’t trust it to be strong.
Atbash and ROT13 ignore the key you enter. Caesar and Rail Fence require it to be an integer (in Rail Fence’s case, higher than 1).
Keep in mind that Rail Fence is a permutation cipher, so trailing newlines are significant characters. For reasons I couldn’t be bothered to troubleshoot, Tk appends a newline when it updates the contents of the textbox, so if you just hit encrypt and decrypt in a row, you won’t get the original message back.
Specifically, a small classical cryptography library.
None of these things are very hard to implement (in fact, two of the hardest bits I haven’t even implemented myself), but it’s good practice for me and some people might hypothetically have a use for it.
I’ve taken a stab at documenting it, but I don’t know how Python documentation works, so forgive me if it’s useless.
Included functions are:
atbash: The Atbash cipher. Optionally allows you to pass your own alphabet.bifid: Delastelle’s bifid cipher.caesar: The Caesar cipher. Optionally allows you to pass your own alphabet, preserves case and letters not in the alphabet.foursquare: The four-square cipher.hill: The Hill cipher, sans decryption function, because I honestly couldn’t figure out how to invert a matrix in a way that preserves integers (protip: it’s not the usual way; I wrote a Gauss-Jordan reduction function before realising that couldn’t work for most key matrices). If you know how to invert it yourself, you can do that and pass it as the key along with the ciphertext, and it’ll decrypt normally. Alternatively, you could tell me how to do it and I’ll fix it.keyword: The keyword cipher. Optionally allows you to pass your own alphabet, preserves case and letters not in the alphabet.playfair: The Playfair cipher. Note that the decryption doesn’t have a way to tell which characters were added during encryption and which were there in the plaintext (though it should be obvious unless your plaintext has a lot of Xs and Qs).railfence: The rail fence cipher. rot13: ROT13. Special case of the Caesar cipher.solitaire: Bruce Schneier’s solitaire cipher.solitaire_prng: The key generator for same.vigenere: The Vigenère cipher previously discussed.vigenere_autokey: Vigenère’s autokey cipher, also previously discussed.Basically, everything in Wikipedia’s “classical ciphers” for which an algorithm was described, minus a few of the more boring ones.
There’s also a Polybius class, which allows for easy construction of Polybius squares, and methods for looking up things in them. It’s just a wrapper around a square matrix and a reverse look-up of the same.
The general principle behind these functions is that they take a message (which is a string) and a key appropriate to the cipher. To decrypt, you use the same function and pass an additional boolean decrypt, set to True.
Preservation of case and punctuation characters and the like is inconsistent, so if that’s important, you’re going to have to look at the code itself. This is because I am nothing if not easily bored.
Anyway, enjoy. I may add to this over time, so if there are ciphers you would like me to implement, let me know.
Memes evolve, and even creationist memes are no exception. It used to be that they denied things like antibiotic resistance in bacteria and pesticide resistance in insects even existed, but now most of the ones that actually end up “debating” sane people seem to agree that “micro-evolution” happens, but “macro-evolution” doesn’t.
The distinction is meaningless, of course, and the argument amounts to nothing more than “I believe in small changes over a small period of time, but not large changes over a large one”. The implication that there is some invisible barrier to speciation, though, still tends to stump many of the people they’re debating, because it’s not necessarily easy to come up with an example that’s simple enough for the wilfully ignorant to understand, and examples of observed speciation in bacteria doesn’t tend to impress people who only barely believe bacteria even exist, much less that it’s meaningful to speak of different species of them.
Ring species, however, are such an example. They demonstrate a gradual process of speciation, and they manage to do it in a way that’s observable in real time.
Put simply, with ring species you have a series of populations along a line or open ring (often a coastline or a river bank, actually). Each of these populations can breed with itself (obviously) and with its slightly different neighboring populations. Perhaps some populations are classified as different subspecies of some overarching species; it’s difficult to point down where one subspecies ends and the next begins, though: the populations basically form a continuum.
The punchline, though: though each population can breed with its neighbors and produce fertile offspring (which, according to a popular definition, makes them the same species), the populations at the ends of the line cannot. They are, in effect, different species.
The canonical example is the herring gull complex around the north pole.1
The ring starts on the shores of the northern North Atlantic with the herring gull Larus argentatus. It can interbreed freely with the American herring gull Larus smithsonianus,2 which occurs, surprisingly, across North America.
The North American herring gull can interbreed with the Vega gull Larus vegae, which is a subspecies of the East Siberian gull (the only other subspecies is the Larus vegae mongolicus, which isn’t relevant to our story). That in turn can interbreed with Heuglin’s gull, Larus heuglini, which can interbreed with the Siberian lesser black-backed gull Larus fuscus heuglini, concluding our trip across northern Siberia.
The Siberian lesser black-backed gull can interbreed with the lesser black-backed gull Larus fuscus, which bring us back to Europe, mostly around the Baltic Sea, though they share some territory with the herring gull.3
These gulls form a species continuum, as you can see, but the herring gull cannot interbreed with the lesser black-backed gull.
It’s hard to tell where one species ends and the next begins, but it’s undeniable that the herring gull and the lesser black-backed gull are different species.
This is just one mechanism of speciation (allopatric speciation; these gulls inhabit much the same niches, so this is nearly entirely down to genetic drift, too), but it’s a plainly obvious one.4
Of course, most creationists would just reply “but they’re all still birds”, which is a level of ignorance against which little can be done.
Some of the more literate ones might point out that not only are they all birds, they’re all still in the genus Larus, moving their non-existent barrier to speciation to a barrier to “genusiation”, but that’s no more to the point than to point out that all new growth on an old tree seems to happen at the bud level, with only new twigs being born, and that no new massive boughs have been added to it in decades.
(Incidentally, there’s an interesting parallel in linguistics, with the dialect or language continuum. Flanders is a good example of it, with most of its many, many dialects being intelligible to its immediate geographic neighbors, but people on one end of the region not being able to understand a damn thing people on the other are saying. It can probably be argued that the Limburgs dialects make Dutch form a language continuum with German.)
1 I’m aware of the Liebers/de Knijff/Helbig paper, and I slightly disagree with its conclusion that the Larus species complex does not represent a ring species. Regardless of how it was formed, it’s a ring now. There may be contexts in which it’s not useful to speak of it as a ring species, but that isn’t relevant here.
If you want to argue about this, by all means do.
2 And, indeed, is almost indistinguishable from it to the untrained eye. The American herring gull is a bit bigger, though.
3 The taxonomic story is actually a bit more complex than that, obviously. For instance, the gulls in the eastern part of the range of the Heuglin’s gull are often considered a separate subspecies (Larus heuglini taimyrensis, or the Taimyr Gull), but some people believe these to just be the result of interbreeding between Heuglin’s gulls and Vega gulls; and of course most species names are in dispute: the Vega gull is sometimes classified as Larus argentatus vegae, and people seem to be confused over what the Siberian lesser black-backed gull actually is.
This is something for taxonomists to squabble over, though, and I’m not convinced it’s a productive use of anyone’s time.
4 If the Liebers paper makes you nervous, there are other, less controversial examples of ring species, including Ensatina salamanders around Central Valley in California, and greenish warblers around the Himalayas.
The Vigenère cipher is a more interesting cipher, in that it uses an actual key, and isn’t trivial to bruteforce. Central to the Vigenère cipher (and, in fact, most classical substitution ciphers) is the tabula recta:
Suppose that you have a message to encrypt starting with “Proletarians of Europe! The war has lasted more than a year.”, and a key “secret”. To encrypt your message, you line the plaintext and the ciphertext up like this:
PROLETARIANSOFEUROPETHEWARHASLASTEDMORETHANAYEAR
SECRETSECRETSECRETSECRETSECRETSECRETSECRETSECRETAnd then you just look up vertical pairs in the table. The first letter to encrypt is P, so you go to the row marked P. The first letter to encrypt it with is S, so you look in the column S of the row P. The resulting ciphertext letter is H. You repeat this for the rest of the letters.
The resulting ciphertext will be:
HVQCIMSVKRRLGJGLVHHIVYIPSVJRWESWVVHFGVGKLTFEAVEKDecrypting is the reverse of this: you go to the column corresponding to the key letter, find the letter H in that column, and then look at what row that letter is in.
Implementing this in Python turns out not to be very hard.
I actually constructed the tabula recta as a lookup table, mostly to see how hard it would be, but you don’t, of course, have to do that. All you need is a way to turn letters into numbers (A = 0, B = 1, &c.) and back, and encryption is as easy as:
ciphertext_letter =
to_letter((to_number(plaintext_letter) + to_number(key_letter)) % 26)
Decryption just changes the + to a -.
This is also how I constructed the tabula recta. The to_letter function is easily accomplished by treating a string of the letters in alphabetical order as an array, and the for the to_number function I just used a reverse dictionary lookup of the same.
The tabula recta then takes literally three lines of code:
lookup = dict([(a, 0) for a in alpha])
for a in lookup:
lookup[a] = dict([(b, alpha[(tonum[a] + tonum[b]) % 26]) for b in alpha])
You really don’t want to know how much longer that would be in Java.
A sample session:
$ ./vigenere.py
[E]ncrypt or [D]ecrypt? E
Phrase to encrypt? Proletarians of Europe! The war has lasted more than a year.
Strip whitespace and punctuation? (Y/n) Y
Key? secret
hvqcimsvkrrlgjglvhhivyipsvjrweswvvhfgvgkltfeavek
$ ./vigenere.py
[E]ncrypt or [D]ecrypt? D
Phrase to decrypt? hvqcimsvkrrlgjglvhhivyipsvjrweswvvhfgvg kltfeavek
Key? secret
proletariansofeuropethewarhaslastedmorethanayear
The Vigenère cipher is less easy to crack than the Caesar cipher, but not much.
If you know the length of the key, the ciphertext simply turns into so many Caesar-shifted ciphertexts, which may be subjected to frequency analysis. Though of course, we’ve already seen how great that turns out if you don’t have a massive amount of text to analyse.
If you don’t know the length of the key, but do have a huge ciphertext, you can look for repeating clusters of letters, which are likely to be bits of plaintext that happen to repeat at intervals equal to some multiple of the key length. If you have a bunch of different ones, you can find the largest common factor of the intervals, and this will likely be the key length. After this you can just use frequency analysis again. This method is known as the Kasiski examination, after the Prussian Friedrich Kasiski (though Charles Babbage also came up with it).
These things aren’t that hard to implement, but also not that interesting.
It should be clear, though, that the Vigenère cipher is a pretty weak one (it certainly doesn’t deserve the name «le chiffre indéchiffrable»), which is ironic, given that Blaise de Vigenère never actually came up with it, and the one he did come up with was actually quite a bit stronger.
Vigenère’s autokey cipher uses the same basic principle of the tabula recta, but instead of a repeating key, it’s going to use a short key to start with, and then for the rest of it use the plaintext.1 Vigenère himself suggested using a single letter of the alphabet for the key, but that’s pretty stupid. Let’s instead use a full word, like so:
PROLETARIANSOFEUROPETHEWARHASLASTEDMORETHANAYEAR
SECRETPROLETARIANSOFEUROPETHEWARHASLASTEDMORETHAAgain, not that hard to implement.
Rather than repeating, the key is now just the provided key plus the ciphertext, which saves a modulo operation in the lookup.
Decryption has an extra step, because every deciphered letter has to be added to the key while the decryption is in progress, but on the whole, it’s very similar to the other “Vigenère” cipher.
Breaking it is rather harder than breaking the other one, though. Frequency analysis isn’t really going to work, though if you use a single letter for the key, like Vigenère suggested, bruteforcing is trivial, and our own example is vulnerable to a dictionary attacks. Both of those are attacks against the key, though, not the cipher.
There are some relatively straightforward attacks possible, one of which is described in the Wikipedia article, and I’m working on those now.
More to come.
1 You’ll remember that this basic principle also returns in a slightly different form in certain block cipher modes of operation in modern cryptography. It’s also used for some types of self-synchronising stream ciphers.
Alright, so it’s about programming again.
I’ve been meaning to write some tools to help me encrypt text using some classical ciphers, and also to break those ciphers. I’ve also been meaning to using Python more, because holy fuck Python is awesome.
Maybe it’s just the fact that I only really have any experience with seriously shit languages (Java, PHP, COBOL; I’ve used other languages, some of which didn’t suck, but never to the extent I’ve been forced to use those three), but after I wrote a Python script to deal with that ball algorithm in a third of the time it took me to write it in Java, which is probably the language I’ve used the most in my life, I redid some of the Project Euler problems, which I’d previously also mostly done in Java (some in Scheme), and I was amazed how much less painful it was. If this puts me on the same level as Randall Munroe I apologise.
Anyway.
The obvious place to start was with the Caesar cipher, which is the one where you take each letter and move it ahead X places in the alphabet. ROT13 is a Caesar cipher with the key set to 13.
There are two obvious attacks here: one is frequency analysis, where you count the number of letters and compare it to a chart of letter frequencies (the most common letter in the ciphertext is likely to be E if it’s an English (or Dutch, for that matter) text, for example); the other is bruteforcing, where you just go through all possible keys. This is possible because there are really only 25 possible keys, plus one (26, or 0) which produces a ciphertext identical to the plaintext.
Frequency analysis seemed overkill, since bruteforcing is so easy and much more fool-proof, especially for short ciphertexts, so I went with an algorithm that encrypts the ciphertexts with all 26 possible keys, and uses a dictionary file to count the number of dictionary words in the resulting texts. The one with the highest number of dictionary words is likely to be the plaintext, so that’s then returned.
You can find the code here.
A sample session (text in bold is entered by the user):
$ ./caesar.py
Phrase to encrypt? All that we see or seem is but a dream within a dream.
Shift by? 12
Encrypted: Mxx ftmf iq eqq ad eqqy ue ngf m pdqmy iuftuz m pdqmy.
Decrypting...
Decrypted: All that we see or seem is but a dream within a dream.
Success!
It’s slow as fuck because of the way it uses the dictionary file, of course (but that shouldn’t be too hard to optimise), and because the decrypting function doesn’t take punctuation into account it won’t count words that might actually be in the dictionary, which can be a problem for really short sentences with a lot of punctuation:
$ ./caesar.py
Phrase to encrypt? A ghost!
Shift by? 11
Encrypted: L rszde!
Decrypting...
Decrypted: L rszde!
Fail!
It also doesn’t work if punctuation and—more importantly—spaces have been stripped, though, as was usual for real-life applications of this cipher, and that’s more of a problem.
The easiest way around this is to let the user identify which the decrypted string is, of course, or you could look at random parts of the text to try to guess where words end and see if anything matches a dictionary word, but that’s a lot of work. It may be easier to solve this using frequency analysis.
Like I said, frequency analysis counts the number of letters in a word and guesses what they’re likely to map to based on how often they normally appear in a text written in the same language as the plaintext.
This sort of attack can break not just Caesar ciphers, but any simple substitution cipher, though for a polygraphic one (that is, one that replaces groups of letters rather than individual letters), you’ll need a polygraph frequency chart, of course. It doesn’t do too well with polyalphabetic substitution ciphers (where a plaintext glyph can map to multiple ciphertext ones), but given enough ciphertext, it can break Vigenère ciphers as well.
I wrote a function that does just this: it counts the number of times each letter appears in a ciphertext (ignoring punctuation and spacing, if any), and then naively replaces the letters in the ciphertext based on a letter frequency chart.
It works a lot faster than having to look up words in a dictionary, but the main problem seems to be finding a good letter frequency chart. ETAOIN SHRDLU is a famous sequence, of course, and if you look at the source you’ll see I used that first. My ciphertext was this speech by Karl Popper, ROT13′d (because it was the first large block of text that I could find that had been written in modern English).
With Linotype’s ETAOIN SHRDLU, the “decrypted” version started off like this:
Dr. Rujior, Dr. Mufg, Lfmtuz fgm Nugiludug
Io ipu 3rm Afjeliq oa Dumtjtgu oa ipu Jpfrluz Egtvurztiq fgm tiz Mufg, Hroauzzor Pouzjpl, fgm fll oa qoe kpo pfvu jodu puru iomfq, T ktzp io uxhruzz dq muuhlq auli ipfgwz aor ipu nrufi pogoer kptjp qoe fru jogaurrtgn og du.
Then I tried the one used by Samuel F. B. Morse to determine which letters should map to the least amount of dots and dashes in his code: E IT SAN HURDM WGVLFBK OPJXCZ YG.
This was the result:
Kr. Ruohlr, Kr. Dufv, Wfdsuc fvd Guvhwukuv
Hl heu 3rd Bfoawhz lb Kudsosvu lb heu Oefrwuc Avsnurcshz fvd shc Dufv, Mrlbucclr Elucoew, fvd fww lb zla iel efnu olku euru hldfz, S isce hl ujmrucc kz duumwz buwh hefvxc blr heu grufh elvlar iesoe zla fru olvburrsvg lv ku.
And finally, the one given by Robert Edward Lewand in Cryptographical Mathematics:
Dr. Rutlor, Dr. Wufg, Jfwiuc fgw Nugljudug
Lo lhu 3rw Aftejlq oa Duwitigu oa lhu Thfrjuc Egivurcilq fgw ilc Wufg, Proauccor Houcthj, fgw fjj oa qoe mho hfvu todu huru lowfq, I mich lo uxprucc dq wuupjq aujl lhfgkc aor lhu nrufl hogoer mhith qoe fru togaurrign og du.
In the first case, it got 47 letters right (not counting how many letters out of the 26 it matched up right, just counting how many letters in our “decrypted” paragraph match the original, ignoring spaces, punctuation, and numbers (out of 209)) and managed to turn “honour” into “pogoer”.
The second only got 25 right. For shame, Mr. Morse.
The third got a whopping 67 right, though “hogoer” isn’t really a word.
Unsurprisingly, the cryptography frequency chart came closest, but none of them did particularly well. Presumably they’d do better with more text, but it still surprised me.
It’s particularly surprising how few vowels they got right: both ETAOIN SHRDLU and the cryptography one got O, and the cryptography one also got I. And that’s it. Morse didn’t get a single one, and none of them got A, E, or U.
I suspect the “decrypted” text would be much easier to use as a starting point for a human code breaker if more vowels had been found. Maybe Popper is just really atypical, I don’t know.
Either way, this is just a taste of what I’ve been doing since my last exam on Friday. More to come, I’m sure.
I was browsing Reddit (with the quality of Slashdot’s editing going from utterly crap to considerably worse, and the discussion going from merely ignorantly libertarian to frothing-at-the-mouth ultra-conservatism, I need a new place to get my tech news from, so I’ve been trying a few other websites), and I came across this article on the “eight ball problem”.
If you’re averse to clicking links, the idea is that you’re given eight balls, one of which is slightly heavier than the others, and a set of scales. Your task is to find the heavier ball in as few weighings as possible.
I’ve been told it’s a popular job interview question, and the author of the article is probably right in saying that most computer science types would naturally try a binary search, which would get you the heavier ball in three weighings.
He then goes on to describe an algorithm that does it in two weighings: first you weigh balls 1 through 3 against 4 though 6. If both sets weigh the same, the heavier ball is either 7 or 8, and you just weigh those against each other. If not, take the set of three that was the heaviest and weigh the first ball against the second. If they weigh the same, the heavier ball is the third. If not, you have found the heavier ball.
This is indeed faster for eight balls, but is it faster in all situations, and how much?
The algorithm can be generalised as follows:
The astute among you may have noticed that while the binary search algorithm will always finish in the same time, this isn’t always true for this other algorithm; its best- and worst-case performance times differ for sets of balls that aren’t neatly divisible by 3 all the way down.
It’s not going to be a huge difference, especially for small amounts of balls, but it’s good to note.
The binary search will always take log2(n) weighings to find the heavier ball, where n is the number of balls. It’s not immediately obvious how many the new algorithm will take, so let’s just write a program simulating it and run it a bunch of times.
While this seems an obvious place to use a functional language, I’m sorry to admit that I don’t yet feel comfortable with Haskell, and Scheme isn’t very legible, so I did it in Java.1
That program outputs data in a gnuplot-friendly format, so let’s use it (gnuplot doesn’t like logarithms in base 2, but you’ll remember from your high school maths that log(n) / log(2) is equivalent):
We noted earlier that the second algorithm won’t always take the same amount of time, so the plotted line is the average time it will take. If you want to see the best- and worst-case performances plotted as well, you can do so yourself. The output data is here.
So yes, it does look like this algorithm is always faster, though binary search should really be fast enough for most purposes, and it’s somewhat easier to implement.
For 10,000 balls, this algorithm takes 8.84 weighings on average (and 9 in the worst case), while a binary search takes 14. For a million, it takes about 12.95 (13 in the worst case), and a binary search takes 20.
Unless someone is killing one of your loved ones for each weighing, I wouldn’t bother.
1 Even as Java goes this is slower than it could be. For legibility’s sake I’m using Ball objects of my own making when I could be using ints or even booleans to represent the balls, for instance, and a getWeight() method call rather than a not-that-much-longer inline calculation. I invite you to write your own, better code in your favorite language.
Update: Like Python, for example. That was surprisingly painless, especially considering that it’s the first thing I’ve ever written in Python. It took maybe a third of the time it took me to write the Java version to write, and it’s not even that much slower, even though it’s an interpreted language.