djfroofy

I has Game

2008-04-06T02:00:15Z

ロボットゲーム(pyweek6のプロジェクト)

pyglet(又OpenGLとmiruも使い)でほとんど純粋Pythonゲーム〇アプリケーション

Twistedを使って、ゲームのアクターとSchedulingを制御する。

Links:

Pyweek details:

http://www.pyweek.org/e/etf/

To play:

  $ hg clone http://hg.enterthefoo.com/roboto Roboto
  $ cd Roboto
  $ python run_game.py (less)

The Problem with Doctests

2008-01-31T14:38:21Z

I used to think python doctests were the greatest thing since sliced bread. I no longer think this and I've returned to the basics (most of the time): plain old unittests. So let me explain why.

This is the crux of the matter: documents are supposed to be documents and readable as such. Too often I see doctests that stray from readability in favor of including chunks of code which achieve testability - test set up, tear down, white space stripping ... For example, let's look at a part of README.txt in zope.event (and the Zope camp definitely embraces doctests):

The package has a list of subscribers.  Application code can manage
subscriptions by manipulating this list.  For the examples here, we'll
save the current contents away and empty the list. We'll restore the
contents when we're done with our examples.

  >>> import zope.event
  >>> old_subscribers = zope.event.subscribers[:]
  >>> del zope.event.subscribers[:]

This isn't documentation! You're wasting time (your time and mine) explaining something to me that has nothing to with how to use the interface. doctest does provide facilities for executing setUp and tearDown functions in regular python modules prior to executing a test, but I seldom see this approach taken. Why? Well I guess this requires more work, and it seems the appeal with doctests is doing less work. I can relate, and even took a crack once at automating specification of setUp/tearDown functions: http://xix.python-hosting.com/file/trunk/xix/utils/test.py .

And while we're on the subject of code I've written, here's another example of a bad doctest:

>>> action.addMessageCategoryCallback(thread1, t1)
>>> action.addMessageCategoryCallback(thread2, t2)
>>> action.addMessageCategoryCallback(metaname, mname)
>>> from loggrok.log import LogStream
>>> stream = LogStream(fname)
>>> stream.messageParser = parser
>>> stream.action = action
>>> for entry in stream:
...     continue
...
====
THREAD1 C
D E
F G
H I
====
THREAD1 J K
====
THREAD2 L M
N
====

WTF? This isn't a document! This is a plain old unittest, disguised as a document! Stop wasting my time!

Okay, so I've been a little harsh. I've tried hard to rant as little as possible on this blog, but today I've fallen short of that goal. But please don't get what I'm saying wrong. I'm not asserting that doctests are useless and you should never use them. (Har har! Get it? "I'm not asserting ...") In fact doctests can be great but I don't think they're effective for fully testing an application - since this will always conflict with writing apt and succinct documentation. What they are good for, if I can suggest, is to verify that examples in a documentation don't contain syntactic errors and are up to date with the current code base. Of course some setUp and tearDown routines may be needed to ensure the examples actually work or don't leave the interpreter in bad state when complete, but there are several approaches to not revealing this in the document (for example, put the setUp/tearDown in comments or use the facilities providing by the doctest library to execute such functions).

On side note, it really isn't worth fussing about, at least on a policy level. If someone on your team is writing unittests (be it doctests or old school unittest), it's best to hold your lip and not engage in bickering about what format they should be in. What matters most is that the application is being tested and the project is approaching good line and combinatorial branch coverage.

World Coordinate Picking

2008-01-30T03:49:26Z

Manipulating an object's world coordinates with OpenGL is tricky. This is an operation which involves translating the screen coordinates to a position in the world. The "tricky" part of this is determining the depth of the object in the range [0,1] to use as the wz argument to gluUnProject (a function that does the hard screen coordinate to world coordinate transformation for us).

Unfortunately, googling for gluUnProject yields some very bad results. For example the NeHe tutorial gives a wretched example using glReadPixels with GL_DEPTH_COMPONENT as the format. This is a really, really stupid idea. First of all, imagine an object is in focus (I've selected it as the target from a drop down menu, or some other way), and then I click on some arbitrary point on the screen. Well, guess what kids? I get the window depth at that point (not the point corresponding to the object I'm interested in). Even if I point directly over the object, it's easy for the mouse to race ahead of the object while dragging it across the screen, and (boom!) the window z depth is 1.0, or the far clipping plane, and my object flies of the friggin' screen. Beautiful!

Other links that crop up literally hard code the wz depth value which only makes sense if you're manipulating an object sitting directly on the near clipping plane.

So, here's how you do it, and it's very simple. To determine the window depth of the object, first do the reverse of gluUnProject (that is, gluProject) based on the coordinates of the object in focus. Here's my final (working) code in python:

        viewport = (GLint * 4)()
        mvmatrix = (GLdouble * 16)()
        projmatrix = (GLdouble * 16)()
        glGetIntegerv(GL_VIEWPORT, viewport)
        glGetDoublev(GL_MODELVIEW_MATRIX, mvmatrix)
        glGetDoublev(GL_PROJECTION_MATRIX, projmatrix)
        wx = GLdouble()
        wy = GLdouble()
        wz = GLdouble()
        sz = GLdouble()
        ex = target.pos.x
        ey = target.pos.y
        ez = target.pos.z
        gluProject(ex, ey, ez, mvmatrix, projmatrix, viewport, wx, wy, sz)
        gluUnProject(x, y, sz,
          mvmatrix, projmatrix, viewport, wx, wy, wz)

        ...
        target.pos.x = wx.value
        target.pos.y = wy.value

Bidirectional Communication over AMP

2007-12-29T18:11:13Z

The following is a simple server that exposes two commands (Echo and Foo). The responder for Foo, in addition to returning an empty dict, will try to call Echo on the client.

ampbidir.py

from twisted.internet import reactor
from twisted.protocols import amp

class Foo(amp.Command):
    response = []

class Echo(amp.Command):
    arguments = [('value', amp.String())]
    response = [('value', amp.String())]


class TheServer(amp.AMP):

    def foo(self):
        def e(v):
            print '[TheServer]', self.transport.getPeer(), 'echoed', v
        self.callRemote(Echo, value='hello').addCallback(e)
        return {}
    Foo.responder(foo)

    def echo(self, value):
        print 'Echo:', value
        return {'value':value}
    Echo.responder(echo)


def main():
    from twisted.internet import reactor
    from twisted.internet.protocol import Factory
    pf = Factory()
    pf.protocol = TheServer
    reactor.listenTCP(1234, pf)
    reactor.run()

if __name__ == '__main__':
    main()

The client will simply start up and call Foo to demonstrate bidirectional support in amp:

client.py

from ampbidir import TheServer, Foo

from twisted.internet.protocol import ClientCreator
from twisted.internet import reactor

client = ClientCreator(reactor, TheServer).connectTCP(
        '127.0.0.1', 1234).addCallback(lambda p: p.callRemote(Foo))
reactor.run()

In the above example, we expose Foo and Echo from the client to the server by creating the client with the protocol TheServer, rather than the generic amp.AMP which exposes nothing, of course.

Hah

2007-12-28T23:43:56Z

from twisted.python import filepath
import mmpython as mmp
import time

EXT = '.mp3'
TEMP = '%2.2d_%s_-_%s_-_%s' + EXT
NESTED = True

def rp(s):
    return s.replace('\x00', '').replace(' ', '_').replace(':','').encode('utf-8')

def job(desc):
    info = mmp.parse(desc.path)
    if not info:
        print 'WARNING: no info found on', desc
        return 'No info found on: %r' % desc
    name = TEMP % (int(info.trackno), rp(info.title), rp(info.artist), rp(info.album))
    print 'Parsed', desc.basename(), 'as', name
    if NESTED:
        fp = filepath.FilePath(rp(info.artist), rp(info.album))
        tgt = tgt_dir
        for c in (rp(info.artist), rp(info.album)):
            tgt = tgt.child(c)
            try:
                tgt.createDirectory()
            except:
                pass
        tgt = tgt.child(name.encode('utf-8'))
    else:
        tgt = tgt_dir.child(name.encode('utf-8'))
    return desc.copyTo(tgt)

def main():
    for desc in [f for f in root.walk() if f.isfile()]:
        job(desc)

if __name__ == '__main__':
    import sys
    root = filepath.FilePath(sys.argv[1])
    try:
        tgt_dir = filepath.FilePath(sys.argv[2])
    except:
        tgt_dir = filepath.FilePath('_ip-recover-%d' % int(time.time() * 1000000))
        tgt_dir.createDirectory()
    main()

Mural Complete

2007-11-13T18:45:17Z

The mural is complete ... After about 9 hours and several bottles of acrylic. So this isn't technical but it will be after I embed micro-controllers in the wall to make the animals' eyes glow. Ok, that might freak my daughter out though. Yes, that's a mountain lion, a bunny and a pig. There is a reason for all these elements.

Erlang and binary

2007-07-31T01:15:29Z

Messing around some with erlang, I've really been impressed with the bit syntax which makes dealing with binary data actually fun. In erlang, binaries can be expressed using the following syntax:

 1> A = <<205>>.
 <<"\315">>

So A is no bound to the binary value 205 - which is by default 1-byte in length. With the following syntax, we can extract the first four bits with out any masking:

 2> <<_:4,B:4>> = A.
 <<"\315">>
 3> B.
 13

We can also deduce the first 4 bits without shifting and/or masking:

  4> <<C:4,_:4>> = A.
  <<"\315">>
  5> C.
  12

So, what's the point? Think about binary protocols (mp3, unicode, ip4, ...). Dealing with this in your run-of-the mill language generally entails the same tedious mechanics - gross amounts of bit-shifting and masking values to arrive at the required values to decode the protocol, which necessitates a cognitive shift from the protocol's specification to the implementation. Using erlang's bit syntax allows me to almost match the protocol verbatim. Take for example, the following function which turns utf-8 encoded binary data (probably read from a file) into a list of code points (or blows up correctly if the data is corrupted):

01. utf8points(Bin) ->
02.    utf8points(binary_to_list(Bin),[]).
03.
04. utf8points([], L) -> lists:reverse(L);
05. utf8points([H|T], L) ->
06.    case <<H>> of
07.        <<2#110:3,  _:5>> -> decode2([H|T], L);
08.        <<2#1110:4, _:4>> -> decode3([H|T], L);
09.        <<2#11110:5,_:3>> -> decode4([H|T], L);
10.        <<0:1,_:7>>       -> utf8points(T, [H|L]);
11.        _                 -> exit({decode_error, {bad_bytes, [H]}})
12.    end.

Ok, so here's a synopsis of the above code fragment:

(Lines 1-2) This function takes the binary (Bin), transforms it to a list of byte values and applies utf8points/2 on it.

(Line 4) We use the list L as an accumulator for the code points. If the list of byte values has been exhausted, we simply return the list reversed - since we are appending to the head of the accumulated list along the way..

(Lines 5-12) The list of bytes has not been exhausted. We consider the first byte of the list to decide whether this bytes alone or in combination with 1-3 bytes following it should be used to determine the code point. If it's one byte, (line 10), we prefix the accumulator and recurse with the remaining bytes. In the other cases we pass control to helper functions (decode2, decode3, decode4), depending on the leading bits of the byte. (2#N is used to represent numbers in base 2).

And decode3 uses some similar logic:

decode3([A,B,C|T], L) ->
    case {<<B>>, <<C>>} of
        {<<2#10:2,_:6>>,<<2#10:2,_:6>>} ->
            <<V:16>> = <<A:4,B:6,C:6>>,
            utf8points(T, [V|L]);
        _ ->
            exit({decode_error, {badbytes, [B,C]}})
    end;
decode3(_, _) ->
    exit(error_eof).

decode2 and decode4 are left as an exercise to the reader ;)

Two CPUs are Fun

2007-04-17T01:26:17Z

cpuinfo from proc:


processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 CPU         T7600  @ 2.33GHz
stepping        : 6
cpu MHz         : 1000.000
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm 
constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm
bogomips        : 4661.07
clflush size    : 64

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 CPU         T7600  @ 2.33GHz
stepping        : 6
cpu MHz         : 1000.000
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm 
constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm
bogomips        : 4655.17
clflush size    : 64

Task count per processor:


$ for cpu in 0 1; do echo cpu $cpu task ct : `ps axf -o psr | egrep "$cpu" | wc -l`; done
cpu 0 task ct : 65
cpu 1 task ct : 53

CPUTIME in seconds on each processor (low load):


>>> o = commands.getoutput('ps axf -o psr,bsdtime')
>>> seconds = {'0':0,'1':0}
>>> for line in o.split('\n'):
...     tks = line.split()
...     if tks[0] == 'PSR': continue
...     ms,sec = tks[1].split(':')
...     psr = tks[0]
...     seconds[psr] = int(ms) * 60 + int(sec)
... 
>>> seconds['0']
5
>>> seconds['1']
196

Ubuntu Dapper/Edgy/Feisty Broadcom Corporation NetXtreme BCM5752 (module bcm43xx) Wireless Nightmare

2007-04-16T12:33:48Z

Do NOT use firmware (at least from cafeugo) or modules for Broadcom wirless cards - you will experience much pain.

Read this ... please: http://www.linuxquestions.org/questions/showthread.php?t=463002

In short:

Blacklist el' bcm43xx
Install windows driver from here using ndiswrapper
And the ndiswrapper module as an alias for your ethernet device (ex. eth1).

My setup:

djfroofy@kieru:~# uname -a
Linux kieru 2.6.20-15-generic #2 SMP Sat Apr 14 00:54:01 UTC 2007 i686 GNU/Linux

djfroofy@kieru:~# lspci
...
09:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5752 Gigabit Ethernet PCI Express (rev 02)
0c:00.0 Network controller: Broadcom Corporation Dell Wireless 1390 WLAN Mini-PCI Card (rev 01)

vimrc Revisited

2006-11-17T16:49:37Z

After upgrading to vim 7 I began to notice some subtle (and annoying) effects on my unwieldy
vimrc configuration due to years of impatient hacking and mindless cutting and pasting from
examples. Auto indentation seemed to be erratic for one thing. So I've scrapped my original and started from scratch doing some research on appropriate settings for python-centric hacker like myself. It seems to be working nicely so far, so I'll post here for anyone else interested:

" For Python, replace existing tabs with 8 spaces - we use ``filetype indent on`` so it
" isn't a problem
au BufRead,BufNewFile *.py set tabstop=8
au BufRead,BufNewFile *,*.py set expandtab
" This is good for non-*nix machine - force proper newline '\n'
au BufNewFile *,*.py set fileformat=unix

set encoding=utf-8
filetype indent on
" Haven't noticed any difference in highlighting but here it is
let python_highlight_all=1

" Haskell specific settings
" Don't automatically insert comment header continuation '--' (it's annoying)
au BufRead,BufNewFile *.hs,*.lhs set formatoptions-=c formatoptions-=o formatoptions-=r
" We have a nice plugin for smart python folding, but for haskell ... we'll
" base it on indentation alone:
au BufRead,BufNewFile *.hs,*.lhs set foldmethod=indent

" This could get annoying, you'd be oh so tempted to waste time cleaning them up
" set list listchars=trail:_

syntax on
colorscheme desert

map gs :%s
" Kill the annoying beeps
set noerrorbells
if has('autocmd')
    autocmd GUIEnter * set vb t_vb=
endif

Continuation Passing Style

2006-11-09T16:26:28Z

So I'm reading Daume's YAHT - quite excellent by the way - and treading along just merrily until the section on continuation passing style (CPS):

  cfold' f z [] = z
  cfold' f z (x:xs) = f x z (\y -> cfold' f y xs)
  cfold f z l = cfold' (\x t g -> f x (g t)) z l

This ... makes ... my brain ... hurt.

Just had to get that out of my system. That feels better now.

Number Theory

2006-11-02T19:40:06Z

Aren't positive integers the damn'dest things? I just noticed today the sum over (1,2,3,4,5,6,7,8,9,10,9,8,7,6,5,4,3,2,1) was 100. And so I tried for (1,2,3,4,5,4,3,2,1) and alas the sum was 25. So here's my proof by MI for the "Mirrored Square" (I'm sure there's some official term for this numerical phenomenon buried in the number theory literature):

WTS
For all positive n:
n^2 = 1 + ... + (n-1) + n + (n-1) + ... + 1
-------------------------------------------

Is it true for 1 and 2?

1 = 1^2
2^2 = 4 = 1 + 2 + 1

Yes.

Let it be true for some positive k-1.

(k-1)^2 = 1 + ... + (k-2) + (k-1) + (k -2) + ... + 1

Showing it is true for k also:

k^2 = ((k - 1) + 1)^2 
    = (k - 1)^2 + 2(k - 1) + 1 
    = (k - 1)^2 + k + (k -1)
    = 1 + ... + (k -1) + k + (k - 1) + ... + 1
--
EP

Sound in Firefox on Ubuntu Dapper

2006-10-04T19:12:25Z

Just some knowledge I've gleened from the forums:

$ sudo apt-get install alsa-oss
$ sudo gedit /etc/firefox/firefoxrc
# Replace FIREFOX_DSP="none" with:
FIREFOX_DSP="aoss"

And to keep the browser from locking up:

$ sudo ln -s /usr/lib/libesd.so.0 /usr/lib/libesd.so.1
$ sudo mkdir -p /tmp/.esd
$ sudo touch /tmp/.esd/socket

JPyping

2006-10-04T01:21:27Z

JPype ... I just don't know where to start - I mean interacting with Java objects from CPython!?

So I've brewed up some simple experiments and was amazed to see that it just freaking works. Starting with a simple java class:

package com.xix;

public class Example {
    private String name;
    public Example() {
        this("anon");
    }
    public Example(String name) {
        setName(name);
    }
    public void setName(String name) {
        this.name = name;
    }
    public String getName() {
        return name;
    }
}

And a little wiring:

import jpype as jp
import os, sys

env = os.environ.get
_iswin = sys.platform in ('win32', 'nt')

JAVA_HOME = env('JAVA_HOME')

if not JAVA_HOME:
    raise RuntimeError, 'Environment variable JAVA_HOME not defined'

JVMLIB = os.path.sep.join([JAVA_HOME] + 'jre/lib/i386/client/libjvm'.split('/'))
_dllext = ('.so', '.dll')[_iswin]
JVMLIB += _dllext

CLASSPATH = env('CLASSPATH')

jp.startJVM(JVMLIB, '-Djava.class.path=%s' % CLASSPATH)

Ok ... so the interface to importing java classes is kind of weird but sensible nonetheless:

>>> from jpype import JPackage
>>> com = JPackage("com")
>>> example = com.xix.Example('drew')

But of course shimming it is 2 cms away. And getter and setter mappings to properties work:

>>> example.name = 'djfroofy'
>>> example.name
u'djfroofy'

Python Decorators

2006-08-07T16:49:15Z

Up until now, the easy way to write Python 2.4 function decorators that takes arguments before the decoratorated function has been a complete mystery to me. Writing a decorator that only takes its wrapped function as the first agument is pretty straight-forward. Alas, just a function that wraps another. Take for example, a function that transforms another into a curryable function:

from xix.utils.python import Curried

# a utility function to give the decorator the same metainfo as the decorated
def _applymeta(wrapper, func):
    wrapper.__dict__.update(func.__dict__)
    wrapper.__doc__ = func.__doc__
    wrapper.__name__ = func.__name__
    return wrapper

# the decorator definition
def curryable(func):
    def decorator(*args, **kwargs):
        curried = Curried(func)
        return curried(*args, **kwargs)
    return _applymeta(decorator, func)

A decorator that takes additional arguments before the wrapped function can be though of as a decorator decorator, and so can involve three levels of nested functions - 1) a function that wraps the decorator (taking the initial argument), 2) the decorator/wrapper, and 3) the inner wrapper function:

def foo(*dpargs, **dkwargs):
    def decorator(func):
        def inner(*pargs, **kwargs):
            print 'pargs:', pargs
            print 'kwargs:', kwargs
            print 'decor pargs:', dpargs
            print 'decor kwargs:', dkwargs
            return func(*pargs, **kwargs)
        return _applymeta(inner, func)
    return decorator

Applying:

@foo(1,'a',k=23)
def mydef(a,b,z=34):
    """Documentation for mydef"""
    print a, b

WPA on Kanotix Linux 2.6.14

2006-08-06T15:09:34Z

Finally got wpa encryption working on my laptop ... after lots of pain. It all boiled down to using generic driver (wext) instead of specific ipw2100/2200 (ipw) which was giving me nasty ioctl errors. Here's my kernel details:

$ uname -a
Linux techwood 2.6.14-kanotix-9 #1 PREEMPT Wed Dec 28 10:17:53 CET 2005 i686 GNU/Linux

First you need to verify the settings on your router (WPA Personal, TKIP encryption algorithm, your passphrase). To generate the passphrase key and append to /etc/wpa_supplicant.conf:

$ wpa_passphrase yourssid yourpassphrase >> /etc/wpa_supplicant.conf

I edited my configuration and arrived at the following:

ap_scan=1
fast_reauth=1
network={
    ssid="myessid"
    pairwise=TKIP
    key_mgmt=WPA-PSK
    psk=deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef
    group=TKIP
}

To bring up the interface using wpa_supplicant:

$ wpa_supplicant -B -c /etc/wpa_supplicant.conf -i eth1 -D wext
$ dhclient eth1

I've also found some issues with eth1 getting recognized with ipw2100 driver and kanotix kernel. My hack solution has been to unload and reload ethernet driver and ipw2100 driver in that order, so I've bundled this all into one script:

#!/bin/sh

killall -9 wpa_supplicant &>/dev/null
ifconfig eth1 down &>/dev/null
# cleanly remove and reinsert mods (my ipw2100 hack)
rmmod tg3
rmmod ipw2100
modprobe tg3
modprobe ipw2100
echo "Starting wpasupplicant ..."
wpa_supplicant -B -c /etc/wpa_supplicant.conf -i eth1 -D wext
sleep 2
dhclient eth1

I'm curious if anyone else has had to jump through similar hoops. Anyhow, I'm happy now.