NAME

lab 103 - python content assist

NOTES

Building on lab 102 I've started to write an acme client for python content-assist in python. I read the code for python's IDLE editor and pulled some pieces out of that to help with my implementation. The AutoComplete.py and CallTip.py modules are copied out of idlelib and reduced to just the functions I needed.

Just as in lab 102 the acme client is implemented using pyxp to connect to the acme file system.

In an acme editor window execute PyAssist. Start typing some python. After typing a dot the client will display available attributes for the module. Continue typing and it will reduce the list to those names with a matching prefix. Press the Tab key to complete a name or prefix depending on whether there is one or more names.

After typing an open parenthesis the client will show the call tip for the function.

The client implements an Import command so that more python modules can be added to the namespace. PyAssist doesn't attempt to parse the edited script. Idlelib includes code that could help to do this. But for now you need to pass a module name as argument to the Import command.

A next step for python integration to acme is to implement a debugger. Python comes with libraries for this too, so a debugger similar to the acme debugger adeb should be a future lab. I might then run edited scripts in a sub shell and inspect that process' namespace similar to the IDLE implementation, so that content-assist adapts to the script being edited.

FILES

inferno-lab/103

Why is it that cfs(4) doesn't have an implicit option to cache only to memory with some specified size -- seems a natural case you'd want to accomodate. I know you could always set up a ramfs and create a disk image, but that adds extra layers and overhead that seem fundamentally unnecessary. Perhaps its just that older systems had precious little ram and this seemed like an unnecessary novelty. Additionally, for BG/P it seems like we might want to add dcache like functionality (if only for a read-only root partition). Of course, in the longer term we'll need a more sophisticated caching methodology anyways.

Traditionally Plan 9 cpu servers mount a file server to get their executables and what not -- however, the cpu command includes file service back to the operating terminal via /mnt/term. With a proper setup I don't see any reason why we can't switch to just using the terminal's file system as the "primary" file system for a cpu server. This eases use of traditional tools (like source control where necessary) and offline access, even allows people to use non-Plan 9 editors if they so desire. Most importantly it circumvents fossil which seems to be causing folks no end of trouble. The host file system can be backed up with venti using p9p, which while not quite as well integrated as fossil can give the basics of functionality that most people use. The interface can be made better and dump mechanisms much more efficient, but I think we'd be better off focusing on that versus "fixing" fossil.

NAME

lab 102 - python acme client

NOTES

A recent post to 9phackers announced Pyxp, another implementation of Styx in Python.

I immediately downloaded Pyxp and tried it out. I had no trouble using it so I started thinking about python clients I could write. Python is still new to me so writing a styx client was an excuse to get more practice.

I started with an acme client. I translated the acmewin limbo module I use for most acme-sac clients to python. Below is a simple example opening a new acme window and doing some operations on it.

from acmewin import Acmewin

win = Acmewin()
win.writebody("hello, world!\n\n\n")
win.tagwrite("Hello")
win.writebody("goodbye")
win.replace("/goodbye/", "GOODBYE")
win.select(",")
win.show()

Remember to export the namespace before trying it out.

% styxlisten -A 'tcp!*!localhost' export /
% textclient.py

I recently saw on Hacker News a repost of Peter Norvig's spelling corrector. I thought this would make an easy first trial of my python acmewin code. I implemented a client to offer spelling suggestions to an editor window. It works somewhat like my earlier acme content assist code. This client opens the event file of another window it is assisting and writes text out to its own window. In this case it offers a suggested spelling for the word currently being typed.

Here's the implementation. Note that this is single threaded and it is not reading the event file of the second window. I haven't gotten that far in the Python book.

#!/dis/python26

import sys
from acmewin import Acmewin
import spell

win = Acmewin(int(sys.argv[1]))
outwin = Acmewin()

while True:
    (c1, c2, q0, q2, flag, nr, r) = win.getevent()
    if c2 in "xX":
        if flag & 2:
            win.getevent()
        if flag & 8:
            win.getevent()
            win.getevent()
        win.writeevent(c1, c2, q0, q2)
        if c2 == "x" and r == "Del":
            outwin.delete()
            break
    if c1 == "K" and c2 == "I":
        ch = r[0]
        if ch in " \t\r\n":
            outwin.replace(",", "")
            continue
        while q0 >= 0 and not (ch in " \t\r\n"):
            sss = win.read(q0, q0+1)
            if not sss:
                # print("empty sss %d" % q0)
                sss = " "
            ch = sss[0]
            q0 -= 1
        if q0 < 0 and not(ch in " \t\r\n"):
            q0 = 0
        else:
            q0 += 2
        ss = win.read(q0,q2)
        lastcorrect = spell.correct(ss)
        outwin.replace(",", lastcorrect)

To run this we need to know the id of the window we are assisting, so we need a wrapper to send the $acmewin environment variable as an arg to the script. For that I have a script called SpellAssist.

#!/dis/sh
$home/python/assist.py $acmewin

Now that I have a simple assist-like client working I'd like to develop it further. I'd like to try having content assist for python inside acme. It should be possible to adapt the python code that implements the IDLE editor to this purpose.

FILES

inferno-lab/102
big.txt the large text file used to train the spelling corrector. Note that the path is hardcoded in spell.py and should be changed locally.

NAME

lab 101 - limbo B+ tree

NOTES

This lab is an implementation of a B+ tree. It is code from a much older lab that was overly complicated and buggy. I pulled the B+ tree code out, tried to fix the bugs and clean it up. The interface is roughly the same as dbm(2). Here is a synopsis.

include "btree.m";
btreem := load Btreem Btreem->PATH;
Datum, Btree: import Btreem;

Btree: adt {
 create: fn(file: string, perm: int): ref Btree;
 open: fn(file: string, flags: int): ref Btree;
 
 fetch: fn(b: self ref Btree, key: Datum): Datum;
 delete: fn(b: self ref Btree, key: Datum): int;
 store: fn(b: self ref Btree, key: Datum, val: Datum):int;

 firstkey: fn(b: self ref Btree): Datum;
 nextkey: fn(b: self ref Btree, key: Datum): Datum;

 flush: fn(b: self ref Btree);
 close: fn(b: self ref Btree);
};

init: fn();

Like Dbm the keys and values are stored as arrays of bytes, and being a B+tree the values are stored only in the leaf nodes. The maximum key and val size is 255 each. The block size is 8192, so the maximum leaf node size is 515 making the minimum branching factor 15. The maximum number of records in an internal nodes assuming an int as key is 630. The delete is not fully implemented; it doesn't merge nodes. Here is some example code listing the full contents of a btree.

sys := load Sys Sys->PATH;
btreem := load Btreem Btreem->PATH;
Datum, Btree: import btreem;

btreem->init();
bt := Btree.open("index.bt", Sys->ORDWR);
for(key := bt.firstkey(); key != nil; key = bt.nextkey(key)){
 v := bt.fetch(key);
 sys->print("%s %s\n", string key, string v);
}

Here is a rough comparison of btree vs. dbm. This simple test is more of a sanity check that the btree doesn't do anything horribly wrong in its implementation (which it did originally, by making a syscall to get the daytime for every block it tried to get).

% awk '{print $1, NR}' < /lib/words | tr -d 'cr' > t1

% >index.bt
% >dbm.pag
% >dbm.dir

% time sh -c 'btree/store < t1' 
0l 4.172r 4.172t

% time sh -c 'dbm/store dbm < t1'
0l 35.25r 35.25t

% ls -l dbm.* index.bt
--rw-rw---- M 6 xcs0998 XCS0998    8192 Jun 04 12:38 dbm.dir
--rw-rw---- M 6 xcs0998 XCS0998 1048576 Jun 04 12:38 dbm.pag
--rw-rw---- M 6 xcs0998 XCS0998 770048 Jun 04 12:36 index.bt

% time sh -c 'btree/list > /dev/null'
0l 5.187r 5.187t

% time sh -c 'dbm/list dbm > /dev/null'
0l 9.438r 9.438t

FILES

inferno-lab/101

NAME

lab 100 - limbo tags

NOTES

I've been playing more with exuberant ctags. It's possible to make ctags recognize limbo source code using regular expressions to identify symbols we want tagged.

Here's a command I defined called ltags. It will search for functions, assuming the function name is not preceded by spaces, and, more reliably, find adt and module definitions:

#!/dis/sh
os -t ( /n/D/ctags57/ctags.exe  -n 
 '--tag-relative=yes'
 '--langdef=limbo' 
 '--langmap=limbo:.b.m' 
 '--regex-limbo=/^([a-zA-Z][a-zA-Z0-9]+)\(.*\)/\1/m/' 
 '--regex-limbo=/([a-zA-Z][a-zA-Z0-9]+) *: *adt */\1/t/'  
 '--regex-limbo=/([a-zA-Z][a-zA-Z0-9]+) *: *module */\1/c/'  
 $* ) 

Try running it under /, and use relative path names:

 
% cd /
% ltags -R appl module

And open the acme client

Ctag /tags

I added a command "Tag" to the Ctag program that will find and print all matching symbols to the name given as argument. Mouse-chording makes this nice to use. Right-clicking on symbol names will open the source file at that location. Browsing the whole limbo source hierarchy is then made quite easy.

FILES

inferno-lab/100

NAME

lab 99 - catching up with inferno-os

NOTES

I had fallen way behind in keeping acme-sac code up to date with inferno-os. I tried to correct that this week by merging changes and adding files from inferno-os.

I wrote a script, part of this labs files, to check the differences between inferno-os and acme-sac. I'm hoping by using this script I'll stay up to date with inferno-os as changes are made. The script compares sub-trees, such as the limbo source code hierarchy, the C source code, the manual pages, and the /lib directory. It prints out commands to diff the files or copy the file over.

% inf
inferno synchronization:
  appl    show changed limbo files
  sys     show changed C source files
  man     show changed man pages
  lib     show changed /lib files
  cmd     show changed sh scripts below /dis
  update  refresh file tree info for local and remote
  adiff   adiff selected file with inferno-os equivalent
  diff    diff selected file with inferno-os equivalent
  missing show files missing from local tree
  added   show files added to local tree
% inf update
% inf sys
diff -u /sys/emu/port/win-x11a.c /n/D/inferno-os/emu/port/win-x11a.c
cp /n/D/inferno-os/emu/port/win-x11a.c /sys/emu/port/win-x11a.c

% 

There is a short list of directories I did not synchronize because I know every file has changed; I consider these a fork, though the changes to charon are not dramatic,

/appl/acme/  /appl/charon/  /acme/

There is a list of directories acme-sac has removed from the inferno-os hierarchy:

/FreeBSD/  /libbio/     /utils/
/Hp/       /libdynld/   /appl/alphabet/
/Inferno/  /libkern/    /appl/collab/
/Irix/     /liblogfs/   /appl/demo/
/NetBSD/   /libnandfs/  /appl/ebook/
/OpenBSD/  /libprefab/  /appl/spree/
/Plan9/    /libtk/      /appl/tiny/
/asm/      /tools/      /appl/wm/

And finally there is a sub-tree we need to actually synchronize:

Linux    appl/grid lib       libkeyring  libmp   module
Nt       appl/lib  lib9      libmath     libsec
appl     appl/math libdraw   libmemdraw  include
appl/cmd emu       libinterp libmemlayer man

After comparing these sub-trees I built a list of files that were intentionally removed and files locally changed. Any file found "missing", where inferno-os added new functionality, was copied over. Files that were not marked as locally changed but nevertheless contained changes were merged, usually overwritten with the latest from inferno-os. There is also a list of files added to acme-sac apart from anything below /acme.

I found it useful to use the side-by-side diff functionality from lab 95. I added some simple interactive merge functionality that worked if you start from the end of the file and stepped backwards through the diffs using a PrevDiff command on the adiff output and calling the command Merge to apply the currently highlighted change. The implementation of this Merge command was pretty simple because it involved just copying the selected text from the source file over to the target. Given acme's filesystem interface this is literally a file copy,

cp /mnt/acme/$b/rdsel /mnt/acme/$a/wrsel

The PrevDiff commands does the work of setting up the selected texts correctly, properly handling the case of added, changed, or deleted text.

The list of files in the changed list should be reduced. Some of these are changes that could be pushed to inferno-os. E.g., diff adds the -u flag, os has changes from rog implementing host file name translation. Some files will need to be continually merged, such as the man page indices.

The list of files and directories removed are mostly just functionality either specific to running inferno as a native os, or the Tk functionality. There shouldn't be anything there people would really miss in an acme-sac environment. Though if there is let me know.

FILES

inferno-lab/99