2008-09-19 00:02 <flipz_out> but can it pogo
2008-09-19 00:15 <shapor> ...
2008-09-19 00:20 <flipz_out> Shapor: # Idle priority is VERY cautious about marking block devices idle. If your foreground tasks are using disk, then your background tasks will become noticeably slower, as they get blocked from touching the disks until Linux knows for sure your foreground tasks have all had a chance at the disk. Most of the times, you don?t care about this anyway, but don?t run a torrent in non-idle class and expect a 20GB copy to finish till the torrent?s done!
2008-09-19 00:20 <flipz_out>  == lame
2008-09-19 00:20 <flipz> http://friedcpu.wordpress.com/2007/07/17/why-arent-you-using-ionice-yet/
2008-09-19 00:21 <flipz> got to be a hint why only phb oriiented vendors provide it by default
2008-09-19 00:21 <flipz> course... maybe that's every vendor ;)
2008-09-19 00:32 <shapor> yeah its not very fine grained
2008-09-19 00:33 <flipz> that VERY is a red flag
2008-09-19 00:34 <shapor> i didn't generate hard data but when i use a combinatino of ionice and nice compressing log files the system seems more responsive
2008-09-19 00:34 <shapor> than if i dont use them
2008-09-19 00:34 <shapor> i dont care how long it takes to compress my log files really
2008-09-19 00:35 <shapor> so if ANY other io wants my disk let it have it
2008-09-19 00:35 <shapor> if the log compression never finishes thats ok 
2008-09-19 00:35 <shapor> it is useful even in its current state
2008-09-19 00:36 <shapor> now i just need a version of cat which puts posix_fadvise in the io even loop so it doesn't piss on my buffer cache either
2008-09-19 00:37 <shapor> cat --dont-piss-on-my-buffer-cache
2008-09-19 00:37 <flipz> mount -ttux3 -oloop foodev /mnt
2008-09-19 00:37 <flipz> we start here.
2008-09-19 00:37 <flipz> wow! we got here
2008-09-19 00:38 <flipz> my cut n paste of mazes junk just worked
2008-09-19 00:38 <shapor> not junk
2008-09-19 00:38 <shapor> junkfs :)
2008-09-19 00:38 <flipz> junkfs reulz
2008-09-19 00:39 <flipz> unlike maze, I did not have to reboot my workstation
2008-09-19 00:39 <flipz> because I ran it under uml
2008-09-19 00:39 <flipz> worth getting that working
2008-09-19 00:40 <shapor> heh
2008-09-19 00:40 <flipz> mount: wrong fs type, bad option, bad superblock on /dev/loop0,
2008-09-19 00:40 <flipz>        or too many mounted file systems
2008-09-19 00:40 <flipz>        (aren't you trying to mount an extended partition,
2008-09-19 00:40 <flipz>        instead of some logical partition inside?)
2008-09-19 00:40 <flipz> ok, let's get out the junk mop
2008-09-19 00:40 <flipz> and make it presentable for tux3 checkin #3
2008-09-19 00:41 <shapor> i thought you were working on extents ;)
2008-09-19 00:41 <flipz> or #4 if you count the lame git checkin
2008-09-19 00:41 <flipz> this is related to extents, just as ketchup is related to ice cream
2008-09-19 00:42 <flipz> maze done good in here
2008-09-19 00:43 <flipz> I particularly like the little hexdump in 7 lines
2008-09-19 00:43 <flipz> 
ACTION cuts it down to 6

2008-09-19 01:00 <shapor> flipz: where did dir.c come from
2008-09-19 01:00 <shapor> looks a lot different from fs/ext2/dir.c in a recent kernel
2008-09-19 01:00 <flipz> dir.c?
2008-09-19 01:00 <flipz> oh
2008-09-19 01:00 <flipz> it's the same
2008-09-19 01:01 <flipz> just marginally cleaned up
2008-09-19 01:01 <flipz> and got rid of the page wanking
2008-09-19 01:01 <flipz> when back to buffer ops as god intended
2008-09-19 01:02 <shapor> changed the interface a bit?
2008-09-19 01:02 <flipz> not really
2008-09-19 01:02 <shapor> ext2_create_entry
2008-09-19 01:02 <shapor> ?
2008-09-19 01:03 <flipz> pretty much the same
2008-09-19 01:03 <flipz> that level isn't implemented
2008-09-19 01:03 <flipz> in tux3
2008-09-19 01:03 <flipz> well
2008-09-19 01:03 <flipz> it's in inode.c
2008-09-19 01:03 <shapor> well i dont see ext2_create_entry in the ext2/dir.c
2008-09-19 01:03 <shapor> in fact lxr says it doesn't exist
2008-09-19 01:03 <flipz> try namei.c
2008-09-19 01:03 <flipz> mknod
2008-09-19 01:03 <flipz> or something
2008-09-19 01:03 <shapor> did you rename it?
2008-09-19 01:03 <flipz> buncha verbosity
2008-09-19 01:04 <shapor> were they passing in a dentry before?
2008-09-19 01:04 <flipz> no
2008-09-19 01:04 <flipz> it's trivial
2008-09-19 01:04 <flipz> um
2008-09-19 01:04 <flipz> ok
2008-09-19 01:04 <flipz> you want to know the name
2008-09-19 01:04 <flipz> justa sec
2008-09-19 01:05 <flipz> ext2_add_link
2008-09-19 01:05 <flipz> dumb name
2008-09-19 01:05 <shapor> yeah thats what i thought
2008-09-19 01:06 <shapor> you changed the interface
2008-09-19 01:06 <flipz> I really didn't change much
2008-09-19 01:06 <flipz> did not want to discover new bugz
2008-09-19 01:06 <shapor> because they create the dentry first
2008-09-19 01:06 <shapor> and pass that
2008-09-19 01:06 <shapor> rather than a filename
2008-09-19 01:06 <flipz> hmm I did a little
2008-09-19 01:06 <flipz> because no dentries in tux3 userspace
2008-09-19 01:06 <shapor> and they call ext2_create seperately
2008-09-19 01:07 <shapor> hmm
2008-09-19 01:07 <flipz> caught me
2008-09-19 01:07 <shapor> perhaps there should be
2008-09-19 01:07 <shapor> to make kernel port easier
2008-09-19 01:07 <flipz> ext2 is not an exemplary model for namespace structure
2008-09-19 01:07 <shapor> hmm
2008-09-19 01:07 <flipz> this is all fs internal
2008-09-19 01:07 <shapor> ok
2008-09-19 01:08 <flipz> might as well drop some of the braindamage
2008-09-19 01:08 <flipz> good call on that though
2008-09-19 01:08 <shapor> i'm just trying to fix a bug in it ;)
2008-09-19 01:08 <flipz> bug!
2008-09-19 01:08 <shapor> i think you did introduce one
2008-09-19 01:08 <shapor> ;)
2008-09-19 01:08 <flipz> happens
2008-09-19 01:15 <shapor> feels like there are too many interfaces in ext2/dir.c
2008-09-19 01:15 <flipz> yes
2008-09-19 01:15 <flipz> a linux meme
2008-09-19 01:16 <flipz> making interfaces looks confusingly like productive work
2008-09-19 01:19 <flipz> now... why did maze put a wait queue inside the bio
2008-09-19 01:19 <flipz> looking forward to the explanation ;)
2008-09-19 01:19 <flipz> 
ACTION unborks

2008-09-19 01:20 <shapor> it seems like a lot of stuff is landing in our inode.c
2008-09-19 01:20 <flipz> sposed to put a pointer to the wait queue there, not the wait queue itself
2008-09-19 01:20 <flipz> sure
2008-09-19 01:20 <flipz> inode.c is a toilet
2008-09-19 01:20 <shapor> heh
2008-09-19 01:20 <flipz> by tradition
2008-09-19 01:20 <shapor> dont flush it!
2008-09-19 01:20 <flipz> might lose something good
2008-09-19 01:22 <shapor> ah so the vfs does indeed hand you a dentry
2008-09-19 01:22 <shapor> not a filename
2008-09-19 01:22 <shapor> man lxr is fucking slow
2008-09-19 01:23 <shapor> i'm going to run my own
2008-09-19 01:23 <shapor> damn europeans much be awake
2008-09-19 01:23 <flipz> good luck installing it
2008-09-19 01:23 <shapor> must even
2008-09-19 01:23 <flipz> let me know how it works out
2008-09-19 01:23 <shapor> hrm the interface is kinda crap
2008-09-19 01:24 <shapor> 
ACTION tries not getting sidetracked making lxr not suck as much

2008-09-19 01:29 <flipz> shapor, know a shell command for writing a few bytes at the beginning of a file without truncating the file?
2008-09-19 01:30 <shapor> reiserfs has some weird looking shit in it
2008-09-19 01:30 <flipz> you don't say
2008-09-19 01:31 <flipz> take a simple idea and make it weird
2008-09-19 01:31 <shapor> flipz: dd ?
2008-09-19 01:31 <flipz> how bout that shell command?
2008-09-19 01:31 <flipz> ah
2008-09-19 01:31 <flipz> didn't know it could do that
2008-09-19 01:31 <shapor> notrunc i think
2008-09-19 01:33 <shapor> conv=notrunc
2008-09-19 01:33 <shapor> lets you plop down data in it without truncating
2008-09-19 01:33 <flipz> dd conv=notrunc if=hello of=foodev
2008-09-19 01:33 <flipz> dd has a really weird command syntax
2008-09-19 01:34 <flipz> root@usermode:~# ./tux3
2008-09-19 01:34 <flipz> we start here.
2008-09-19 01:34 <flipz> wow! we got here
2008-09-19 01:34 <flipz> super =  68 65 6C 6C 6F 0A 00 00 00 00 00 00 00 00 00 00
2008-09-19 01:34 <flipz> mount: Not a directory
2008-09-19 01:34 <flipz> with maze's 'art' fixed
2008-09-19 01:35 <flipz> the number of right things in maze's little hack _vastly_ outnumbers the wrong things
2008-09-19 01:35 <flipz> but the wrong things are doozers ;)
2008-09-19 01:36 <shapor> "It is rumored to have been based on IBM's JCL, and though the syntax may have been a joke[1], there seems never to have been any effort to write a more Unix-like replacement."
2008-09-19 01:36 <shapor> from the wikipedia dd page
2008-09-19 01:36 <shapor> http://en.wikipedia.org/wiki/Dd_(Unix)
2008-09-19 01:36 <flipz> :p
2008-09-19 01:36 <flipz> longest running joke in unix
2008-09-19 01:37 <shapor> dd deprecated?
2008-09-19 01:37 <shapor> i think not
2008-09-19 01:37 <shapor> flipz: we should fix it :)
2008-09-19 01:38 <flipz> right, if only because we own the name
2008-09-19 01:38 <shapor> yup
2008-09-19 01:38 <flipz> ddcp
2008-09-19 01:38 <shapor> nah too long
2008-09-19 01:38 <shapor> and its not cp
2008-09-19 01:38 <flipz> dd --oldbroken
2008-09-19 01:38 <shapor> dd2
2008-09-19 01:38 <flipz> dd --muchbetter
2008-09-19 01:39 <shapor> ddd
2008-09-19 01:39 <shapor> or how about just "d"
2008-09-19 01:40 <flipz> dd with a symlink
2008-09-19 01:42 <shapor> hardlink
2008-09-19 01:42 <flipz> mandatory
2008-09-19 01:42 <shapor> provide legacy compatability if the argv[0] is dd
2008-09-19 01:42 <shapor> otherwise new hawtness
2008-09-19 01:43 <flipz> root@usermode:~# ./tux3
2008-09-19 01:43 <flipz> we start here.
2008-09-19 01:43 <flipz> wow! we got here
2008-09-19 01:43 <flipz> super =  68 65 6C 6C 6F 0A 00 00 00 00 00 00 00 00 00 00
2008-09-19 01:43 <flipz> root@usermode:~# mount
2008-09-19 01:43 <flipz> /dev/ubda on / type ext2 (rw)
2008-09-19 01:43 <flipz> proc on /proc type proc (rw)
2008-09-19 01:43 <flipz> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
2008-09-19 01:43 <flipz> /root/foodev on /mnt type tux3 (rw,loop=/dev/loop0)
2008-09-19 01:43 <flipz> that's enough for tonight
2008-09-19 01:43 <shapor> sweet
2008-09-19 01:43 <flipz> almost ;)
2008-09-19 01:43 <shapor> so i'm trying to get a backup of your git tree up on github.com
2008-09-19 01:43 <flipz> how'd you clone it?
2008-09-19 01:43 <shapor> they already have linus's tree
2008-09-19 01:44 <flipz> I failed
2008-09-19 01:44 <shapor> so i forked it
2008-09-19 01:44 <flipz> always forget how
2008-09-19 01:44 <shapor> now i'm just trying to push your changes in to it
2008-09-19 01:44 <flipz> just clone mine
2008-09-19 01:44 <shapor> i dont think i can
2008-09-19 01:44 <flipz> don't rebase to anything
2008-09-19 01:44 <flipz> well
2008-09-19 01:44 <flipz> I'll fix that
2008-09-19 01:44 <flipz> tomorrow
2008-09-19 01:44 <shapor> you need to have the git service running
2008-09-19 01:44 <shapor> maybe you do
2008-09-19 01:44 <flipz> I do
2008-09-19 01:44 <flipz> it's just configged borkly
2008-09-19 01:45 <flipz> git ui braindamage as much as anything
2008-09-19 01:45 <flipz> nothing is obvious
2008-09-19 01:45 <shapor> telnet phunq.net 9418
2008-09-19 01:45 <shapor> yeah you do
2008-09-19 01:45 <flipz> mercurial is altogether more usable in this and other ways
2008-09-19 01:46 <shapor> we should get the whole vfs running in user space
2008-09-19 01:46 <shapor> would be killer for testing
2008-09-19 01:46 <flipz> I'm milding interested in doing a dentry like thing
2008-09-19 01:46 <flipz> but we have fuse for that, really
2008-09-19 01:46 <shapor> fuse is... 
2008-09-19 01:46 <shapor> ugh
2008-09-19 01:46 <flipz> we just need to use it better
2008-09-19 01:46 <shapor> yeah
2008-09-19 01:46 <shapor> true
2008-09-19 01:46 <flipz> we're really fitting sideways into it right now
2008-09-19 01:46 <shapor> yeah its gross
2008-09-19 01:46 <flipz> I'm amazed anything at all works
2008-09-19 01:47 <shapor> the bug i was fixing 
2008-09-19 01:47 <shapor> is trying to create a file with a name which is too long
2008-09-19 01:47 <shapor> returns an error
2008-09-19 01:47 <shapor> that its too long
2008-09-19 01:47 <flipz> as it should
2008-09-19 01:47 <shapor> but creates it anyway
2008-09-19 01:47 <flipz> oh, bad
2008-09-19 01:47 <shapor> with the name truncated
2008-09-19 01:47 <shapor> and no inode
2008-09-19 01:47 <flipz> naughty
2008-09-19 01:47 <shapor> its fucked
2008-09-19 01:47 <shapor> heh
2008-09-19 01:47 <flipz> I doubt that was my idea
2008-09-19 01:47 <shapor> its a case you never get in dir.c
2008-09-19 01:48 <shapor> because it checks when it creates the dentry
2008-09-19 01:48 <flipz> nice catch
2008-09-19 01:48 <shapor> in the vfs
2008-09-19 01:48 <shapor> since we dont use it
2008-09-19 01:48 <flipz> lamissimo
2008-09-19 01:48 <shapor> yeah
2008-09-19 01:48 <flipz> "always check your inputs"
2008-09-19 01:48 <shapor> yeah
2008-09-19 01:48 <flipz> that one musta got quietly slipped by ted
2008-09-19 01:48 <shapor> subtle due to a minor interface change
2008-09-19 01:49 <shapor> and its not liek dentries have fixed sized strings in d_name
2008-09-19 01:49 <shapor> so there is no hard maximum
2008-09-19 01:49 <shapor> its just supposed be be checked
2008-09-19 01:49 <shapor> i dunno the limit is rediculously short
2008-09-19 01:50 <shapor> i think 255 bytes maybe
2008-09-19 01:50 <shapor> why not allow for long filenames
2008-09-19 01:50 <flipz> that's considered long
2008-09-19 01:51 <shapor> i suppose
2008-09-19 01:51 <shapor> silly limitation
2008-09-19 01:51 <flipz> useful silly limitation
2008-09-19 01:51 <flipz> of course it comes from wanting to represent the length with a byte
2008-09-19 01:51 <shapor> so you can have fixed size dentries?
2008-09-19 01:51 <flipz> fixed size?
2008-09-19 01:52 <shapor> i'm asking
2008-09-19 01:52 <shapor> is that the reason?
2008-09-19 01:52 <shapor> oh i see
2008-09-19 01:52 <shapor> they certainly aren't fixed size
2008-09-19 01:53 <shapor> hm one byte lengths
2008-09-19 01:53 <shapor> true, useful
2008-09-19 01:53 <flipz> qstr
2008-09-19 01:53 <shapor> yeah
2008-09-19 01:53 <shapor> was looking at that earlier
2008-09-19 01:53 <flipz> len is int
2008-09-19 01:54 <flipz> it's checked somewhere but you're right
2008-09-19 01:54 <flipz> ext3 is violating, not checking it
2008-09-19 01:54 <flipz> ext2
2008-09-19 01:54 <shapor> __d_path or something
2008-09-19 01:54 <flipz> playing fast and loosey goosey
2008-09-19 01:55 <shapor> er no
2008-09-19 01:55 <shapor> where do the dentries get created?
2008-09-19 01:55 <shapor> i guess i should look top-down
2008-09-19 01:55 <shapor> start with sys_open
2008-09-19 01:56 <shapor> rather than bottom up
2008-09-19 01:56 <flipz> somewhere in path_walk
2008-09-19 02:01 <shapor> ah yeah, just got there
2008-09-19 02:01 <flipz> 3440        objp = ____cache_alloc(cache, flags);
2008-09-19 02:01 <flipz> :p
2008-09-19 02:01 <shapor> damn thats twisty
2008-09-19 02:01 <flipz> guy who invented slab also invented zfs
2008-09-19 02:02 <shapor> eh?
2008-09-19 02:02 <flipz> course I doubt he wrote four underbars there
2008-09-19 02:02 <flipz> true
2008-09-19 02:03 <flipz> this is by way of checking whether kmalloc returns ERR_PTR or just NULL on error
2008-09-19 02:03 <flipz> seems to be the latter
2008-09-19 02:03 <shapor> http://lxr.linux.no/linux+v2.6.26.5/fs/namei.c#L869
2008-09-19 02:03 <flipz> maze naively assumed otherwise, of course maze is right and we are wrong
2008-09-19 02:04 <flipz> but he have to match linux fart for fart
2008-09-19 02:05 -!- kushal(~kushal@121.246.36.162) has joined #tux3
2008-09-19 02:07 <shapor> so i got all the way down to http://lxr.linux.no/linux+v2.6.26.5/fs/dcache.c#L1241
2008-09-19 02:07 <shapor> but i can't find the damn length check 
2008-09-19 02:07 <flipz> let me know ;)
2008-09-19 02:07 <flipz> try get_name
2008-09-19 02:09 <shapor> bah better to do this during the day when europeans are sleep and lxr is fast 
2008-09-19 02:10 <flipz> heh
2008-09-19 02:10 <flipz> you're poking in the right place
2008-09-19 02:10 <shapor> by the time lxr loads i've lost my train of thought
2008-09-19 02:10 <flipz> right
2008-09-19 02:10 <flipz> it would be useful to install it
2008-09-19 02:10 <flipz> then you can teach me
2008-09-19 02:11 <flipz> involves postgres & mod_perl
2008-09-19 04:15 -!- kushal_(~kushal@121.246.36.194) has joined #tux3
2008-09-19 05:13 -!- kushal(~kushal@121.246.36.194) has joined #tux3
2008-09-19 07:52 -!- tim_dimm(~timothyhu@cpe-76-90-98-247.socal.res.rr.com) has joined #tux3
2008-09-19 08:34 -!- RazvanM(~RazvanM@dazzler.isi.jhu.edu) has joined #tux3
2008-09-19 09:27 -!- guile(~guile@89-159-217-245.rev.numericable.fr) has joined #tux3
2008-09-19 09:28 -!- hirofumi(~hirofumi@210.171.168.39) has joined #tux3
2008-09-19 09:35 -!- tim_dimm(~timothyhu@cpe-76-90-98-247.socal.res.rr.com) has joined #tux3
2008-09-19 10:46 -!- Kirantpatil(~kiran@122.167.197.109) has joined #tux3
2008-09-19 10:46 -!- Kirantpatil(~kiran@122.167.197.109) has left #tux3
2008-09-19 11:09 <MaZe> actually whether I had used IS_ERR and ERR_PTR correctly was something else I'd been meaning to ask ;-)
2008-09-19 11:12 <shapor> flipz: "Ext3cow was designed as a platform for regulatory compliance, and has been used to implement secure deletion, authenticated encryption, and incremental authentication. See the publications page for more details."
2008-09-19 11:13 <shapor> http://www.ext3cow.com/Publications.html
2008-09-19 11:13 <MaZe> about idle - not lame - idle class is not meant to affect anything else using io, neither b/w wise nor latency wise - hence it has to be very conservative, if that's not what you want... don't use idle (and even then idle can still impact performance of non-idle tasks...)
2008-09-19 11:14 <MaZe> and ionice has more classes than just idle
2008-09-19 11:15 <shapor> exactly two more
2008-09-19 11:15 <MaZe> although it's not as powerful as it could/should be
2008-09-19 11:15 <shapor> although i think idle is the most appealing one
2008-09-19 11:16 <shapor> its common to want to do io intensive tasks in the background like backups or whatnot
2008-09-19 11:16 <MaZe> yeah, using kvm now
2008-09-19 11:16 <shapor> ionice'ing a kvm session?
2008-09-19 11:16 <MaZe> mind you of course, all my printk's are non-multithreaded-printk compatible - who cares ;-) [for now]
2008-09-19 11:18 <MaZe> I put the wq in the bio, cause I needed something to wait on... was there something else I could wait on, and wake up from the endbio func?
2008-09-19 11:18 <MaZe> uhm, what's wrong with just putting the wq there? what use is the extra level of indirection?
2008-09-19 11:19 <MaZe> If you do install your own lxr - pass links to it ;-)
2008-09-19 11:19 <MaZe> we don't need all kversions
2008-09-19 11:19 <MaZe> dd can write bytes without trunc
2008-09-19 11:20 <MaZe> ah, there it is in the log - still catchingup
2008-09-19 11:20 <shapor> ;)
2008-09-19 11:20 <MaZe> hey what did you expect... I have no bloody idea what I'm doing ;-) [about the wrong things being doozers]
2008-09-19 11:20 <shapor> hrm it would be cool if lxr could be tied to a git repo
2008-09-19 11:21 <MaZe> and dd is weird... but it works and is everywhere...
2008-09-19 11:21 <shapor> or is it already
2008-09-19 11:22 <shapor> you know what is annoying is the number of clicks you need to do to download anything from sourceforge
2008-09-19 11:22 <shapor> ah yeah it talks to git
2008-09-19 11:25 <MaZe> yeah I was thinking I should be checking both for errors and null...
2008-09-19 11:26 <MaZe> kvm, and ionice, no was referring to running my tests in kvm, like flips is in uml
2008-09-19 11:27 <MaZe> clicks - yeah agreed
2008-09-19 11:27 <MaZe> Ah, caught up....
2008-09-19 11:27 <MaZe> seems you guys had a productive night
2008-09-19 11:27 <MaZe> mine was as well
2008-09-19 11:27 <MaZe> first time in a long time that I'm not sleepy before noon
2008-09-19 11:27 <shapor> i was going to ask what the problem was with putting the wq in the bio as well
2008-09-19 11:28 <shapor> whats the difference if you put a pointer there
2008-09-19 11:28 <MaZe> well, I need both the wq, and a bool
2008-09-19 11:28 <MaZe> so I put in a pointer to a struct with both
2008-09-19 11:28 <MaZe> (also should probably have an error return field in there)
2008-09-19 11:30 <MaZe> ok, back to work
2008-09-19 11:56 <MaZe> +<----->bio->bi_io_vec[bio->bi_vcnt] = (struct bio_vec){ 
2008-09-19 11:56 <MaZe> +<-----><------>.bv_page = virt_to_page(buf), 
2008-09-19 11:56 <MaZe> +<-----><------>.bv_offset = offset_in_page(buf), 
2008-09-19 11:56 <MaZe> +<-----><------>.bv_len = SB_SIZE }; 
2008-09-19 11:56 <MaZe> +<----->bio->bi_size = SB_SIZE; 
2008-09-19 11:56 <MaZe> +<----->bio->bi_end_io = end_io_read; 
2008-09-19 11:56 <MaZe> +<----->bio->bi_private = &mz; 
2008-09-19 11:56 <MaZe> +<----->bio->bi_vcnt = 1; 
2008-09-19 11:57 <MaZe> either that should be bio->bi_io_vec[0] = ...
2008-09-19 11:57 <MaZe> or bio->bi_vcnt++;
2008-09-19 11:59 <MaZe> plus putting the wq on the stack is stack bloat - isn't that bad if we want 4k stacks?
2008-09-19 12:00 <MaZe> actually, adding some sort of debug stack depth tracking might be useful.
2008-09-19 12:00 <MaZe> record deepest spot on stack ever hit in your code
2008-09-19 12:00 <MaZe> hmm, maybe the kernel already does that automatically
2008-09-19 12:07 -!- nataliep(~nataliep@207.47.98.129.static.nextweb.net) has joined #tux3
2008-09-19 12:12 -!- tim_dimm(~timothyhu@cpe-76-90-98-247.socal.res.rr.com) has joined #tux3
2008-09-19 13:16 <flipz> maze, ping
2008-09-19 13:17 <flipz> maze, your ERR_PTR is mostly wrong, where you've applied it to functions that only return ptr or null
2008-09-19 13:17 <flipz> entirely wrong to be precise ;)
2008-09-19 13:19 <flipz> maze, putting the work queue in the bio is inherently fragile, the bio can disappear
2008-09-19 13:19 <flipz> put a pointer to the work queue in the bio
2008-09-19 13:20 <MaZe> right, the wq is in a pointer in the bio
2008-09-19 13:20 <MaZe> yeah checking whether err-ptr is required or not was a todo
2008-09-19 13:20 <MaZe> but of course that's actually not documented anywhere
2008-09-19 13:20 <MaZe> useful, like at the top of said functions
2008-09-19 13:21 <MaZe> the way you've done it, you've got double indirection - I've got single
2008-09-19 13:21 <flipz> maze, it would be very cool if lxr could be tied to a repo... think about versioned indexes :)
2008-09-19 13:21 <MaZe> heh
2008-09-19 13:21 <flipz> you either indirect on the wait, or indirect on the complete
2008-09-19 13:22 <flipz> the wait indirection will be executed more often than the complete
2008-09-19 13:23 <flipz> bio->bi_vcnt++ would be an improvement
2008-09-19 13:24 <flipz> the mz goes on the stack anyway
2008-09-19 13:24 <flipz> if we're that tight for stack space we should not be doing 4K stacks
2008-09-19 13:24 <flipz> (which people are slowly learning)
2008-09-19 13:31 <flipz> Iceweasel can't find the server at m.a.z.e.pl.
2008-09-19 13:38 <flipz> maze, by the way, a wait queue is tiny
2008-09-19 13:38 <flipz> just a spinlock and a list
2008-09-19 13:38 <MaZe> yeah my university is migrating to a new building
2008-09-19 13:38 <MaZe> why does the mz go on stack if it's kmalloc'ed?
2008-09-19 13:39 <flipz> my mz wan't kmalloced
2008-09-19 13:39 <MaZe> oh
2008-09-19 13:39 <flipz> can't see your original code any more
2008-09-19 13:39 <flipz> good to post that kind of thing to the list
2008-09-19 13:39 <flipz> it was a nice hack
2008-09-19 13:39 <flipz> very nice
2008-09-19 13:39 <MaZe> which hack?
2008-09-19 13:39 <flipz> junkfs
2008-09-19 13:39 <MaZe> oh
2008-09-19 13:40 <flipz> too bad bio is such a bloaty interface
2008-09-19 13:40 <flipz> not easy to make useful helpers for it
2008-09-19 13:42 <MaZe> I have ;-)
2008-09-19 13:42 <flipz> int bioio(int rw, dec_t dev, sector_t sector, unsigned size, 
2008-09-19 13:42 <flipz> 	endio_t endio, void *private, unsigned vecs, struct page *page, 
2008-09-19 13:42 <flipz> 	unsigned off, unsigned len);
2008-09-19 13:42 <flipz> :p
2008-09-19 13:43 <MaZe> yeah, should probably call it something like synchronous_bio_io
2008-09-19 13:44 <flipz> can shell it as syncbio
2008-09-19 13:44 <MaZe> or synchbio standing for synch[ronous]_b[io]_io
2008-09-19 13:44 <MaZe> don't want sync since that means something else
2008-09-19 13:44 <flipz> not really
2008-09-19 13:44 <flipz> it's just a part of a sync
2008-09-19 13:44 <MaZe> well, it won't sync to disk
2008-09-19 13:44 <MaZe> oh, wait it will
2008-09-19 13:44 <flipz> it will
2008-09-19 13:44 <MaZe> uhm, even if it's an lvm volume?
2008-09-19 13:44 <flipz> syncbio is the one
2008-09-19 13:45 <flipz> yes
2008-09-19 13:45 <MaZe> right it will
2008-09-19 13:45 <MaZe> the page cache is above this level
2008-09-19 13:45 <flipz> things get screwy when virtual block devices cache
2008-09-19 13:45 <flipz> which some do
2008-09-19 13:45 <MaZe> I'm still not quite clear on how to do barriers and permit reordering in the elevator at this level
2008-09-19 13:45 <flipz> and theyget screwy
2008-09-19 13:45 <flipz> barriers are a big mess
2008-09-19 13:46 <flipz> mostly we just close our eyes and try to do simple things
2008-09-19 13:46 <flipz> but anyway
2008-09-19 13:46 <MaZe> as you can tell ... I don't do/like simple
2008-09-19 13:46 <flipz> a barrier is a flag on any bio
2008-09-19 13:46 <flipz> bad idea actually
2008-09-19 13:46 <MaZe> I like powerful - shoot yourself in the foot things ;-)
2008-09-19 13:46 <flipz> barrier should be separate bio
2008-09-19 13:46 <MaZe> but a barrier should be more like a pointer to another bio(s) which should be first
2008-09-19 13:46 <MaZe> this write must happen after those writes
2008-09-19 13:46 <flipz> maybe
2008-09-19 13:47 <flipz> a new barrier api would be a nice contribution
2008-09-19 13:47 <MaZe> no reason for barriers between fs'es on two partitions on the same bdev
2008-09-19 13:47 <flipz> current one is teh suck
2008-09-19 13:47 <MaZe> and I don't think you need barriers on read...
2008-09-19 13:47 <flipz> you do
2008-09-19 13:47 <flipz> I'll even remember why
2008-09-19 13:47 <flipz> not badly
2008-09-19 13:47 <MaZe> why? something net related?
2008-09-19 13:47 <flipz> but its the same as memory ops
2008-09-19 13:48 <flipz> need all combinations if you look hard enough
2008-09-19 13:48 <MaZe> oh, but then it's a these rights must hit disk before this read
2008-09-19 13:48 <MaZe> s/rights/writes/
2008-09-19 13:48 <flipz> that kind of thing
2008-09-19 13:48 <MaZe> barriers between readS?
2008-09-19 13:48 <flipz> barries between reads... hmm
2008-09-19 13:48 <flipz> I'd think no
2008-09-19 13:48 <flipz> probably tackling something at the wrong level
2008-09-19 13:48 <MaZe> I can see writes -> barrier -> writes/reads
2008-09-19 13:49 <MaZe> I don't see reads -> barrier -> reads, nor reads -> barrier -> writes
2008-09-19 13:49 <MaZe> although I guess what exactly should happen if read, write to same sector gets reordered... hmm.
2008-09-19 13:49 <flipz> the arrow directions are ambiguous
2008-09-19 13:49 <MaZe> arrows pointing out time
2008-09-19 13:49 <MaZe> flow
2008-09-19 13:49 <flipz> commas do that ;)
2008-09-19 13:50 <MaZe> will we really have to fix the bdev interface first?
2008-09-19 13:50 <flipz> I can see reads/barrier/writes
2008-09-19 13:50 <flipz> but not a strong case
2008-09-19 13:50 <flipz> the bdev barrier interface?
2008-09-19 13:50 <flipz> yes, it's naive
2008-09-19 13:50 <MaZe> well, and the prio interface
2008-09-19 13:51 <flipz> there is none
2008-09-19 13:51 <MaZe> get it all kind of nice and usable
2008-09-19 13:51 <flipz> the prio ideas are just a hack in one elevator option
2008-09-19 13:51 <MaZe> we (I?) need a bdev interface which is aio read/write scatter/gather with priorities htb-like and barriers
2008-09-19 13:52 <flipz> true
2008-09-19 13:52 <flipz> be happy to work on it with you
2008-09-19 13:52 <MaZe> a lot of it is there
2008-09-19 13:52 <MaZe> a lot isn't
2008-09-19 13:52 <flipz> I have plenty of apps
2008-09-19 13:52 <flipz> starting with media...
2008-09-19 13:53 <MaZe> how exactly barriers should work is an interesting question
2008-09-19 13:53 <flipz> yes
2008-09-19 13:53 <MaZe> you don't want them to be too strong
2008-09-19 13:53 <flipz> or awkward
2008-09-19 13:53 <MaZe> but strong enough to implement the consistency the fs needs
2008-09-19 13:53 <flipz> you want it to solve the primary problem well, which is journalling
2008-09-19 13:53 <MaZe> exactly
2008-09-19 13:54 <MaZe> and it has to take into consideration real world disks
2008-09-19 13:54 <MaZe> and the fact they spin/seek - something to be very aware of when working on this, since impacts priorities much
2008-09-19 13:54 <MaZe> and you might desire consistency x-dev
2008-09-19 13:54 <MaZe> need to write to journal dev before hitting base dev
2008-09-19 13:54 <flipz> maze, notice there is nothing read-specific about your endio
2008-09-19 13:55 <MaZe> *cute*
2008-09-19 13:55 <flipz> needs a different name
2008-09-19 13:55 <MaZe> I know
2008-09-19 13:55 <MaZe> hey it was a hack ;-)
2008-09-19 13:55 <flipz> not any more
2008-09-19 13:55 <MaZe> hehe
2008-09-19 13:55 <MaZe> right
2008-09-19 13:56 <flipz> maze, I don't know what I was going on about with your bio private pointer, your usage is fine
2008-09-19 13:56 <flipz> on the stack is more leet
2008-09-19 13:57 <flipz> kmallocs are bad things
2008-09-19 13:57 <MaZe> I don't know
2008-09-19 13:57 <flipz> fragment
2008-09-19 13:57 <flipz> fragile
2008-09-19 13:57 <MaZe> stack is small nowadays
2008-09-19 13:57 <flipz> not that small
2008-09-19 13:57 <MaZe> yeah, I've wanted to see exactly how much stack space I actually have
2008-09-19 13:57 <MaZe> for my leet new fs idea, I actually need to be very careful
2008-09-19 13:58 <flipz> sure
2008-09-19 13:58 <MaZe> since with both a net layer and an fs layer it might get tight
2008-09-19 13:58 <flipz> but this is on the other side of too careful
2008-09-19 13:58 <MaZe> mayhaps
2008-09-19 13:58 <MaZe> I'm still new ;-)
2008-09-19 13:58 <flipz> you are?
2008-09-19 13:59 <flipz> I think you're past 50 percentile in hacking skills of people who call themselves that
2008-09-19 13:59 <flipz> kernel hacking
2008-09-19 13:59 <flipz> another couple months will get you past 90
2008-09-19 13:59 <MaZe> I still have no idea about anything yet ;-)
2008-09-19 13:59 <flipz> you think anybody else does?
2008-09-19 13:59 <flipz> how'd we get all that crap in kernel if anybody had a clue?
2008-09-19 14:00 <flipz> oh one thing, there are a few null statements in your code that you may not think are there
2008-09-19 14:01 <flipz> extra semicolons
2008-09-19 14:05 <MaZe> oh, well I like semicolons
2008-09-19 14:05 <MaZe> think every } should be followed by a ;
2008-09-19 14:06 <MaZe> heading to grab lunch
2008-09-19 14:06 <MaZe> (except in } else {)
2008-09-19 14:06 <MaZe> and C/C++ just has bad syntax with semicolon
2008-09-19 14:06 <MaZe> s
2008-09-19 14:07 <flipz> I assume they'll be gone ;)
2008-09-19 14:08 <flipz> extra parents and curlies are also frowned at, but extra semicolons are cause for shouting
2008-09-19 14:08 <flipz> extra parens I mean
2008-09-19 14:08 <flipz> extra parents are probably ok, particularly in utah
2008-09-19 14:11 <MaZe> let em shout
2008-09-19 14:12 <MaZe> hmm, I wonder
2008-09-19 14:12 <MaZe> is it true that removing a semicolon will either
2008-09-19 14:12 <MaZe> a) result in code functioning the exact same way as before
2008-09-19 14:12 <MaZe> or
2008-09-19 14:12 <MaZe> b) result in a compile failure
2008-09-19 14:13 <MaZe> probably noy
2008-09-19 14:13 <flipz> extra semicolons make the code more fragile
2008-09-19 14:13 <flipz> you can get a big surprise if somebody adds a seemingly innocuous conditional
2008-09-19 14:14 <flipz> in theory there is no effect on generated code
2008-09-19 14:14 <flipz> in practice, theory and practice are different
2008-09-19 14:15 <MaZe> yeah while I know ';' is a statement seperator, I much prefer to think of them as end-of-statement markers
2008-09-19 14:15 <MaZe> hmm
2008-09-19 14:15 <flipz> closet pascal groupie ;)
2008-09-19 14:15 <MaZe> maybe I got that backwards
2008-09-19 14:15 <flipz> well
2008-09-19 14:15 <MaZe> I love pascal syntax
2008-09-19 14:15 <flipz> soor
2008-09-19 14:15 <flipz> sorry
2008-09-19 14:16 <flipz> yes, backwards
2008-09-19 14:16 <flipz> but still a closet pascaller I think
2008-09-19 14:16 <MaZe> whatever - everything should end with a ';'
2008-09-19 14:16 <flipz> semicolons are stupid
2008-09-19 14:16 <flipz> should be optional
2008-09-19 14:16 <flipz> designers of c are/were stupid
2008-09-19 14:16 <flipz> but since they are there, have to use them lindentally
2008-09-19 14:16 <MaZe> imho should be required ;-)
2008-09-19 14:17 <flipz> every line should be required to have two semicolons, one at the beginning, one at the end
2008-09-19 14:17 <MaZe> because you need them anyway, - can't live without em
2008-09-19 14:17 <MaZe> nah
2008-09-19 14:17 <MaZe> every statement should end with a ';'
2008-09-19 14:18 <MaZe> might be at the end of line, might be in the middle, might extend into the next line
2008-09-19 14:18 <MaZe> whitespace shouldn't matter (although could cause compiler warnings)
2008-09-19 14:18 <flipz> e l s e
2008-09-19 14:19 <flipz> anway
2008-09-19 14:19 <flipz> let's not go there ;)
2008-09-19 14:19 <MaZe> else should always be } else {
2008-09-19 14:19 <MaZe> you either have a simple if (blah) something;
2008-09-19 14:19 <flipz> not considered lindenty to have curlies around single statements
2008-09-19 14:19 <MaZe> or an if () { ... } else { ... };
2008-09-19 14:19 <flipz> not saying I think that's good or bad, it's just not lindenty
2008-09-19 14:20 <MaZe> yeah, I know
2008-09-19 14:20 <MaZe> my personal opinion, is:
2008-09-19 14:20 <MaZe> either it's short and sweet fits on a line if (something) something;
2008-09-19 14:21 <MaZe> or should be the full multi-line if () {\n ...\n } else {\n ...\n };\n
2008-09-19 14:21 <flipz> my personal opinion is, if it's written in C is going to look ugly and there is little you can do about it
2008-09-19 14:21 <MaZe> possibly without the else clause if not needed
2008-09-19 14:21 <flipz> break your heart trying
2008-09-19 14:21 <MaZe> true
2008-09-19 14:21 <bh> folks
2008-09-19 14:22 <MaZe> yes, fixing C is something I've thought of, codenamed 'the language advanced', a curious mix of pascal/c/c++/java/asm/gnu-isms/lisp
2008-09-19 14:22 <MaZe> but besides thinking about it never got anywhere
2008-09-19 14:22 <MaZe> (never tried)
2008-09-19 14:22 <MaZe> bh: hey
2008-09-19 14:22 <flipz> your brainpower is needed more badly elsewhere ;)
2008-09-19 14:22 <MaZe> hehe
2008-09-19 14:23 <MaZe> but if you write the language first, you can then write the kernel in a language which doesn't suck...
2008-09-19 14:23 <flipz> you can and nobody will care
2008-09-19 14:23 <MaZe> agreed
2008-09-19 14:23 <MaZe> still an interesting exercise
2008-09-19 14:23 <flipz> a disappear for years exercise
2008-09-19 14:23 <MaZe> true
2008-09-19 14:24 <MaZe> hence the 'haven't ever tried' part
2008-09-19 14:24 <flipz> save it for when you're old
2008-09-19 14:24 <flipz> show those whippersnappers
2008-09-19 14:24 <MaZe> I'm hoping someone else will do it
2008-09-19 14:24 <flipz> they will
2008-09-19 14:24 <MaZe> or I'll get some smart/bright friends and students to do it
2008-09-19 14:24 <flipz> there's always somebody with enough time on their hands to write an os from scratch
2008-09-19 14:25 <flipz> they get 15 minutes of slashdot fame and a nice job where they can stew
2008-09-19 14:25 <MaZe> hehe
2008-09-19 14:25 <MaZe> if/when I go back to school to get my phd, I've been thinking about leading a course for some of the best'n'brightest with design and implementation of a language or os as the topic
2008-09-19 14:26 <flipz> you can practice here
2008-09-19 14:26 <MaZe> anyway, back to earth
2008-09-19 14:26 <flipz> you're already TA at tux3u
2008-09-19 14:26 <flipz> sure
2008-09-19 14:26 <flipz> got to think about the next level for junkfs/tux3fs
2008-09-19 14:26 <MaZe> right
2008-09-19 14:27 <MaZe> right now I'm trying to think of what I want from the mm subsystem for my fs
2008-09-19 14:27 <flipz> it's cool how tux3fs is both ramfs and diskfs now, hmm?
2008-09-19 14:27 <MaZe> hehe
2008-09-19 14:27 <flipz> that's the most instructive thing so far
2008-09-19 14:27 <flipz> re vfs
2008-09-19 14:28 <MaZe> and at which layer of the vfs (generics for most ops or not) the interfaces need to happen
2008-09-19 14:28 <flipz> linux kinda gets it right, it's just warty
2008-09-19 14:28 <flipz> also... error trapping
2008-09-19 14:29 <flipz> I'd like to see a stack unwinding/resource recovery discipline
2008-09-19 14:29 <MaZe> also wondering if implementing reads by userspace (with appropriate aligned buffer) by unmap and map in ro cow pages from page cache or somewhere else would be appropriate and fast and/or sow
2008-09-19 14:29 <MaZe> slow
2008-09-19 14:29 <flipz> it would be appropriate even on linux
2008-09-19 14:29 <MaZe> and if there's any race there
2008-09-19 14:29 <flipz> there are cases where it's slow, but in general it's powerful
2008-09-19 14:29 <flipz> just linux isn't orgainized that way
2008-09-19 14:30 <flipz> linux has a loopy approach
2008-09-19 14:30 <flipz> very naive
2008-09-19 14:30 <MaZe> would get zero copy reads, and most of the time you don't write over that data
2008-09-19 14:30 <flipz> as in... too many loops
2008-09-19 14:30 <flipz> yes, and it gets even more fun with net + disk + buffer + both ways
2008-09-19 14:30 <MaZe> exactly
2008-09-19 14:30 <MaZe> hence I'm thinking of this as a two layer fs to begin with
2008-09-19 14:30 <flipz> that's why nobody has been crazy enough to attempt it
2008-09-19 14:30 <flipz> look a splice
2008-09-19 14:30 <flipz> simple thing
2008-09-19 14:31 <flipz> big disaster
2008-09-19 14:31 <MaZe> I'm not actually sure where splice is atm?
2008-09-19 14:31 <MaZe> what happened there? last I knew there was an exploit and fix and exploit and fix...
2008-09-19 14:31 <flipz> freesearch lxr
2008-09-19 14:32 <flipz> it's a feature build on a base of jello
2008-09-19 14:33 <MaZe> there does appear to be a way to get your own page to trigger on all sorts of page operations, so that's good
2008-09-19 14:33 <flipz> oh yes
2008-09-19 14:33 <flipz> it's fun
2008-09-19 14:33 <MaZe> meant code not page
2008-09-19 14:33 <flipz> "stupid page tricks"
2008-09-19 14:34 <MaZe> yeah, but I'm guessing it's needed for decent cache coherency
2008-09-19 14:34 <MaZe> even if it will mean locking will effectively end up being page (not byte-range) based
2008-09-19 14:35 <MaZe> anyway, I figure it's important to know what's possible, to know what can be later implemented, and to design the possibility in from the start
2008-09-19 14:35 <flipz> you'll be getting into vm soon enough
2008-09-19 14:36 <flipz> you can help me with the variable page rewrite if you like
2008-09-19 14:36 <flipz> linus said he would open 2.7 if I did that hack
2008-09-19 14:36 <MaZe> I've realized that the xattr interface can probably be used as a nice ioctl layer for the fs
2008-09-19 14:36 <flipz> it can?
2008-09-19 14:36 <MaZe> yeah like setting fs.tux3.option = something on an inode
2008-09-19 14:37 <MaZe> and then reading it back
2008-09-19 14:37 <MaZe> have the stuff be auto-generated
2008-09-19 14:37 <flipz> ioctls would not be pleasant for that
2008-09-19 14:37 <MaZe> and have options for stuff like type of optimizations to be used on this file or etc
2008-09-19 14:37 <flipz> we have ddlink for that
2008-09-19 14:37 <MaZe> I think xattr is nice here - although haven't looked at ddlink
2008-09-19 14:37 <flipz> reiser5 ;)
2008-09-19 14:37 <MaZe> hmm?
2008-09-19 14:37 <flipz> ddlink is cool
2008-09-19 14:38 <flipz> really cook
2008-09-19 14:38 <flipz> cool
2008-09-19 14:38 <MaZe> reiser5? what's with reiser4?
2008-09-19 14:38 <flipz> "dead"
2008-09-19 14:38 <MaZe> is reiser even being worked on?
2008-09-19 14:38 <flipz> slowly
2008-09-19 14:38 <flipz> very slowly
2008-09-19 14:39 <MaZe> is reiser 4 done? stable? dropped?
2008-09-19 14:39 <flipz> quasi stable
2008-09-19 14:40 <flipz> should be merged, under a different name imho
2008-09-19 14:41 <flipz> chris mason was one of the big driving forces on reiser, at least reiser 3, and he's entirely devoted to btrfs now
2008-09-19 14:41 <flipz> which is something like reiser 3.5
2008-09-19 15:00 <MaZe> ah
2008-09-19 15:00 <MaZe> I came up with a few interesting network fs related ideas last night
2008-09-19 15:00 <MaZe> was a very productive bath ;-)
2008-09-19 15:01 <flipz> works for me too, showers though
2008-09-19 15:01 <flipz> something about that running water
2008-09-19 15:01 <flipz> settles the lame ideas, let's bouyant ones float to the top
2008-09-19 15:11 <flipz> maze,
2008-09-19 15:11 <flipz>  	while (vecs--)
2008-09-19 15:11 <flipz> 		bio->bi_io_vec[bio->bi_vcnt++] = va_arg(args, struct bio_vec);
2008-09-19 15:13 <flipz> int bio(int rw, dev_t dev, sector_t sector, bio_end_io_t endio, void *private, unsigned vecs, ...)
2008-09-19 16:57 <bh> what are you folks going to finished the file system ?
2008-09-19 16:57 <bh> what=when
2008-09-19 16:57 <bh> are were there yet ?
2008-09-19 16:58 <bh> 
ACTION grins

2008-09-19 17:31 <flipz> gregkh is an idiot
2008-09-19 17:32 <flipz> oh was that public
2008-09-19 17:32 <flipz> http://dustinkirkland.wordpress.com/2008/09/18/whats-behind-gregkhs-latest-rant/
2008-09-19 17:32 <flipz> wouldn't be so bad if he could design, code or debug
2008-09-19 17:33 <flipz> bh, we're getting closer
2008-09-19 17:33 <flipz> the kernel port is getting a little attention
2008-09-19 17:33 <flipz> needs a lot more
2008-09-19 17:56 <flipz> true fact: the linux kernel makefile is 1600 lines long
2008-09-19 18:01 <flipz> 541 KBUILD_CFLAGS += $(call cc-option,-Wdeclaration-after-statement,) <- this is the line we kill to enable inline decls
2008-09-19 18:01 <flipz> I guess we are going to do taht
2008-09-19 18:01 <flipz> for now until just before merge
2008-09-19 18:09 <flipz> sk8 oclock
2008-09-19 18:09 <flipz> one could say sk8teen oclock
2008-09-19 18:18 -!- BSD(~bandan@70-4-203-156.area3.spcsdns.net) has joined #tux3
2008-09-19 18:26 -!- BSD(~bandan@70-4-203-156.area3.spcsdns.net) has joined #tux3
2008-09-19 19:02 -!- BSD(~bandan@70-4-203-156.area3.spcsdns.net) has joined #tux3
2008-09-19 19:46 -!- BSD(~bandan@68-244-245-217.area3.spcsdns.net) has joined #tux3
2008-09-19 19:57 -!- konrad(~konrad@D-128-208-53-208.dhcp4.washington.edu) has joined #tux3
2008-09-19 20:42 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #tux3
2008-09-19 20:51 -!- RazvanM(~RazvanM@pool-151-196-118-156.balt.east.verizon.net) has joined #tux3
2008-09-19 21:20 <flipz> Results 1 - 10 of about 283,000 for tux3.
2008-09-19 21:20 <flipz> up 100k in a day
2008-09-19 21:20 <flipz> wonder what happened
2008-09-19 21:21 <Bushman> 100k...hits?
2008-09-19 21:21 <flipz> musta been the waking post
2008-09-19 21:21 <flipz> 100k up in one day, yes
2008-09-19 21:21 <Bushman> damn
2008-09-19 21:21 <flipz> the internet loves wanking I guess
2008-09-19 21:21 <Bushman> hey, did you guys want a faster lxr? 
2008-09-19 21:21 <flipz> very much
2008-09-19 21:22 <Bushman> ok, i just got one going on my home quad, i gotta tweak postgres and we should be ready to rock
2008-09-19 21:22 <flipz> excellent
2008-09-19 21:22 <Bushman> of course bandwidth might be a problem
2008-09-19 21:22 <flipz> your admin skillz rock
2008-09-19 21:22 <flipz> shapor can fix that
2008-09-19 21:22 <Bushman> shap can fix anything, that bastard! :)
2008-09-19 21:23 <flipz> truth
2008-09-19 21:23 <Bushman> do you need the free text searches?  that's some extra software that i'd have to get/configure 
2008-09-19 21:23 <flipz> oh yes
2008-09-19 21:23 <flipz> the whole enchilada
2008-09-19 21:23 <flipz> freetext is essential
2008-09-19 21:23 <Bushman> ok, i'll go play with that
2008-09-19 21:24 <flipz> thanks much
2008-09-19 21:24 <flipz> have to run out before whole foods closes
2008-09-19 21:24 <Bushman> didnt know, i just started dicking around with it to remind myself what sysadmining was like in linux ;)
2008-09-19 21:24 <flipz> or I don't get my sushi tonight
2008-09-19 21:24 <Bushman> go get sushi, that's a moral imperative
2008-09-19 21:24 <flipz> it's a mess, isn't it.  LXR install I mean
2008-09-19 21:24 <Bushman> it aint pretty, but than again i dont do normal...anything
2008-09-19 21:25 <flipz> that's a good sign
2008-09-19 21:25 <flipz> bbiaf
2008-09-19 21:26 <MaZe> lol
2008-09-19 21:43 <shapor> Bushman: what kind of bandwidth do you have?
2008-09-19 21:43 <Bushman> 
ACTION puts head down in shame

2008-09-19 21:43 <Bushman> cable modem
2008-09-19 21:44 <shapor> its good enough
2008-09-19 21:44 <shapor> most requests are only a few 10's of kb probably
2008-09-19 21:44 <Bushman> i can 'pimp my apache' and turn on compression since it's text it should be alright
2008-09-19 21:44 <shapor> by looking at the code cpu is gonna be the bottle neck first ;)
2008-09-19 21:45 <Bushman> yea the db design makes baby jesus cry 
2008-09-19 21:45 <Bushman> so i gotta go through postgres configs first, which can take hours to do right, that shit is complicated
2008-09-19 21:46 <shapor> i'm i can get a dual Dual Core Woodcrest
2008-09-19 21:46 <shapor> for $120/mo
2008-09-19 21:46 <shapor> i might do that
2008-09-19 21:47 <shapor> since i'm finally more than breaking even with my current servers
2008-09-19 21:47 <Bushman> the database for a whole 2.6.26.5 tree is about 1.1gb, so i'll just shove the whole thing into memory
2008-09-19 21:47 <shapor> how much ram does it have?
2008-09-19 21:47 <Bushman> braking even?  what are you hosing?
2008-09-19 21:47 <Bushman> 6gb ;)
2008-09-19 21:47 <shapor> a few sites
2008-09-19 21:48 <Bushman> it's my Matlab cruncher ;)
2008-09-19 21:48 <shapor> well a few paying sites
2008-09-19 21:48 <Bushman> persiankitty is back? :)
2008-09-19 21:48 <shapor> haha
2008-09-19 21:48 <shapor> then various freebees like zumastor.org
2008-09-19 21:49 <shapor> Bushman: you installed lxrng right?
2008-09-19 21:49 <shapor> thats the one running on lxr.linux.no
2008-09-19 21:49 <Bushman> lxrng?  i just found lxr-devel
2008-09-19 21:49 <shapor> i think that ones out of date
2008-09-19 21:49 <Bushman> it's like 0.9.5
2008-09-19 21:49 <shapor> but if it works thats cool
2008-09-19 21:50 <shapor> there are annoying bugs in the one running on lxr.linux.no
2008-09-19 21:50 <shapor> like it will show the same result multiple times
2008-09-19 21:50 <shapor> see http://lxr.linux.no/ at the bottom of the page
2008-09-19 21:50 <shapor> step by step instructions for setting it up too ;)
2008-09-19 21:50 <Bushman> well i havent seen the web part of it yet, the first run through the code indexing just ended like 10 mins ago
2008-09-19 21:51 <shapor> ah cool
2008-09-19 21:51 <Bushman> took a while
2008-09-19 21:51 <shapor> too bad i dont have any phat hardware at home
2008-09-19 21:51 <shapor> i have 15/15 fios
2008-09-19 21:51 <Bushman> should i ship you a box? :)
2008-09-19 21:52 <shapor> how many watts ? ;)
2008-09-19 21:52 <shapor> if i were to guess it probably sounds like a helicopter
2008-09-19 21:52 <shapor> never knew you to have an quiet boxes
2008-09-19 21:52 <Bushman> ups says about 90 sitting idle with cpu frequency throttling, 140+ at full boogie
2008-09-19 21:53 <shapor> so 100 on average
2008-09-19 21:53 <Bushman> it's a bulb
2008-09-19 21:53 <shapor> so about $8/mon 
2008-09-19 21:54 <Bushman> are you all green, or just being a cheapass?
2008-09-19 21:54 <shapor> little from column a...
2008-09-19 21:54 <Bushman> dont have to answer, i know the answer ;) 
2008-09-19 21:54 <shapor> hm the place i used to live had free electricity and fios available
2008-09-19 21:55 <shapor> should build a datacenter in it
2008-09-19 21:55 <Bushman> or a grow house ;)
2008-09-19 21:55 <shapor> ok u turn
2008-09-19 21:55 <Bushman> is that a 'weeds' reference?
2008-09-19 21:55 <shapor> get me a prius
2008-09-19 21:56 <Bushman> so i can sneak up on mofos real quiet
2008-09-19 21:56 <shapor> 'so i can sneak up on motherfuckers'
2008-09-19 21:56 <shapor> hah
2008-09-19 21:56 <Bushman> that's a bit spooky
2008-09-19 21:56 <shapor> annnyway
2008-09-19 21:57 <shapor> bandwidth shouldnt be a problem
2008-09-19 21:57 <shapor> if it is, ship it to me ;)
2008-09-19 21:57 <Bushman> cant we just set it up on one of your real servers?  
2008-09-19 21:57 <shapor> sure
2008-09-19 21:57 <shapor> but i dont want to kill the cpus
2008-09-19 21:57 <shapor> we'll see how it goes on cable modem
2008-09-19 21:57 <Bushman> i can put it up on marcintology, but that's just a normal shared web account
2008-09-19 21:57 <shapor> i can run dyndns for you if you dont have it already
2008-09-19 21:58 <Bushman> oh i got dns for it  ;)
2008-09-19 21:58 <shapor> lxr.tux3.org ?
2008-09-19 21:58 <shapor> ooh could we pull from ddtree too?
2008-09-19 21:58 <Bushman> ok, let's do that than
2008-09-19 21:58 <shapor> that would be slick!
2008-09-19 21:58 <shapor> flipz: you like that idea?
2008-09-19 21:58 <Bushman> well first i wanna see if i can  get it working nicely 
2008-09-19 21:58 <shapor> have the ddtree lxr'ed ? ;)
2008-09-19 21:58 <Bushman> flipz is doing sushi
2008-09-19 21:59 <shapor> ah ok well i'm out for a bit too
2008-09-19 21:59 <Bushman> i sushi'ed m'self for lunch yesterday so i should be good for a day or two before i'm gonna start jonsing again
2008-09-19 21:59 <Bushman> good, i cant work with you making me laugh
2008-09-19 22:08 <MaZe> http://blogs.pcworld.com/staffblog/archives/007783.html
2008-09-19 22:08 <MaZe> take a look at the windows ad at the bottom...
2008-09-19 22:09 <MaZe> some enlightened folk at microsoft snuck in penguins...
2008-09-19 22:09 <MaZe> they're everywhere
2008-09-19 22:12 <MaZe> if needs be I can probably stick lxr on an athlon64 at my univ in poland
2008-09-19 22:12 <MaZe> or, maybe I could host a second copy from home off of comcast
2008-09-19 22:13 <flipz> ddtree.tux3.org
2008-09-19 22:14 <MaZe> no functionen
2008-09-19 22:14 <MaZe> Host ddtree.tux3.org not found: 3(NXDOMAIN)
2008-09-19 22:15 <flipz> it was just a suggestion
2008-09-19 22:15 <flipz> we can make it happen any time
2008-09-19 22:15 <flipz> haha
2008-09-19 22:16 <flipz> nice redmond penquins
2008-09-19 22:16 <flipz> are they that clueless or are there subversives inside m$?
2008-09-19 22:19 <MaZe> I'm guessing subversion from inside
2008-09-19 22:19 <MaZe> but it looks like windows out onto the future to me, penguins everywhere
2008-09-19 22:22 <MaZe> ugh, reading a notebook review and someone is claiming gigabit wired is overkill...
2008-09-19 22:22 <MaZe> what the hell are they drinking?
2008-09-19 22:22 <MaZe> or smoking
2008-09-19 22:23 <flipz> "The thing that gets us out of bed every day is the prospect of creating pathways above, around and through walls." msft marketdroid
2008-09-19 22:23 <flipz> sheesh
2008-09-19 22:23 <flipz> I'm glad I'm not him
2008-09-19 22:23 <flipz> or them
2008-09-19 22:23 <flipz> what gets me out of bed is sheer effort of will
2008-09-19 22:23 <flipz> and the prospect of some french roast
2008-09-19 22:25 <MaZe> heh, for me it's usually the buzzer and a sense of duty
2008-09-19 22:25 <flipz> when the family gets back tomorrow it will be a four year old jumping on me
2008-09-19 22:26 <flipz> "time to get up and play daddy"
2008-09-19 22:27 <flipz> "An approach dedicated to engineering the absence of anything that might stand in the way" -- my gawd who came up with that one, ballmer?
2008-09-19 22:27 <MaZe> ugh
2008-09-19 22:27 <flipz> engineering the absence - new msft slogan
2008-09-19 22:31 <MaZe> i'm all for engineering the absence
2008-09-19 22:31 <MaZe> of msft
2008-09-19 22:32 <Bushman> is the stupid question hour still on? 
2008-09-19 22:32 <MaZe> yep
2008-09-19 22:33 <Bushman> vm.swappiness, wanna explain it to me, what's it really do, what rules of thumb i wanna use to determine it, etc  
2008-09-19 22:33 <MaZe> oh, one of those
2008-09-19 22:34 <flipz> it's an andrewism
2008-09-19 22:34 <Bushman> did i pick a touchy topic? :)
2008-09-19 22:34 <flipz> take a vm that just plain doesn't swap very well and bolt knobs on it
2008-09-19 22:34 <flipz> akpm is a friend
2008-09-19 22:34 <flipz> but the linux vm has been dire for a few years
2008-09-19 22:34 <shapor> `http://kerneltrap.org/node/3000
2008-09-19 22:34 <flipz> swappiness is one of the attempted bandaids
2008-09-19 22:36 <Bushman> what's wrong with knobs?  better to have them than to not have them, isnt it?
2008-09-19 22:36 <flipz> better to have it work
2008-09-19 22:36 <flipz> than to give up and offer a knob which also doesn't work
2008-09-19 22:37 <flipz> like those old televisions
2008-09-19 22:37 <flipz> you had a color control that ranged from "very green" to "very red" with "very blue" in between
2008-09-19 22:38 <flipz> anyway
2008-09-19 22:38 <flipz> I'm taking a sabbatical from vm
2008-09-19 22:38 <flipz> so I am licensed to throw turds
2008-09-19 22:42 <shapor> Bushman: the vm operates in a delicate balance right now with knobs pulling in 4 dimensions
2008-09-19 22:42 <shapor> this breaks more than you think
2008-09-19 22:43 <shapor> i've seen memory recursion deadlocks with AoE and ddsnap
2008-09-19 22:43 <shapor> as well as other wacky behavior like:
2008-09-19 22:43 <shapor> http://pengaru.com/~swivel/pop_comparisons/04-26-2006/
2008-09-19 22:44 <MaZe> I don't even want to think about the vm
2008-09-19 22:44 <MaZe> let's stick with fixing fs and bdev
2008-09-19 22:44 <Bushman> Vito sounds like he could use to do some PCA to reduce the dimensionality in his quest for performance
2008-09-19 22:46 <shapor> its not a lot to ask of a modern server to be able to parse silly protocols like pop at wire speed
2008-09-19 22:46 <shapor> and cache some stuff
2008-09-19 22:47 <shapor> thats a case of the os clearly pissing in your cheerios
2008-09-19 22:48 <shapor> evicting >2GB of buffer cache at a time due to brain damage in the vm
2008-09-19 22:49 <Bushman> shapor:  you're spoiled, you need to work with windows for a while ;)
2008-09-19 22:49 <shapor> no thanks
2008-09-19 22:49 <Bushman> once i reinstalled a SCSI driver and all my fonts went bold
2008-09-19 22:49 <Bushman> wanna explain that one
2008-09-19 22:51 <flipz> mmm the nigiri was fine, now time to look into the sake issue
2008-09-19 22:51 <MaZe> lol
2008-09-19 22:52 <flipz> bushman, use after free?
2008-09-19 22:52 <flipz> in the registry?
2008-09-19 22:53 <Bushman> that was a long time ago, i dont remember.  i saw that and i thought i had some bad sushi or something ;) 
2008-09-19 22:55 <Bushman> i'm still working on the freetext indexing, do we need just straight text/html parsing, or do we want more like PDF or .doc? 
2008-09-19 22:55 <MaZe> there's no such thing as bad sushi
2008-09-19 22:55 <flipz> careful there ;)
2008-09-19 22:55 <Bushman> MaZe:  let it sit out in the sun for few hours, then you'll experience bad sushi 
2008-09-19 22:56 <flipz> bushman, I didn't know lxr had that knob
2008-09-19 22:56 <flipz> but straight text, yes
2008-09-19 22:56 <Bushman> it's not lxr, it's swish, the text indexer
2008-09-19 22:56 <flipz> right, so lxr doesn't recommend a config for swish?
2008-09-19 22:56 <flipz> I thought they did
2008-09-19 22:57 <Bushman> kinda, but if i'm doing it by hand, might as well pimp it out a bit 
2008-09-19 22:57 <flipz> anyway, text
2008-09-19 22:58 <Bushman> plus Shap tells me that i grabbed a slightly different code than what the norwiegian lxr site is running 
2008-09-19 22:58 <flipz> remember when it was glimpse?
2008-09-19 22:58 <MaZe> ıʞsʍoʞʎzɔuǝż ˙ɐ ɾǝıɔɐɯ - who can read this?
2008-09-19 22:58 <flipz> a kinda sort open indexer
2008-09-19 22:58 <flipz> university project
2008-09-19 22:58 <Bushman> the lxr manual says glimpse doesnt support all the functions it wants, so i went with swish
2008-09-19 22:58 <flipz> they tried to close it up and make some dough, nobody used it, finally somebody wrote swishe and nobody remembers glimpse
2008-09-19 22:59 <Bushman> i've used this russian indexer/search engine called mnogosearch before, if i'm really bored i might see if i can use that here
2008-09-19 23:00 <flipz> one of the things we plan to get happening with tux3 is proper incremental indexintg
2008-09-19 23:04 <shapor> why are we leaking the sb inode map in inode.c test?
2008-09-19 23:04 <shapor> ==31367== 8,160 (8,040 direct, 120 indirect) bytes in 1 blocks are definitely lost in loss record 4 of 7
2008-09-19 23:04 <shapor> ==31367==    at 0x4A1B858: malloc (vg_replace_malloc.c:149)
2008-09-19 23:04 <shapor> ==31367==    by 0x401E44: new_map (buffer.c:452)
2008-09-19 23:04 <shapor> ==31367==    by 0x40A088: new_inode (inode.c:128)
2008-09-19 23:04 <shapor> ==31367==    by 0x40B53C: make_tux3 (inode.c:493)
2008-09-19 23:04 <shapor> ==31367==    by 0x40BA21: main (inode.c:554)
2008-09-19 23:04 <MaZe> because it's broken ;-)
2008-09-19 23:04 <flipz> shapor, we want to know
2008-09-19 23:05 <shapor> does seem like the test is broken
2008-09-19 23:05 <shapor> doesn't*
2008-09-19 23:06 <flipz> want hints to track it down?
2008-09-19 23:06 <flipz> put exit(1) somewhere and see if you put it before or after the leak
2008-09-19 23:06 <shapor> smrt
2008-09-19 23:06 <MaZe> question:
2008-09-19 23:06 <MaZe> if a page-fault gets triggered
2008-09-19 23:07 <MaZe> due to the page being not present or write to read-only page
2008-09-19 23:07 <MaZe> from user space
2008-09-19 23:07 <MaZe> what context do you end up in the kernel? is that considered process context or interrupt context?
2008-09-19 23:07 <flipz> process
2008-09-19 23:07 <flipz> but not process
2008-09-19 23:07 <flipz> it's non-interrupt kernel
2008-09-19 23:08 <MaZe> lol, so what can/can you not do - how does it differ from process and interrupt?
2008-09-19 23:08 <MaZe> can you sleep?
2008-09-19 23:08 <MaZe> any doc pointers?
2008-09-19 23:08 <flipz> yes, you have to in order to read in the page
2008-09-19 23:08 <MaZe> right - fair point
2008-09-19 23:09 <MaZe> and of course another thread can trigger the same page fault again before you read it in, so you need to lock appropriately
2008-09-19 23:09 <MaZe> or that might happen automagically
2008-09-19 23:09 <flipz> http://lxr.linux.no/linux+v2.6.26.5/arch/x86/mm/fault.c
2008-09-19 23:09 <MaZe> so how many different types of contexts do we have?
2008-09-19 23:10 <MaZe> process? kthread? fault handler? interrupt? anything else?
2008-09-19 23:10 <shapor> wtf if i put an exit(1) right before the return 0; in main it doesn't detect a leak
2008-09-19 23:10 <MaZe> try exit(0)?
2008-09-19 23:10 <MaZe> does it always leak the same way?
2008-09-19 23:10 <MaZe> maybe instead of exit(1) do goto return 0 at end of main?
2008-09-19 23:11 <shapor> same
2008-09-19 23:11 <Bushman> cant you printf a bunch of usual suspects?
2008-09-19 23:11 <shapor> exit(0) also no error
2008-09-19 23:11 <flipz> maze, http://lxr.linux.no/linux+v2.6.26.5/Documentation/exception.txt#L266
2008-09-19 23:11 <MaZe> maybe it bypasses the check code?
2008-09-19 23:11 <shapor> i'll try moving the return up instead
2008-09-19 23:12 <MaZe> right that's the dealing with exceptions in kernel space
2008-09-19 23:12 <flipz> shapor, right, valgrind does that
2008-09-19 23:12 <MaZe> and using them to detect unauthorized reads, etc
2008-09-19 23:12 <flipz> maze, process, kthread and fault handler are all the same
2008-09-19 23:12 <Bushman> shap: you got ida pro handy?
2008-09-19 23:13 <MaZe> oh really? so basically there's just 2: process/kthread/fault vs interrupt
2008-09-19 23:13 <flipz> the only real difference with process is it has a user address space to work with
2008-09-19 23:13 <flipz> mm
2008-09-19 23:13 <flipz> that's actually just a flag bit
2008-09-19 23:13 <MaZe> and the doc you referred to is about faults triggered from the kernel
2008-09-19 23:13 <flipz> like having a file table
2008-09-19 23:13 <MaZe> ah
2008-09-19 23:14 <MaZe> right, but since we don't have any pointers passed in as parameters from userspace that doesn't really much matter
2008-09-19 23:14 <flipz> maze, the doc tells you about do_page_fault
2008-09-19 23:15 <MaZe> that it gets called?
2008-09-19 23:15 <MaZe> I know that ;-)
2008-09-19 23:15 <MaZe> but do_page_fault does special stuff if the page fault got triggered with eip in kernel space
2008-09-19 23:16 <shapor> ah i see whats going on
2008-09-19 23:16 <MaZe> although I guess we can trigger page fault-ins for user pages from kernel space as well
2008-09-19 23:17 <shapor> with the leak
2008-09-19 23:17 <MaZe> so the two cases shouldn't really differ
2008-09-19 23:17 <Bushman> how would that change anythign?  if you need to get a page, gotta fetch it, regardless where EIP is pointing at, isnt it?   
2008-09-19 23:18 <flipz> we don't support faults in kernel space
2008-09-19 23:18 <MaZe> you don't want to trigger sigsegv from kernel space
2008-09-19 23:18 <flipz> only object is to oops properly
2008-09-19 23:18 <flipz> used to panic on that
2008-09-19 23:18 <MaZe> you return EFAULT instead
2008-09-19 23:18 <Bushman> ok so how do you prevent that?
2008-09-19 23:19 <MaZe> wait, so what happens if userspace syscall writes and the buffer I passed in is swapped out?
2008-09-19 23:19 <MaZe> I'd always assumed the page-ins would happen via fault, do we actually map the memory in in some other way?
2008-09-19 23:23 <flipz> maze, ok it took me a while to remember
2008-09-19 23:23 <flipz> when a fault occurs, unlike an interrupt it isn't interrupting some random process
2008-09-19 23:23 <MaZe> right
2008-09-19 23:23 <flipz> it faults in the process that needs to work, so the fault handler uses that context
2008-09-19 23:24 <flipz> it just has to do a little register fiddling and play with the intruction pointer
2008-09-19 23:24 <flipz> in some cases, parsing the instruction stream
2008-09-19 23:24 <flipz> dimly remembering this from the last time I did it, many years ago
2008-09-19 23:24 <flipz> fault semantics on x86 are utter crap
2008-09-19 23:24 <MaZe> so basically a fault triggered from userspace is almost like a syscall
2008-09-19 23:24 <flipz> yes
2008-09-19 23:25 <MaZe> besides the fact it can happen anywhere, and needs some special asm on entry/exit to deal with the weird semantics
2008-09-19 23:25 <flipz> http://lxr.linux.no/linux+v2.6.26.5/arch/sh/kernel/cpu/sh5/entry.S#L1134
2008-09-19 23:25 <flipz> for example
2008-09-19 23:25 <MaZe> how about a fault triggered on access from within the kernel?
2008-09-19 23:26 <flipz> oops if you're lucky
2008-09-19 23:26 <flipz> panic if not
2008-09-19 23:26 <MaZe> and what if its triggered from irq context?
2008-09-19 23:26 <flipz> death
2008-09-19 23:26 <flipz> try it ;)
2008-09-19 23:26 <flipz> it's easy
2008-09-19 23:26 <MaZe> so how does the kernel guarantee the userspace memory accessed by syscalls is present?
2008-09-19 23:27 <flipz> it goes delving into page table entries and things
2008-09-19 23:27 <MaZe> do we grab and release locks on the userspace memory before and after the copy (and if not present call the pagein handlers manually?)
2008-09-19 23:27 <flipz> well
2008-09-19 23:27 <flipz> it doesn't need to be present
2008-09-19 23:27 <flipz> because it can fault
2008-09-19 23:27 <MaZe> huh?
2008-09-19 23:27 <flipz> well
2008-09-19 23:27 <flipz> sorry
2008-09-19 23:27 <flipz> not in kernel ;)
2008-09-19 23:27 <Bushman> double huh
2008-09-19 23:27 <flipz> it does the fault by hand
2008-09-19 23:27 <flipz> see get_user_pages
2008-09-19 23:28 <MaZe> right, so like I said above at 11:27:10
2008-09-19 23:28 <flipz> don't have timestamps on
2008-09-19 23:28 <MaZe> (11:27:10 PM) MaZe: do we grab and release locks on the userspace memory before and after the copy (and if not present call the pagein handlers manually?)
2008-09-19 23:29 <flipz> we don't grab locks on user memory
2008-09-19 23:29 <flipz> not sure what the question is
2008-09-19 23:29 <MaZe> not on struct page *?
2008-09-19 23:29 <flipz> no
2008-09-19 23:29 <flipz> not for that
2008-09-19 23:29 <flipz> we take ref counts
2008-09-19 23:29 <MaZe> ain't that the same thing?
2008-09-19 23:29 <flipz> nonzero ref count holds a page in memory
2008-09-19 23:29 <flipz> no
2008-09-19 23:30 <MaZe> but it prevents the page from disappearing from under us? right? so it's like a lock - except others can access it as well, right
2008-09-19 23:30 <MaZe> see why lock is an inappropriate name here
2008-09-19 23:30 <MaZe> I meant lock in the sense of lock into ram
2008-09-19 23:30 <flipz> it's not at all like a lock
2008-09-19 23:30 <flipz> it's a refcount
2008-09-19 23:31 <flipz> a lock is a serializer, and recount is a don't kill me
2008-09-19 23:31 <flipz> http://lxr.linux.no/linux+v2.6.26.5/mm/memory.c#L962 <- follow_page, doing the job of mm hardware by hand
2008-09-19 23:32 <MaZe> does refcount = 0 immediately result in memory being destroyed?
2008-09-19 23:32 <flipz> yes
2008-09-19 23:32 <MaZe> or will it only get evicted if need be at that point?
2008-09-19 23:32 <flipz> now... depends whether it's anon or not
2008-09-19 23:33 <flipz> anon has to be swept up, page cache is immediately freed at that point
2008-09-19 23:33 <flipz> don't quote me, I used to hack that stuff ;)
2008-09-19 23:33 <flipz> but it's been a while
2008-09-19 23:33 <MaZe> so normally a processes mapping holds a refcount on it's memory?
2008-09-19 23:34 <MaZe> but that doesn't work
2008-09-19 23:34 <flipz> check __free_page
2008-09-19 23:34 <MaZe> there have to be 2 layers here
2008-09-19 23:34 <MaZe> one virtual - what a process needs, one physical - what's in memory
2008-09-19 23:34 <flipz> that puts the page back on the buddy as soon as count hits zero I believe
2008-09-19 23:34 <MaZe> probaby vma vs page
2008-09-19 23:34 <flipz> so... treatment of inodes by the vfs is rather different
2008-09-19 23:35 <flipz> there is one refcount on a page for each pointer to it, basically
2008-09-19 23:35 <flipz> including one for the lru, I tried to implement, andrew never took the patch
2008-09-19 23:35 <flipz> but definitely one for the page cache
2008-09-19 23:35 <flipz> and one for each page table entry pointing at it
2008-09-19 23:35 <flipz> there aren't really two layers in linux
2008-09-19 23:35 <flipz> unlike freebsd
2008-09-19 23:36 <flipz> it's all one layer
2008-09-19 23:36 -!- amey(~amey@116.73.35.180) has joined #tux3
2008-09-19 23:36 <flipz> vma just specifies some access rights to memory regions
2008-09-19 23:37 <MaZe> hmm
2008-09-19 23:37 <flipz> you have lots of time to get that sorted
2008-09-19 23:37 <flipz> has no real impact on fs development
2008-09-19 23:39 <MaZe> am I right in assuming that during kernel startup
2008-09-19 23:39 <MaZe> a fragment of physical memory is reserved for a big honking array 
2008-09-19 23:39 <MaZe> of struct pages - one per page of physical memory in the system? including high-mem and all that?
2008-09-19 23:40 <MaZe> possibly with some hacks for discontig mem
2008-09-19 23:40 <flipz> yes
2008-09-19 23:40 <flipz> it's rather crude
2008-09-19 23:40 <MaZe> and struct page as struct address_space * mapping in it
2008-09-19 23:40 <MaZe> s/as/has/
2008-09-19 23:41 <MaZe> which appears to be inode related
2008-09-19 23:41 <flipz> very much so
2008-09-19 23:41 <flipz> we make every page header have that field even if its anon where the field has no use
2008-09-19 23:42 <MaZe> how big is a struct page - around 50 bytes?
2008-09-19 23:42 <flipz> address_space, mapping, and page cache are different names for the same thing by the way
2008-09-19 23:42 <flipz> stupidly sloppy terminology
2008-09-19 23:42 <flipz> less I think
2008-09-19 23:43 <flipz> it's been heavily sqzd
2008-09-19 23:43 <flipz> maybe 50 on 64 bit
2008-09-19 23:43 <MaZe> so we basically throw 1% of memory out for accounting purposes.
2008-09-19 23:43 <flipz> go into junkfs and printf(... sizeof(struct page))
2008-09-19 23:43 <flipz> much more than that
2008-09-19 23:44 <MaZe> and that's not even including the cpu pagetables
2008-09-19 23:44 <flipz> dentry and inode cache are really extravagant
2008-09-19 23:44 <flipz> it's not lean and mean
2008-09-19 23:44 <flipz> only compared to even worse kernels
2008-09-19 23:45 <MaZe> 56
2008-09-19 23:45 <flipz> 64 bit, right?
2008-09-19 23:45 <MaZe> yes
2008-09-19 23:46 <flipz> multiply 56 times 1 TB / 4096
2008-09-19 23:46 <MaZe> kind of wicked how you can use junkfs as a code injector
2008-09-19 23:46 <flipz> that's the point
2008-09-19 23:46 <flipz> side door into the kernel for dodgy people
2008-09-19 23:46 <MaZe> 14 GB
2008-09-19 23:46 <flipz> so... imagine the suck when we scan that
2008-09-19 23:47 <flipz> the sound of sucking is the only thing you hear from that computer
2008-09-19 23:47 <MaZe> scan it for what?
2008-09-19 23:47 <flipz> anything
2008-09-19 23:47 <flipz> freeable memory
2008-09-19 23:47 <MaZe> why would we want to scan it?
2008-09-19 23:47 <MaZe> oh
2008-09-19 23:47 <MaZe> so there's no heap structure of memory or anything like that
2008-09-19 23:47 <flipz> nope
2008-09-19 23:47 <flipz> it's the crudest imaginable system
2008-09-19 23:47 <MaZe> oh, that is indeed quite a vacuum
2008-09-19 23:48 <MaZe> while 1 TB is still rare
2008-09-19 23:48 <flipz> it was only recently that linus allow vma to be a tree isntead of a linear list
2008-09-19 23:48 <MaZe> 32-128G is perfectly reasonable nowadays
2008-09-19 23:49 <MaZe> and at that point we have almos 1-2G of struct page's
2008-09-19 23:49 <flipz> 1 tb is right around the corner
2008-09-19 23:49 <MaZe> argh
2008-09-19 23:49 -!- amey(~amey@116.73.35.180) has left #tux3
2008-09-19 23:49 <MaZe> so basically another part that wasn't designed
2008-09-19 23:49 <flipz> indeed
2008-09-19 23:50 <flipz> that comment applies to almost all the parts
2008-09-19 23:50 <flipz> s/almost//
2008-09-19 23:50 <MaZe> do other kernels get this even worse?
2008-09-19 23:50 <flipz> yes
2008-09-19 23:50 <flipz> unbelievable?
2008-09-19 23:50 <MaZe> yes
2008-09-19 23:50 <flipz> but true
2008-09-19 23:50 <flipz> possible exception of, oh, qnx
2008-09-19 23:51 <MaZe> I'd always assumed the kernel to be this awesome C/assembler layer of wicked algos and data structures
2008-09-19 23:51 <flipz> haha
2008-09-19 23:51 <flipz> welcome, you're a kernel hacker now
2008-09-19 23:51 <MaZe> evrything optimized and tuned to hell and back
2008-09-19 23:51 <flipz> to hell, not back
2008-09-19 23:51 <MaZe> lol
2008-09-19 23:52 <flipz> some bits are ok
2008-09-19 23:52 <MaZe> yeah
2008-09-19 23:52 <flipz> some bits are pretty damm amazing
2008-09-19 23:52 <MaZe> and I'm assuming it is in general getting better over time
2008-09-19 23:52 <flipz> but most bits are just plain crap
2008-09-19 23:52 <flipz> hard to say
2008-09-19 23:52 <flipz> it's getting bigger
2008-09-19 23:53 <MaZe> yes, I've noticed
2008-09-19 23:53 <flipz> I'm not sure its getting faster, seems to be regressing a little
2008-09-19 23:53 <MaZe> but I've assumed that hasn't been core functionality
2008-09-19 23:53 <MaZe> more just new drivers
2008-09-19 23:53 <MaZe> new filesystems
2008-09-19 23:53 <MaZe> etc
2008-09-19 23:53 <flipz> also core
2008-09-19 23:53 <flipz> all the big iron stuff
2008-09-19 23:53 <flipz> from sgi and ibm
2008-09-19 23:54 <MaZe> hmm
2008-09-19 23:54 <flipz> buffer.c and filemap.c get longer and longer
2008-09-19 23:54 <MaZe> so 2.7 happens when we get rid of bh?
2008-09-19 23:54 <flipz> things like mpage.c appear
2008-09-19 23:54 <flipz> linus said the variable sized page patch would be enough to open 2.7
2008-09-19 23:54 <flipz> that was a while ago
2008-09-19 23:54 <MaZe> thing is, I'm not sure I would want 2.7 to happen
2008-09-19 23:54 <MaZe> 2.5 was an utter mess
2008-09-19 23:55 <flipz> don't know how much he treats an email as a promise ;)
2008-09-19 23:55 <MaZe> and 2.6. up to 2.6.7 or so was junk
2008-09-19 23:55 <MaZe> going through that again would be painful
2008-09-19 23:55 <flipz> 2.6 is kinda starting to stink
2008-09-19 23:55 <flipz> it was fresh and new once
2008-09-19 23:56 <flipz> well
2008-09-19 23:56 <flipz> it's different
2008-09-19 23:56 -!- hirofumi(~hirofumi@210.171.168.39) has joined #tux3
2008-09-19 23:56 <flipz> in 2.3/4/5 we had a desparate situation
2008-09-19 23:56 <flipz> nobody had a kernel that worked properly
2008-09-19 23:56 <MaZe> otoh, it does seem like there's a lot of deep in the core stuff that should be changed
2008-09-19 23:56 <flipz> complaints about paging artifacts every day on lkml
2008-09-19 23:57 <flipz> for years
2008-09-19 23:57 <flipz> that was bad
2008-09-19 23:57 <flipz> it just didn't work
2008-09-19 23:57 <flipz> it's better now, where 2.6 may suck but it works
2008-09-19 23:57 <flipz> that's a good base to step out and do some housecleaning
2008-09-19 23:57 <MaZe> how much of this is sucky code, or bad algos, and how much is it just being np-complete or unsolvable problems
2008-09-19 23:58 <flipz> most of it is just sucky code
2008-09-19 23:58 <flipz> nearly all
2008-09-19 23:58 <flipz> we know about impossibility
2008-09-19 23:58 <flipz> don't count that
2008-09-19 23:59 <flipz> also, we have much better processes for bug tracking and nailing regressions
2008-09-19 23:59 <flipz> back in the day it was just linus and some text mode mailer