Back to Home / #mythtv / 2008 / 05 / Prev Day | Next Day
#mythtv IRC Logs for 2008-05-28

---Logopened Wed May 28 00:00:18 2008
00:00-!-jhulst [n=jhulst@unaffiliated/jhulst] has quit ["Konversation terminated!"]
00:52-!-joobie_ [n=joobie@] has joined #mythtv
01:00-!-foxbuntu_vm1 [] has quit [Read error: 104 (Connection reset by peer)]
01:00-!-foxbuntu_vm1 [] has joined #mythtv
01:05-!-joobie [n=joobie@] has quit [Read error: 110 (Connection timed out)]
01:07-!-joobie [n=joobie@] has joined #mythtv
01:20-!-joobie_ [n=joobie@] has quit [Read error: 110 (Connection timed out)]
01:31-!-Loto_ is now known as Loto
01:44-!-foxbuntu_vm1 [] has quit [Read error: 104 (Connection reset by peer)]
01:45-!-foxbuntu_vm1 [] has joined #mythtv
01:50-!-gnome42 [] has quit [Remote closed the connection]
01:52-!-Dave123 [] has joined #mythtv
02:25-!-joobie_ [n=joobie@] has joined #mythtv
02:27-!-grokky [] has quit [Read error: 110 (Connection timed out)]
02:44-!-iamlindoro [n=mcnamara@] has quit []
02:46-!-joobie [n=joobie@] has quit [Read error: 110 (Connection timed out)]
02:49-!-Dibblah [] has joined #mythtv
03:09-!-dekarl1 [] has joined #mythtv
03:14-!-Netsplit <-> quits: MaverickTech, gbee, mace, nsaspook, meshugga_, Captain_Murdoch, stuarta, feiner, Hannibal-, lsobral, (+71 more, use /NETSPLIT to show all of them)
03:14-!-Netsplit over, joins: dekarl1, Dibblah, joobie_, Dave123, foxbuntu_vm1, melunko_, xris, Tanthrix, naeo, gbee (+71 more)
03:14-!-Netsplit <-> quits: MaverickTech, gbee, mace, nsaspook, meshugga_, Captain_Murdoch, stuarta, feiner, Hannibal-, lsobral, (+71 more, use /NETSPLIT to show all of them)
03:15-!-Netsplit over, joins: dekarl1, Dibblah, joobie_, Dave123, foxbuntu_vm1, melunko_, xris, Tanthrix, naeo, gbee (+71 more)
03:24-!-robthebob [] has joined #mythtv
03:26-!-dekarl [] has quit [Read error: 110 (Connection timed out)]
03:32-!-siXy [] has joined #mythtv
03:38-!-xris [] has quit []
03:49-!-grokky [] has joined #mythtv
03:52-!-will00 [] has joined #mythtv
03:52-!-will00 [] has left #mythtv []
04:04-!-joobie_ [n=joobie@] has quit [Read error: 110 (Connection timed out)]
04:04-!-robthebob [] has quit [Read error: 113 (No route to host)]
04:05-!-cecil [n=cecil@] has joined #mythtv
04:08-!-tomimo [] has quit [Read error: 113 (No route to host)]
04:20-!-cesman [n=cecil@pdpc/supporter/sustaining/cesman] has quit [Read error: 110 (Connection timed out)]
04:26-!-naeo [n=asl@] has quit [Read error: 110 (Connection timed out)]
04:27-!-Agrajag- [] has quit [Read error: 110 (Connection timed out)]
04:36-!-joobie [] has joined #mythtv
04:49-!-foxbuntu_vm1 [] has quit ["Leaving"]
04:59-!-Dibblah [] has quit [Read error: 113 (No route to host)]
05:06-!-aneiane [] has left #mythtv []
05:06-!-_BleedAway [] has joined #mythtv
05:18-!-BleedAway [] has quit [Read error: 110 (Connection timed out)]
05:18-!-_BleedAway is now known as BleedAway
05:24-!-joobie [] has quit [Read error: 113 (No route to host)]
05:24-!-joobie_ [] has joined #mythtv
06:10-!-Agrajag- [] has joined #mythtv
06:11-!-joobie_ [] has quit ["This computer has gone to sleep"]
06:18-!-zaheerm [] has quit [Read error: 104 (Connection reset by peer)]
06:28-!-joobie_ [] has joined #mythtv
06:30-!-Dibblah [] has joined #mythtv
06:31-!-siXy [] has quit ["bye!"]
06:47-!-naeo [n=asl@] has joined #mythtv
06:56-!-siXy [i=siXy@] has joined #mythtv
07:15-!-siXy [i=siXy@] has quit [Read error: 104 (Connection reset by peer)]
07:20-!-siXy [i=siXy@] has joined #mythtv
07:41-!-CDev [] has joined #mythtv
07:44-!-Bob24 [] has joined #Mythtv
07:44<Bob24>could someone please help me with this question
07:45<Bob24>is it possible to setup a Twinhan remote in Mythbuntu?
07:58<gbee>actually #ubuntu-mythtv or #lirc
07:58<Bob24>k thanks
08:05<Bob24>theres no one ther
08:12<Bob24>hello anyone here
08:29<gbee>there are plenty of people in #mythtv-users or #ubuntu-mythtv - this is the developers channel
08:39-!-melunko_ [n=hmelo@] has quit [Read error: 110 (Connection timed out)]
08:44-!-grokky [] has quit []
08:44<GreyFoxx>You can pay for your SD account in 2 month increments now ? Nifty for users who want more than 7 days but not a full year
08:52-!-joobie_ [] has quit ["This computer has gone to sleep"]
09:05<stuarta> /away busy at work
09:23-!-Cardoe [n=Cardoe@gentoo/developer/Cardoe] has joined #mythtv
09:39-!-abqjp [] has joined #mythtv
09:41-!-Bob24 [] has quit []
09:50-!-Anduin1 [] has quit [Connection timed out]
10:20-!-reynaldo [] has joined #mythtv
10:23-!-j-rod [n=jarod@nat/redhat-us/x-7d6dde32f054ce7d] has joined #mythtv
10:29-!-jmk [n=jmk@] has joined #mythtv
10:43-!-melunko_ [n=hmelo@] has joined #mythtv
10:51-!-gnome42 [] has joined #mythtv
11:13-!-reynaldo [] has quit ["Lost terminal"]
11:21-!-meshugga_ [] has quit [Read error: 110 (Connection timed out)]
11:25-!-otwin [n=otwin@] has joined #mythtv
11:28-!-jpabq [] has quit [Remote closed the connection]
11:28-!-beandog [n=steve@gentoo/developer/beandog] has joined #mythtv
11:39-!-siXy [i=siXy@] has quit ["bye!"]
11:47-!-Anduin [] has joined #mythtv
11:55-!-meshugga [] has joined #mythtv
12:02-!-stoth [] has joined #mythtv
12:09-!-meshugga [] has quit [Remote closed the connection]
12:14-!-xris [n=xris@] has joined #mythtv
12:17-!-Chutt [] has quit [Remote closed the connection]
12:30-!-meshugga [] has joined #mythtv
12:32-!-beandog [n=steve@gentoo/developer/beandog] has quit ["Leaving"]
12:37-!-beandog [n=steve@gentoo/developer/beandog] has joined #mythtv
12:53-!-timgonzales [n=timgonza@] has joined #mythtv
13:12-!-cattelan [] has quit ["Leaving"]
13:12-!-cattelan [] has joined #mythtv
13:21-!-famicom [] has joined #mythtv
13:24-!-cattelan [] has quit ["Leaving"]
13:24-!-cattelan [] has joined #mythtv
13:30-!-timgonzales [n=timgonza@] has left #mythtv []
13:33-!-Wonka [i=produzie@madwifi/support/wonka] has joined #mythtv
13:38-!-jgarvey [] has joined #mythtv
13:45-!-czth_ [n=dbrobins@nat/microsoft/x-1b3fc4f548298fb0] has joined #mythtv
13:46-!-mobrien [] has quit [Read error: 113 (No route to host)]
13:52-!-Anduin [] has quit [Connection timed out]
13:52-!-Anduin [] has joined #mythtv
14:00-!-czth [n=dbrobins@nat/microsoft/x-7e2ee37c7a8480d8] has quit [Connection timed out]
14:11-!-cecil is now known as cesman
14:12-!-cattelan [] has quit ["Leaving"]
14:25-!-jk1joel [] has quit [Remote closed the connection]
14:32-!-cattelan [] has joined #mythtv
14:39-!-Wonka [i=produzie@madwifi/support/wonka] has left #mythtv ["*plink*"]
14:56-!-_charly_ [] has quit []
14:57-!-_charly_ [] has joined #mythtv
15:08-!-robthebob [] has joined #mythtv
15:25-!-xris [n=xris@] has quit []
15:33-!-robthebob [] has quit [Read error: 104 (Connection reset by peer)]
15:36-!-stoth [] has left #mythtv ["Leaving"]
15:37-!-xris [n=xris@] has joined #mythtv
15:40-!-Loto [n=Loto@xbmc/user/Loto] has quit [Remote closed the connection]
15:41-!-jk1joel [] has joined #mythtv
15:42-!-Loto [n=Loto@xbmc/user/Loto] has joined #mythtv
15:45-!-lsobral [n=sobral@] has quit [Remote closed the connection]
15:46-!-robthebob [] has joined #mythtv
15:57-!-PointyPumper [i=Pintlezz@] has quit [Read error: 104 (Connection reset by peer)]
16:05-!-kormoc [n=kormoc@unaffiliated/kormoc] has quit []
16:15*xris ponders just linking mythweb's ajax APIs to google's new hosting service. heh
16:59-!-kormoc [n=kormoc@unaffiliated/kormoc] has joined #mythtv
17:05-!-joobie_ [] has joined #mythtv
17:10-!-robthebob [] has quit [Read error: 113 (No route to host)]
17:13-!-joobie_ [] has quit ["This computer has gone to sleep"]
17:18-!-bkero [n=bkero@osuosl/staff/bkero] has joined #mythtv
17:18<bkero>Howdy folks
17:20<bkero>MythTV's server seems to have SSH flapping about
17:20<stuarta>how so, you shouldn't have access to it's ssh server?
17:20*bkero is with the Open Source Lab.
17:21<bkero>We host it, and monitor it's connectivity.
17:21*stuarta pokes Snow-Man
17:21<bkero>SSH availability is being finnicky
17:21<gbee>yeah, he knows, he just didn't know who you were :)
17:21*stuarta apologises
17:21<bkero>np sir
17:21<stuarta>too many -users pop in here
17:22<gbee>good luck with Snow-Man, I've been pestering him to regen the ssh keys for a week
17:23<kormoc>gbee, way to make me feel secure!
17:23<bkero>I think it's either load, or someone's hammering trying to crack
17:23<gbee>the ssh issue account for the failed svn up
17:23<stuarta>probably the latter if the keys haven't been regenerated
17:24<gbee>can't ssh in at the moment
17:24<kormoc>stuarta, it's not that bad if it's just the host keys
17:24<stuarta>no, but still crap
17:24<kormoc>stuarta, it opens up to man in the middle, but noo viable
17:24<kormoc>*not too
17:24<stuarta>however, our usernames aren't that difficult to work out
17:24<gbee>kormoc: well weak host keys still suggest to an attacker than no remedial action has been taken on that machine, so they are looking for weak user keys
17:25<kormoc>gbee, indeed
17:25<stuarta>and some of us use debian and could possibly have generated keys on the crap ssh version
17:25<gbee>my key is fine, but I can't speak for others
17:25<stuarta>heh, i've done mine too
17:25*kormoc starts using his ssh rainbow tables... for educational purposes of course...
17:25<kormoc>my key's never not been fine :P
17:26<stuarta>only my throwaways suffered
17:26<gbee>never used a debian based distro for more than a couple of hours :)
17:26<kormoc>for fun (and profit!) I scanned my keys with the ssh bad ones and verified none of mine were affected
17:27<gbee>all the same, I wasn't complacent and I checked all my keys anyway
17:27<kormoc>it's likely load on the machine and not a ssh scan. www is down as well
17:28<gbee>plus the host keys on all servers to which I routinely connect - which is how I came to notice the problem with
17:28<bkero>So wait it out then?
17:28<gbee>xris: ping
17:28<stuarta>bkero: Snow-Man is the person you really need
17:28<kormoc>bkero, I'd give it some time to see if it will clear up, given I don't think any of us can access it anyway
17:28<stuarta>however Chutt may be able to help
17:29<kormoc>if it doesn't clear up on it's own here in a few, it might need physical access to figure out what's up, and we don't have a serial console, do we?
17:29<gbee>Snow-Man is officially in charge, Chutt is the project lead
17:30<gbee>in charge of the server I mean
17:30<bkero>If you don't, we can see about getting you a serial console.
17:30<kormoc>bkero, that'd be rockin da sockin
17:32<xris>gbee: busy, 5 mins
17:32<stuarta>what you do with your socks, please keep to yourself
17:33*bkero socks in place
17:33*stuarta whacks kormoc with a flimsy sock
17:34*kormoc sock-alanches stuarta
17:34*stuarta hides behind the washing machine
17:35<gbee>xris: np, just thought I'd ping you as you're the only one of us aside from Snow-Man and Chutt with any sort of real privledges on the server
17:35-!-whoDat_ [] has joined #mythtv
17:35<xris>what's going on?
17:35<kormoc>xris, server is flooded/down it seems
17:35<gbee>(22:20:00) bkero: MythTV's server seems to have SSH flapping about
17:35<kormoc>xris, know of any non-ssh way to access the box?
17:36<xris>it's pretty pegged, though
17:36<xris>we can have the osl guys restart it for us if we need to, though
17:36<xris>but that's up to Snow-Man
17:36<xris>and this is why xen is better than vservers...
17:36<gbee>Snow-Man seems to be AWOL
17:36<kormoc>xris, OSU ( bkero ) is who notified us :)
17:37*stuarta chuckles
17:37<kormoc>xris, cpu load?
17:37<xris>I can't even get into the parent server
17:37<xris>ssh is still "connecting"
17:37<kormoc>it'll time out
17:37<kormoc>none of the machines are actually working it seems
17:37<xris>vservers don't isolate cpu like xen does
17:38<whoDat_>anyone here in east cleveland area?
17:38<kormoc>but you can put process cpu caps on tho
17:38<xris>kormoc: but apparently Snow-Man hasn't. heh
17:38<xris>whoDat_: -users question maybe?
17:39<gbee>xris: bkero is offering a serial console, given that Chutt is MIA and Snow-Man doesn't seem to be around, do we wait or ... ?
17:39<xris>I can possibly get in via serial console...
17:39<bkero>Does it have a getty running on ttyS1? :/
17:39<stuarta>it'll still take a while if it's being hammered
17:39<xris>bkero: on that note, I have no idea. probably not, actually
17:39<whoDat_>xris lol if you say so
17:40<xris>I wonder how much damage it'd cause to just reboot the box.
17:40<gbee>bkero: any way on your end to identify/isolate a DDOS? Assuming that's what we're looking at?
17:40<kormoc>xris, safe reboot, not much
17:41<stuarta>maybe at the router level
17:41<xris>kormoc: assuming it has acpi/apm/whatever enabled..
17:41<stuarta>drop a few packets here and there
17:41-!-aneiane [] has joined #mythtv
17:41<gbee>stuarta: that's what I'm thinking, but I don''t know much about our arrangement with OSU
17:41<bkero>gbee: With a little work I can search through cacti and look at network traffic
17:42<xris>you guys think it's a ddos and not just a flooded process?
17:42<kormoc>bkero, if you hace the average connection time graphs, see if that jumped though the roof
17:42<bkero>We'd be notified if it did, but I'll look
17:42<kormoc>xris, personally, I think it's a process, but doesn't hurt to see what other things might be going on while we wait :)
17:42<bkero>If it is an ssh attack, it wouldn't be that bandwidth intensive anyway though
17:43<stuarta>just tie ssh up in knots
17:43<kormoc>stuarta, but we lost apache too
17:43<stuarta>probably a side effect
17:43<stuarta>if ssh takes too much cpu, apache will get starved
17:43<gbee>xris: I suspect it's either a ssh attack (as opposed to a proper ddos) or a load issue, but we're in the dark right now until someone has access to the server :)
17:43<kormoc>DDOS attacts typically manifest themselves as a ton of very very long open connections, each connection attempts to hold it open as long as it can
17:44<xris>well, I *do* have isaac's phone number...
17:44<xris>brb, "goodbye party for office mgr.
17:44<kormoc>Which is why the average connection open time is a simple way to see if it's just too much traffic or really a DDOS
17:44<gbee>we've had times when apache has been crippled by load, but I can't remember a time when apache and ssh have been down for this long at the same time
17:45<stuarta>good old /. effect
17:45<gbee>that's a point, have we been slashdotted today? :)
17:46*gbee accepts that committing isn't possible tonight and decides to watch some recordings instead
17:46<xris>yeah, this is more than the usual apache-load issues.
17:46<stuarta>no reason, we haven't released anything special
17:46<xris>my ssh connection is still trying. heh
17:47<kormoc>xris, I'm getting 0 bits after quite awhile here, don't think it's gonna actually work :P
17:47<xris>weird. "connection established" and then it just hangs
17:47<xris>kormoc: you're connecting to svn, though, right? the actual machine address is alcor
17:47<gbee>mine just fails to connect
17:47<bkero>I've gotta head to class for about an hour
17:48<bkero>I'll be back, basic` should be helping you
17:48-!-basic` [n=basic@osuosl/staff/basic] has joined #mythtv
17:48<kormoc>xris, svn aye
17:48<xris>I actually get "connection established" with I run `ssh -v`, but it doesn't get much further than that
17:48<kormoc>Howdy basic`
17:48<gbee>heh, though it does refuse immediately if I supply a bogus username
17:48<xris>loads my public keys and then nothing
17:48<xris>which does *feel* like a load issue
17:49<stuarta>sometimes it's faster to log in without the ssh keys if the load is too high (ie. use passwords)
17:49<gbee>at the end of the day it doesn't matter much I suppose, we're just waiting on Snow-Man or Chutt
17:50<basic`>does it prompt for a password?
17:50<basic`>or timeout before that
17:50<xris>stuarta: -v shows the same info with or without keys
17:51<kormoc>xris, you have three levels of -v to play with btw, but shouldn't really change much
17:53<xris>yeah, doesn't help much
17:53<xris>does show that I'm not getting the host cert back, though.
17:53<xris>just the connection, and then nothing
17:54<basic`>ssh_exchange_identification: Connection closed by remote host
17:54<basic`>is that what everyone is getting?
17:54<kormoc>basic`, nope, mine hangs just after the tcp connection and gets 0 data
17:55-!-jgarvey [] has quit ["Leaving"]
17:55<basic`>very odd
17:55<xris>basic`: same as kormoc for me
17:56<basic`>was openssh upgraded recently?
17:56<gbee>basic`: no, is just hanging there - gives me the ident error, though it might always have done -
17:56<basic`>strange, i can get to the password prompt for
17:57<stuarta>ooo it just logged me in
17:57<stuarta>top - 21:57:15 up 93 days, 1:06, 2 users, load average: 86.45, 124.38, 124.37
17:58<gbee>me too
17:58<xris>ooh, I got into svn
17:58<xris>and alcor
17:58<xris>bkero: you do something?
17:58<basic`>still nothing with www?
17:58-!-PointyPumper [i=Pintlezz@] has joined #mythtv
17:58<basic`>bkero left for class, he'll be back in ~50 minutes
17:59<kormoc>we have a metric ton of apache threads
17:59<basic`>wait, looks like is accessible now too?
17:59<xris>I'm into the master server, though. going to restart apache for good measure
17:59<kormoc>they're hanging bout a little too long
17:59<basic`>is apache eating all the resources or something else?
17:59<gbee>loads dropping off fast, no processes using excessive resources
17:59<stuarta>apache is iobound
18:00<gbee>ooo missed that
18:00<xris>stuarta: on www or svn?
18:01<xris>my vserver process hung when I tried to get in as root
18:01<kormoc>apache wasn't IO bound according to my last top update, but it's frozen now
18:01<gbee>wouldn't hurt to put some more ram in
18:01<kormoc>all the swap is used, if you bounced apache, it could be swap cleaning
18:01-!-Cardoe [n=Cardoe@gentoo/developer/Cardoe] has quit ["Leaving"]
18:02<kormoc>ps ax | grep apache2 | wc -l returned 195 processes a little while ago
18:02<kormoc>Did anyone put a cap on the number of threads to spawn?
18:02<xris>I didn't get in to bounce apache yet
18:02<xris>kormoc: that's be up to Snow-Man
18:03<xris>"vserver enter" is taking a LONG time
18:03<basic`>anything strange in the logs?
18:03<kormoc>down to 22 apache processes
18:04<basic`>did it go down again?
18:04<kormoc>it's messing with the swap, we're maxed on ram and swap, and that's what's killing everything
18:04<basic`>PROBLEM: is CRITICAL
18:04<kormoc>the OOM Should have killed something by now tho
18:05<gbee>hence my comment about more RAM, it's those apache processes
18:07<xris>I'm still waiting for my vserver command to run so I can get into svn as root..
18:07<xris>Snow-Man doesn't believe in sudo, so that's the only way.
18:07<gbee>I'll leave it to the big boys
18:09*xris twiddles thumbs
18:09<basic`>how much ram does it have? i'm guessing something needs adjusting before more ram is added
18:11<kormoc>2 gb + 2 gb swap
18:12<kormoc>we should have 6 slots open for registered/ecc ddr2
18:12<basic`>any way to check the logs?
18:12<kormoc>not until it's cleared up a tad
18:15<xris>I'm still locked out
18:15<xris>second ssh attempt to alcor is still hung
18:15<basic`>well it appears to have gone down again ~10m ago
18:16<basic`>at least that's when nagios caught it
18:17<clever>ive had to deal with oom freezes before too
18:17<clever>my 'main' server(133mhz 64mb ram) got frozen with oom for 48 hours once
18:18<clever>and i wasnt anywhere near close enough to boot the reset button
18:18<clever>but ssh managed to bearly work after 48 hours and then i got it rebooted
18:18<clever>and i later discovered it was apache slowly creeping until it ate everything
18:18-!-Chutt [] has joined #mythtv
18:23-!-jmk [n=jmk@] has quit ["Leaving"]
18:25<xris>I'm afraid to just reboot the machine
18:25<xris>but I still can't get in as root.
18:25<stuarta>it's out of swap
18:25<stuarta>Swap: 2097144k total, 2097144k used, 0k free,
18:26<xris>sounds like a reboot is pretty much necessary
18:26*stuarta ponders the why
18:26<basic`>yeah, can you not kill apache?
18:26<xris>not from where I can log in
18:26<xris>it uses vservers, and the only way I can get root is to go through the vserver command.
18:26<xris>apache lives inside of the vservers
18:26<basic`>ah, interesting
18:27<basic`>Snow-Man has root though?
18:27<xris>presumably.. but he also primarily uses the vserver command
18:27<xris>I can't ssh in anymore, though, so it doesn't matter much.
18:27<xris>I wonder if there are any processes on the parent virt that I can kill..
18:28<xris>assuming I can actually get my one ssh session to respond.
18:28<xris>any of you people here still logged in via ssh? can you please log out?
18:28<stuarta>i can log off if that'll help
18:28<stuarta>that's a bit dead
18:29-!-Dave123 [] has quit ["Leaving"]
18:29<stuarta>killed it off
18:30<stuarta>basic`: is there nothing you can do to temp drop the http traffic on the floor?
18:30<basic`>stuarta: i might be able to do something, let me see if it's possible
18:32<xris>still spinning
18:33<xris>I don't want to ~. my session because I don't know if it would let me back in at that point
18:33-!-nnewton [n=nnewton@osuosl/staff/nnewton] has joined #mythtv
18:33<clever>i had trouble even loging in withmy system, it would assume i took over 5mins to enter my pw and hangup
18:33<clever>before it even gave a pw prompt
18:34<nnewton>clever: I hear all the root admins are away?
18:35<stuarta>nnewton: Snow-Man is awol
18:35<stuarta>however xris is around
18:35<nnewton>k, so does anyone here have the ability to restart apache?
18:35<gbee>xris has root
18:35<gbee>but can't log in due to the load from apache
18:35<clever>moar ulimit next time!
18:36<stuarta>so ideally we need to drop the http traffic
18:36*xris votes for putting kormoc in charge of the next server
18:36*clever votes justinh, he will realy bitch about it:P
18:36<nnewton>stuarta: doing that externally would require us putting filters in place, which would need us to ping our uplevel provider which isn't really in scope of this issue
18:37<nnewton>we can however kick the box which will clear out the load and let you guys get in
18:37<stuarta>k, that's all inhouse at our office :)
18:37<xris>nnewton: yeah, I just don't want to do that without checking with the "senior" admins.
18:37<gbee>showoff ;)
18:37<nnewton>stuarta: we have less flexability there :)
18:37<clever>i have sysrq on so i can force an oom kill easily
18:38<xris>Chutt: ping
18:38<nnewton>clever: hehe, and hope it picks the right thing
18:38<stuarta>gbee: i will admit, it's nice being at the same level as an ISP
18:38*xris just noticed that Chutt logged in
18:38<gbee>huh, when did Chutt rejoin? He wasn't here earlier
18:38<stuarta>high level peerings and whatnot
18:38<clever>nnewton: even if it kills the wrong thing, it will recover enough for a proper reboot
18:38<nnewton>clever: not if it kills sshd
18:39<stuarta>1-0 to nnewton
18:39<clever>cant sysrq from the net
18:39<clever>sysrq needs psysical access to the keyboard
18:39<clever>or serial
18:39<gbee>does no-one have contact details for Snow-Man aside from Chutt?
18:40<gbee>clever: ok, so not helping with our current situation then ;)
18:40<stuarta>not ever read your kernel dmesg?
18:40<clever>i pipe the dmesg from my master backend out serial
18:40<clever>to a winblows box with a serial window open
18:40<stuarta>iirc he wrote some of the firewalling kernel modules
18:41<stuarta>hence his email appears when the module is loaded
18:41<nnewton>k guys, so if you decide to do a reboot let bkero/basic` know and they can do that (or someone else in the office)
18:41<xris>I can call isaac
18:42<nnewton>good luck :)
18:42-!-nnewton [n=nnewton@osuosl/staff/nnewton] has left #mythtv []
18:42<clever>if your close enough to psysicaly reboot it, you could sysrq it
18:42<gbee>if I dig through my archived emails I probably can find Snow-Man's address, just wondered if anyone had it handy and whether he's likely to respond to an email any more than IRC
18:42<xris>wow, my control-C finally registered.
18:43<stuarta>it may be untangling itself
18:43<xris>I probably have his email address somewhere.
18:43<clever>xris: yes its amazing when it finaly responds
18:43<stuarta>as the oom-killer rampages
18:43<clever>even a ps aux|sort -nk5 takes a few hours
18:43<xris>clever: that's *all* that happened. I now have a prompt, and a \n waiting to be acknowledged
18:43<stuarta>quick, type and hope
18:43<xris>stuarta: I did
18:44<clever>you could echo a leter into /proc/ to trigger oom remotely
18:44<clever>might be faster then ps aux|sort then kill an hour later
18:44<gbee>I assume that Chutt logged on then promptly went to make coffee or answered the phone :)
18:49<gbee>actually I favour the theory that he connected, saw the discussion and decided to pretend he's not there
18:50<stuarta>i lean toward the irc client autoreconnect theory myself
18:51-!-Dave123 [] has joined #mythtv
18:56<xris>no isaac by phone -- will leave message
18:59<gbee>I'm off to bed, good luck with it
19:04<bkero>Woo, the government is stimulating me!
19:11-!-Plugh [] has joined #mythtv
19:12<xris>sent an email to Snow-Man
19:12<xris>just in case
19:12-!-Plugh [] has left #mythtv []
19:19-!-grokky [] has joined #mythtv
19:24-!-reynaldo [] has joined #mythtv
19:27-!-famicom [] has quit ["Leaving"]
19:27-!-famicom [] has joined #mythtv
19:33<Chutt>why wouldn't you guys just reboot the machine?
19:33*Chutt doesn't understand the difficulty here
19:34<bkero>Chutt: If it's load we don't want to mess up your builds
19:34<Chutt>nothing to mess up
19:35<bkero>Do you have the authority to reboot it?(if so I'll go do it)
19:35<xris>Chutt: I'm more concerned about potential file system corruption
19:35<Chutt>trac likely just got stuck in a memory leak loop as usual
19:35<xris>dunnno what wacky stuff Snow-Man has set up
19:35<Chutt>everything important is raided
19:36<xris>raid doesn't protect against some of that.
19:37-!-joobie [n=joobie@] has joined #mythtv
19:37<Chutt>bkero, lemme know when you've rebooted it, please.
19:38<Chutt>ah, it's up.
19:38<bkero>Chutt: :)
19:38<Chutt>bkero, thank you =)
19:40<Chutt>everything appears to be working fine.
19:40<Chutt>i'll just blame it on trac, since that's what always takes down the machine
19:42<clever>more ulimit!
19:42<clever>i still havent thrown that at my 64mb apache box:P
19:43<bkero>apparmor apache :P
19:51-!-beandog [n=steve@gentoo/developer/beandog] has quit ["Leaving"]
19:59-!-PointyPumper [i=Pintlezz@] has quit [Read error: 110 (Connection timed out)]
20:02-!-melunko_ [n=hmelo@] has quit [Read error: 113 (No route to host)]
20:13-!-jhulst [n=jhulst@unaffiliated/jhulst] has joined #mythtv
20:23-!-jpabq [] has joined #mythtv
20:37-!-jhulst [n=jhulst@unaffiliated/jhulst] has quit ["Konversation terminated!"]
20:40*Snow-Man sighs.
20:44<Snow-Man>trac is a likely culprit.
20:46<bkero>Snow-Man: howdy
20:46<Snow-Man>thanks for rebooting it.
20:46<bkero>Sure thing
20:46<bkero>I just didn't want to screw your stuff up, so I waited
20:46<Snow-Man>In general it should be safe to do.
20:47<Snow-Man>trac tends to be a big, and once it's deep into swap, rebooting's probably safer than trying to do something else, heh.
20:47<kormoc>Snow-Man, have you thought about using ulimit to limit the apache user?
20:48<Snow-Man>we've done a few things to try and minimize the impact, havn't set a ulimit yet though, no.
20:48<kormoc>It should be fairly easy to deploy
20:49<bkero>You can set the number of child processes/worker threads too
20:49<Snow-Man>bkero: that we've done.
20:49<Snow-Man>bkero: the problem, in general, isn't the number of children or anything
20:50<Snow-Man>bkero: it's that each one grows to be stupidly large.
20:50<Snow-Man>We've already got max req. per child set down to 1000
20:50<kormoc>Snow-Man, one other thing I wondered was why the OOM killer doesn't start in?
20:50<Snow-Man>and it's using prefork, etc.
20:51<clever>when i had similar problems i ran gnome-panel thru X11 forwarding, so i could monitor the swap/ram 24/7
20:51<clever>and then i was able to catch it before it became unstable and track it down
20:51<Snow-Man>kormoc: I'm not sure it didn't?
20:51<Snow-Man>but it's not like it's just one process.
20:52<clever>using cacti&snmp you could do the same thing with less bandwidth&cpu&ram
20:52<kormoc>Snow-Man, true. Perhaps I'm just thinking they grew slower then they do
20:52<Snow-Man>eh, it takes a long time for them to get to be pigs
20:52<Snow-Man>but the OOM tends to come into the game *very* late
20:53<Snow-Man>and apache is going to respawn them if they die anyway
20:53<kormoc>ahh, fair 'nuff
20:53<clever>like when your 1000 miles away from the reset button for 2 weeks:P
20:53<clever>took the system 48 hours to recover that time
20:56*xris should go home.
20:56<xris>and in fact.. :)
20:56-!-xris [n=xris@] has quit []
20:57-!-xris [n=xris@] has joined #mythtv
20:57<Snow-Man>guess I'll do some upgrades on the box.
20:58<Snow-Man>so, be aware that the host key will likely change.
20:58-!-xris [n=xris@] has quit [Client Quit]
21:00-!-jhulst [n=jhulst@unaffiliated/jhulst] has joined #mythtv
21:01<clever>you could save the ssh key if you wanted
21:03<clever>recently i got a patch because the ssh key gen was poor
21:03<clever>made certain keys too much, had to regen mine
21:03<kormoc>clever, if it's a week host key, it should be changed
21:03<clever>no way to get arround that
21:04<clever>just securely tell the users about the new key
21:04<clever>gpg sign or something
21:04<Snow-Man>erm, if it's any of those keys, it shouldn't be used, period.
21:04<Snow-Man>I just told them.
21:05<clever>i think the problem is, you can test the public hostkey against the list of probable keys
21:05<clever>then fake being the host
21:05<clever>but you would still need to intercept the tcp stream also
21:07<bkero>Have you ever thought of using something like apparmor to limit memory usage?
21:09*Snow-Man shrugs.
21:09<Snow-Man>I think we're really just hoping trac will fix itself some day. :D
21:12<Snow-Man>trac's busted now, goodie.
21:12<Snow-Man>that'll mean it won't ever use too much memory at least.
21:20-!-xris [] has joined #mythtv
21:23-!-foxbuntu_vm [] has joined #mythtv
21:32<Snow-Man>ta-da, trac fixed.
21:32<Snow-Man>ugly ass fix, but it works.
21:33<bkero>trac = bandaids
21:33<clever>i should peel that bandaid off my elbow and see if the memleak is stoped:P
21:34-!-foxbuntu_vm [] has quit [Read error: 104 (Connection reset by peer)]
21:34-!-foxbuntu_vm [] has joined #mythtv
21:34<Chutt>i'll update trac + its dependencies this weekend
21:34<Chutt>their big 'trac has huge memleaks' bug is closed as fixed
21:34<Snow-Man>moving to 0.11 too?
21:34<clever>sounds fun, took me a day to figure out the sqlite->sqlite3 upgrade
21:34<clever>which now seems dead simple
21:34<Snow-Man>and to python2.5?
21:35<Chutt>Snow-Man, whatever current svn is
21:35<Chutt>is python2.5 in debian yet?
21:35<Chutt>is stuff built against it?
21:35<Snow-Man>that's what I was just beating back with a stick
21:35<Snow-Man>heh, libapache2-mod-python is built against it, the current version is sid
21:35<Chutt>last time i checked, most of the stuff trac needed (svn bindings, database bindings, mod-python) weren't
21:35<Snow-Man>which meant shit broke, so I had to forcibly downgrade it to the prior version
21:36<Snow-Man>I'm pretty sure it's all there now..
21:36<Chutt>ah, cool.
21:36<Snow-Man>I don't do alot of python stuff tho, so don't just take my word on it. :)
21:36<Snow-Man>They've been pushing real hard to get everything to 2.5 tho
21:36<Chutt>yeah, i'll check
21:37<Snow-Man>this was me about 2 minutes ago:
21:37<Snow-Man>===# dpkg --force-depends -i /var/cache/apt/archives/libapache2-mod-python_3.3.1-3_amd64.deb
21:37<Snow-Man>Preparing to replace libapache2-mod-python 3.3.1-3+b1 (using .../libapache2-mod-python_3.3.1-3_amd64.deb) ...
21:37<Snow-Man>note that '+b1' is a binary-only rebuild which moves it to 2.5.
21:42-!-PointyPumper [n=pintlezz@] has joined #mythtv
21:44<Snow-Man>going to upgrade the main host part of the box now, though not the kernel.
21:45<Snow-Man>that should be done at some point, but there isn't a kernel in Debian which has the vserver patch atm.
21:45<Snow-Man>mostly because the vserver patch is still on .22, heh.
21:45<Chutt>did you upgrade the www vserver?
21:45<Snow-Man>not yet, no.
21:46<Snow-Man>that's the.. 'fun' one..
21:46<Chutt>bah, it's only apache
21:46<Snow-Man>and mailman
21:46<Chutt>oh right
21:46<Snow-Man>I can give it a whirl tho, I suppose.
21:46<Chutt>i can try this weekend
21:47<Chutt>does upgrading automatically regenerate those weak keys?
21:47<Snow-Man>and should also automatically start rejecting user keys which are weak
21:50<clever>and spews a warning telling you so:)
21:52-!-kormoc [n=kormoc@unaffiliated/kormoc] has quit []
21:53<Snow-Man>the main box was old enough to not have a bad key. :D
21:53<Snow-Man>fyi, the upgrade of the main box is done.
21:53<clever>i think its just a chance thing
21:54<clever>half the keys it makes are predictable
21:54<Snow-Man>yea, so, no, not really.
21:54-!-melunko_ [n=hmelo@] has joined #mythtv
21:55<bkero>the openssl-blacklist should autoreject weak keys
21:55<bkero>Yay repeating
21:55<Snow-Man>I assume you mean openssh-blacklist. :)
21:55<clever>that doesnt mean 100% of the keys made by old code are weak
21:56<Snow-Man>no, just 100% of the keys made by the specific Debian-modified code in question are weak.
21:56<Snow-Man>Which ran for a couple of years..
21:56<Snow-Man>keys made older than that are, in general, fine.
21:56<bkero>and dsa keys that ever interacted with a debian system :P
21:57<clever>ahh debian made a tweak that broke it
21:57<Snow-Man>where have you been, exactly?
21:57<Snow-Man>and why are you still talking?
21:57<clever>that explains why older systems havent harmed it
21:57<clever>i saw the msg about it from a ubuntu upgrade(debian based)
21:58<clever>but i didnt dig into the problem much
21:58<bkero>Ya'll should've been using dropbear. :P
21:59<clever>whats to say they dont have twice as many bugs?:P
21:59<bkero>It very well could
21:59<bkero>But they're not publicly disclosed like some. :P
21:59<Snow-Man>gee, great
21:59<Snow-Man>bugs only the bad guys know about
21:59-!-jhulst_ [n=jhulst@unaffiliated/jhulst] has joined #mythtv
21:59<Snow-Man>those are the *best*
22:00<clever>now i ant check for problems myself!
22:00<clever>or fix them!
22:00<bkero>I'm just making a joke you turds.
22:00<clever>just like windows kernel bugs:P
22:01-!-jmk_ [] has quit ["Leaving"]
22:01<clever>ive got a palm based ssh client, and it warns me that it cant make good random numbers and may be unsecure
22:01<Snow-Man>hah, 'turds'
22:01<clever>and its natrualy got low ram/cpu
22:01-!-jhulst [n=jhulst@unaffiliated/jhulst] has quit [Read error: 104 (Connection reset by peer)]
22:02<Snow-Man>a'ight, welp, I'm done for the night on that stuff.
22:11-!-whoDat_ [] has left #mythtv []
22:14-!-jhulst_ [n=jhulst@unaffiliated/jhulst] has quit ["Konversation terminated!"]
22:15-!-jmk [] has joined #mythtv
22:16-!-adante_ [] has joined #mythtv
22:32-!-adante [] has quit [Read error: 101 (Network is unreachable)]
22:32-!-adante_ is now known as adante
22:40-!-jhulst_ [n=jhulst@unaffiliated/jhulst] has joined #mythtv
22:48-!-Netsplit <-> quits: adante, janneg, GreyFoxx, splat1, dekarl1, phrag, abqjp, otwin, Honk
22:57-!-Netsplit over, joins: adante, otwin, abqjp, Honk, dekarl1, phrag, janneg
22:57-!-naeo [n=asl@] has quit ["If you try to fail, and succeed, which have you done?"]
22:57-!-Netsplit over, joins: GreyFoxx, splat1
23:09-!-jhulst__ [] has joined #mythtv
23:09-!-jhulst_ [n=jhulst@unaffiliated/jhulst] has quit [Read error: 104 (Connection reset by peer)]
23:11-!-jhulst__ is now known as jhulst
23:14-!-joobie [n=joobie@] has quit ["Leaving"]
23:16-!-streamtrade [n=jsass@] has joined #MythTV
23:35-!-jhulst_ [n=jhulst@unaffiliated/jhulst] has joined #mythtv
23:41-!-jhulst_ [n=jhulst@unaffiliated/jhulst] has quit [Read error: 104 (Connection reset by peer)]
23:43-!-jhulst [n=jhulst@unaffiliated/jhulst] has quit [Read error: 113 (No route to host)]
23:52-!-otwin_ [n=otwin@] has joined #mythtv
23:56-!-foxbuntu_vm [] has quit [Read error: 104 (Connection reset by peer)]
23:57-!-foxbuntu_vm [] has joined #mythtv
23:58-!-Winkie [] has joined #mythtv
---Logclosed Thu May 29 00:00:46 2008