Back to Home / #linode / 2006 / 03 / Prev Day | Next Day
#linode IRC Logs for 2006-03-01

---Logopened Wed Mar 01 00:00:33 2006
00:38|-|interferon [~user@gastly-03.dynamic.rpi.edu] has quit [Quit: ERC Version 5.1.2 (CVS) $Revision: 1.809 $ (IRC client for Emacs)]
00:45<iggy>anybody having network issues?
00:46<gpd>not on mine
00:46<taupehat>all good here
00:49<asynch>i *thought* i was having network issues, but it turns out my linode was powered down (as of ~5am 2/28, afaict) -- never had that problem before.
00:50<iggy>it's strange... the initial connection setup takes forever, after that it seems to be okay
00:50<asynch>what kind of connection?
00:51<iggy>I've tried ssh and imap
00:51<asynch>hmm. sometimes those apps have dns reverse lookups on connect -- might be dns?
00:52<iggy>I disabled rdns lookups on my mail server
00:53<asynch>and sshd? RevMappingCheck iirc
00:55<asynch>gotta run, gl
01:30|-|Newsome [~sorenson@d53-64-13-210.nap.wideopenwest.com] has quit [Quit: Linux: Now with employee pricing!]
01:57|-|cow [Ap0ll0@modemcable160.99-83-70.mc.videotron.ca] has joined #linode
01:57|-|funkycow [Ap0ll0@modemcable160.99-83-70.mc.videotron.ca] has quit [Read error: Connection reset by peer]
02:56<linbot>New news from forums: Anti-SPAM Revisited in Email/SMTP Related Forum <http://www.linode.com/forums/viewtopic.php?t=2122>
---Logclosed Wed Mar 01 04:17:06 2006
---Logopened Wed Mar 01 04:39:20 2006
04:39|-|mikegrb [~michael@mail.thegrebs.com] has joined #linode
04:39|-|Ekipa kanalu #linode: Wszystkich: 44 |-| +op [0] |-| +voice [0] |-| normalnych [44]
04:39|-|VS_ChanLog [~stats@ns.theshore.net] has joined #linode
04:39|-|mode/#linode [+o mikegrb] by ChanServ
04:39<Megaversal>oooo
04:39<Megaversal>it came back
04:39|-|heidi [~heidi@mail.thegrebs.com] has joined #linode
04:39<Megaversal>i'm getting pings
04:39|-|efudd [~jason@forever.broked.net] has joined #linode
04:39|-|Kanal #linode zsynchronizowany w 32 sekundy
04:40<graham>Me too. Joy ! That's your sleep written off Megaversal then :-)
04:40<darkbeholder>maybe i looked after it came back up, i havent been at my computer for the past 4 or 5 hrs
04:40<Megaversal>who cares, i can sleep soundly!
04:40<Megaversal>night!
04:40<darkbeholder>:)
04:40|-|Megaversal [~dave@pool-71-107-253-96.lsanca.dsl-w.verizon.net] has quit [Quit: Leaving]
04:40<darkbeholder>night
04:41|-|chris [~chris@nullcode.org] has joined #linode
04:41|-|tierra [~tierra@ibaku.net] has joined #linode
04:41|-|SupaZubon [~crack@frotz.zork.net] has joined #linode
04:41|-|caker [~caker@ns.theshore.net] has joined #linode
04:43|-|BB [~chris@adsl.chrisburton.info] has joined #linode
04:43<BB>are we having fun at theplanet again?
04:44<graham>There was. Seems like it's back now
04:45<BB>i still cant get there via level3
04:45<graham>I couldn't earlier but I can now.
04:46<darkbeholder>could it have been a problem with one of the links and not the planet?
04:47<darkbeholder>cause i was on linode.com before when you siad it wasnt up
04:47<BB>I doubt it
04:48<darkbeholder>i dunno, the nets a weird place
04:48<graham>Guess it must of been. It is indeed :-)
04:48<BB>indeed it is, sadly the part around theplanet falls down more oftern than it should
04:49|-|graham [~graham@194-203-209-41.kbcfp.net] has quit [Quit: Leaving]
04:49|-|gpd [~gpd@70.85.16.173] has joined #linode
04:50<darkbeholder>i dunno, i find it more annoying when telstra screws up and no one in australia can access any hosts that arnt in australia
04:52<BB>heh, I have less people to blame when I run my access ISP and have multiple providers to chose from :)
04:54<darkbeholder>that would be nice but telstra owns all the lines and cables in au and when they screw up everyone in the country is screwed (well except those ppl who dont use the net ever)
04:56<BB>heh, I best warn my m8 who wants to move there ;)
05:03<darkbeholder>:)
05:03<darkbeholder>it doesnt happen that often
05:03<darkbeholder>its just really annoying when it does
05:03<BB>i can imagine
05:03<darkbeholder>probably happened 2 or 3 times in the past 3 years
05:04<darkbeholder>basically nothing is hosted here so it means the net just doesnt work
05:08|-|jekil [~alessandr@europa48.univ.trieste.it] has joined #linode
06:35|-|linville [~linville@azure.tuxdriver.com] has joined #linode
07:35|-|jekil [~alessandr@europa48.univ.trieste.it] has quit [Quit: Leaving]
07:46|-|Redgore [~Redgore@195.38.74.82] has joined #linode
08:06|-|thin [thin@69.46.24.28] has joined #linode
08:07<thin>are there any Xen linodes available?
08:12<tsi>maybe if xen3 weren't broken right now...
08:13|-|jimcooncat [~jim@216-220-225-50.midmaine.com] has joined #linode
08:16<jimcooncat>From 5:15 am to 5:40 am this morning my linode on host 50 was unaccessable on port 110 -- however I have uptime from Feb 1. Any clues what happened?
08:16<Redgore>checked your pop server is up ?
08:16|-|jim [~jim@82-254.119-70.tampabay.res.rr.com] has joined #linode
08:17<jimcooncat>it straightened out all by itself, and I have no special monitoring software installed yet
08:17|-|thin [thin@69.46.24.28] has left #linode []
08:17|-|jim [~jim@82-254.119-70.tampabay.res.rr.com] has quit [Quit: ]
08:17<Redgore>if it was just port110 not responding then itll be down to the pop3 server software you use not the linode itself
08:18<jimcooncat>I was wondering if there was network problems. If my pop server was down it would have probably stayed down until I got in this morning
08:19<Redgore>I dont know of any, but Ive only been online for about half hour
08:19<jimcooncat>unless the leprechauns were fixing it behind me back
08:19<jimcooncat>thanks Redgore, just curious
08:20|-|jimcooncat [~jim@216-220-225-50.midmaine.com] has left #linode []
08:37|-|cout [~cout@c-68-58-222-12.hsd1.sc.comcast.net] has quit [Quit: Changing server]
08:37|-|cout [~cout@c-68-58-222-12.hsd1.sc.comcast.net] has joined #linode
08:45<linbot>New news from forums: Host 40 in Sales Questions and Answers <http://www.linode.com/forums/viewtopic.php?t=2132>
09:52|-|jekil [~alessandr@82.52.168.202] has joined #linode
10:55|-|Dreamer3 [~dreamer3@0-1pool145-251.nas72.chicago3.il.us.da.qwest.net] has quit [Ping timeout: 480 seconds]
10:57|-|Dreamer3 [~dreamer3@0-1pool174-165.nas82.chicago3.il.us.da.qwest.net] has joined #linode
11:22|-|Redgore [~Redgore@195.38.74.82] has quit [Ping timeout: 480 seconds]
11:38|-|Redgore [~Redgore@195.38.74.82] has joined #linode
11:40|-|jekil [~alessandr@82.52.168.202] has quit [Read error: Connection reset by peer]
12:02|-|Redgore [~Redgore@195.38.74.82] has quit [Ping timeout: 480 seconds]
12:02|-|Redgore [~Redgore@195.38.74.82] has joined #linode
12:04<Spads>130.70-85-94.reverse.theplanet.com is pissing me off
12:04<Spads>knocking on my box's door
12:05<tsi>what a bizarre reverse
12:05<@mikegrb>so send a log snippet to abuse@theplanet.com
12:06[~]tsi checks rfc to be reminded if numbers are ok for hostnames
12:07<Spads>will do
12:08<@mikegrb>tsi: I believe it is rfc compliant, and even if not, it is certainly in common practice
12:08<tsi>that doesn't mean i like it
12:08[~]tsi looks for some other reason to formally complain
12:09<@mikegrb>it isn't meant to be left in place, it is generic rdns until a customer sets their own, like li-NN-NNN.members.linode.com
12:09<@mikegrb>er liNN-NNN\
12:10<taupehat>sheesh
12:11|-|Redgore [~Redgore@195.38.74.82] has quit [Ping timeout: 480 seconds]
12:12<Spads>right
12:13<Spads>I get a lot of these: reverse mapping checking getaddrinfo for 130.70-85-94.reverse.theplanet.com failed - POSSIBLE BREAKIN ATTEMPT
12:13<Spads>is that a forward/reverse matchup check or something?
12:13[~]taupehat relies upon a default DROP policy and some ssh trickery to not have to spend a lot of time on such matters
12:13<taupehat>btw
12:14<taupehat>the DROP target in iptables is also not rfc-compliant
12:14<taupehat>but I'm sure as heck going to continue using it
12:15<Spads>I use fail2ban
12:15<Spads>all it causes is logspam for me
12:15<taupehat>yep
12:15<Spads>because I don't even use passwords onthis system
12:15<Spads>it's all ssh keys and OPIE
12:15<taupehat>oh
12:15<taupehat>heh
12:15<Spads>but I should let the planet know about this guy
12:20<Spads>anyway, I bet theplanet just leaves canaries in there for things like this
12:20<Spads>http://www.google.com/search?q=reverse-mapping-checking-getaddrinfo-failed+POSSIBLE-BREAKIN-ATTEMPT <-- lots of reverse.theplanet.com entries
12:21<Spads>http://www.google.com/search?q=reverse+mapping+checking+getaddrinfo+failed <-- even more
12:21<@mikegrb>yes
12:21<@mikegrb>it's funny how many people post saying wtf does this error mean
12:21<@mikegrb>when the error message says exactly what it means
12:22<Spads>well
12:22<taupehat>how is the breakin supposed to work?
12:22<Spads>the question *I* have is why the failure indicates a breakin
12:22<Spads>bad DNS is rampant, for a start
12:22<taupehat>part of the confusion, btw, is that a lot of the error messages that say "OMG HAX" are false positives
12:22<Spads>and yeah
12:22<taupehat>so people tend to ignore that part
12:22<@mikegrb>because anyone can set their rdns to some.host.you.trust.com
12:22<Spads>so like, this IP has a reverse that does not exist as forward
12:22<Spads>that's pretty common
12:22<Spads>mikegrb: *anyone*?
12:23<@mikegrb>yes anyone
12:23<Spads>I thought you needed to have the netblock authority delegated to you
12:23<@mikegrb>you can set rdns PTRs to anything
12:23<Spads>man, my bath is probably cold now
12:24<Spads>anyway, I'm thinking of filtering out these warnings
12:24<Spads>because it's usually shit like this
12:24<@mikegrb>this wasn't a breakin attempt?
12:24<Spads>it was
12:24<@mikegrb>so how is that a false positive?
12:25<Spads>the diagnosis was wrong
12:25<Spads>first of all, I saw the breakin attempt in the ssh login failures
12:25<taupehat>mikegrb: I was referring to other software
12:25<Spads>second, it wasn't like he set his rDNS to zoot.zork.net
12:25<@mikegrb>ok sure, I'll bet the ssh developers get right on that
12:25<Spads>in order to abuse trust with that machine
12:25<Spads>get right on what?
12:26<@mikegrb>how is ssh supposed to no what domains you trust when you see them in the logs?
12:26<linbot>New news from forums: Is Linode for me? in Sales Questions and Answers <http://www.linode.com/forums/viewtopic.php?t=2124>
12:26<Spads>I don't think you're reading me correctly
12:26<Spads>I'm saying that I've never seen useful information from this log entry
12:26<@mikegrb>I don't think you understand the reason the message is there
12:26<Spads>so I'm just going to have logcheck stop telling me about them
12:26<Spads>because I learn nothing new by them
12:26<Spads>and I have no trust on a host/ip level with anywhere
12:27<Spads>since I don't use passwords or hostname-based auth, tell me what useful information this log entry could show me
12:28<Spads>and note that i plan to leave in the login failures
12:28<Spads>I'm genuinely curious
12:29<Spads>because I've been taught that rDNS is an amusing fiction for the past ten years
12:30<Spads>mikegrb: no really, am I missing something?
12:32<Spads>I don't want to filter it out if it's likely to give me useful security warnings that will help me take action.
12:32<@mikegrb>not everyone understands the fallibility of rdns, some people could see authentications failures from $random-name and think, oh that's just $random-user, he logs in from there and must have forgotten his passwrd
12:32<@mikegrb>etc
12:32<Spads>aha
12:32<@mikegrb>for you, sure there may not be much point
12:32<Spads>okay, good
12:32<@mikegrb>but the purpose of the messages are to warn people who don't understand DNS not to trust the name
12:32<Spads>yeah
12:33<Spads>okay, thanks for the explanation. I can see the usefulness for that scenario.
12:38<caker>http://svk.elixus.org/ <-- decentralized SVN
12:46|-|Redgore [~Redgore@195.38.74.167] has joined #linode
12:54|-|Redgore_ [~Redgore@195.38.74.167] has joined #linode
12:56|-|Redgore [~Redgore@195.38.74.167] has quit [Ping timeout: 480 seconds]
12:57|-|mode/#linode [+o caker] by ChanServ
13:09<Spads>haha
13:09<Spads>wow
13:09<Spads>http://svk.elixus.org/?SVKAntiFUD <-- "svk only makes use of the 2 underlying layers of Subversion, which are considered solid"
13:09<Spads>huh
13:13|-|Redgore_ [~Redgore@195.38.74.167] has quit [Read error: Connection reset by peer]
13:14|-|Redgore [~Redgore@195.38.74.167] has joined #linode
13:18<@caker>!weather 37211
13:18<linbot>caker: Temperature: 73°F / 23°C | Humidity: 47% | Pressure: 29.87in / 1011hPa | Conditions: Mostly Cloudy | Wind Direction: South | Wind Speed: 14mph / 22km/h; This Afternoon - Partly cloudy. Highs in the mid 70s. Southwest winds 10 to 15 mph with gusts to 25 mph. as of 11:48 am CST on March 1, 2006;
13:18<@caker>yummy
13:20<Viza>lucky
13:20<Viza>!weather 11229
13:20<linbot>Viza: Temperature: 39°F / 4°C | Humidity: 28% | Pressure: 29.89in / 1012hPa | Conditions: Clear | Wind Direction: West | Wind Speed: 12mph / 18km/h; This Afternoon - Mostly sunny. Highs in the upper 30s. Northwest winds 10 to 15 mph. as of 10:20 am EST on March 1, 2006;
13:22<Beirdo>!weather 00646
13:22<linbot>Beirdo: Temperature: 82°F / 28°C | Humidity: 66% | Pressure: 30.01in / 1016hPa | Conditions: Partly Cloudy | Wind Direction: ENE | Wind Speed: 15mph / 24km/h; Tonight - Variably cloudy. Isolated showers in the evening. Lows 73 lower elevations ranging to 62 higher elevations. East winds 5 to 15 mph. Chance of rain 20 percent. as of 3:09 PM AST on March 1, 2006;
13:22<Beirdo>:)
13:22<Beirdo>I wanna move already!
13:22<Redgore>!weather EGOS
13:22<linbot>Redgore: Temperature: 30°F / -1°C | Humidity: 80% | Pressure: 29.65in / 1004hPa | Conditions: Partly Cloudy | Wind Direction: WSW | Wind Speed: 13mph / 20km/h | Updated: 6:50 PM GMT;
13:23<Redgore>ill swap :P
13:23<Beirdo>hehe
13:23<Beirdo>!weather CYYZ
13:23<linbot>Beirdo: Temperature: 28°F / -2°C | Humidity: 55% | Pressure: 29.97in / 1015hPa | Conditions: Partly Cloudy | Wind Direction: West | Wind Speed: 8mph / 13km/h | Updated: 2:00 PM EST;
13:23<Beirdo>that's where I am now
13:23<Beirdo>well, actually
13:23<Beirdo>!weather CYTZ
13:23<linbot>Beirdo: Temperature: 30°F / -1°C | Humidity: 47% | Pressure: 29.97in / 1015hPa | Conditions: Clear | Wind Direction: WSW | Wind Speed: 12mph / 18km/h | Updated: 2:00 PM EST;
13:29|-|Redgore [~Redgore@195.38.74.167] has quit [Quit: A geek without purpose - http://martlev.com | SMDC-Network IRC - irc.smdc-network.org]
13:33|-|Redgore [~Redgore@195.38.74.167] has joined #linode
13:35|-|jekil [~alessandr@82.52.168.202] has joined #linode
13:42|-|Redgore [~Redgore@195.38.74.167] has quit [Quit: Lost terminal]
13:44|-|Redgore [~Redgore@195.38.74.167] has joined #linode
14:35|-|roadmr [~roadmr@201.144.60.151] has joined #linode
14:48|-|roadmr [~roadmr@201.144.60.151] has quit [Quit: Leaving]
14:57|-|besonen_ [~besonen@dsl-db.pacinfo.com] has quit [Read error: Connection reset by peer]
15:04|-|Sgeo [~Sgeo@ool-18bf61f7.dyn.optonline.net] has joined #linode
15:10|-|besonen [~besonen@dsl-db.pacinfo.com] has joined #linode
15:23<djayc>mikegrb: You around?
15:26<djayc>Is there a way to mount rw in lish?
15:35<Spads>Yes and no.
15:35<Spads>lish itself only has like six commands
15:35<Spads>but if you connect to a running linode
15:35<Spads>and can acquire root
15:35<Spads>you can often remount your volume
15:37<djayc>yeah I have root
15:37<djayc>mount says its mounted rw
15:37<djayc>but it's definitely ro
15:38<Spads>ah
15:39<Spads>so mount uses /etc/mtab
15:39<Spads>which may not have been written to before it went ro
15:39<Spads>cat /proc/mounts
15:39<Spads>rootfs / rootfs rw 0 0
15:39<Spads>^-- mine
15:39<djayc>yeah looks like it worked though
15:39<djayc>I just mount -o remount'ed and it looks like its working
15:39|-|jekil [~alessandr@82.52.168.202] has quit [Read error: Connection reset by peer]
15:39<Spads>how did you come to be ro?
15:39<djayc>its just how it came up
15:40<Spads>boot problem?
15:40<Spads>I had a problem with my ext3 going ro on regular occasions
15:40<djayc>nah looks like it got hijacked somehow so I need to rebuild one of my linodes :-/
15:40<@mikegrb>no
15:40<Spads>and it turned out to need bad fscking
15:40<@mikegrb>he is rectricted to lish access only
15:40<Spads>ahhhh
15:40<@mikegrb>and since inittab stuff didn't run, it didn't get remounted rw
15:40<Spads>huh
15:41<djayc>mike: Is there anything I can look for to see how it happened?
15:41<djayc>mike: While I'm getting my data
15:41<@mikegrb>how what happened?
15:41<djayc>mike: How it got rooted
15:41<@mikegrb>I tried to help you with that last time, so that you could fix the problem
15:41<@mikegrb>but you didn't want any help
15:41<@mikegrb>said the problem was fixed
15:42<djayc>mikegrb: I didn't say I didn't want help.. I said that I would change all my passwords and stuff..
15:42<Spads>heh
15:42<Spads>an important last step
15:42<djayc>mikegrb: You have to understand that I didn't get your tickets for days later.. I don't normally check that address.. and this linode has been up for quite a while with no problems so it was never a concern.. thats why I was very surprised
15:44<@mikegrb>you told me in your words.. you were "sure you had found the problem and corrected it" as I continued to tell you that wasn't the problem
15:44<djayc>mike: Where are you getting that from? I have the ticket open right here
15:44<djayc>you told me there were brute force attacks coming from my linode..
15:44<djayc>I asked what IPs
15:45<djayc>you told me "various other computers on the internet"
15:45<@mikegrb>it is not our fault you don't check that address, it is the address you gave us to contact you, if we should have used another address to contact you then you should have placed that other address in your account info
15:45<@mikegrb>sure, further up
15:45<djayc>then I asked for some more information.. and I asked you which computers so I can try and investigate
15:46<djayc>and you said "Many IPs.. etcetc.. due to your failure to respond we are shutting down"
15:46<djayc>I never once said I didn't want help trying to figure out how it happened
15:46<@mikegrb>correct, your linode was continueing to violate our terms of service and so in accordance with those terms your linode was shutdown
15:46<djayc>Yeah but its like .. you shut it down while we were discussing what the problem was
15:46<@mikegrb>You continued to tell me that this particular account had the password brute forced
15:47<djayc>I was like.. Really? What IPs .. you said "a lot" and I said "Give me some so I can check logs and stuff" and you're like "There are a lot.. I'm shutting it down"
15:47<@mikegrb>and I repeatedly told you that was not the case
15:47<djayc>well whatever.. it was a discussion
15:47<djayc>I had one account with a weak password and the brute force attacks stopped at the one in alphabetical order that got attacked
15:48<@mikegrb>obviously that was not the problem
15:48<djayc>I'm sure thats how it originally got taken over.. I was just wondering what was done to the system after I had take then web apps down and changed the password
15:48<djayc>I didnt know if they installed some kind of root kit or something
15:48<djayc>thats what I was asking about
15:48<@mikegrb>I would be money that isn't how it originally was compromised
15:48<@mikegrb>as I have quite a bit of experience in investigating such things
15:49<djayc>well that's what I was asking you to give me some suggestions for.. I just assumed since I had that one account with the weak password (weak as in easily brute forced) that's how it got cracked
15:49<@mikegrb>the account was not suscipptable to a random bruteforce attack, the only way that account would have been brute forced is if someone was specifically targetting your Linode, which, as I told you, was highly doubtful
15:49<djayc>regardless, I never said I didn't want your help.,.. just because I disagreed doesn't mean I was rejecting everything you were saying.. if you check the ticket dialog I asked you for suggestions numerous times on how to diagnose it
15:50<@mikegrb>I kept telling you that wasn't the problem and suggesting we look for the real problem
15:50<@mikegrb>you kept telling me that you were "sure" that was the problem
15:50<@mikegrb>just as you said there
15:50<@mikegrb>at any rate it doesn't matter
15:50<@mikegrb>you didn't want my help then, now you realize you are in over your head and want my help now, it isn't going to happen
15:51<djayc>I've never had anything but good support when dealing with caker.. I don't get why you are so rude about this.. show me where I didn't want your help?
15:52<djayc>I mean whats the big deal if I'm telling you had a user account with the password the same as the username with some random digits
15:52<djayc>it was a weak account
15:52<djayc>it could have easily have been targetted
15:52<djayc>I had a website on there that was getting like 500,000 hits a day
15:52<@mikegrb>you said you have the ticket in front of you, do you not see where I kept telling you where that account was not the problem and you keep telling me you were sure it was the problem?
15:52<djayc>someone could have easily targetted it
15:53<djayc>yeah I see what you were saying
15:53<djayc>but that wasn
15:53<djayc>but how it got hacked wasn't what I was asking you about
15:53<@mikegrb>you are welcome to speak to caker if you wish, but I will not reenable your Linode until it has been reinstalled
15:53<djayc>the fact that after I changed all the accounts, and even brought down the web apps, how it remained hacked..
15:53<djayc>I'm not asking you to re-enable it
15:53<djayc>I'm going to get my data off of it and rebuild it
15:54<djayc>I was just asking if you could help me take a look before I destroy it to learn and use that knowledge in the future
15:54<@mikegrb>it remained hacked because you didn't take care of the problem in the first place, what is so hard to underrstand about that?
15:54<djayc>but I guess under your thinking it'll be better for me to rebuild it, make the same mistake again possibly, and do this all over again
15:54<djayc>you suggested webapp, I thought brute force
15:54<djayc>I hardened the passwords and removed the webapps
15:54<djayc>I catered to both suggestions
15:54<djayc>THATS why I'm curious how it was still launching attacks
15:55<Spads>djayc: the fact that it remained hacked is a pretty good indication that the attacker had some other vector into your system, either an installed backdoor or some service that you didn't think to take down.
15:55<@mikegrb>if you had shown interest rather then tell me I was wrong and you were right, I would have been happt to help you
15:55<djayc>spads: Yeah, thats what I was trying to figure out before I took it down, but this is not my field, and thats why I was asking for help
15:55<@mikegrb>I have spent many hours on the phone with customers as we both watched a console session and I took them through the prossess of forensic analysis
15:56<djayc>mike: Does it matter how it originally got taken that much? I'm not trying to make this an ego war.. regardless of how it was originally taken over I told you I would have liked help just to learn what caused the problem in the first place
15:56<djayc>mike: Rather the continued problem.. that's all
15:56<djayc>mike: If you don't want to help, it's cool.. I'm fine rebuilding the thing and trying to figure it out on my own.. I was just asking, that's all
15:56<@mikegrb>and I tried to help you learn what caused the problem in the first place
15:57<@mikegrb>you /refused/ to listen to me
15:57<djayc>How did I refuse to listen to you? You said webapps, I took them down
15:58<Spads>djayc: if you want help in setting your system up anew, there are plenty of customers here (many full to overflowing with opinion) who could counsel you on establishing a more secure system.
15:58<encode>perhpas this discussion with djayc and mikegrb would be better in some private message / channel?
15:58<Spads>yeah
15:59<Spads>I'd be happy to actually discuss the hacking with djayc, or how best to set up a linode in future, but the current discussion seems to be at an impasse.
16:00<djayc>Spads: I'm just now getting my data together on there..
16:00<djayc>Spads: But what I'd really like to know is how the box continues to remain hacked.. I tried running chkrootkit on it and it didn't find anything
16:01<Spads>djayc: okay, so i'll drop a little history here, to perhaps give you some broader perspective
16:01<Spads>djayc: in the 1970s Ken Thompson, one of the original authors of Unix, wrote a paper called "On Trusting Trust"
16:02<Spads>djayc: in it, he revealed a backdoor that he had put in *all* versions of Unix, allowing anyone to log in with root privileges. It was a harmless feature he'd put in when he only expected a few installations, and it was an easy way for him to troubleshoot and help out his friends to whom he'd given the tapes.
16:03<Spads>djayc: but he was playing mischief, and actually put code in the C compiler that checked to *make sure* that this backdoor was in Unix. And if the admin tried to take it out, the compiler would PUT IT BACK IN!
16:03<Spads>Aha you say, but you have the source code to the compiler, too! We can take this check-and-replace code out of the compiler and THEN fix the login programs!
16:04<Spads>But he had forseen this, and added code to look for the check-and-replace code inthe compiler, and put THAT back in if it was missing
16:04<djayc>ah
16:04<djayc>er.. hah
16:04<djayc>thats nuts
16:04<Spads>now, this is not the mechanism I suspect, but it's an illustration of how deep security compromise can go
16:04<djayc>right..
16:04|-|Ciaran [~Ciaran@host86-132-95-128.range86-132.btcentralplus.com] has joined #linode
16:04<Spads>how do you know that your version of chkrootkit wasn't hacked by a rootkit?
16:04<Spads>how do you know that it doesn't depend on a rootkitted library that is lying to it?
16:04<Ciaran>Hi.
16:04<djayc>I got it afterwards.. but that's a good point..
16:05<Spads>the chkrootkit tool is a helpful first pass
16:05<Spads>but if you've been truly compromised, the trust you have in your tools has to be shaken to the point where you take more difficult tactics
16:05<Spads>for example, after this paper was released, someone went in with a debugger on another system and released a non-rootkitted version of Ken Thompson's C compiler
16:06<Spads>actually I think the compiler was Dennis Ritchie's, but you get the point.
16:06<djayc>mike: Are you still there?
16:06<djayc>spads: Yeah I gotcha
16:07<Ciaran>Spad: If you're referring to Reflections on Trusting Trust, I believe the rootkitted version was only released internally, not to real Unix users. But I could be wrong. Anyway, it doesn't detract from your point.
16:07<Spads>another problem is that a worm isn't the sort of thing that chkrootkit is meant to look for. Many worms don't even run as root. they just want to steal system resources, which are available to regular users. You may be part of a botnet compiling statistics about available systems and access methods, waiting for someone to do an ordinary user-level DDOS attack on someone else with your net connection
16:07<Spads>so running chkrootkit to find a worm is like trying to use antibiotics against a virus: the tools and the problem don't match up.
16:08<Sgeo>Hi Ciaran
16:08<Spads>Ciaran: ah, okay. the paper did describe resorting to an assembler, though.
16:08<Ciaran>Hiya Sgeo. :)
16:08<Spads>Ciaran: and people did report looking in the process table and finding programs running as "ken" after
16:08<Spads>but that may have just been hysteria :)
16:09<Ciaran>Spads: Ah. It's quite possible that I'm wrong, then.
16:09<Ciaran>I'm just going from what I heard.
16:09<djayc>I think I found the problem
16:09<djayc>actually I know I did
16:10<Ciaran>I was compromised on my Linode recently, but I've fixed it now. I'm fairly sure there's nothing left behind.
16:10<djayc>awstats.pl .. I installed it a long time ago
16:10<djayc>old version
16:10<djayc>I grepped the logs for it
16:10<djayc>and I can see the attack
16:10<djayc>wgets some files
16:10<djayc>etc etc..
16:11<Ciaran>I got compromised by an old phpBB version. It wasn't mi ne, it was one of my hostees'. They didn't use it anyway, so I moved it out of the webspace and renamed it.
16:11<Spads>yeah, that may have been the vector
16:11<djayc>yep.. that would be it
16:11<Spads>Ciaran: yeah, phpBB is frightening
16:11<Ciaran>The script was actually quite interesting to look at. It had two parts to it.
16:11<Ciaran>The first part is the spreader. It uses search engines to search for vulnerable versions of phpBB.
16:11<Sgeo>Is there a better alternative to a Linode for some background python processing and web hosting
16:12<Ciaran>Once the spreader was installed on a vulnerable machine, it used wget to download an irc zombie bot.
16:12<Spads>Sgeo: Not really :)
16:12<Ciaran>^IRC
16:12<Spads>Sgeo: you may find services that don't give you root access, or that just give you a user account on some many-usered machine
16:13<Sgeo>Are any of those services free?
16:13<Ciaran>From there, of course, anything goes. I think my Linode was taking part in a UDP DDoS attack when I saw it. That was frightening, and I didn't waste any time in using "iptables -A OUTPUT -j DROP" and the same for INPUT. (note for people who don't know - that command will cut off all communications from your box)
16:13<Spads>Sgeo: not likely.
16:13<Spads>Sgeo: unless you have a friend who has a linode and likes having users on
16:13<Sgeo>Cheaper than a Linode 80?
16:13<Spads>Sgeo: I don't know. You'd have to shop around.
16:14<Spads>Ciaran: why not just take down your interface?
16:14<Spads>I'd have just shut down eth0
16:14<Ciaran>Spads: I thought of that afterwards. I should have done.
16:14<Spads>haha
16:14<Beirdo>djayc, just curious, what version of awstats.pl?
16:14<Ciaran>Spads: At the time I just wanted to do anything.
16:14<Spads>yeah, I know
16:14[~]Sgeo remembers Ciaran taking something down with that command
16:15|-|Redgore [~Redgore@195.38.74.167] has quit [Quit: A geek without purpose - http://martlev.com | SMDC-Network IRC - irc.smdc-network.org]
16:15<Ciaran>Hehe. Yeah, my Linode box hosts an IRC server for a private IRC network which Sgeo and I are part of. :D
16:16<djayc>Beirdo: I had 6.1 on there, but I think anything prior to 1-25-06 is vulnerable
16:16<Sgeo>iiuc, you're the one providing webaccess then?
16:16<djayc>oh no
16:16<djayc>not 1-25-06
16:16<djayc>disregard that
16:16<Beirdo>heh
16:16<djayc>anyways, it was 6.1 on there
16:16<djayc>although the binar said 6.2
16:16<Beirdo>6.1 is pretty old for sure
16:16<djayc>yeah
16:16<Ciaran>Sgeo: My machine provides webaccess to the IRC chat, yeah.
16:17<djayc>I installed it to play with it a while back and kind of forgot about it.. then I remember hearing about a awstats exploit.. so I grepped the logs and voila
16:17<djayc>which would be why me removing all my webapps didn't stop the attack.. because it was awstats.pl which is not one of my webapps so I forgot about it ;-)
16:18|-|linville [~linville@azure.tuxdriver.com] has quit [Quit: Leaving]
16:19<Spads>djayc: well, that's *part* of it
16:19<Ciaran>I think AWStats and phpBB are probably *the* two scripts most often compromised.
16:19<djayc>spads: it wget'ed and installed bind and stuff... I think it's prob better to just rebuild the linode.. it's no biggie
16:19<djayc>yeah.. no more awstats, and no more phpbb ;-)
16:19<Spads>yeah
16:20<Spads>wow, it installed bind?
16:20<djayc>I had phpbb on there but I kept up to date with that
16:20<djayc>yeah you wanna a capture of the log?
16:20<djayc>you can see what it did
16:20<Spads>yeah, I'm really curious now
16:20<Spads>I don't do many postmortems nowadays
16:20<djayc>:)
16:20<djayc>k one sec..
16:20<Ciaran>The first time I got compromised, I re-installed the OS. I was looking for an excuse to migrate from Gentoo to Debian anyway, and it seemed as good an excuse as I was likely to get.
16:20<djayc>I'm in the middle of a tar right now
16:20<Spads>no worries
16:20<Spads>yeah
16:21<Spads>djayc: what distribution do you use?
16:21<djayc>debian
16:21<Ciaran>When I was on Gentoo I hardly ever updated since not only did it take ages but I knew I probably wasn't making things better for the others on host36.
16:22<djayc>but I installed awstats outside of apt
16:22<djayc>for whatever reason
16:22<Ciaran>Don't get me wrong. I absolutely adore Gentoo's customisability. I use it on my home computer. It's awesome. But it just doesn't work to have it on a Linode.
16:22<Ciaran>s/work/make sense/
16:23<Spads>heh
16:23[~]Sgeo wishes he was capable of the patience and skill that it would take to use Gentoo at home
16:23<Spads>I have been impressed with ubuntu's insistance on making all the core packages just install quietly
16:23[~]Sgeo uses (K)ubuntu at home
16:23<djayc>I've used Gentoo too.. same idaea.. I just got annoyed after a wahile
16:23<djayc>I use Ubuntu for my desktops and debian for servers
16:23<Spads>I have an ubuntu install with cron-apt set up to actually install the new packages it finds instead of just download them
16:23<djayc>I can't wait for Dapper Drake to come out so I can play with XGL
16:24<Spads>I don't do it on a critical server, but it's interesting to watch for problems
16:24<Sgeo>cron-apt?
16:26|-|schultmc [~schultmc@zealot.progeny.com] has quit [Quit: Client exiting]
16:26|-|schultmc [~schultmc@zealot.progeny.com] has joined #linode
16:26<Spads>Sgeo: it's a debian/ubuntu tool that runs apt-get out of cron. by default it just updates package lists and downloads ready packages into a cache area, but you can set it up to actually upgrade your system on a regular schedule
16:26<Spads>Sgeo: it's just that that upgrade goes smoother in ubuntu than in debian proper, because the packages in debian tend to ask a lot of detailed questions
16:27<Spads>whereas ubuntu has done a good job of replacing that stuff with reasonable defaults
16:27<Ciaran>Talking of apt, why is security.debian.org always SO SLOW?
16:27<Ciaran>Gah.
16:27<Spads>yeah
16:29<djayc>Debian was like the first linux I heard of and for some reason I never tried it or its off shoots up until fairly recently.. it's weird
16:29<Ciaran>Hehe. I had tried Debian once before, but I really didn't give it a very fair test. Thankfully I kept that in mind.
16:30[~]Sgeo 's first experience with linux was with Knoppix
16:31<Ciaran>My very first experience with Linux was Mandrake Linux 6.1.
16:33<Ciaran>Or, as I should say, Linux-Mandrake 6.1. That's what it was called back then.
16:34<djayc>My first was Slackware linux that I downloaded off a BBS way back
16:34<djayc>hah
16:34<djayc>and I was like.. what.. the hell.. is this
16:34<Spads>likewise!
16:34<djayc>it was broken into like a billion 1.44meg disks ;-)
16:34<Spads>it took me forever to install X
16:34<djayc>I don't even think I got that far Spads ;-)
16:34<Spads>like I think it was 2000 before i had X on a home system
16:34<Spads>nono, I never did X on slackware
16:35<Ciaran>Hehe. Slackware was what I moved on to other Mandrake. To b ehonest, when I installed Mandrake first, it was only as an occasional thing to dabble in and fool around with. I don't think I ever got as far as installing a GUI.
16:35<Ciaran>^be honest
16:35<Ciaran>Or actually, wait. Yes, I did.
16:35<Ciaran>I just never used it at the time, since the computer I was using was so slow.
16:35<Spads>heh
16:36<Ciaran>Besides, it was weird and unfamiliar.
16:36<Spads>working at SuSE was what pushed me to Debian
16:37<Ciaran>So yeah. Slackware was next after some time. It was recommended by a friend. It was, um, interesting to say the least.
16:37<Ciaran>Fast forward a few years and I'm back on Mandrake, but this time 9.0. This time I was using it far more than I was before, but still not a lot. It didn't help that my parents used the same computer I did and couldn't stand Linux.
16:38<Ciaran>So it was nice when they finally got their own computer. I began spending a lot more time in Linux.
16:40<Ciaran>I upgraded to Mandrake 9.1, but before I did so, I decided to have a quick go at the latest Slackware at the time, which I believe was also 9.1. It didn't take long for me to scurry back to Mandrake. I was spoiled by hotplug at the time, and Slackware didn't seem to have any hardware detection. Getting my USB mouse to work took ages for me at the time.
16:40<Ciaran>(I know a lot more about Linux now than I did then, of course.)
16:41<Ciaran>Mandrake 9.1 was niiice. It had its own Galaxy theme, which I loved.
16:41<Ciaran>Oh yeah. When I first used Mandrake 9.0, I had set it to use a Windows-like theme. I think it went quite some way to allaying my fears.
16:42<Ciaran>I gradually weaned myself off of it as time went by.
16:42<Spads>heh
16:42<Ciaran>So yeah, I was with Mandrake 9.1 for a few years. I then switched to Gentoo, and I've been using it since. A few years, at least.
16:43<Ciaran>I experimented lately with Debian on my home system, but I prefer Gentoo, I think. But as I say, Debian's perfect for a Linode.
16:44<Ciaran>Horses for courses, as they say.
16:44<Spads>http://zork.net/wiki/ZorkNetHardwareHistory <-- and now it's running Debian on a linode
16:45<Spads>I should update that
16:46<Ciaran>frotz! Nice.
16:46<Spads>haha yes.
16:49<Spads>there have been three or four main zork.net systems, always called just "zork" by the users
16:49<Spads>foo.zork.net, zork.zork.net, gaspar.zork.net, and frotz.zork.net
16:49<Spads>gaspar was so named posthumously
16:50<Spads>because it became a backup server, and that was the spell that made the most sense
16:51<npmr>hmmm
16:52<npmr>1.5% growth year to date in the linode customer base
17:08<anderiv>heh - gotta sending out a less-than-clear wallop to 30,000 IRC users and then having to correct yourself.
17:08<anderiv>***gotta love
17:08<Sgeo>Hmm?
17:16<djayc>mike: I'm ready to get my data off of the linode..
17:17<@mikegrb>reboot and you can transfer it
17:17<djayc>alright, thanks
17:26<djayc>pretty good transfer rates..
17:26<djayc>getting about 1MB/s
17:30<djayc>mike: I shut it back down again, thanks
17:54|-|harshy [~harshy@cpe-65-24-72-253.columbus.res.rr.com] has joined #linode
17:57[~]taupehat pulls his new X2100 out of the box =]
17:58<Spads>http://zork.net/~nick/tech-arg.txt <-- finally i found this in the logs. Proof POSITIVE that I helped write that "Things to say when you're losing a technical argument" file that has been net.flotsam for the past five years
18:22|-|Spads [~crack@dsl081-246-246.sfo1.dsl.speakeasy.net] has quit [Remote host closed the connection]
18:25|-|Spads [~crack@dsl081-246-246.sfo1.dsl.speakeasy.net] has joined #linode
18:45|-|harshy [~harshy@cpe-65-24-72-253.columbus.res.rr.com] has quit [Quit: Ex-Chat]
18:52|-|Spads [~crack@dsl081-246-246.sfo1.dsl.speakeasy.net] has quit [Quit: leaving]
18:52|-|Spads [~crack@dsl081-246-246.sfo1.dsl.speakeasy.net] has joined #linode
19:15[~]gpd twigs that Spads is SupaZubon / |^__^| - why all the name changes?
19:24|-|jekil [~alessandr@host204-174.pool8252.interbusiness.it] has joined #linode
19:26<Spads>gpd: I got bored with the katamari face
19:26<Spads>and people got mad because of it being hard to type
19:26<Spads>so I switched to SIGNALS PASSED AT DANGER
19:26<Spads>but SupaZubon is still the persistent irc process on my linode, and I'm the local laptop irssi
19:29|-|jekil [~alessandr@host204-174.pool8252.interbusiness.it] has quit [Read error: Connection reset by peer]
19:38<linbot>New news from forums: Xen update time? in Xen Testing <http://www.linode.com/forums/viewtopic.php?t=2118>
19:45<tierra>mikegrb: I just realized I still had my old CC info on my account (an expired CC now). Since billing just ran (my annual payment is due this month), do I need to run through that manual payment ("click to charge" next to my balance) after I updated my card or is updating my card enough and it'll automatically get around to charging it?
19:51|-|nybble [~nybble@d150-157-11.home.cgocable.net] has joined #linode
20:01<gpd>Spads: interesting - Spads is new to me: http://www.hse.gov.uk/railways/spads.htm
20:01<gpd>but I like the IT connotations you bring to it :)
20:02<Spads>haha
20:04<@caker>tierra: updating the cc info is enough -- it'll attempt to charge it tomorrow morning
20:04<tierra>alright, thanks caker
20:05<@caker>np
20:13<Sgeo>CC? What's... oh, Credit Card
20:13<Sgeo>I was thinking "Creatures COmmunity"
20:15<gpd>!acronym cc
20:15<linbot>gpd: No definitions found.
20:15<gpd>!acronym spads
20:15<linbot>gpd: No definitions found.
20:15<Eman>community college?
20:15<gpd>linbot are you well after your upgrade?
20:21|-|harshy [~harshy@cpe-65-24-72-253.columbus.res.rr.com] has joined #linode
20:57|-|nybble [~nybble@d150-157-11.home.cgocable.net] has quit [Quit: Leaving]
21:10|-|flatronf700B [~flatronf7@202.75.186.154] has joined #linode
21:27|-|gpd [~gpd@70.85.16.173] has left #linode []
21:28|-|gpd [~gpd@70.85.16.173] has joined #linode
21:55|-|FireSlash [~FireSlash@0-3pool251-221.nas19.kansas-city2.mo.us.da.qwest.net] has joined #linode
22:01|-|FireSlash [~FireSlash@0-3pool251-221.nas19.kansas-city2.mo.us.da.qwest.net] has quit [Quit: Leaving]
22:17|-|Eman [~go@dyn216-8-131-190.ADSL.mnsi.net] has quit [Read error: Connection reset by peer]
22:18|-|Eman [~go@dyn216-8-131-190.ADSL.mnsi.net] has joined #linode
22:59|-|VS_ChanLog [~stats@ns.theshore.net] has left #linode [Rotating Logs]
22:59|-|VS_ChanLog [~stats@ns.theshore.net] has joined #linode
22:59|-|Dreamer3 [~dreamer3@0-1pool174-165.nas82.chicago3.il.us.da.qwest.net] has quit [Ping timeout: 480 seconds]
23:00|-|Dreamer3 [~dreamer3@0-2pool194-253.nas82.chicago3.il.us.da.qwest.net] has joined #linode
23:28|-|Sgeo [~Sgeo@ool-18bf61f7.dyn.optonline.net] has quit [Remote host closed the connection]
23:37|-|darkbeholder [darkbehold@nmathe02.res.csu.edu.au] has quit [Quit: bbl]
---Logclosed Thu Mar 02 00:00:23 2006