Back to Home / #linode / 2018 / 06 / Prev Day | Next Day
#linode IRC Logs for 2018-06-21

---Logopened Thu Jun 21 00:00:02 2018
00:01-!-acald3ron [~acald3ron@189.202.79.21.cable.dyn.cableonline.com.mx] has quit [Remote host closed the connection]
00:02-!-Rboreal_Frippery [~gV$e9dV5D@69.136.102.83] has quit [Ping timeout: 480 seconds]
00:15-!-eagles051388 [~Thunderbi@c178-234.i02-5.onvol.net] has joined #linode
00:15-!-eagles051388 is "eagles0513875" on #linode
00:21-!-eagles051387 [~Thunderbi@81.171.85.155] has quit [Ping timeout: 480 seconds]
00:23-!-eagles051388 [~Thunderbi@c178-234.i02-5.onvol.net] has quit [Ping timeout: 480 seconds]
---Logclosed Thu Jun 21 00:30:30 2018
---Logopened Thu Jun 21 00:30:40 2018
00:30-!-mikegrb_ [~mikegrb@orion.thegrebs.com] has joined #linode
00:30-!-Irssi: #linode: Total of 331 nicks [19 ops, 0 halfops, 0 voices, 312 normal]
00:31-!-purrdeta_ is now known as purrdeta
00:31-!-fullstop_ is now known as fullstop
00:31-!-CornishPasty_ is now known as CornishPasty
00:31-!-eichin_ is now known as eichin
00:31-!-zifnab_ is now known as zifnab
00:31-!-Alan__ is now known as Alan_
00:31-!-neersighted_ is now known as neersighted
00:31-!-Eugene [~eugene@kashpureff.org] has quit [Ping timeout: 480 seconds]
00:31-!-Netsplit synthon.oftc.net <-> charm.oftc.net quits: schwa, jr_net, atrus, d1b, Dianoga, noah, retro|blah, @dmonschein, mjevans, Patches, (+22 more, use /NETSPLIT to show all of them)
00:31-!-mode/#linode [+o dmonschein] by ChanServ
00:31-!-Netsplit over, joins: ChasedSpade, dmonschein
00:31<dwfreed>there goes OFTC's fremont server
00:31<greggypoo>my traceroute to fremont gets bounced between a bunch of dfw routers at cogentco
00:31-!-CornishPasty is now known as Guest5958
00:31-!-neersighted is now known as Guest5957
00:31-!-zifnab is now known as Guest5960
00:31-!-Peng [~mnordhoff@fin.mattnordhoff.net] has joined #linode
00:31-!-Peng is "Matt Nordhoff" on #tor-relays #powerdns +#Corsair #https-everywhere #linode-beta @#english #help #tor #moocows #python #opendns #oftc #linode #cherrypy
00:32-!-encode [~encode@mail.coadcorp.com] has quit [Ping timeout: 480 seconds]
00:32-!-Schroeder [1000@0001604e.user.oftc.net] has quit [Ping timeout: 480 seconds]
00:32-!-newton [james@0001510e.user.oftc.net] has quit [Ping timeout: 480 seconds]
00:32-!-Kassandry [~Kassandry@kassandry.net] has quit [Ping timeout: 480 seconds]
00:32-!-TecnoBrat [~tecnobrat@173.255.254.92] has quit [Ping timeout: 480 seconds]
00:32-!-jme [~oftc-webi@cpe-67-248-50-106.nycap.res.rr.com] has joined #linode
00:32-!-jme is "OFTC WebIRC Client" on #linode
00:32-!-arno_ [~arno@li587-95.members.linode.com] has quit [Ping timeout: 480 seconds]
00:32<dwfreed>mine dies after HE sjc2
00:32-!-DanielNM [user82500@0001518c.user.oftc.net] has quit [Ping timeout: 480 seconds]
00:32-!-stotch [~stotch@perrin.stotch.com] has quit [Ping timeout: 480 seconds]
00:32-!-ckuehl [~ckuehl@raziel.ckuehl.me] has quit [Ping timeout: 480 seconds]
00:32-!-mattmcc [~matt@2600:3c01::f03c:91ff:fedb:2ed0] has quit [Ping timeout: 480 seconds]
00:33-!-Netsplit over, joins: Cypher100
00:33-!-Archon810 [~oftc-webi@c-69-181-99-27.hsd1.ca.comcast.net] has joined #linode
00:33-!-FastLizard4 [fastlizard@ridley.fastlizard4.org] has quit [Ping timeout: 480 seconds]
00:33-!-rsdehart [~rob@eightpence.com] has quit [Ping timeout: 480 seconds]
00:33-!-levlaz [~levlaz@2600:3c01::f03c:91ff:fe62:b9ec] has quit [Ping timeout: 480 seconds]
00:33<Archon810>Fremont down?
00:33-!-Archon810 is "OFTC WebIRC Client" on #linode
00:33<dwfreed>yes
00:33<Archon810>damnit
00:33-!-Irssi: Join to #linode was synced in 171 secs
00:33-!-Netsplit over, joins: Asandari
00:33-!-hanksims [~oftc-webi@47.208.95.143] has joined #linode
00:33-!-hanksims is "OFTC WebIRC Client" on #linode
00:33-!-yepperoni [~yepperoni@li285-6.members.linode.com] has joined #linode
00:33-!-yepperoni is "yepps" on #linode
00:33<hanksims>Fremont down?
00:33<Archon810>yes
00:33-!-WindPower is "4096R/974EE250 | https://perot.me/" on #tor #tails #monkeysphere #linode #https-everywhere #freedombox #cryptoparty #cryptodotis #bitlbee
00:33-!-Netsplit over, joins: WindPower, Patches
00:33<hanksims>Thank you.
00:33<Zimsky>nonsense
00:33<Zimsky>everything is fine
00:34<dwfreed>...
00:34-!-Roedy is "Roedy" on #tor-project #linode #tor #openvas #munin #debian #OpenBSD
00:34-!-Netsplit over, joins: Roedy
00:34<Zimsky>
00:34-!-eam [~oftc-webi@c-76-126-228-69.hsd1.ca.comcast.net] has joined #linode
00:34-!-eam is "OFTC WebIRC Client" on #linode
00:34<Zimsky>https://status.linode.com/
00:34<Zimsky>see even the status page shows it
00:35<dwfreed>the status page has to be manually updated
00:35<Zimsky>no dwfreed
00:35<Zimsky>the status page would never lie
00:35-!-BlkStormy [~oftc-webi@74.82.132.99] has joined #linode
00:35-!-BlkStormy is "OFTC WebIRC Client" on #linode
00:35-!-mode/#linode [+q *!*@0001cb66.user.oftc.net] by dwfreed
00:35<nyancat>roses are red, violets are blue, fremont is down, this channel will do
00:35-!-Hydroxide1 [~oftc-webi@bas3-montreal28-70-53-157-223.dsl.bell.ca] has joined #linode
00:35-!-Hydroxide1 is "OFTC WebIRC Client" on #linode
00:35<eam>ah, fremont is down?
00:35<dwfreed>yes
00:35<nyancat>Seems to be having some connectivity issues at present.
00:36<eam>great, just came to confirm it wasn't me. Have a good evening all
00:36-!-krayon [~krayon@0001b1cd.user.oftc.net] has joined #linode
00:36-!-krayon is "Krayon" on #linode #suckless
00:36-!-ChasedSpade is now known as Guest5966
00:36-!-ChasedSpade [Centrixx@00026597.user.oftc.net] has joined #linode
00:36-!-ChasedSpade is "Chris Tyrrel" on #linode
00:36<nyancat>cheers mate
00:36<Hydroxide1>yup, basically same for me. good luck with the fix.
00:36-!-eam [~oftc-webi@c-76-126-228-69.hsd1.ca.comcast.net] has quit []
00:36<Archon810>Called Linode support, all the admins are looking into it
00:36<Archon810>https://status.linode.com/ will be updated in a few mins
00:37<jme>it's updated now
00:37-!-enc0de [~nathan@ppp59-167-135-129.static.internode.on.net] has joined #linode
00:37-!-enc0de is "Nathan Coad" on #linode
00:37<nyancat>they have ninjas working overnight.
00:38<greggypoo>i tried but i couldn't think of a better poem than nyancat's
00:38<linbot>New news from status: Connectivity Issues - Fremont <https://status.linode.com/incidents/2hd9pdyl81p5>
00:39-!-Guest5966 [Centrixx@00026597.user.oftc.net] has quit [Ping timeout: 480 seconds]
00:39<Woet>Zimsky: you were wrong, the status page clearly shows issues
00:40<dwfreed>Zimsky can't speak at the moment
00:40<enc0de>yeah i'm having issues connecting to my linode in fremont for about the last 15 minutes
00:40-!-james-123456789 [~oftc-webi@61-69-98-198.mel.static-ipl.aapt.com.au] has joined #linode
00:40-!-james-123456789 is "OFTC WebIRC Client" on #linode
00:40<nyancat>enc0de: Alright. see https://status.linode.com/incidents/2hd9pdyl81p5
00:40<enc0de>hopefully nothing too major and things are up again soon
00:40<enc0de>yeah saw that
00:41-!-derek [~derek@ds.bbido.com.yang.red] has joined #linode
00:41-!-derek is "derek" on #linode
00:42-!-CompWizrd [compwiz@dhcp-108-170-188-212.cable.user.start.ca] has joined #linode
00:42-!-CompWizrd is "compwiz" on #debian #debian-tech #linode
00:42-!-Dianoga [~dianoga7@3dgo.net] has joined #linode
00:42-!-Dianoga is "dianoga7" on #linode
00:42-!-noah is "this is noah" on #debian
00:42-!-noah [~noah@oof.n0ah.net] has joined #linode
00:42-!-levlaz [~oftc-webi@136.24.36.82] has joined #linode
00:42-!-levlaz is "OFTC WebIRC Client" on #linode
00:42<levlaz><3 hugops to whoever is on call right now.
00:42-!-relidy [~relidy@dev.servers.relidy.net] has joined #linode
00:42-!-relidy is "Relidy" on #linode
00:42-!-jimmy-test [~oftc-webi@61-69-98-198.mel.static-ipl.aapt.com.au] has joined #linode
00:42-!-jimmy-test is "OFTC WebIRC Client" on #linode
00:42-!-james-123456789 [~oftc-webi@61-69-98-198.mel.static-ipl.aapt.com.au] has quit []
00:43-!-schwa [~laptopdud@pool-108-24-44-55.cmdnnj.fios.verizon.net] has joined #linode
00:43-!-schwa is "purple" on #linode
00:46-!-mlabs [~oftc-webi@2001:8000:10de:9a00:7817:6d70:e291:528c] has joined #linode
00:46-!-mlabs is "OFTC WebIRC Client" on #linode
00:47<mlabs>hi all - any eta on freemont
00:47<dwfreed>it'll be fixed when it's fixed
00:47<mlabs>gee thnx
00:47<dwfreed>the status page has ways of notifying you when there are updates
00:47-!-fpfpfdev [~oftc-webi@121.99.130.115] has joined #linode
00:47-!-fpfpfdev is "OFTC WebIRC Client" on #linode
00:48-!-Affliction [affliction@2402:1f00:8101:135::] has joined #linode
00:48-!-Affliction is "Affliction" on #linode
00:48<fpfpfdev>so how bad is the outage
00:48<mlabs>thnx for the tip dwfreed but our mail is impacted so it's a pita
00:48<dwfreed>time spent trying to figure out an eta is time not spent fixing the problem
00:48-!-Zachary [~oftc-webi@2600:1700:1150:96aa:9828:62a5:3b88:e105] has joined #linode
00:48-!-Zachary is "OFTC WebIRC Client" on #linode
00:48<dwfreed>fpfpfdev: it appears that all of fremont is inaccessible
00:48-!-jasonlshelton [~oftc-webi@ip72-199-249-163.sd.sd.cox.net] has joined #linode
00:48-!-jasonlshelton is "OFTC WebIRC Client" on #linode
00:49<fpfpfdev>any idea of cause yet? like tractorhitthecable cause or just general network being bad
00:49<hanksims>Power outage at the datacenter
00:49<hanksims>https://imgur.com/a/4i9uElW
00:49<levlaz>oh fun, that brings back a lot of great memories.
00:49<dwfreed>fmt1, anybody?
00:49<fpfpfdev>oh ugh
00:49<Peng_>fmt is up
00:50<Peng_>fmt1 is up* !!!
00:50<Peng_>bad typo Peng_
00:50<levlaz>\o/
00:50<fpfpfdev>is it really
00:50<Peng_>hanksims: That's not proof the data center lost power though. I mean, it's likely, but. Could also have taken out the routers but not (many of) the servers.
00:50<hanksims>https://imgur.com/a/4djMRml
00:51-!-archon810m [~androirc@c-69-181-99-27.hsd1.ca.comcast.net] has joined #linode
00:51-!-archon810m is "Android IRC Client" on #linode
00:51<Peng_>fpfpfdev: Yes. fmt1 is really up, and it really was a bad typo
00:51<fpfpfdev>whats fmt1- one of the data centers?
00:51<fpfpfdev>because I can't reach the servers I use there
00:52<Peng_>fpfpfdev: It's a Hurricane Electric data center Linode used to use but no longer users because it sucked
00:52<fpfpfdev>ah
00:52<Peng_>It's a useless ironic observation on my part
00:52<fpfpfdev>so now we're on suck centre #2
00:52<fpfpfdev>?:)
00:53<Peng_>Yes.
00:53-!-Zachary [~oftc-webi@2600:1700:1150:96aa:9828:62a5:3b88:e105] has quit [Remote host closed the connection]
00:53<Peng_>No, it's a lovely but only *almost* perfect data center. ;D
00:54<jme>assuming it's still there ;-
00:54<dwfreed>nyancat: you've been beaten: < furrywolf> oh where, oh where, did my linode go? it all started when I noticed my connection geting slow, with my packets unable to flow. checking the status page, it seems you have seen that this is so, but waiting is still causing me woe. any estimate on when it will resume its glow, if anyone should perchance to know?
00:54-!-valentic [~oftc-webi@c-73-162-255-242.hsd1.ca.comcast.net] has joined #linode
00:54-!-valentic is "OFTC WebIRC Client" on #linode
00:54<fpfpfdev>l33t rhymes
00:54<nyancat>dwfreed: i shall recoup my loss
00:54<ChasedSpade>Beautiful
00:55<nyancat>and emerge victorinyan
00:55-!-g4e [~oftc-webi@124-149-164-117.dyn.iinet.net.au] has joined #linode
00:55-!-g4e is "OFTC WebIRC Client" on #linode
00:55<Woet>dwfreed: why not? I didn't see any channel op taking any action.
00:56<linbot>New news from community: i can't opencart my website <https://www.linode.com/community/questions/16953>
00:56-!-circlenet [~oftc-webi@23-25-121-89-static.hfc.comcastbusiness.net] has joined #linode
00:56-!-circlenet is "OFTC WebIRC Client" on #linode
00:57-!-kenyon_ [~oftc-webi@2605:e000:1c0d:236:222:4dff:fe83:cd4a] has joined #linode
00:57-!-kenyon_ is "OFTC WebIRC Client" on #linode
00:57<dwfreed>everyone with op in this channel is either: 1) not working and thus probably sleeping; 2) wading through the fresh mountain of tickets; or 3) working on trying to fix the problem in Fremont
00:59-!-Q3Man [~oftc-webi@ip68-108-160-170.lv.lv.cox.net] has joined #linode
00:59-!-Q3Man is "OFTC WebIRC Client" on #linode
01:00<nyancat>dwfreed: eat shit.
01:00<Q3Man>so... anyone have any Fremont insights? any news on if its power or connectivity? Trying to determine how bad my night is going to be....
01:00<nyancat>https://www.irccloud.com/pastebin/ebSXivzg/
01:00<MrPPS>whenfremont.jpg
01:00<MrPPS>:D
01:00<nyancat>also i might be drunk.
01:00<greggypoo>+1 nyancat
01:01-!-Gary [~oftc-webi@119.93.9.133] has joined #linode
01:01-!-Gary is "OFTC WebIRC Client" on #linode
01:01-!-fpfpfdev [~oftc-webi@121.99.130.115] has quit [Quit: Page closed]
01:02<dwfreed>nyancat: that doesn't quite rhyme very well
01:02<circlenet>We don't see the BGP networks advertised
01:02<nyancat>i let the liquor talk.
01:02<circlenet>for Freemont
01:02<nyancat>this is what happened.
01:02-!-Zachary [~oftc-webi@2600:1700:1150:96aa:6571:71b4:7cfe:7765] has joined #linode
01:02-!-Zachary is "OFTC WebIRC Client" on #linode
01:02<dwfreed>circlenet: the current theory is power outage
01:03<nyancat>dwfreed: I'm quite aware that it doesn't rhyme, but you know what, there was an attempt, and I will not be beaten at my own game
01:03<dwfreed>I mean, it's not a successful attempt unless it rhymes
01:03-!-cc_ [~oftc-webi@114.94.176.62] has joined #linode
01:03-!-cc_ is "OFTC WebIRC Client" on #linode
01:03<circlenet>Thx
01:03<nyancat>challenge accepted.
01:03<nyancat>brb
01:04<Hydroxide1>nyancat: if we're analyzing closely, "while I ping tested, many a query" has the wrong scansion - The Raven is mostly in trochaic octameter, including the original of that line. nice work though!
01:05<nyancat>I tried
01:05<nyancat>lol
01:05<Woet>what the fuck am I reading
01:05-!-Gary [~oftc-webi@119.93.9.133] has quit []
01:05<nyancat>Woet: someone trying to show me up
01:05<nyancat>and i'm responding with drunken stupor
01:05-!-cc_ [~oftc-webi@114.94.176.62] has quit []
01:05<nyancat>and failing miserably
01:05-!-vyanky [~oftc-webi@43.243.80.163] has joined #linode
01:05-!-vyanky is "OFTC WebIRC Client" on #linode
01:05<vyanky>Hi
01:05<nyancat>vyanky: Good evening.
01:06<vyanky>my linode is not rebooting
01:06<archon810m>I really hope the servers are still up and just don't have network
01:06<nyancat>vyanky: what region is your linode located in?
01:06<Hydroxide1>vyanky: if it's in fremont, known outage, they're working on it
01:06<hanksims>vyanky: Don't reboot if you're in Fremont
01:06<nyancat>as per Hydroxide1, please reference https://status.linode.com/incidents/2hd9pdyl81p5
01:06-!-transfervans [~oftc-webi@122-62-136-119-fibre.bb.spark.co.nz] has joined #linode
01:06-!-transfervans is "OFTC WebIRC Client" on #linode
01:06<vyanky>Fremont, CA, USA
01:07<nyancat>Indeed.
01:07<transfervans>Hi All
01:07<nyancat>Please pay close attention to the link I've just pasted for your convenience.
01:07<vyanky>is there any problem in Fremont, CA, USA
01:07<nyancat>transfervans: Good evening.
01:07<Q3Man>has anyone said if its power or connectivity?
01:07<nyancat>vyanky: please review the link I've just pasted. https://status.linode.com/incidents/2hd9pdyl81p5
01:07<vyanky>ok
01:07<nyancat>Q3Man: There's been no further direction on exactly what's happened or what will happen
01:08<Peng_>archon810m: Hopefully. They may be up. They may not be. Sometimes *some* of them are up.
01:08<Q3Man>ok.. assuming nuclear attack by N. Korea until proven otherwise...
01:08<Hydroxide1>Q3Man: dwfreed said above that the current theory is a power outage, but I think the people who will give an official answer to that are busy fixing the problem now
01:08<hanksims>PG&E is down at their datacenter -- or what I believe to be their datacenter -- but the outage might have just taken out routers.
01:08<vyanky>We are aware of connectivity issues affecting Linodes in our Fremont data center and are currently investigating. We will continue to provide additional updates as this incident develops.
01:08-!-Rboreal_Frippery [~gV$e9dV5D@69.136.102.83] has quit [Ping timeout: 480 seconds]
01:08<kenyon_>reminds me of 2015... https://status.linode.com/incidents/2rm9ty3q8h3x
01:08<jasonlshelton>why is there no backup generator at a big data center?
01:09<Q3Man>heck.. reminds me of a few months ago
01:09<hanksims>Yes indeed.
01:09-!-TEAMGEN [~oftc-webi@192.34.78.244] has joined #linode
01:09-!-TEAMGEN is "OFTC WebIRC Client" on #linode
01:09<hanksims>They do have backup generators. Sometimes things go wrong.
01:09<archon810m>These big dumb backup generators are always failing
01:09<TEAMGEN>tf happened
01:09<jasonlshelton>hanksims: clearly
01:09<greggypoo>i would imagine there is a back up generator but it is having a bad day
01:09<dwfreed>hanksims: your PG&E map does include where fmt2 is located, which is where Linode's Fremont facility is
01:09<Peng_>TEAMGEN: Don't know yet.
01:10<Peng_>Aren't there six backup generators
01:10-!-Gary [~oftc-webi@119.93.9.133] has joined #linode
01:10-!-Gary is "OFTC WebIRC Client" on #linode
01:10<hanksims>https://imgur.com/a/4djMRml
01:10<Hydroxide1>TEAMGEN: if your linode is in Fremont, CA, USA, keep an eye on https://status.linode.com/incidents/2hd9pdyl81p5 for official updates, and here for random blather/chatter. they're working on the outage
01:10<Peng_>Also, sometimes a UPS or ATS goes
01:10<kenyon_>there were apparently 8 generators according to that 2015 outage report
01:10<hanksims>dwfreed: Well, there ya go. The time was about right, too.
01:10<Peng_>kenyon_: Ah. Thanks. :D
01:10<Q3Man>.. but only 4 lights!
01:11<enc0de>8 generators, but how many of them were plugged in and fueled up?
01:11<levlaz>kenyon_: that was a good night -- I was on call at Linode back then. Didn't sleep for two days :)
01:11<Peng_>enc0de: All of them probably
01:11<dwfreed>hopefully
01:11<jasonlshelton>so, what you are all saying is that right when there was a power outage, all of the generators and backup batteries failed all at once......now there's a coincidence
01:11<dwfreed>they didn't all fail
01:11-!-Gary [~oftc-webi@119.93.9.133] has quit []
01:12<Q3Man>backup batteries only last 10-15 minutes under the best of conditions
01:12<dwfreed>all of HE's fmt2 core routers are still responding
01:12-!-gary [~oftc-webi@119.93.9.133] has joined #linode
01:12-!-gary is "OFTC WebIRC Client" on #linode
01:12<hanksims>jasonlshelton: They're usually very transparent with their postmortems. So we'll see.
01:12<kenyon_>clock.fmt.he.net still respons
01:12<Peng_>kenyon_: That's in fmt1 :D
01:12<dwfreed>that's in fmt1
01:12<dwfreed>heh
01:12-!-gary [~oftc-webi@119.93.9.133] has quit []
01:13<jasonlshelton>hanksims: yeah, i'm just kinda being a punk, I know they are working hard
01:13<Q3Man>send the intern down to IRC to let us know what's going on :)
01:13<kenyon_>tserv3.fmt2.ipv6.he.net responds :P
01:13<kenyon_>only on legacy IP though
01:13<Peng_>D:
01:14<nyancat>https://www.irccloud.com/pastebin/YTiIVuh3/
01:14<nyancat>@ dwfreed this any better?
01:14-!-Guest3826 is now known as jrhunt
01:14<TEAMGEN>nyancat: thats beautiful
01:15<dwfreed>kenyon_: you mean tserv1?
01:15<dwfreed>huh
01:15<nyancat>the last two sentences do not make any sence
01:15<nyancat>*sense
01:15<kenyon_>same for tserv1
01:15<vyanky>when Fremont data center will start again?
01:16<dwfreed>when it's fixed
01:16<dwfreed>nyancat: line 2 and 4 still do not rhyme
01:16-!-dyous87 [~oftc-webi@pool-141-155-32-186.nycmny.fios.verizon.net] has joined #linode
01:16-!-dyous87 is "OFTC WebIRC Client" on #linode
01:17<dyous87>hello all. are there any linode reps here who could give me an eta on the fremont datacenter outage?
01:17<kenyon_>finally my ssh sessions are starting to time out
01:18<dwfreed>nyancat: also line 4, 5, and 6 all have to rhyme
01:18<dwfreed>dyous87: when it's fixed
01:18<jme>i'd say the outage has already arrived
01:18<circlenet>I have some other equipment in a cage at HE Freemont it's working
01:18<dwfreed>jme: hah
01:19<dwfreed>circlenet: fremont2?
01:19<circlenet>but the provider *might* have extra batterys
01:20<dyous87>my servers in freemont have been down ~45 minutes now. i know there is an outage, does anyone have any additional info on the situation?
01:20<TEAMGEN>fugg
01:20<TEAMGEN>should have set up a failover
01:20<Hydroxide1>dyous87: only what's at the status link
01:20<dwfreed>dyous87: we only have theories; current theory is power outage
01:20<Affliction>dyous87: There was a rumor of a power failure, no idea if the linodes themselves are affected.
01:21<circlenet>Yeah atleast some things are running at fm2
01:23<dwfreed>STATUS: PG&E is assessing the cause at the outage location.
01:23<dwfreed>LAST UPDATED: JUN 21, 1:14 AM
01:23<dwfreed>from PG&E's outage map
01:24<dwfreed>Linode posted this to the status page about 6 minutes ago: "Our team is still in communications with our upstream provider to determine the cause of this outage. We are continuing our investigation and will provide additional updates as the issue develops."
01:24<Peng_>Does HE have their VoIP system in fmt2 now?
01:25<enc0de>doesn't sound like it will be fixed quickly
01:26<TEAMGEN>:|
01:26<circlenet>I can ping 216.218.252.250 which is an HE router my traffic passes through at fmt
01:26<circlenet>doesn't mean the whole building has power
01:27<circlenet>it might be on a battery and will pop off in a but
01:27<circlenet>bit
01:27-!-greggypoo [~greg@c-68-45-64-213.hsd1.nj.comcast.net] has quit [Quit: ircing from a laptop, what is the world coming to]
01:27<Peng_>Surely the batteries would be dead by now
01:28<Peng_>Unless they run the routers on a separate, overprovisioned UPS
01:28<circlenet>Could be
01:28<hanksims>Something seems to be happening with my Linodes. ssh timing out.
01:29<circlenet>BGP just came back up
01:29<Peng_>\o/
01:29<circlenet>I'm seeing the networks in my router tables now
01:29-!-fpfpfdev [~oftc-webi@121.99.130.115] has joined #linode
01:29-!-fpfpfdev is "OFTC WebIRC Client" on #linode
01:30<circlenet>Interesting I see the networks from GTT now but not from Tata. Still no access but something changed.
01:31<levlaz>dwfreed: I was in the middle of packaging up that mysql-utilities thing on my linode when the outage happened. I hope the disk is still ok because naturally I didn't commit any of my changes. :)
01:32-!-SleePy [~SleePy@00018c73.user.oftc.net] has joined #linode
01:32-!-SleePy is "SleePy" on #linode
01:32<dwfreed>levlaz: lol
01:32-!-atrus [~atrus@nickurak.ca] has joined #linode
01:32-!-atrus is "Jeremy Nickurak" on #linode
01:32-!-FastLizard4 [~fastlizar@ridley.fastlizard4.org] has joined #linode
01:32-!-FastLizard4 is "These violent delights have violent ends" on #linode
01:32<Peng_>Hey those are Fremont people joining
01:33<Affliction>no ping reply from mine yet
01:33<TEAMGEN>same
01:33-!-ajmitch_ [ajmitch@vps.ajmitch.net.nz] has joined #linode
01:33-!-ajmitch_ is "Andrew Mitchell" on #debian-ubuntu #debian-ipv6 #ubuntu-expats #linode #debian-cli
01:34<Peng_>Soon :D
01:34-!-vincent_ [~oftc-webi@223.93.174.130] has joined #linode
01:34-!-vincent_ is "OFTC WebIRC Client" on #linode
01:34-!-G [~njones@ruru.jnet.net.nz] has joined #linode
01:34-!-G is "Nigel Jones" on #linode #virt #oftc #moocows #powerdns @#luukland.chat
01:35-!-dru [~oftc-webi@c-73-241-89-34.hsd1.ca.comcast.net] has joined #linode
01:35-!-dru is "OFTC WebIRC Client" on #linode
01:35-!-encode [~encode@2600:3c01::f03c:91ff:fe96:d82a] has joined #linode
01:35-!-encode is "Access Denied" on #linode
01:35-!-vincent_ [~oftc-webi@223.93.174.130] has quit []
01:35-!-jr_net [~irc@2600:3c01::f03c:91ff:fee7:2259] has joined #linode
01:35-!-jr_net is "jr_net" on #linode
01:35<dru>are we still down in fremont?
01:35<archon810m>I just got an email from my server
01:36<archon810m>Woo
01:36<Peng_>Some folks just came up
01:36-!-stotch [~stotch@perrin.stotch.com] has joined #linode
01:36-!-stotch is "stotch" on #linuxfs #linux #linode #kernelnewbies #ext4
01:36<enc0de>yeah i can get to my linode now
01:36<enc0de>\o/
01:36<fpfpfdev>not mine
01:36<hanksims>Some but not all of my sites up
01:36<dru>was it power outage or network outage or both?
01:36<Affliction>enc0de: was it rebooted?
01:36<levlaz>dru: power outage per status page
01:37<dru>I can ping my nodebalancer but not my nodes
01:37<archon810m>Woo didn't reboot
01:37<Peng_>\o/
01:37<archon810m>2 out of 4 db servers are up
01:38<archon810m>And with my luck, 0 of 5 web boxes
01:38<enc0de>nope my linode wasn't rebooted
01:38<Peng_>\o/
01:38<archon810m>And it's not a master db either
01:38<archon810m>I fear the other 7 were rebooted
01:38<archon810m>Happened this way last outage
01:38<TEAMGEN>rip
01:39<Peng_>/o\
01:39<dru>yikes, it took an hour to figure out network was down due to power outage?
01:39<Peng_>No
01:40<Peng_>Well
01:40<archon810m>Linode manager still says all 9 servers are up
01:40<archon810m>There's hope
01:40<circlenet>Linode DNS server are back up
01:40<Peng_>Everyone was busy
01:40<circlenet>at FM2
01:40-!-wpidea [~oftc-webi@122.243.143.140] has joined #linode
01:40-!-wpidea is "OFTC WebIRC Client" on #linode
01:40<jme>it was dark
01:40<levlaz>They were probably waiting for HE to respond as well. They only have remote hands.
01:40<dwfreed>archon810m: if the host is not up, the status is not updated
01:41<archon810m>:-/
01:42<wpidea>Hi guys, my VPS are down, are yours running?
01:42<TEAMGEN>no
01:42<Peng_>!status
01:43<linbot>Peng_: I am connected to OFTC as linbot.
01:43<Peng_>oops
01:43<Peng_>!linstatus
01:43<linbot>Think something is wrong? Check out https://status.linode.com
01:43<dwfreed>Peng_: I'd fix that, but I don't have the access :/
01:43<wpidea>oh,god, Fremont again?
01:43<dwfreed>it only happens once every 7 years
01:44<dwfreed>or months
01:44<dyous87>1 of my db servers is back up, still waiting on my app server
01:44<Peng_>dyous87: What's the uptime?
01:44<dwfreed>fwiw, neither the speedtest server nor the lish gateway are accessible at the moment
01:44-!-Zachary [~oftc-webi@2600:1700:1150:96aa:6571:71b4:7cfe:7765] has quit [Remote host closed the connection]
01:45<Peng_>Oh dear
01:45<dyous87>@Peng_ 33 days uptime, wasn't rebooted. service returned to it about 7 minutes ago
01:45<Peng_>Great :-)
01:46-!-Val_ [~oftc-webi@c-73-162-255-242.hsd1.ca.comcast.net] has joined #linode
01:46-!-Val_ is "OFTC WebIRC Client" on #linode
01:46<Hydroxide1>mine is back now - thanks for working on recovery! good luck with cleanup, postmortem, and remediation. 'night all
01:46-!-Hydroxide1 [~oftc-webi@bas3-montreal28-70-53-157-223.dsl.bell.ca] has quit []
01:46<dwfreed>also OFTC's ircd host running in Fremont is still down
01:46<BlkStormy>1/5 are back up for me
01:47<archon810m>No changes for me. 2/9 up
01:47<jrhunt>looks like pg&e cleared the outage from their map
01:48<dwfreed>jrhunt: there's still an outage, it's just much smaller, and HE is no longer in the affected area
01:49<dwfreed>the nice thing is that PG&E's outage map is browser-timezone-aware
01:49<dwfreed>so I don't have to remember to add 3 hours
01:49<jrhunt>ah, there it is. their refresh apparently doesn't always redraw properly. whatevs.
01:50<jrhunt>it sounds like kvm hosts are being powered back on?
01:50-!-eagles051387 [~Thunderbi@81.171.58.197] has quit [Quit: eagles051387]
01:50<archon810m>Someone seeing fresh boots?
01:50<Affliction>most people have reported no reboot
01:50<Peng_>Has *anyone* reported a reboot?
01:50<Affliction>Mine hasn't come back yet
01:50<Peng_>Three or four people have said no reboots and nobody else has specified.
01:50<levlaz>im on fremont964 -- nothing yet
01:50<Peng_>I'm not optimistic, but I don't think it's been confirmed yet
01:53<BlkStormy>The 1/5 linodes that have come back for me did not restart
01:54<archon810m>Do other locations have this many serious issues?
01:54<archon810m>I don't really follow their uptimes
01:54<jrhunt>is fremont a particularly problematic datacenter?
01:54<TEAMGEN>seems like it
01:54<jrhunt>i've been in ATL forever and don't recall having lots of issues
01:54<Peng_>Not really.
01:55<Peng_>jrhunt: HURRICANE IRMA
01:55<jrhunt>it could just be that i have a client with monitoring dialed in quite tight on this particular setup
01:55<Peng_>These kinds of incidents are so rare that it's hard to say, without detailed information, which data centers are "better" and which are luckier.
01:56<Peng_>but Atlanta is terrible
01:56<Peng_>:D
01:56<jrhunt>again, ATL is my personal site, and i really don't care much for its availability compared to sleep and other fun things to do not on computers
01:57<Affliction>fremont used to have a reputation
01:57<Peng_>They literally moved Fremont to a new building. Less of a reputation since.
01:58<Affliction>Personally, I haven't noticed it being particularly awful since ~2011
01:58-!-zoltan_ [~oftc-webi@2600:1700:1150:96aa:e5ed:6cf5:7f6c:9275] has joined #linode
01:58-!-zoltan_ is "OFTC WebIRC Client" on #linode
01:58<Affliction>And 2011 was before the move.
01:58<Affliction>Perhaps I just have low expectations!
01:58<dwfreed>shit happens
01:59<dwfreed>it's when shit happens too often that it's time to move
01:59<Affliction>Pretty much
01:59<archon810m>Fremont is getting it all over again
01:59<dwfreed>I don't think "too often" is anywhere near being reached with fmt2
02:01<jrhunt>we got bit back in feb by a generator autostart failure
02:01<jrhunt>that was fun
02:02<Peng_>Who's we?
02:02<Peng_>like Fremont?
02:02<dwfreed>fmt2
02:02<Peng_>Oh
02:02<Peng_>I don't remember that :P
02:02-!-jasonlshelton [~oftc-webi@ip72-199-249-163.sd.sd.cox.net] has quit [Quit: Page closed]
02:02<dwfreed>i do
02:02-!-zoltan_ [~oftc-webi@2600:1700:1150:96aa:e5ed:6cf5:7f6c:9275] has quit [Remote host closed the connection]
02:02<Affliction>Woo
02:03<Affliction>1738 packets transmitted, 5 received, 99% packet loss, time 1778153ms
02:03<dwfreed>the generator's starter battery was dead, iirc
02:03<Affliction>hm, port 22, connection refused. Think I got rebooted
02:03<dwfreed>likely
02:03<jrhunt>we = my instances in fremont
02:03-!-dru [~oftc-webi@c-73-241-89-34.hsd1.ca.comcast.net] has quit [Quit: Page closed]
02:04<Peng_>Lish is up over IPv4
02:04<Affliction>Yeah, linode manager says I got rebooted.
02:04<dwfreed>yep, just came back
02:05<dwfreed>speedtest is still down and charm is still down
02:07<BlkStormy>2/5 linodes up for me now
02:07-!-BlkStormy [~oftc-webi@74.82.132.99] has quit [Quit: Page closed]
02:07-!-BlkStormy [~oftc-webi@74.82.132.99] has joined #linode
02:07-!-BlkStormy is "OFTC WebIRC Client" on #linode
02:07<dwfreed>"Lassie initiated boot: My Debian 3.0r1 (Small) Profile" oh man this Linode is old
02:07<Affliction>heh
02:09<fpfpfdev>0/2 :(
02:09-!-kenyon_ [~oftc-webi@2605:e000:1c0d:236:222:4dff:fe83:cd4a] has quit [Quit: Page closed]
02:09-!-Val_ [~oftc-webi@c-73-162-255-242.hsd1.ca.comcast.net] has quit [Remote host closed the connection]
02:10<jrhunt>do linode power-on ops count as "migration" operations from the point-of-view of the linode manager interface?
02:10<dwfreed>no
02:10<archon810m>A migration would involve moving your data
02:11<jrhunt>interesting
02:11<wpidea>I can reach my vps via putty, but website is still not accessible
02:11<dwfreed>because storage is generally locally attached, Linodes are fixed to a specific host
02:11<jrhunt>if i click the "free upgrade avail!" it says there's 215 migrations ahead of me in queue
02:11<jrhunt>seems like an odd time for 200+ people to decide to upgrade :/
02:11-!-nyanev [~oftc-webi@149.62.205.146] has joined #linode
02:11-!-nyanev is "OFTC WebIRC Client" on #linode
02:11<dwfreed>wpidea: is your Linode's webserver running
02:11<hanksims>They took away the WE ARE AWARE OF THE PROBLEM ON THIS LINODE message on my manager page, but one of my machines still ain't up
02:12<Affliction>I guess those people didn't know about the upgrade and only just now logged in?
02:12<Peng_>They might be scheduling migrations for people on iffy hardware?
02:12-!-json [~oftc-webi@110.54.146.244] has joined #linode
02:12-!-json is "OFTC WebIRC Client" on #linode
02:12<Peng_>E.g. if a server came back up but lost a drive
02:12<dwfreed>jrhunt: doesn't for me; people log into the manager to see why their linode is down and see there's a free upgrade
02:12<json>re fremont.. data center operator issues?
02:12<dwfreed>also what Peng_ said
02:12<dwfreed>json: power outage
02:13<nyanev>@linode this is lame, Our servers are down for 2 hours
02:13-!-valentic [~oftc-webi@c-73-162-255-242.hsd1.ca.comcast.net] has quit [Remote host closed the connection]
02:13<dwfreed>nyanev: shit happens
02:13<Peng_>No one's happy about it, but you'll probably be up soon
02:13<Affliction>nyanev: they slowly seem to be coming back, mine just did.
02:13<dwfreed>Peng_: i'm happy about it
02:14<Peng_>dwfreed: I'm glad everyone is running new kernels now, but this was a little extreme
02:14<nyanev>Peng_: I hope so
02:14<dwfreed>Peng_: rofl
02:14<Affliction>Yeah, there is that little element of excitement on an outage. "Will I have to restore from a backup!"
02:14<TEAMGEN>what backup?
02:15<hanksims>Aw damn, I guess they rebooted me.
02:15-!-Q4Man [~oftc-webi@ip68-108-160-170.lv.lv.cox.net] has joined #linode
02:15-!-Q4Man is "OFTC WebIRC Client" on #linode
02:15<Affliction>Yeah, there's a nice hole in my munin graphs
02:16<TEAMGEN>should i do the free upgrade right now?
02:16<TEAMGEN>jk
02:16<hanksims>They've rebooted my Linode. Praying they didn't stick me in the goddamn free upgrade queue
02:17<archon810m>I'm so looking forward to glusterfs eating up all the CPU for the next 4 hours after boot
02:17<wpidea>my websites are back now
02:18<TEAMGEN>awesome
02:18<json>mine went up.. then went out again..
02:18<dwfreed>charm and speedtest are still down :(
02:18<TEAMGEN>still no response from mine
02:19<nyanev>frfrom mine too
02:19<json>misery loves company.. @linode =)
02:20-!-valenitc [~oftc-webi@c-73-162-255-242.hsd1.ca.comcast.net] has joined #linode
02:20-!-valenitc is "OFTC WebIRC Client" on #linode
02:21<dwfreed>woo, charm has a boot job now
02:21-!-MJCS [~script2@li1001-24.members.linode.com] has joined #linode
02:21-!-MJCS is "Moo" on #linode
02:21-!-wpidea [~oftc-webi@122.243.143.140] has quit [Quit: Page closed]
02:22<archon810m>How have none of my 7 booted yet
02:22-!-yhager [~yuval@d50-92-255-81.bchsia.telus.net] has joined #linode
02:22-!-yhager is "Yuval Hager" on #linode
02:24-!-mattmcc [~oftc-webi@50-76-106-26-static.hfc.comcastbusiness.net] has joined #linode
02:24-!-mattmcc is "OFTC WebIRC Client" on #linode
02:24<dwfreed>and charm is back
02:25-!-arielp [~arielp@0001aa68.user.oftc.net] has joined #linode
02:25-!-arielp is "arielp" on #linode
02:25<archon810m>1 just came up
02:26-!-Guest5958 is now known as CornishPasty
02:28-!-jimmy-test [~oftc-webi@61-69-98-198.mel.static-ipl.aapt.com.au] has quit [Quit: Page closed]
02:29-!-Q4Man [~oftc-webi@ip68-108-160-170.lv.lv.cox.net] has quit [Remote host closed the connection]
02:30-!-kenyon [kenyon@darwin.kenyonralph.com] has joined #linode
02:30-!-kenyon is "Kenyon Ralph" on #ipv6 #munin #debian-ipv6 #linode #debian
02:31-!-mjevans [~mjevans@li984-246.members.linode.com] has joined #linode
02:31-!-mjevans is "mjevans" on #ceph #linode
02:32-!-lilytrade [~oftc-webi@159.89.210.33] has joined #linode
02:32-!-lilytrade is "OFTC WebIRC Client" on #linode
02:32<lilytrade>hi
02:33<TEAMGEN>hello
02:33<lilytrade>my server down
02:33-!-Q3Man [~oftc-webi@ip68-108-160-170.lv.lv.cox.net] has quit [Remote host closed the connection]
02:33<TEAMGEN>yeah
02:33<TEAMGEN>outage
02:33<TEAMGEN>1 server just came back for me
02:33<lilytrade>when come back
02:33<TEAMGEN>!linstatus
02:33<linbot>Think something is wrong? Check out https://status.linode.com
02:33<Affliction>Power outage at fremont, they seem to slowly be coming back up
02:33<TEAMGEN>sweeeet
02:33<TEAMGEN>finally one is up
02:33-!-Q4Man [~oftc-webi@ip68-108-160-170.lv.lv.cox.net] has joined #linode
02:33-!-Q4Man is "OFTC WebIRC Client" on #linode
02:33<yhager>dashboard says server is up, but lish says it is down..
02:33<lilytrade>why so slow
02:34<Q4Man>"There are 301 Linode migrations queued in front of you"
02:34<TEAMGEN>lel
02:34<dwfreed>because there were probably a lot of hosts that went down and need to be checked that they're okay
02:34<lilytrade>It affect a lot of owers there
02:34<lilytrade>orders
02:36<dwfreed>yhager: that can happen; I probably can't say why without getting in trouble
02:36<dwfreed>yhager: it'll be updated soon
02:36<lilytrade>it take so long time
02:36<lilytrade>my 3 site all affected
02:36<yhager>dwfreed: thanks. don't get in trouble for me.. I guess I'll wait for things to stabilize
02:37<jme>ya gonna let a little trouble scare ya? ;-
02:37<dwfreed>jme: it's a first rule of fight club kind of thing
02:37<jme>rules? what are those?
02:38<Peng_>the first rule of Fight Club is the GRU publishes Fight Club's listserv on WikiLeaks
02:38<dwfreed>in this case it's protected by a legally binding agreement
02:38<fpfpfdev>comon already been down for a while now
02:38-!-mlabs [~oftc-webi@2001:8000:10de:9a00:7817:6d70:e291:528c] has quit [Quit: Page closed]
02:38-!-EG [~oftc-webi@106.0.59.164] has joined #linode
02:38-!-EG is "OFTC WebIRC Client" on #linode
02:39-!-circuitdigest [~oftc-webi@117.204.242.51] has joined #linode
02:39-!-circuitdigest is "OFTC WebIRC Client" on #linode
02:39<jme>i see - like the stormy daniels thing ;-
02:39<dwfreed>yeah, but I didn't get any hush money :(
02:39-!-circuitdigest [~oftc-webi@117.204.242.51] has quit [Remote host closed the connection]
02:39<jme>or hopefully the other part
02:40<dwfreed>lol
02:42<lilytrade>fuck slow
02:44-!-lilytrade [~oftc-webi@159.89.210.33] has quit [Quit: Page closed]
02:44<TEAMGEN>rage quit
02:46-!-Kassandry [~Kassandry@kassandry.net] has joined #linode
02:46-!-Kassandry is "Kassandry" on #ovirt #linode
02:46<dwfreed>speedtest is up
02:47-!-serverDown [~oftc-webi@203.192.202.130] has joined #linode
02:47-!-serverDown is "OFTC WebIRC Client" on #linode
02:47-!-circlenet [~oftc-webi@23-25-121-89-static.hfc.comcastbusiness.net] has quit [Quit: Page closed]
02:47<vyanky>any update on Fremont data center?
02:47<serverDown>Server id down ?
02:47<dwfreed>vyanky: things are coming back up
02:47<vyanky>ok thanks for update
02:47<dwfreed>serverDown: https://status.linode.com/
02:47<TEAMGEN>waiting on one more
02:48<fpfpfdev>1/2up
02:48<fpfpfdev>\o\
02:48<TEAMGEN>both mine are up now
02:49<TEAMGEN>gl guys
02:49<TEAMGEN>have a good night
02:49-!-TEAMGEN [~oftc-webi@192.34.78.244] has quit [Quit: Page closed]
02:49<BlkStormy>3/5
02:49<nyanev>0/3 :(
02:50<jrhunt>1/7 up, whee!
02:50-!-ckuehl [~ckuehl@2604:5500:c2c7:0:4ecc:6aff:fe8c:d243] has joined #linode
02:50-!-ckuehl is "Chris Kuehl" on #linode #debian-printing #debian-openstack #debian-gnome #debian-boot
02:51-!-serverDown [~oftc-webi@203.192.202.130] has quit [Remote host closed the connection]
02:51-!-nate [~NBishop@c-73-10-154-113.hsd1.nj.comcast.net] has joined #linode
02:51-!-nate is "Nathan" on #linode #php
02:53<FastLizard4>1/3 here, but at least the one that's up is my important one :P
02:53-!-purrdeta [sid201815@2001:67c:2f08:4::3:1457] has quit [Ping timeout: 480 seconds]
02:53-!-schwa [~laptopdud@pool-108-24-44-55.cmdnnj.fios.verizon.net] has quit [Ping timeout: 480 seconds]
02:54-!-Q4Man [~oftc-webi@ip68-108-160-170.lv.lv.cox.net] has quit [Remote host closed the connection]
02:55-!-igufi [sid205746@0001a331.user.oftc.net] has quit [Read error: Connection reset by peer]
02:57-!-retro|blah [retrograde@000196da.user.oftc.net] has joined #linode
02:57-!-retro|blah is "retrograde inversion" on #linode
02:57<Archon810>4/9 up
02:57<Archon810>5/9 up
02:57<Archon810>6/9 now
02:57-!-Eugene [~eugene@jack.kashpureff.org] has joined #linode
02:57-!-Eugene is "Eugene E. Kashpureff Jr" on #linode
02:58<nate>unfortunately only got one at fremont and it's still down
02:58-!-purrdeta [~sid201815@id-201815.tooting.irccloud.com] has joined #linode
02:58-!-purrdeta is "Alex" on #linode #Corsair
02:58-!-igufi [sid205746@0001a331.user.oftc.net] has joined #linode
02:58-!-igufi is "igufi" on #linode
02:58*jrhunt so jealous of Archon810 right now
02:58-!-synfinatic [~synfin@mail.synfin.net] has joined #linode
02:58-!-synfinatic is "Just Me" on #tor #seul #linode
02:58-!-levlaz [~oftc-webi@136.24.36.82] has quit [Remote host closed the connection]
02:59<fpfpfdev>2/2
02:59-!-jme [~oftc-webi@cpe-67-248-50-106.nycap.res.rr.com] has quit [Quit: Page closed]
03:00-!-JE66 [~oftc-webi@cpe-104-32-155-134.socal.res.rr.com] has joined #linode
03:00-!-JE66 is "OFTC WebIRC Client" on #linode
03:00<BlkStormy>4/5
03:01-!-TecnoBrat [~tecnobrat@173.255.254.92] has joined #linode
03:01-!-TecnoBrat is "Brian" on #linode
03:01<jrhunt>2/7
03:01<Archon810>8/9
03:01<Archon810>replication broke on all slaves
03:01<Archon810>damnit
03:02-!-nyanev [~oftc-webi@149.62.205.146] has quit [Quit: Page closed]
03:09-!-endurant [sid181723@192.184.10.9] has quit [Ping timeout: 480 seconds]
03:11<yhager>mine is up too
03:12-!-endurant [sid181723@tooting.irccloud.com] has joined #linode
03:12-!-endurant is "geo" on #linode @#hello
03:12<nate>anyone want to trade me their up linode? I'd love to get back to sleep already lol
03:12<json>0/2
03:14<FastLizard4>2/3!
03:15<json>1/2
03:15<json>2/2.. ciao!
03:16-!-json [~oftc-webi@110.54.146.244] has quit [Quit: Page closed]
03:16-!-valenitc [~oftc-webi@c-73-162-255-242.hsd1.ca.comcast.net] has quit [Remote host closed the connection]
03:17<Archon810>9/9
03:18<Archon810>it's been real, y'all
03:18<Archon810>time to fix replication and sleep
03:18-!-Archon810 [~oftc-webi@c-69-181-99-27.hsd1.ca.comcast.net] has quit [Quit: Page closed]
03:18<FastLizard4>yay 3/3
03:18<Peng_>:D
03:19-!-Guest5942 [~oftc-webi@175.158.49.119] has quit [Quit: Page closed]
03:20<BlkStormy>Just 1 more and I can go back to bed...
03:22-!-devilspgd [znc@wtf.thedave.ca] has joined #linode
03:22-!-devilspgd is "Dave" on #tor #linode #https-everywhere
03:25-!-BlkStormy [~oftc-webi@74.82.132.99] has quit [Quit: Page closed]
03:25-!-mattmcc_ [~matt@li516-48.members.linode.com] has joined #linode
03:25-!-mattmcc_ is "Matt McClanahan" on #linode
03:27-!-mattmcc [~oftc-webi@50-76-106-26-static.hfc.comcastbusiness.net] has quit [Quit: Page closed]
03:27-!-mattmcc_ is now known as mattmcc
03:27-!-schwa [~laptopdud@pool-108-24-113-231.cmdnnj.fios.verizon.net] has joined #linode
03:27-!-schwa is "purple" on #linode
03:28-!-JE66 [~oftc-webi@cpe-104-32-155-134.socal.res.rr.com] has quit [Remote host closed the connection]
03:28-!-fpfpfdev [~oftc-webi@121.99.130.115] has quit [Quit: Page closed]
03:34-!-nate [~NBishop@00013625.user.oftc.net] has quit [Remote host closed the connection]
03:35-!-nate [~NBishop@c-73-10-154-113.hsd1.nj.comcast.net] has joined #linode
03:35-!-nate is "Nathan" on #linode #php
03:39-!-arielp [~arielp@0001aa68.user.oftc.net] has quit [Quit: WeeChat 2.1]
03:40<darwin>if anyone runs Slackware on a linode with IPv6, do you recall if it's usually best to leave /etc/udev/rules.d/70-persistent-net.rules or stop IPv6, delete that file, restart? Linode staff had me do that at one time, which restored my IPv6 connection, but later, it may have been best to leave that file...
03:42-!-w3bcrawler [uid63635@2001:67c:2f08:1::f893] has quit [Quit: Connection closed for inactivity]
03:44<darwin>my linode is at Fremont and is back up, but now I have to figure out how to restore that connectivity
03:44<darwin>on every reboot, IPv6 is down
03:46<grawity>when does it come back up?
03:47-!-nate [~NBishop@00013625.user.oftc.net] has quit [Remote host closed the connection]
03:50<dwfreed>darwin: pastebin the following: 'ip addr show' 'sysctl -a | grep ipv6'
03:50<darwin>ok
03:51-!-frail_ [frail_@2600:3c01:e001:3930::3] has joined #linode
03:51-!-frail_ is "frail_" on #linode
03:54<darwin>it's a lot of output: https://pastebin.com/mpyc4LCY
03:57<dwfreed>max_addresses
03:58<dwfreed>you've statically configured a lot of addresses, so the kernel skips doing SLAAC
03:58<darwin>I'm not sure what all that means
03:58<darwin>there's no automatic IPv6 on Slackware
03:59<grawity>there is
03:59<grawity>unless slackware deliberately remove it from their kernels
03:59<grawity>which is... fairly unlikely imho
03:59<darwin>well not more than one address
03:59<dwfreed>I'm assuming slackware has an /etc/sysctl.conf ? if so, put 'net.ipv6.conf.eth0.max_addresses=32' in it, and do 'sysctl net.ipv6.conf.eth0.max_addresses=32' as a command as well
04:00<darwin>yes, it has that
04:00<darwin>wait, no it doesn't
04:00<darwin>I saw something that looked similar
04:01<dwfreed>do you have any initscripts that call 'sysctl --system' or similar?
04:02<dwfreed>relatedly, if you run 'sysctl --system' does it work? (work here means no output)
04:02<darwin>yes, and some check for the file you mentioned
04:02<darwin>so I guess I could make it with just that
04:03<darwin>I added that and did the command
04:04-!-rsdehart [~rob@eightpence.com] has joined #linode
04:04-!-rsdehart is ""St. Asaph the Caffeinated, POEE"" on #linode
04:05-!-vyanky [~oftc-webi@43.243.80.163] has quit [Remote host closed the connection]
04:07-!-rsdehart [~rob@eightpence.com] has quit [Quit: Lost terminal]
04:07<darwin>I think it's working now! :)
04:08<dwfreed>yep, responds here
04:09<darwin>thanks for explaining that; I'll look up max_addresses
04:10-!-transfervans [~oftc-webi@122-62-136-119-fibre.bb.spark.co.nz] has quit [Remote host closed the connection]
04:10-!-archon810m [~androirc@c-69-181-99-27.hsd1.ca.comcast.net] has quit [Quit: AndroIRC - Android IRC Client ( http://www.androirc.com )]
04:13-!-underscorenorth [~oftc-webi@198-91-150-195.cpe.distributel.net] has joined #linode
04:13-!-underscorenorth is "OFTC WebIRC Client" on #linode
04:14<underscorenorth>Good morning.
04:14-!-yhager [~yuval@d50-92-255-81.bchsia.telus.net] has quit [Ping timeout: 480 seconds]
04:14<Peng_>Hi
04:15<underscorenorth>That was pretty wild haha
04:17<linbot>New news from community: [ Fremont Power Outage ]ⁿ <https://www.linode.com/community/questions/16954>
04:17<grawity>darwin: tbh the address can be configured statically as well, not just via SLAAC
04:17<grawity>darwin: but it still needs to be present in order for your /64 to work
04:18<dwfreed>^
04:18-!-Phil [~oftc-webi@196.44.191.3] has joined #linode
04:18-!-Phil is "OFTC WebIRC Client" on #linode
04:18-!-Phil [~oftc-webi@196.44.191.3] has quit []
04:19-!-underscorenorth [~oftc-webi@198-91-150-195.cpe.distributel.net] has quit [Quit: Page closed]
04:23-!-runf0r3st [~runf0r3st@c-76-25-24-44.hsd1.co.comcast.net] has quit [Ping timeout: 480 seconds]
04:23<Woet>everyone move your Linodes to fremont now
04:23<Woet>statistically its less likely to be affected by the net outage
04:23<Woet>next *
04:24<darwin>I don't know what the difference is. I just use an unofficial script to bring up addresses, that people wrote in the Slackware Usenet newsgroup
04:24<darwin>which I modified
04:24*dwfreed beats Woet with the troll bat
04:24<Woet>at least it's not a +q
04:24<Woet>like some other losers get
04:25<dwfreed>his mistake was trolling when customers needed real answers
04:25<Woet>his?
04:25<Woet>HIS?
04:25<dwfreed>s/his/their/
04:25<dwfreed>besides, there are no girls on the internet
04:26<Woet>what about bhanks
04:26<darwin>LoL
04:26-!-mode/#linode [-q *!*@0001cb66.user.oftc.net] by dwfreed
04:27-!-cbernard [~oftc-webi@79.97.74.182] has joined #linode
04:27-!-cbernard is "OFTC WebIRC Client" on #linode
04:29<cbernard>I just added a volume to my Linode using the Manage Volumes option at the dashboard, but does anyone know the pricing for external volumes? It's not listed on the pricing page of the site.
04:29<bmartin_>.10 a gb per month
04:30-!-rsdehart [~rob@eightpence.com] has joined #linode
04:30-!-rsdehart is ""St. Asaph the Caffeinated, POEE"" on #linode
04:31-!-runf0r3st [~runf0r3st@c-76-25-24-44.hsd1.co.comcast.net] has joined #linode
04:31-!-runf0r3st is "realname" on #ceph #ceph-devel #linode #biz #sepia #privacytools.io #moocows
04:32<cbernard>thanks
04:36<bmartin_>No problem!
04:38<dwfreed>bmartin_: nice tail
04:38<bmartin_>yeah apparently I need to fix my name or I was let go
04:38<bmartin_>either way
04:38<dwfreed>you're probably at work, so I doubt it was the latter
04:44-!-Shentino [~Shentino@68-186-69-103.dhcp.knwc.wa.charter.com] has quit [Remote host closed the connection]
04:44-!-Shentino_ [~Shentino@68-186-69-103.dhcp.knwc.wa.charter.com] has joined #linode
04:44-!-Shentino_ is "realname" on #linode
04:57-!-chris_ [~oftc-webi@173.82.5.8] has joined #linode
04:57-!-chris_ is "OFTC WebIRC Client" on #linode
04:57<chris_>Hi
04:58-!-chris_ [~oftc-webi@173.82.5.8] has left #linode []
05:00-!-dyous87 [~oftc-webi@pool-141-155-32-186.nycmny.fios.verizon.net] has quit [Remote host closed the connection]
05:16<Woet>Hi
05:26-!-peter [~oftc-webi@41.90.237.109] has joined #linode
05:26-!-peter is "OFTC WebIRC Client" on #linode
05:26-!-peter [~oftc-webi@41.90.237.109] has quit []
05:28-!-EG [~oftc-webi@106.0.59.164] has quit [Remote host closed the connection]
05:28-!-bmartin_ [~bmartin@li1749-198.members.linode.com] has left #linode [Textual IRC Client: www.textualapp.com]
05:28-!-bmartin_ [~bmartin@li1749-198.members.linode.com] has joined #linode
05:28-!-bmartin_ is "Blake Martin" on #linode #outerspace
05:32-!-StygianBlues [~StygianBl@74-194-176-205.chstcmtk03.res.dyn.suddenlink.net] has quit [Read error: Connection reset by peer]
05:32-!-StygianBlues [~StygianBl@74-194-176-205.chstcmtk03.res.dyn.suddenlink.net] has joined #linode
05:32-!-StygianBlues is "StygianBlues" on #ooni #lxde #linode #cryptoparty
05:41-!-bmartin_ is now known as bmartin
05:41-!-zhonghuali [~oftc-webi@103.106.98.115] has joined #linode
05:41-!-zhonghuali is "OFTC WebIRC Client" on #linode
05:41<bmartin>bmartin is back
05:44<zhonghuali>cannot connect to my vps probably due to wrong configuration in /etc/fstab file, is there a way to rescue?
05:44<grawity>reboot to the Finnix rescue image
05:44<Peng_>zhonghuali: How do you know it's /etc/fstab?
05:45<zhonghuali>Just guess.
05:45<dwfreed>it's best not to guess
05:46<dwfreed>log into lish and start investigating
05:46<dwfreed>!lish
05:46<linbot>LISH allows you to perform certain actions without having to log in to the Linode Manager. LISH's primary function is to allow you to access your Linode's console, even if networking is disabled. https://www.linode.com/docs/networking/using-the-linode-shell-lish
05:46<dwfreed>bmartin: /msg ChanServ op #linode
05:46-!-mode/#linode [+o bmartin] by ChanServ
05:47<dwfreed>\o/
05:47<@bmartin><3
05:48-!-darwin [d@melik.windwireless.net] has quit [Quit: Hail to the forest glade! Hail Sherwood!]
05:48<zhonghuali>I tried to reboot the rescue, but still cannot connect with glish. no terminal popup .
05:49<dwfreed>finnix doesn't use glish as far as I know
05:49<grawity>connect with the SSH one then
05:49<dwfreed>you have to use regular lish
05:55<zhonghuali>I created a new volume, and mounted it with /etc/fstab. then I removed the new volume. but forget to edit /etc/fstab. after rebooting, I cannot connect it via all ways.
05:57<zhonghuali>In the Rescue page, there still is a volume (with name 'data' in it).
06:00<zhonghuali> [ TIME ] Timed out waiting for device dev-di…csi\x2d0Linode_Volume_data.device. [DEPEND] Dependency failed for File System C…sk/by-id/scsi-0Linode_Volume_data. [DEPEND] Dependency failed for /mnt/data. [DEPEND] Dependency failed for Local File Systems. Starting ebtables ruleset management... Starting Set console font and keymap...
06:00<zhonghuali>[ TIME ] Timed out waiting for device dev-di…csi\x2d0Linode_Volume_data.device.
06:01<zhonghuali>this is the logview by lish.
06:06<Zimsky>Woet: oh damn gee whizz I feel like a downright ninny now
06:06<dwfreed>O.o
06:10<Woet>dwfreed: remember what I PMed you the other day?
06:10<Woet>dwfreed: now would be a great time to apply that
06:13<dwfreed>You have not PMed me
06:15<Woet>shit
06:19-!-Jasminder [~oftc-webi@124.253.4.100] has joined #linode
06:19-!-Jasminder is "OFTC WebIRC Client" on #linode
06:27-!-Jasminder [~oftc-webi@124.253.4.100] has quit [Quit: Page closed]
06:35-!-cbernard [~oftc-webi@79.97.74.182] has quit [Quit: Page closed]
06:38-!-Solomon [~oftc-webi@154.160.14.231] has joined #linode
06:38-!-Solomon is "OFTC WebIRC Client" on #linode
06:39-!-Solomon [~oftc-webi@154.160.14.231] has quit []
06:44-!-Affliction [affliction@2402:1f00:8101:135::] has quit [Quit: Read error: Connection reset by beer]
07:14-!-cnf [~cnf@dd47b0e01.static.telenet.be] has joined #linode
07:14-!-cnf is "Frank Rosquin" on #linode #openconnect
07:35-!-Sanju [~oftc-webi@182.75.41.202] has joined #linode
07:35-!-Sanju is "OFTC WebIRC Client" on #linode
07:37-!-Sanju [~oftc-webi@182.75.41.202] has quit []
07:39-!-Sanju [~oftc-webi@182.75.41.202] has joined #linode
07:39-!-Sanju is "OFTC WebIRC Client" on #linode
07:44-!-Sanju [~oftc-webi@182.75.41.202] has quit [Quit: Page closed]
07:56-!-Schroeder [1000@charliebrown.outwardhosting.com] has joined #linode
07:56-!-Schroeder is "Schroeder the Piano Guy" on #linode #moocows
08:06-!-troy [~troy@00012f14.user.oftc.net] has quit [Quit: WeeChat 2.1]
08:06-!-cnf [~cnf@dd47b0e01.static.telenet.be] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
08:17<linbot>New news from community: How to remove linode ipaddess from spam database <https://www.linode.com/community/questions/16955>
08:27-!-cnf [~cnf@dd47b0e01.static.telenet.be] has joined #linode
08:27-!-cnf is "Frank Rosquin" on #linode #ceph
08:33-!-DanielNM [user46904@0001518c.user.oftc.net] has joined #linode
08:33-!-DanielNM is "-=[ Galatians 2:20 ]=-" on #linode #bitlbee @#mvlug
08:33-!-Q3Man [~oftc-webi@ip68-108-160-170.lv.lv.cox.net] has joined #linode
08:33-!-Q3Man is "OFTC WebIRC Client" on #linode
08:50-!-enc0de [~nathan@ppp59-167-135-129.static.internode.on.net] has quit [Quit: leaving]
08:50-!-anomie [~anomie@00018802.user.oftc.net] has joined #linode
08:50-!-anomie is "Anomie" on #linode
08:51-!-acald3ron [~acald3ron@189.202.79.21.cable.dyn.cableonline.com.mx] has joined #linode
08:51-!-acald3ron is "Armando" on #linode #debian-next #debian-mx #debian-es #debian
08:58-!-Cristian [~oftc-webi@94.204.46.145] has joined #linode
08:58-!-Cristian is "OFTC WebIRC Client" on #linode
08:59-!-Cristian [~oftc-webi@94.204.46.145] has quit []
09:23-!-VetraNyx [~oftc-webi@213162073097.public.t-mobile.at] has joined #linode
09:23-!-VetraNyx is "OFTC WebIRC Client" on #linode
09:24<VetraNyx>hello, as i am not getting any response over mail.. what happened to my abuse report of 96.126.103.245?
09:24<VetraNyx>solved? or still in progress?
09:26<@bmartin>you send an abuse report to abuse@linode.com?
09:27<VetraNyx>sure
09:27<VetraNyx>with mail vetranyx @ aol.com
09:27<@bmartin>Where did you send this abuse report?
09:28<@bmartin>Did you send it to abuse@linode.com? If not please do with any pertinent logs and we can take a look
09:28<VetraNyx>have sent there
09:28<@bmartin>ok excellent someone will be looking into this
09:29<VetraNyx>no no... it is an old one but i never got a response if solved or not ;)
09:29<@bmartin>what was the nature of the abuse report? Usually unless it's copyright there isn't a follow up once the customer has resolved the issue.
09:30<VetraNyx>IP 96.126.103.245
09:30-!-bartwe [~bwerf@bwerf.xs4all.nl] has joined #linode
09:30-!-bartwe is "Bart van der Werf" on #linode
09:30<bartwe>o hai, trying out my ipv6 stuff
09:31<VetraNyx>last answer from jun 11
09:31<VetraNyx>Hello, We have yet to hear back from our customer. Once we have more information, we will update you
09:31<@bmartin>Can you follow up via abuse@linode.com they are going to have access to more details than I will
09:31<bartwe>the ipv6 address listed on the publics ip section can be reached from linodes in the same datacenter
09:31<bartwe>but not from linodes in another datacenter
09:31<bartwe>and not from the public internet
09:32<VetraNyx>i tried - but i guess my mails hit spam
09:32<bartwe>maybe i'm missing a detail on how ipc6 is supposed to work
09:32<bartwe>c/v
09:32<Peng_>bartwe: It's supposed to already Just Work
09:33<VetraNyx>@bmartin sent a new one now
09:33<bartwe>Peng_ that is what i was thinking, but its not ;)
09:33<Peng_>bartwe: Can you pastebin "ip -6 a" and "ip -6 r"?
09:33<@bmartin>VetraNyx I have someone looking into this now on my end
09:34<VetraNyx>Thanks. Just curious what was the outcome, as that abuse was quite a messy one
09:34<VetraNyx>@bmartin perhaps someone can check if my mail has arrived or hit spam ;)
09:35<bartwe>Peng_:https://pastebin.com/651X0nzh
09:35<@sjacobs>VetraNyx: i can confirm we got it. we're looking into it and will have a response after we take a closer look.
09:37<Peng_>bartwe: That looks... not actually wrong
09:37<bartwe>Peng_: two linodes at the 'London, England, UK' datacenter can ping to it, others can't
09:37-!-cnf [~cnf@dd47b0e01.static.telenet.be] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
09:38<bartwe>i think the london site might have ipv6 misconfigured
09:38<VetraNyx>@sjacobs the Ip is populated again btw
09:39<VetraNyx>@sjacobs so most likely my report was denied
09:39<@bmartin>Or the customer took appropriate action and remedied the situation
09:40<Peng_>bartwe: Maybe. Have you double checked the firewall?
09:40<bartwe>i have yeah
09:41<bartwe>machine a and b can ping eachother both in london, home, c and d can't, home, c, and d can ping eachother, i am using the public ipv6 not the private/lan one
09:41<bartwe>traceroute stops at 2001:668:0:3:ffff:1:0:2ce
09:41<bartwe>but dunno what machine is that
09:42<Peng_>It just started working.
09:43<VetraNyx>@bmartin could also be. Anyway, looking forward for a reply which clears that up ;)
09:43<bartwe>... it did
09:43<Peng_>I was pinging it and boom. https://mtr-atlanta.mnrd.us/?c=924e0ca2
09:43<@bmartin>VetraNyx we are looking at it now to make sure and you should have a reply shortly
09:43<bartwe>(did a linode engineer pick up on this conversation perhaps ? if so, thanks for the fix ;)
09:44<Peng_>Technically this IRC conversation probably did pass through the London routers at some point. Maybe the router is sentient. :D
09:44<bartwe>haha
09:44<frailtyy>Don't say that.
09:44<frailtyy>I don't want the computers coming for me.
09:46<VetraNyx>@bmartin thanks :)
10:00-!-Q3Man [~oftc-webi@ip68-108-160-170.lv.lv.cox.net] has quit [Remote host closed the connection]
10:01-!-bartwe [~bwerf@bwerf.xs4all.nl] has quit []
10:06-!-hanksims [~oftc-webi@47.208.95.143] has quit [Quit: Page closed]
10:14-!-veritaseum [~oftc-webi@5.62.39.247] has joined #linode
10:14-!-veritaseum is "OFTC WebIRC Client" on #linode
10:14<veritaseum>hello
10:14<veritaseum>I need help!
10:15<Peng_>Hello
10:15<veritaseum>I need help!
10:15<Woet>I need help!
10:15<veritaseum>I lost access to my email
10:15-!-veritaseum [~oftc-webi@5.62.39.247] has quit []
10:26-!-cnf [~cnf@dd47b0e01.static.telenet.be] has joined #linode
10:26-!-cnf is "Frank Rosquin" on #linode #openconnect #ceph
10:27-!-Whiskey`Wonka [~oftc-webi@47-36-190-137.dhcp.mdfd.or.charter.com] has joined #linode
10:27-!-Whiskey`Wonka is "OFTC WebIRC Client" on #linode
10:30<Whiskey`Wonka>hello. anyone know if there was a event that caused some linodes to reboot last night around 22:28 pst ?
10:31<Whiskey`Wonka>Sorry I mean 21:25 pst
10:32<@sjacobs>Whiskey`Wonka: https://status.linode.com/incidents/2hd9pdyl81p5
10:32-!-acald3ron [~acald3ron@189.202.79.21.cable.dyn.cableonline.com.mx] has quit [Remote host closed the connection]
10:34<Whiskey`Wonka>So linode doesnt run redundant power? thats.... oh my
10:35<Woet>Whiskey`Wonka: Linode doesn't run the DC, and I'm sure the DC does.
10:35<Woet>Whiskey`Wonka: apparently it failed for whatever reason
10:35<Peng_>Not everything went wrong
10:36<@bmartin>Yes, we do. However, even with these in place, there was still a partial power outage and our teams are working with the data center to determine the "why?" behind that. This information isn't available to us at the moment, and for that we apologize, but as soon as we do have more details to share, a postmortem will be made available on our Status page.
10:36<Peng_>er
10:36<Peng_>Not everything went down*
10:36<Woet>wait, Linode uses the HE datacenter?
10:36<Peng_>fmt2
10:37<Whiskey`Wonka>bmartin: thanks for the info.
10:37<@bmartin>No problem!
10:37-!-cnf [~cnf@dd47b0e01.static.telenet.be] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
10:37<Woet>HE is not known for their.. outstanding quality, are they?
10:37<Woet>or did things change in the past few years
10:38<Woet>weren't they like the ultra budget guys, undercutting even cogent?
10:38<Whiskey`Wonka>Over all, I was just worried that I didnt have a ticket for a scheduled reboot and didnt see anything related, and i keep forgeting the status page
10:38<Whiskey`Wonka>Woet: for IP services HE has been far more reliable in most markets for me then c*
10:38<@bmartin>We also could have communicated better. We appreciate your patience and also your understanding
10:38<Woet>interesting
10:38<Whiskey`Wonka>cheap as hell sure
10:39<Whiskey`Wonka>bmartin: in fairness linodes hardware level failures have been darn rare (for me)
10:39<@bmartin><3
10:45-!-cps [~cps@c-73-133-107-201.hsd1.md.comcast.net] has quit [Ping timeout: 480 seconds]
10:45<Whiskey`Wonka>well at least this DC is still standing. the last DC event I remember right now was the power room failure in houston
10:46<Peng_>Thankfully Linode was 200 miles from that
10:46<Peng_>though using the same provider!
10:48<Whiskey`Wonka>I wasnt using linode much back then
10:50<Whiskey`Wonka>that particular even took down both my email servers, cause I didnt know that my backup server (hosted to a partner compay) /was in the same damn DC/
10:50<Whiskey`Wonka>company*
10:51<Peng_>Oh no :(
10:52<Whiskey`Wonka>Oh Yes <coolaid man enters power room>
10:52<csnxs>heh
10:53<Whiskey`Wonka>I learned real fast to not trust IT people about where crap is hosted and to get proof.
10:58<Whiskey`Wonka>welp, thanks for your time and info. time to go assemble a tower and upset the neighbors. Something about playing Sabaton loud enough to hear it over the welding grinding ect, they just dont like.
11:00<linbot>New news from status: Scheduled Network Maintenance - Frankfurt <https://status.linode.com/incidents/y4cy2dhxs20b>
11:17-!-cps [~cps@c-73-133-107-201.hsd1.md.comcast.net] has joined #linode
11:17-!-cps is "Chris Smolinski" on #linode
11:24-!-james [~oftc-webi@240e:e8:f0fc:f78a:f560:e502:5b03:325c] has joined #linode
11:24-!-james is "OFTC WebIRC Client" on #linode
11:25-!-james is now known as Guest5991
11:25-!-Guest5991 [~oftc-webi@240e:e8:f0fc:f78a:f560:e502:5b03:325c] has quit []
11:36-!-acald3ron [~acald3ron@189.202.79.21.cable.dyn.cableonline.com.mx] has joined #linode
11:36-!-acald3ron is "Armando" on #linode #debian-next #debian-mx #debian-es #debian
11:47-!-cps [~cps@c-73-133-107-201.hsd1.md.comcast.net] has quit [Quit: cps]
11:57-!-Rboreal_Frippery [~gV$e9dV5D@69.136.102.83] has joined #linode
11:57-!-Rboreal_Frippery is "realname" on #linode
12:06-!-BigBox [~oftc-webi@96-85-213-54-static.hfc.comcastbusiness.net] has joined #linode
12:06-!-BigBox is "OFTC WebIRC Client" on #linode
12:06<BigBox>Morning! I was wondering if Linode offers the ability to create/destroy VM's via an API at all?
12:07<@ctarquini>BigBox: Sure do! https://developers.linode.com and https://linode.com/api (deprecated) should get you on your way
12:07<@mcintosh>!point ctarquini
12:07<linbot>mcintosh: Point given to ctarquini. (2)
12:07<@bmartin>So fast
12:08<BigBox>Awesome
12:08<BigBox>Thanks!
12:08<BigBox>!point ctarquini
12:08<linbot>BigBox: Point given to ctarquini. (3)
12:09-!-BigBox [~oftc-webi@96-85-213-54-static.hfc.comcastbusiness.net] has quit []
12:13<csnxs>!lick ctarquini
12:13<linbot>csnxs: Point given to ctarquini. (4)
12:38-!-namero999 [~oftc-webi@ip5b417526.dynamic.kabel-deutschland.de] has joined #linode
12:38-!-namero999 is "OFTC WebIRC Client" on #linode
12:38<namero999>hello
12:38<@bmartin>How can we help you?
12:39<namero999>I joined to ask a question about the feaseability of something I would like to achieve on linode
12:39<@bmartin>1ask
12:39<@bmartin>!ask
12:39<linbot>If you have a question, feel free to just ask it -- someone's always willing to help. If you don't get a response right away, be patient! You may want to read http://alexfornuto.com/how-to-ask-for-help-on-irc/
12:40-!-nix-7 [~nix@2a01:7e00::f03c:91ff:feae:891d] has quit [Quit: R: Tape loading error]
12:40<namero999>basically I would like to reach my linode via an IP, but have the outbound traffic of such linode go through a different interface/ip. on other cloud providers I managed to do so with the concept of "elastic ip" or "floating ip" but I'm not sure linode has such a concept
12:40<namero999>my specific use case is this
12:41<namero999>I have a squid, and my app goes through it when it performs requests towards external service providers
12:41<namero999>assuming that my squid has ip x.x.x.1
12:42<namero999>I would like that the service provider sees my traffic as a different ip, x.x.x.9 for instance
12:42<@sjacobs>you could most definitely set up a squid proxy.
12:42<namero999>the squid proxy is already up and running
12:42<@sjacobs>each Linode is going to have it's own unique static ip address. it's not going to change, but each Linode will have a unique one.
12:43<namero999>yep so far so good
12:44<@sjacobs>any other questions to see if what you are trying to do is feasable?
12:44<namero999>yes, I can maybe elaborate a bit more. I want to understand if I can achieve the "feature parity" with my other cloud providers, specifically aws and digital ocean. with a "floating ip" I basically have 2 networks interfaces on the box. I managed to configure squid to use the second interface for outbound traffic.
12:45<namero999>the reason is so that
12:45<namero999>I can always keep the "real" ip of the machine to access it myself
12:45<namero999>while being able to assign a different ip on the outbound traffic
12:45<@sjacobs>oh. so you just need two ip addresses per Linode?
12:45-!-runf0r3st [~runf0r3st@c-76-25-24-44.hsd1.co.comcast.net] has quit [Ping timeout: 480 seconds]
12:45<namero999>yes. I know about the failover ip on linode
12:46<namero999>but I don't know if I can "change" it on demand like I do with elastic ip, where I can just "detach", get another one, and "attach".
12:46<namero999>no problem to automatize everything with ansible on my side
12:46<namero999>just want to know if I can achieve this setup on linode
12:47<@sjacobs>the second ip address would need approval from support, which requires technical justification. if you provide some more details as to why you need it in a ticket, you may get it.
12:47<ericoc>you get one IPv4 address per linode. you don't get to simply burn through addresses and unassign and reassign to get a new one at will however... you _can_ swap IPv4 addresses between linodes if you have more than a single linode, but you don't get unlimited numbers of IPv4 addresses per linode
12:47<@sjacobs>if you could use ipv6, though, a /64 pool of addresses would let you swap addresses all day.
12:47<ericoc>a "floating" IPv4 address is sort of just a gimmick/buzzword by other providers
12:48<ericoc>an IP address is an IP address, calling it "floating" just makes it sound cooler
12:48<namero999>I might need 2/3 different ip addresses per year, and only one "per time" (so, no 2/3 concurrently)
12:49<ericoc>why do you need to cycle through multiple IP addresses?
12:50-!-nix-7 [~nix@li350-218.members.linode.com] has joined #linode
12:50-!-nix-7 is "seven"" on #linode #debian
12:50-!-scorche [~scorche@184.103.193.106] has quit [Read error: Connection reset by peer]
12:50-!-cps [~cps@c-73-133-107-201.hsd1.md.comcast.net] has joined #linode
12:50-!-cps is "Chris Smolinski" on #linode
12:50-!-scorche [~scorche@184.103.152.184] has joined #linode
12:50-!-scorche is "_" on #linode
12:50<namero999>in order to protect myself from faulty rate limits on one of the providers (from which we got the permission to do such an operation because we have the ability to do so, while they inherited a certain infrastructure which is less prone to change)
12:52<Peng_>No offense to you, but that is so gross, wow
12:52<namero999>so if every tot or so the ip changes, we don't risk being cut off (especially during spikes). but I would like not to have change also my app configuration. so I would like to reach my proxy via a static ip, but having flexibility to change the outbound traffic ip
12:52<Woet>you have permission but you still need to change IP to get around their rate limits?
12:52<Woet>that doesnt add up
12:52<Woet>anyways, thats not correct justification, so you can't do that.
12:52<Woet>:)
12:52<namero999>meh, I've seen wierder stuff :) anyway, it's like that. I change ip, everyone is happy.
12:53<ericoc>you might allegedly have "permission" from some provider (who is still rate limiting you??), but i wouldn't blame linode for not giving you permission to do that...
12:53<Peng_>Woet: It adds up. "We configured fail2ban wrong but you have permission to ban-evade."
12:53<Woet>Peng_: lol.
12:53-!-newton [james@0001510e.user.oftc.net] has joined #linode
12:53-!-newton is "james" on #linode
12:53<namero999>basically only "when it's needed" which can be said to be 2/3 time a year, following certain spikes.
12:53<@sjacobs>fail2fail2ban
12:53<namero999>so far the solution was to spin a new node
12:54<namero999>I managed to achieve this on our DO fleet, would love to do something similar on linode, just for "feature parity" so that I can forget about differencies. on linode it's only a box
12:55<namero999>squid has the tcp_outgoing_address that does the trick. just need this second ip, and possibly, a way to cycle it, ideally via api
12:55<ericoc>hahaha, you're definitely not going to get an api to burn ip addresses
12:56<namero999>a rate limit of "once per year" would be more that enough for this use case :)
12:56-!-runf0r3st [~runf0r3st@c-76-25-24-44.hsd1.co.comcast.net] has joined #linode
12:56-!-runf0r3st is "realname" on #ceph #ceph-devel #linode #sepia #privacytools.io #moocows
12:57<Peng_>I don't suppose they support IPv6
12:57<namero999>also why burn? (genuine question). if you "release" an ip once you are done, is it burned?
12:57<ericoc>if you got it blacklisted from a service? it's absolutely burned
12:59<ericoc>you're literally "releasing" it because you misused and subsequently ruined it
13:01<namero999>it's blacklisted by that service but it's not that you can't use it anymore. I get a new IP address from my internet provider each day, but I don't blame my neighboor for the ip I get the day after :)
13:02<@sjacobs>have we established they don't accept ipv6?
13:02<millisa>i think we have established that they dont really want namero999's traffic since they've tried to ban his traffic repeatedly.
13:03<namero999>it's not established but I am ready to bet they are not (ancient, ugly stuff there....) anyway will check.
13:03<namero999>in the unlikely case they do.... where should I look into, to get closer to my solution?
13:03<namero999>(thanks!)
13:05<@mcintosh>namero999: you can spin up second linode and swap IPs
13:05<@mcintosh>spin up a second linode, even
13:05<namero999>interesting. will look into it
13:05<namero999>spinning up a new node is what we do now
13:06<namero999>what this requires, is that the proxy configuration of our backend propagates to acknowledge the new node (and aventually decomission the old one)
13:07<namero999>the improvement here would be to always keep the same node (so that no new configuration needs to propagate to the backend) while changing the outbound ip.... maybe some DNS wizardry might help, but it's not my territory
13:13-!-cnf [~cnf@2a02:1807:3920:400:8c6a:fe46:6018:c957] has joined #linode
13:13-!-cnf is "Frank Rosquin" on #openconnect #linode
13:13-!-cnf [~cnf@2a02:1807:3920:400:8c6a:fe46:6018:c957] has quit []
13:16-!-arno [~arno@li587-95.members.linode.com] has joined #linode
13:16-!-arno is "arno" on #linode
13:38<linbot>New news from community: How do I switch from a commercial SSL to Let's Encrypt? <https://www.linode.com/community/questions/16956>
13:44-!-mrrobot [~doctor@176.246.102.226] has joined #linode
13:44-!-mrrobot is "= doctor" on #linode
13:46<mrrobot>hello everyone
13:47<mrrobot>hello
13:47<millisa>greetings
13:48<mrrobot>how you doing?
13:49-!-mrrobot [~doctor@176.246.102.226] has quit [Quit: leaving]
13:53<VetraNyx>@bmartin @sjacobs i have replied to the mail again
13:53<VetraNyx>perhaps you find that information here interesting.
13:59<VetraNyx>in short words... https://www.virustotal.com/#/ip-address/96.126.103.245 - was used earlier for quite some fraud domains - com.com has moved them now to their own IP - on Amazon (52.33.196.199) and sell them - as far as i can see and say, they have been responsible for many compromised domains - and now where they are defunct.. either reported or taken offline, they make money out of them.
14:01<VetraNyx>just pick a random domain from the list, look on abuseipdb - and then open the domain.. the end-redirect is .gen.xyz - where they are offered for sale then
14:03-!-user33 [~oftc-webi@45.32.186.178] has joined #linode
14:03-!-user33 is "OFTC WebIRC Client" on #linode
14:03<user33>Hello.
14:03<millisa>Hi.
14:03<user33>I am looking for the email address of the marketing/PR team at Linode. Can anyone assist?
14:04<millisa>support@linode.com is usually pretty good about getting things to the right place
14:05<user33>kinda feels spammy to send to there, but okay. ty
14:06-!-Q3Man [~oftc-webi@wsip-174-69-184-230.lv.lv.cox.net] has joined #linode
14:06-!-Q3Man is "OFTC WebIRC Client" on #linode
14:08-!-user33 [~oftc-webi@45.32.186.178] has quit [Quit: Page closed]
14:24-!-levlaz is "levlaz" on #debian
14:24-!-levlaz [~levlaz@2600:3c01::f03c:91ff:fe62:b9ec] has joined #linode
14:26-!-Whiskey`Wonka [~oftc-webi@47-36-190-137.dhcp.mdfd.or.charter.com] has quit [Remote host closed the connection]
14:36-!-runf0r3st [~runf0r3st@c-76-25-24-44.hsd1.co.comcast.net] has quit [Read error: Connection reset by peer]
15:04-!-Q3Man [~oftc-webi@wsip-174-69-184-230.lv.lv.cox.net] has quit [Quit: Page closed]
15:04-!-Q3Man [~oftc-webi@wsip-174-69-184-230.lv.lv.cox.net] has joined #linode
15:04-!-Q3Man is "OFTC WebIRC Client" on #linode
15:11<Eugene>Every day I'm Linodin'
15:12*Q3Man wasn't Linodin' around Fremont last night for a few hours
15:13<millisa>did anyone find what the official excuse was from HE this time?
15:22<Peng_>Linode hasn't announced it yet
15:25<Q3Man>Can we engage in rampant speculation?
15:25<millisa>my money is on ats failure
15:25<millisa>it's their goto move
15:26<Q3Man>my biggest problem is that just a few months ago the same thing happened at fremont.... did they not make any changes after that disaster?
15:27<millisa>maybe the changes are what broke it
15:31-!-Rboreal_Frippery [~gV$e9dV5D@69.136.102.83] has quit [Ping timeout: 480 seconds]
15:31<Peng_>You can be unlucky as hell twice
15:31<Peng_>Or totally incompetent *and* unlucky :D
15:32<millisa>this was the one from Feb: https://status.linode.com/incidents/2088k0rt4rb9 that says 'autostart controller had failed'; which probably wasn't the ats failing I guess
15:32<millisa>https://i.imgur.com/6rZ8g8R.jpg
15:33<Peng_>:D
15:35<Peng_>Changing the subject to focus on me, that's why I'm so upset about the Atlanta Hurricane Irma outage! The data center is run by clowns, but it was actually a weird ATS failure, which was probably totally unpredictable and not their fault. So I can't blame them. >:|
15:36<Q3Man>I can understand crazy failures... Having the same failure twice in 4 months just looks like they don't care about actually fixing the problem
16:09<nyancat>The official excuse was "power outage" but it doesn't get any more specific than that
16:10-!-joolsr [~oftc-webi@nc.joolsr.life] has joined #linode
16:10-!-joolsr is "OFTC WebIRC Client" on #linode
16:11<joolsr>is the linode manager beta client still available ?
16:11<millisa>cloud.linode.com you mean?
16:12<joolsr>ah ok i'll give it a try
16:13<joolsr>there was a link from the old manager but i couldnt see it
16:18-!-joolsr [~oftc-webi@nc.joolsr.life] has quit [Quit: Page closed]
16:27-!-Krzysztof [~oftc-webi@91.238.216.5] has joined #linode
16:27-!-Krzysztof is "OFTC WebIRC Client" on #linode
16:28-!-Krzysztof [~oftc-webi@91.238.216.5] has quit []
16:33-!-VetraNyx [~oftc-webi@213162073097.public.t-mobile.at] has quit [Quit: Page closed]
16:49-!-Shentino_ [~Shentino@68-186-69-103.dhcp.knwc.wa.charter.com] has quit [Remote host closed the connection]
17:01-!-anomie [~anomie@00018802.user.oftc.net] has quit [Quit: Leaving]
17:04-!-DanielNM [user46904@0001518c.user.oftc.net] has quit [Quit: Client excited]
17:04-!-Shentino [~Shentino@68-186-69-103.dhcp.knwc.wa.charter.com] has joined #linode
17:04-!-Shentino is "realname" on #linode #tux3
17:26-!-DanielNM [user25793@0001518c.user.oftc.net] has joined #linode
17:26-!-DanielNM is "-=[ Galatians 2:20 ]=-" on #linode #bitlbee @#mvlug
17:30-!-Rboreal_Frippery [~gV$e9dV5D@69.136.102.83] has joined #linode
17:30-!-Rboreal_Frippery is "realname" on #linode
17:32<nyancat>Question - Would linode ever restrict outbound traffic on a virtual server?
17:33<Eugene>Historically, only the Atlanta datacenter was subject to outbound packet filters, as a byproduct of using the datacenter-provided IP transit. That is no longer the case since Linode runs their own ASN.
17:34<Eugene>However, "would they ever do X" is a very, very loaded question. Abuse cases would definitely be subject to limiting
17:34<nyancat>My proxy scrape seems to be producing way more "bad" proxies as of late, I've tested this by collecting samples and running them through an external judge, and approximately 40-50% of tests yielded positive results
17:36<nyancat>Then again, my issue could simply be explained as the end-user restricting traffic from Linode, in which case there's not much I can do short of attempting to switch the IP
17:37<Eugene>Good luck with that
17:38<nyancat>Thanks! It might also be the script I'm running locally to judge - It's years old and hasn't been maintained much
17:39<nyancat>(Just so we're clear - These proxies aren't being used for abuse, they're being processed and sent for listing on RBL)
17:40<nyancat>(Just so we're clear - These proxies aren't being used for abuse, they're being processed and sent for listing on RBL)
17:40<nyancat>er, sorry for the double message.
17:41<nyancat>meant to up+enter on my terminal and accidentally did so here.
17:49-!-cnf [~cnf@2a02:1807:3920:400:8c6a:fe46:6018:c957] has joined #linode
17:49-!-cnf is "Frank Rosquin" on #ceph #linode
18:38-!-cnf [~cnf@2a02:1807:3920:400:8c6a:fe46:6018:c957] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
18:49-!-neoa [~oftc-webi@70.44.178.148.res-cmts.tnk.ptd.net] has joined #linode
18:49-!-neoa is "OFTC WebIRC Client" on #linode
18:50-!-neoa [~oftc-webi@70.44.178.148.res-cmts.tnk.ptd.net] has quit []
18:53-!-acald3ron [~acald3ron@189.202.79.21.cable.dyn.cableonline.com.mx] has quit [Remote host closed the connection]
19:47-!-namero999 [~oftc-webi@ip5b417526.dynamic.kabel-deutschland.de] has quit [Quit: Page closed]
19:54-!-Shentino [~Shentino@68-186-69-103.dhcp.knwc.wa.charter.com] has quit [Remote host closed the connection]
20:06-!-Shentino [~Shentino@68-186-69-103.dhcp.knwc.wa.charter.com] has joined #linode
20:06-!-Shentino is "realname" on #linode #tux3
20:21-!-Shentino [~Shentino@68-186-69-103.dhcp.knwc.wa.charter.com] has quit [Remote host closed the connection]
20:23-!-Shentino [~Shentino@68-186-69-103.dhcp.knwc.wa.charter.com] has joined #linode
20:23-!-Shentino is "realname" on #linode #tux3
20:25-!-fstd_ [~fstd@xdsl-84-44-228-209.netcologne.de] has joined #linode
20:25-!-fstd_ is "fstd" on #gentoo #linuxfs #gcc #awesome #oftc #vserver #osm #linode #debian #kernelnewbies
20:30<linbot>New news from community: DNS problem: NXDOMAIN looking up A for domain <https://www.linode.com/community/questions/16957>
20:32-!-fstd [~fstd@xdsl-87-78-93-14.netcologne.de] has quit [Ping timeout: 480 seconds]
20:32-!-fstd_ is now known as fstd
20:39-!-cachemoney [~oftc-webi@pool-108-39-199-71.eriepa.dsl-w.verizon.net] has joined #linode
20:39-!-cachemoney is "OFTC WebIRC Client" on #linode
20:40-!-cachemoney [~oftc-webi@pool-108-39-199-71.eriepa.dsl-w.verizon.net] has quit []
20:47-!-acald3ron [~acald3ron@189.202.79.21.cable.dyn.cableonline.com.mx] has joined #linode
20:47-!-acald3ron is "Armando" on #linode #debian-next #debian-mx #debian-es #debian
20:48-!-yhager [~yuval@d50-92-255-81.bchsia.telus.net] has joined #linode
20:48-!-yhager is "Yuval Hager" on #linode
20:54<Q3Man>hmm.. anyone notice that those forum links from linbot don't actually work anymore?
20:54<Peng_>Which links?
20:54<Peng_>https://www.linode.com/community/questions/16957 works for me
20:55<Q3Man>ah.. its this web irc client... it's embedding the ending > as part of the link
20:55<Peng_>Oh. :X
20:56<linbot>New news from zombo: Welcome <https://html5zombo.com/>
20:56<millisa>The impossible is possible
21:06-!-You're now known as mikegrb
21:14-!-yhager [~yuval@d50-92-255-81.bchsia.telus.net] has quit [Ping timeout: 480 seconds]
21:24-!-Q3Man [~oftc-webi@wsip-174-69-184-230.lv.lv.cox.net] has quit [Remote host closed the connection]
21:38<@mcintosh>The unattainable is unknown at Linodecom.
21:46-!-mpusr [~oftc-webi@2601:204:cd00:690:5132:9239:1f60:290] has joined #linode
21:46-!-mpusr is "OFTC WebIRC Client" on #linode
21:47<Woet>stability at fremont is pretty unattainable apparently
21:47-!-mpusr [~oftc-webi@2601:204:cd00:690:5132:9239:1f60:290] has quit []
21:49-!-Q3Man [~oftc-webi@ip68-108-160-170.lv.lv.cox.net] has joined #linode
21:49-!-Q3Man is "OFTC WebIRC Client" on #linode
21:50<staticsafe>oof
21:52<Peng_>Woet: you don't know that for sure
22:12-!-Q3Man [~oftc-webi@ip68-108-160-170.lv.lv.cox.net] has quit [Quit: Page closed]
22:50<retro|blah>...oh.
22:55<retro|blah>Well, good to know that my Linode rebooting was NOT due to a localized issue. Though that leaves me guessing where my compiles left off last night. Hm
22:56<Zimsky>you don't stream your tmux sessions to another server and take snapshots every minute?
22:56-!-genetrix [~oftc-webi@cpc87627-slou5-2-0-cust924.17-4.cable.virginm.net] has joined #linode
22:56-!-genetrix is "OFTC WebIRC Client" on #linode
22:57-!-genetrix [~oftc-webi@cpc87627-slou5-2-0-cust924.17-4.cable.virginm.net] has quit []
23:00<@mcintosh>!boo Woet
23:00<linbot>mcintosh: Point taken from woet! (17)
23:10<linbot>New news from community: why now i can not ssh to my domain in china <https://www.linode.com/community/questions/16958>
23:34-!-yhager [~yuval@d50-92-255-81.bchsia.telus.net] has joined #linode
23:34-!-yhager is "Yuval Hager" on #linode
23:40-!-technoid_ [~technoid@50-26-187-10.abilcmtk01.res.dyn.suddenlink.net] has quit [Remote host closed the connection]
23:44-!-yhager [~yuval@d50-92-255-81.bchsia.telus.net] has quit [Ping timeout: 480 seconds]
23:46-!-acald3ron [~acald3ron@189.202.79.21.cable.dyn.cableonline.com.mx] has quit [Remote host closed the connection]
---Logclosed Fri Jun 22 00:00:04 2018