Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Thread: had an all day (9 hr) problem at work....

  1. #1
    Raw Wave wizardPC's Avatar
    Join Date
    Jun 2000
    Location
    Nashville
    Posts
    2,635

    Angry had an all day (9 hr) problem at work....

    that should've taken no more than 45 minutes.










    If, that is, the dumb **** I talked to on the hardware side of our business had actually SCROLLED DOWN to check the problem instead of just assuming everything was okay.

    Allright, so here's the deal: I work for a company that supplies hardware and software for call centers. The hardware (Automatic Call Distributors, or ACDs) takes the calls and distributes them, then the software takes information from the ACDs and allows managers to track average handle time, number of calls, and other vital statistics. I support the software side.

    So I get a call this morning about 15 minutes into my shift from a "premiere" customer saying that they haven't received information from the ACD since Sunday. This is a pretty typical call, so I get some information from him and dial in. I'm in there about 15 minutes when I realize that there is a problem with one of the servers that runs our software communicating with the ACD. Usually this is because some idiot in IT changed firewall settings without asking why those ports were open in the first place. I try to use telnet to connect to the port, and I immediately get a "connection to host closed" error, which is a pretty good indicator that the server can't talk to the ACD on the right port. The other 12 ACDs they have don't have this problem. So I call up their IT department:

    --------------------------------------------------------------
    "IT, this is Bob"

    "Hey Bob, this is (wizard) from (software vendor). I've got a problem with the Adapter Server talking to the Chicago ACD. I can ping it from Adapter, but it can't connect to port 7000."

    "Try using 'telnet 192.168.1.1 7000' to see if it can connect."

    "I did that, and it can't. Has anything changed on the firewall in the last couple days? Any network issues?"

    "There's no firewall between those two machines, and I can telnet into 7000 with no problem from here. I don't know what to tell you, man."

    "Umm...Thanks." I'm thinking this is a network issue since only one of the 13 ACDs is affected, but this guy doesn't get it. "I'm gonna see what else I can find out, and I'll call you back if I find out anything."

    --------------------------------------------------------------

    So I get off the phone and tell my supervisor what's going on. He says what I was thinking, that this was a lazy IT guy and it's probably a network issue. To cover all our bases, my supervisor wants me to call our hardware guys and verify that nothing is wrong with it. So I call our hardware guys:

    ------------------------------------------
    "ACD support, this is Frank."

    "Hey Frank, this is (wizard) from (software support). I've got a problem with Adapter connecting to this ACD. I don't know anything about the hardware side of this--can you dial in and check it out?"

    "Sure....Looks like everything is okay. RTB is fine, ACD is up and running. Adapter should have no problems connecting. It's probably a network issue."

    "Thanks Frank. That's what we thought, we just wanted to verify that the hardware is working like it should."
    ------------------------------------------

    So I shoot an email to the IT guy asking him to check the routing tables from the Adapter's subnet over to the ACD, because not a single one of the (20 or so) servers our software is running on can connect to it.

    He writes back saying that the routing is fine, and it's "obviously something wrong with your hardware." My supervisor asks me to try and get him to put a packet sniffer on the Adapter server so we can see if there is any traffic being returned when we try to telnet to the ACD.

    At this point, I've been pulling my hair out for about 7 hours trying to prove to the IT guy that it's a network issue. He totally freaks out at the request for a packet sniffer--I mean WAY over the line stuff in this email.

    So me and my supervisor finagle access to the ACD. I'm gonna netstat from there (the ACDs run a Windows-based OS), then try to telnet from Adapter, then netstat again to compare and see if there is a connection even being attempted.

    I run the first netstat, and there are 954 connections to port 7000 from RTB. There should be like six, and RTB is supposed to max out at 200. No one has ever seen more than 80 connections at once. This is one of the things that our ACD support guy, Frank, told me he checked. The entire software support staff that is still at work (about 7 of us) is baffled, because we've never heard of such a thing happening. I call the ACD guy back, because he's in california and it's not near as late there as it is here.

    --------------------------------------
    "ACD support, this is Frank."

    "Hey Frank, it's (wizard), we talked several hours ago about (customer's) ACD?"

    "Yeah, what's up?"

    "If RTB has 954 connections to 7000, is that going to cause this problem?"

    "I'm sorry, I don't think I heard you right--Did you say 900?!"

    "No, I said 954. And they're all ESTABLISHED connections, not WAIT or LISTEN."

    "That can't be right. RTB can only handle 200 or so..."

    "Well, I did a netstat -n and piped it to a text file. Imported that into Excel and I count 954."

    "Hang on....(dials in)....I just ran it and I only count 14."

    "Are you on the right ACD? Because I've run this twice and---"

    "Oh, wait, here they are. I guess I just didn't give netstat enough time to give me the entire list."
    --------------------------------------

    Then the ******* doesn't even fix the problem like he's supposed to. He emails me the directions and says hes "going to let (me) handle it." Oh, and this is no small deal--if I screw it up, the call center goes down. As in they can't take calls.

    So basically, on this guy's word, I stuck to the story of it being a network problem and not our problem. Now I look like the dumbass because I've spent all day telling their IT guy about how they need to fix their damn network because nothing's wrong on our end, only to find out that something is wrong on our end, and the guy who was supposed to catch and fix the problem in the first place isn't going to take responsibility for it.

    Anyway, so it's like 8pm at the customer site when we finally get all this crap figured out, which means it's too late for me to do the work tonight. So I get to call IT guy in the morning and take a bunch of crap from him, reset the ACD, and hope I don't crash the system (which could possibly cascade to the other 12 ACDs this Adapter server is connected to, causing a company-wide outage!).

    Who wants to be me?
    Debt as of 1/1/05: $34,354.48
    Debt as of July 4, 2007: $0.00 explanation
    I'M DEBT FREE!!
    I'm now a reasonably successful gunblogger.

  2. #2
    Raw Wave BoyNextDoor's Avatar
    Join Date
    Aug 2004
    Location
    Walden, NY
    Posts
    2,115
    all i can say is the typical fingerpointing sheme. Hardware vs. Software... And this time you ended up to be the a$$ as in your avatar. Sorry dude, but hey. Tomorrows another day. And if not, the day after. Lol. hope everything works out though...

  3. #3
    Registered User
    Join Date
    May 2003
    Posts
    1,736
    Quote Originally Posted by wizardPC
    Who wants to be me?
    No one, not with that much debt.

    If you have a fit girl then I might consider


  4. #4
    Raw Wave wizardPC's Avatar
    Join Date
    Jun 2000
    Location
    Nashville
    Posts
    2,635
    Quote Originally Posted by Chris31
    No one, not with that much debt.

    If you have a fit girl then I might consider

    I logged 60 hrs last week, I ain't even got time for a fat girl, much less a fit one
    Debt as of 1/1/05: $34,354.48
    Debt as of July 4, 2007: $0.00 explanation
    I'M DEBT FREE!!
    I'm now a reasonably successful gunblogger.

  5. #5
    Raw Wave BoyNextDoor's Avatar
    Join Date
    Aug 2004
    Location
    Walden, NY
    Posts
    2,115
    so the cage'll have to wait right?

    cheers

  6. #6
    Registered User
    Join Date
    May 2003
    Posts
    1,736
    Quote Originally Posted by wizardPC
    I logged 60 hrs last week, I ain't even got time for a fat girl, much less a fit one
    I think Ill leave myself as...well myself and you as yourself

  7. #7
    Raw Wave wizardPC's Avatar
    Join Date
    Jun 2000
    Location
    Nashville
    Posts
    2,635
    Quote Originally Posted by BoyNextDoor
    so the cage'll have to wait right?
    cage money being used to pay off student loans
    Debt as of 1/1/05: $34,354.48
    Debt as of July 4, 2007: $0.00 explanation
    I'M DEBT FREE!!
    I'm now a reasonably successful gunblogger.

  8. #8
    Raw Wave BoyNextDoor's Avatar
    Join Date
    Aug 2004
    Location
    Walden, NY
    Posts
    2,115
    Quote Originally Posted by wizardPC
    I logged 60 hrs last week
    Damn that's some crazy *** disciplin you have there... Me wants it too. So i could get out of debd as early as december 1... DAMN! You da man!

    Let's collect for cage then...

  9. #9
    MySQL Error Scouse Monkey's Avatar
    Join Date
    Mar 2004
    Location
    Bristol
    Posts
    9,727
    Quote Originally Posted by Chris31
    No one, not with that much debt.

    If you have a fit girl then I might consider

    well he got less debt that me!

  10. #10
    cheap custom title JC-S60's Avatar
    Join Date
    Mar 2004
    Location
    Ghent - Belgium
    Posts
    1,863
    Well, I wasn't even considering being you, with or without debt

    jk!

Page 1 of 2 12 LastLast

Similar Threads

  1. Lilliput TouchKit problem solved!
    By AAVfReaK in forum LCD/Display
    Replies: 4
    Last Post: 11-07-2010, 09:29 AM
  2. Not your typical flicker problem
    By JPuma in forum LCD/Display
    Replies: 17
    Last Post: 05-30-2007, 10:28 AM
  3. Prison vs Work
    By ACHIEVER in forum Off Topic
    Replies: 20
    Last Post: 04-15-2005, 02:53 PM
  4. night skin day skin problem.
    By fallnangel in forum FrodoPlayer
    Replies: 2
    Last Post: 02-02-2005, 09:08 AM
  5. cannot get lilli touch to work anymore
    By delussional77 in forum LCD/Display
    Replies: 12
    Last Post: 10-12-2004, 01:52 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •