In the spine-tingling finale of IT Horror Stories, host Jonathan Crowe is joined by a fearless lineup of IT veterans who have stared into the digital void and lived to tell the tale.
In the spine-tingling finale of IT Horror Stories, host Jonathan Crowe is joined by a fearless lineup of IT veterans who have stared into the digital void and lived to tell the tale. Guests Kallum Kyle, Senior Sysadmin at Storable; Josh Adcock, Director of Client Services at The Tech Doctor; and Adam Walter, Founder of Humanize IT, recount their most harrowing experiences from the trenches of tech. They relive moments when end users were their only hope, when vulnerable software lurked like ghosts on forgotten networks, and when automation backfired so badly it cost them their jobs. These stories aren’t just frightening—they’re all true!
Timestamps:
01:46 - Kallum's tale
14:36 - Josh's horror story
24:54 - Adam's woeful experience
[00:00:00] Jonathan: Hello. Please come in, join me. I'm Jonathan Crowe, director of Community at Ninja One, and this is IT Horror Stories. Tonight, the lights are dimmed, the servers are humming suspiciously, and something sinister is lurking in the logs. This isn't just any episode, this is the season finale of IT Horror stories.
[00:00:26] And to mark the occasion, we've summoned a few, especially haunting tales from our most fearless contributors: listeners just like you. We called upon a handful of courageous IT pros and sys admins to relive their most harrowing tech terrors stories of outages, breaches, and configuration's gone horribly wrong, but it's not all doom and gloom.
[00:00:50] In the aftermath of disaster, there's always a glimmer of hope. Today we sit down with three IT leaders. Kallum Kyle, senior sys admin at Storable. Josh Adcock, Director of Client Services at the Tech Doctor. And Adam Walter, founder of Humanize IT. Before we get into our stories, I have three questions for you.
[00:01:11] Dear listener, what's scarier? Having to deal with end user problems or having to rely on end user help as your only solution? Having vulnerable software on your network or grinding down business critical operations without it? Losing your job because you automated your way out of it, or staying stuck in that job forever?
[00:01:36] So grab your flashlight, keep an eye on the firewalls, and whatever you do, don't ignore that blinking red light.
[00:01:46] For our forest story everyone. I've got Kallum Kyle, Senior Systems Administrator at Storable. Win Admins overlord.
[00:01:56] Kallum: Moderator. Not quite an overlord. My name is still yellow. It's not red for the admins yet.
[00:02:01] Jonathan: Not yet. Not yet yet. Working on it. And Microsoft MVP across several categories.
[00:02:09] Kallum: You say several. I say two.
[00:02:10] I am a Microsoft MVP in Intune, and recently awarded PowerShell earlier this month.
[00:02:16] Jonathan: Congratulations for that, Kallum. I really appreciate you coming on to relive your horror story. And I understand that turnabout is fair play here because you recently did this to other people. So now I get to do this to you.
[00:02:31] Kallum: Yes. Yes. Uh, the Midwest Management Conference or MMS, as a lot of people know it. That was in May of 2025. I hosted a session. I won't say I presented a session because it was a lot of. Audience participation on it. Horror stories where we invited every single person in the audience to get up and tell their tale and had an absolutely amazing time with it.
[00:02:55] And being able to share stories about the worst things that have happened to us professionally are, it's extremely helpful for both. Feeling like, Hey, I didn't take down the government of Ottawa. That's fine. My, my mistake's not that bad. And being able to meet people who have been in the trenches with you and make those connections and network in the way that is fun and not in the way that sysco does it.
[00:03:24] So it's great fun.
[00:03:27] Jonathan: You know, I, I think, I think of the phrase, there's, there's always a bigger, oops. You know?
[00:03:32] Mm-hmm.
[00:03:33] Jonathan: Someone has always done something worse. I think that's great. I think it's also super cool that you put that together and you gave people an opportunity to get up on stage and practice feeling comfortable with doing that and public speaking, talking about something scary for, for most, I think normal people, that's a scary thing to do.
[00:03:54] So the fact that you're able to do that and get them to talk about these horror stories, but help them overcome that fear. Um, get used to that and give 'em practice. That's just really cool and so good for you for doing that. Now let's turn the tables on. You tell us about, tell us about your horror story.
[00:04:12] I understand, I, we, we talked about this a little bit. Of course. You don't just have one, you have many, but we're gonna focus on one today, and I understand it's, it's one of the more recent ones. So maybe set the scene for us what's going on before we get into the hour where the real horror strikes.
[00:04:31] Kallum: I had been working with this company for about a year, so I still felt like I was getting my legs under me as the, the primary like Microsoft person.
[00:04:41] I have a very small team and I am the Microsoft person who, if you're, if you're watching the video here, I have an Azure tattoo, so I am the Microsoft person at my job and we are a fully remote company. We're split as to whether we're on Maxs or PCs. So about half of the organization is mine, that I am managing these.
[00:05:06] This fleet is my baby. And then the first person that we get affected was our senior IT security person. Was the first person who reported an issue to us. So
[00:05:23] Jonathan: When you're getting tickets and you see, I'm sure you probably have the people that, okay, yes, you got another ticket from them. That makes sense.
[00:05:32] When you see it coming in from your senior security person, does that hit a little bit different? So I,
[00:05:38] Kallum: in his defense, he did not open a ticket. He went into Slack and said, Hey, can I have my bill for recovery course? He
[00:05:44] Jonathan: didn't open a ticket.
[00:05:47] Kallum: So he asked for his BitLocker recovery key and we're like, okay, sure Bob.
[00:05:53] We'll call him Bob. Sure. Bob, here's your BitLocker recovery key. Just, did you do anything that would make BitLocker trigger? No, not that I can think of. Okay. Hey, I am still not able to get in. Okay. And then that's when we start seeing reports from other people who are not on the east coast start rolling in as they're starting and it's like, oh no, this is, this is a lot of, this is a lot of people for an organization of our size to be having this problem.
[00:06:30] And that is about when I sort of popped into WIN admins. Which, if you're not familiar, WIN Admins is a Discord server with almost 15,000 users worldwide to say like, Hey, what's going on? And it was just an absolute flames, everybody trying to figure out what is going on. And if you. If you feel like your team is too big to get anything done, imagine that across countries.
[00:07:02] Jonathan: Well, in those moments to try to figure it out. Those moments q those moments of panic. I mean, it's, I, you know, q the, the, the, the G of community where, you know, Donald Glover is coming in and everything's on fire. But seeing that, you know, there's such a rush of. Everyone trying to sh find information, trying to share as quickly as possible.
[00:07:21] Sometimes information is not always pointed in the same direction. It really turns into just a, an insane atmosphere to be operated in, in any state, let alone like a calm, cool and collected rational state. Do you think those things, obviously it's helpful, but do you think, you know, when you're in that moment, this is also like, let's talk more about like the, the remote aspect of, of this too, because.
[00:07:46] You know, as opposed to other offices where you see these reports kind of all happening within, you know, a department or within your bill. Like you're seeing these things happening across, as you mentioned, different geos, and then you're going in and you're seeing this. Okay, this is a huge widespread event.
[00:08:04] What's going through your head there? How are you feeling about this?
[00:08:08] Kallum: I didn't mess anything up. That's the main thing that immediately went through my head when I saw colleagues in like Germany and Iceland in the UK also having this problem. It's like, okay, this wasn't my fault. Okay. So I don't have to worry about that portion of it.
[00:08:26] So
[00:08:27] Jonathan: you are, but you are powerful when it comes to your fleet. Yes, but you are not, that is not, not your responsibility. I did
[00:08:35] Kallum: not this,
[00:08:35] Jonathan: yeah,
[00:08:36] Kallum: I have messed other things up, but this one was not me. But seeing that it's, there are so many very highly intelligent, very, very technically minded people in the communities that I work in, and being able to say like, okay.
[00:08:57] I am also not the only one who doesn't know how to fix this. And being able to work together to sort of crowdsource something, one person finds an article and 70 companies take use of one person. Being able to find a piece of research is so, so nice to have.
[00:09:20] Jonathan: How are you, I guess, dividing your attention here in these moments where.
[00:09:25] You're drawn into these, these, uh, like when admins, um, you're drawn into finding out, okay, what's going on? What is the world telling, telling me right now? Um, how much information can I get? You're the person that your company's going to for answers. I'm imagining to you. You know, you, you've got people, what's going on?
[00:09:43] What can I do? How do you, how do you kind of manage and, and start to triage in your mind where your attention's going?
[00:09:50] Kallum: So. We have a fairly good setup where I work and that is that our manager runs comms, so I ignore anything it if, if it is not from somebody that has tagged VIP on Slack. When we get into something like that, because the VIPs are that people that are on my team, they will either be bringing me information or.
[00:10:16] Asking for information to disseminate because there are a lot of people asking them. So that way I only have to say something once and it gets spread. So it's sort of like opposite phone tree at that point. There are six people who can talk to Cal right now and only those six, anybody else will get ignored.
[00:10:37] Jonathan: Yeah, do not disturb. You go into a place where you know, you know it's a reliable place. People can be sharing information that is lit up. People are starting to, starting to find things. They're sharing helpful resources, what happens next?
[00:10:52] Kallum: At that point, we were very much so in like there is, there is a point of emergency where you just tell the company like, look, we know we'll let you know when we know something else.
[00:11:10] We are lucky in that. We have these now. So every single person on our team or on our company has to have a phone for MFA, right? So for their multifactor, they have to use their phone. We don't use UV keys or anything like that. So they're signed into their slack on their phone. We can still communicate with each other.
[00:11:32] We can let everybody know if something is down without them having to be able to log into their company computer to see that. That's the point that we get to is we know there's a problem. We're investigating, and we hate to see that on status pages for like major service providers. Like Google says, Hey, there's a problem with these, we're investigating.
[00:11:54] It's like, well, tell us what the problem is. Uh, we don't know what the problem is. That's sort of, we're trying to figure it out. We're working on it.
[00:12:02] Jonathan: The timeframe here, you know, it feels like forever these things compare to how they used to roll, though. I mean, information gets around a lot quicker because of all these things we've been talking about.
[00:12:13] And so you start to get information about, okay. There, I'm trying to remind myself about how the, the, the timeline of events here.
[00:12:21] Kallum: I know it was the day before I was supposed to go on vacation.
[00:12:24] Jonathan: Of
[00:12:25] Kallum: course it was.
[00:12:29] So
[00:12:29] Jonathan: did you end up being able to go on this vacation? Yeah. Is it the spoiler?
[00:12:34] Kallum: Yeah. I made it.
[00:12:38] Jonathan: How, how did you make it? How did you all resolve this?
[00:12:43] Kallum: How do we resolve this? Well, there was the official guide for resolving it that got sent out same day, thankfully. So everybody knows you boot in a safe mode, delete one file out of the source folder, and when the definitions update on its next run, it pulls down the fixed version of that binary, right?
[00:13:03] So we wrote up documentation in our confluence for our users as to how to do this themselves. And sent them an email with their BitLocker key with like the full step-by-step documentation on it. And when we opened the floodgates, because we had initially said, Hey, we're aware of this. Don't send tickets on it.
[00:13:34] We opened the flood gates and said, Hey. If you have this problem, like this problem, here are screenshots of the problem, send us a ticket. And we sent out the documentation and I think we had to handhold something like seven users total through it.
[00:13:54] Jonathan: That's amazing. Chalk one up for having faith in humanity.
[00:13:57] Right. Well, Kallum thank you so much for taking time to share. Y how you s your company navigated that horror story that impacted so many people. Very cool approach. And just wanna say thanks again for, for doing everything you do, not just here sharing your story, being such a great community advocate and person who.
[00:14:21] Is really helping, um, everyone. Encourage everyone to be sharing their information. Everyone who's listening, you can find Kallum and win admins. Kallum thanks so much for joining us. Really appreciate it.
[00:14:31] Kallum: Thanks for having me. Happy to be here.
[00:14:36] Jonathan: We are back with our second story, and this time we have guest Josh Adcock, director of Client Services at the tech Doctor. Josh, thanks for taking time to come here and relive a dark, scary moment of your past. Well, thanks for having me. Even though you're dragging all the trauma out,
[00:14:57] you know, I will say, when, when you and I talked about you coming on the show, you seem pretty excited and, and saying that you have quite a few of these stories. So yes, we're asking to relive dark times, but sounds like you're having some fun with him.
[00:15:12] Josh: Oh, definitely. Always. You gotta laugh, right? Oh, well, hey, what do you, what else is there to do if you don't collect stories of horribly?
[00:15:20] Timed things and horribly done things and reminisce on them other than just sit and stare at reports all day. So, you know, gotta have something to keep you entertained throughout the day.
[00:15:33] Jonathan: Well, let's get into it. Before you share the details, let's set the scene a little bit. Let's lean in. The fire's crackling.
[00:15:42] What's going on in this day before the bad thing strikes?
[00:15:46] Josh: This particular day was pretty simple and basic day. I just kind of coming into work first thing in the morning, doing my normal reading emails, going through everything. And you know, one of the things that I do on a pretty regular basis is go through the CC emails and everything for.
[00:16:02] You know, different vulnerabilities and what may affect me and my environment or my clients, just to kind of get an idea there of what's going on. That's what I was doing. When I noticed that they released a pretty severe vulnerability for MSMQ and later on in that same day, I happened to notice that it was enabled in almost every one of our environments with an on-prem server.
[00:16:26] So after doing a little of. At the time, I considered diligent research. Just kinda looked through and everything I found told me that MSMQ was horribly antiquated. Nobody really used it for anything anymore. It hasn't been a standard tool in a really long time. So me, on a Monday morning, decided to make the desysion that I was just gonna go with that as the answer and just throw together a quick PowerShell script and run it across everything in our ninja environments, which, you know, I.
[00:16:58] No better than to do. But it's Monday morning, what am I gonna do? It's awesome. I mean,
[00:17:04] Jonathan: hey, like aliens, right? The safest thing to do. Nuke it from orbit. Just get rid of it.
[00:17:10] Pull it out.
[00:17:12] Jonathan: Why run the risk?
[00:17:13] Josh: So, you know, we, we ran that and most of the day continued on as a normal day until I received a phone call from the one and only 24 7 clinic that we have.
[00:17:25] That is always. Very frantic. When anything doesn't work like they can't print it is a entire meltdown. So you can imagine how frantic they were when no one could access the one tool they use for literally everything in the entire. Business. I guess what really made this a nightmare more than anything was the thought that this is probably gonna be pretty simple.
[00:17:51] All I have to do is go in and turn MSMQ back on on the server and everything will be golden. And I quickly realized that was not the case. There was a whole lot more involved into enabling MSFQ and ultimately ended in having to reinstall the application from the server to all 45 of their workstations throughout the whole thing before any single workstation would work again.
[00:18:18] Uh, that was two very miserable days,
[00:18:24] Jonathan: two days. So this one Monday morning desysion. Legacy software. Outdated. No one's using this thing. Let's be safe. This came from a good place. You're trying to keep your, your, your clients safe, right? And lo and behold, so tell us a little bit about what those calls are like.
[00:18:41] You're saying they're frantic, right? It's a 24 7 place. It needs to be online, needs to have access to things. Do you have. Multiple people yelling at you at this
[00:18:51] Josh: point? No. With this client particularly if anything goes wrong, it is generally the owner of the business that is immediately on the phone. And then it's one of those clients where if they don't necessarily like our answer or our way of being expedient about it, it's immediately a phone call to the person that owns our company.
[00:19:10] And then that's when it really becomes super frantic. You know, it's, it's a very long time client. It's been a client for a very long time, whole. Owner to owner relationship between the business. And of course, you know when, when that call gets made to the next level, it's always far more, I don't know if frantic is the word, but.
[00:19:33] We will go with frantic because it's always far more frantic to that call the point the owner is calling another owner than the initial call coming in tense.
[00:19:42] Yeah, so
[00:19:43] Josh: generally, you know, it's, in this situation it was something that the, the owner was not necessarily aware of what had happened yet 'cause he wasn't at the office while we were doing everything.
[00:19:55] So it, it turns up the intensity level quite a bit when you receive that phone call and no one is fully aware of the whole situation other than me stuck there trying to figure out how to put it all back together.
[00:20:10] Jonathan: And so you're pretty much on your own fixing this thing. You're able to do it. What changes after that?
[00:20:15] I mean, aside from you're gonna make extra sure, but. Yeah. Do you have new policy now? Is there, is that, that just a painful lesson learned? Is there anything else that changed after that?
[00:20:26] Josh: Yeah, that definitely generated some new guardrails, so to speak, being put together. We also generally, anytime we have any sort of major incident or, or big thing that occurs like that, that that affects everybody and, and gets the higher levels involved at that level, we generally tend to, you know, sit down with the client afterwards and have kind of a debrief.
[00:20:49] Let you know, give a couple days for tempers and emotions and everything to just kind of go away and fade into the background so we can sit down and and game plan in the future. What is to occur? It's funny 'cause that one particular clinic tends to be a great source of horror stories. Purely just because it is 24 7 and it's, you know, they have one server on location, so you're supposed to, you know, reboot that server relatively frequently, but you can't really reboot the server because their whole operation is running on the server.
[00:21:25] And you know, we've had some instances try to reboot it, thinking it's gonna be a normal 15, 20 minute thing, and five hours later the server's still not coming back on and just stuck configuring an update. Lots of stories have come from there, but. It's also been something that has been very helpful for us overall to go through those trials and tribulations because they give us a guiding path for.
[00:21:50] The future and also kind of inspire you not to take that moment when you, you say to yourself, I know better than to do this, but it's gonna be fine. 'cause I've, you know, bent this rule 150 times and the one time you don't makes all the other 150 not worth it. Yeah,
[00:22:09] Jonathan: yeah, yeah. Yep. Never let a good crisis go to waste.
[00:22:12] It sounds like you've taken a positive spin on it and. You're doing, uh, making a constructive path forward from it. Now, I have one question for you, Josh, and that is after you're able to get things restored, you know, we're still dealing with this vulnerability. I'm just waiting for the sequel to drop. Is there another horror story?
[00:22:31] I mean, you can imagine that would be the icing on the cake, right? I mean, ultimately
[00:22:35] Josh: there's not. Much. I mean, the piece of software is kind of required because of some of those situations where like you may know what the correct move is and it may be replacing something that someone is super used to.
[00:22:50] But at the end of the day, it's not the desysion I get to make. So, you know, you just kind of have to monitor and. Do what you can to try to make sure that that vulnerability isn't being exploited. You know, firewall rules in place and just kind of keeping everything, keeping an eye on everything and paying a little extra attention to that particular environment.
[00:23:11] But yeah, in that particular situation, at least there, there really wasn't a whole lot we could do aside from just get it back to functioning and kind of go from there.
[00:23:21] Jonathan: It's not unusual for folks to have more than one story. And you all obviously are no exception, but it sounds like from this story too, it's, it's, it's really more about how you're able to respond to 'em.
[00:23:34] You're not gonna prevent them all together. There's going to be another one. It's gonna be a surprise whenever it happens. Oh, for sure.
[00:23:40] Josh: There's this, there's two things that you can almost guarantee in it that I've learned, and that's every day is gonna be a Monday. And horror stories are gonna happen, especially in the MST world.
[00:23:54] 'cause I mean, that's essentially what we do. We get paid to deal with horror stories, and I, that's honestly what makes me love what I do, because it is something different every day.
[00:24:05] Jonathan: You know, ironically, it sounds like the, the mundane, the lack of these quote unquote horror stories is the real scary thing.
[00:24:14] Josh: In a way. Yeah, absolutely. It's, there's nothing worse than coming in for a whole, well, I say nothing worse. Sometimes it feels like there's nothing worse than coming into like a whole eight to 10 hour shift and not one ticket comes in the phone doesn't ring one time. You just kind of sit here trying to remember all those things that you've been trying to do because you've been so busy.
[00:24:33] Yeah. And could never get around to do it.
[00:24:37] Jonathan: Well Josh, thank you for coming in and for reliving that tale. Really appreciate you and we'll talk with you again soon.
[00:24:47] Josh: Absolutely appreciate it. Thanks for having me.
[00:24:54] Jonathan: Moving on to our next IT horror story, we have Adam Walter, co-founder of Humanized It. Also a frequent guest and DM on our own show backups and Bandwidth. Adam, thank you for being brave and coming here to share one of your many, many, many IT horror stories.
[00:25:16] Adam: Find me. Find me at any conference event and ask me for a horror story.
[00:25:19] I'll give you a new one every time. I do have my lucky dice here, so maybe I'll have people you roll for a, a story. I like that. All of an encounter sheet maybe, and tell people like, what kind of story do you want today?
[00:25:32] Jonathan: Give us a little context about your background because it's interesting. You've seen, you have your, obviously your, your.
[00:25:39] Horror stories that you can share that show that you have the legit the signs of a sys admin, but you've also set in a few different roles in it and yeah, what I think is interesting. You've had leadership roles and now you're working with MSPs, but you've been on the internal side too. Tell us a little bit about your background.
[00:25:57] Yeah.
[00:25:57] Adam: Bachelor of Science, computer science, I went to college on an art scholarship actually, and a athletic scholarship. So finding a nerd in computer science that did all three of those. I'd love to meet one other person, though I worked for the state patrol as a desktop support technician. I was the only one for the entire state of Nebraska, so I had to cover all 500 miles by myself.
[00:26:21] We had a small team, you know, like server admins and things like that back then. These are in T four days, people. Uh, Corll word. Perfect. These should all resonate with you. Went from there to, uh, assist admin job. That actually will be the focus of my horror story today for a marketing firm, actually an athletic marketing firm.
[00:26:41] And the next job after that, I worked myself out of a job there actually, which will be the result, which of a fun little hook there for the story. And the next job was I worked for a, did cybersecurity and sys admin work for a bank marketing firm. And that was really fun. I got some great horror stories there that are just on your toes kind of things.
[00:27:05] And then I went from there to consulting and contract work for major corporations is as a generalist. They, they wanted somebody who just knew every aspect of it and could command teams. That was my first real leadership roles, and then decided that I missed small business and came back and started a consulting company and then kind of flew into like in 2022, buying a software company that I was doing gap analysis with.
[00:27:32] To build up what you now know is humanize it. A great software for account managers to manage their clients and show their value and find gaps in, in their technology stacks.
[00:27:42] Jonathan: So. What an amazing journey in starting off, adding more and more stakes to the, to the oops that are happening. The size of the oops, and the oops can get bigger and bigger, right?
[00:27:58] You're finding what it looks like to do things right and wrong at these other levels. They're getting bigger and bigger. And then finally you come, uh, back to working with, well, computers, technical stuff isn't hard enough. You're gonna start. Managing, dealing with people and providing leadership challenges, um, addressing those and then, uh, really to cap it off, now you're working with MSPs, the horror Agree.
[00:28:21] Just a different, different group of people. I love, I love me some MSPs. Let's go back to you're, you're, you're horror story now that you wanna share with us. This one in
[00:28:33] Adam: particular was at the athletic. Oh, at the athletic marketing firm. Yeah. They, they did athletic materials for certifications and they were more of an education slash marketing, kinda helping people do strength training.
[00:28:45] What, what was the role like, I mean, were you the only IT person? Were there others involved? The, the set, the scene here, it's 2007 people. If you've got more than one sys admin, you must be, you know, just huge rolling in employees. Yeah. Finding one sys admin back then was really hard. You had a lot of wannabes, a lot of people who just didn't understand.
[00:29:05] And so I came into a role where I was replacing a sys admin who was moving back. So this happened a lot. You back then is that, you know, you. Your sys admin left and you couldn't find anybody to replace 'em, so you double their salary and try to make them come back. And that happened here. That's how this story starts.
[00:29:21] Adam Walter first job outside of government, I, I had been moonlighting kind of as a, as a small consultant for a while. Could not make an, the IT consultancy work full-time. I couldn't, like I did the math and it just didn't work. And I realized that a lot of you did the same thing where you knew it wouldn't work, but you did it anyways and you just didn't sleep for three years.
[00:29:44] Jonathan: Yep. Yep. And so here we have fresh off that you're, you're, you're going into this job. You're, you're young, you gotta bounce in your step. So how long have you been at this, uh, this job before this horror story takes place?
[00:29:56] Adam: I wanna say it takes about a month into the, into the job. Okay. Somewhere within that first month, I, I discovered this.
[00:30:01] I was let go from that job within eight months of being hired. What we're about to hear here lays the foundation for how I worked myself out of a job. Let's hear it. And so, yeah, I come right in. I have worked with some really great, so the, the people I worked under at the, at the state patrol were really great server and network admins that I've been learning from.
[00:30:24] Very, very talented people. And you just assume that all environments are like yours. So I walk into this next environment. I'm like, I'm a little intimidated, right? 'cause now I'm just the only guy with a couple developers, a SQL admin, A PHB developer and me. And so I'm like, okay, there's nothing to fall back on here.
[00:30:42] And like, it's not like, it's not today where you can like go out anywhere and ask questions. You go to Google and answer. But still kind of experts Exchange is about it. You had Gs, that's about it. Yeah. Yeah. Lycos you, you start my job and I start getting my first ticket. And one of the first tickets was, Hey, we've had this issue being going on for a couple months now, and I just wanted to see, like, they, they email us and then they get a rejection notice after about three days.
[00:31:11] Like, Hey, your, your email was not allowed. And this is when smart filters just became like in a play. I think we were using Proofpoint, like early Proofpoint this time. The, the previous, this admin had configured it. And put it in place. Really cool. 'cause now instead of having an on-prem email filter who had a cloud-based, cloud wasn't a thing back then.
[00:31:33] It was just a smart filter at Proofpoint that all your email would route through and then come down to you. And I loved it. It was a really cool concept. I'd never used one like that before. And so I'm chasing this ticket, trying to figure out what's going on. We must have some filters in there. By the way, we had a couple emails that were being blocked and I found out why we due to foul language and because it was a strength and conditioning firm, the word snatch kept getting blocked,
[00:32:06] hang snatch. And so I was like, oh, cool, this must fix the issue. So I fixed the issue and, um, give yourself a pat on the back. Pen back. Sure. Those emails start coming through and they're still getting rejection notices after about three days. And I'm like, gosh, what is this? Or is it, was it, maybe it was four hours, some standard window of rejection, right?
[00:32:29] And they sent me the things like, we'll try again in four hours, or we'll try again in 12 hours. And it kept doing that. And finally it says, failed delivery. And so I'm like, okay, something is blocking. So I'm looking through my filters again, trying to figure out what's going on. And I'm getting on with Proofpoint.
[00:32:43] I'm asking them all sorts of questions, and I start finding in logs that Proofpoint is the one that is receiving the email, then rejecting it. It's not going anywhere. I'm like, okay, what's the deal here? It's being held up and never delivered, and something like, okay, we're getting closer here. And Proofpoint can't communicate with my exchange server.
[00:33:07] I'm like, okay, something's wrong with my exchange. So I'm going through my exchange, figuring it out, troubleshooting, and my exchange server never sees the connection request is what I, I, I figure it out like, God, what is this? It's gotta be a firewall issue. So go to the firewall, right? And I look for my Proofpoint, my firewall rules, and sure enough there are Proofpoint allow rule in there and I go and I check it out.
[00:33:32] And I look at the Proofpoint, you know, allowed ips that are at registered at Proofpoint, and I look at the ones that are on the, on the firewall. They're the same. IP ranges are allowing through, but I'm seeing rejection notices on the firewall. The previous sys admin put in the IP address is 1 7 2 8 4 0 0
[00:33:59] through. Or 1 7 2 16 0 0 16. So he allowed the 16 address, right? He allowed the 16 ips. Sure. He did not know how to read cider notation. It was supposed to be a slash 16. He put a rule for 16 IP addresses in rather than a slash 16. So 90% of emails were getting rejected. 90. They came from any IP other than zero through 16 was getting rejected.
[00:34:31] 'cause our firewall said these are not allowed. And it took me a second because that, that, that 16 zero through 16, not slash 16, zero through 16, was the rule that was created. Yep. It took me a second to kind of correlate. So the next rule down was a was a zero through 24. The next one down was like a zero through 16.
[00:34:48] I'm like, why did he choose those ranges? And I realized he did not know how to read cider, had no idea what cider notation was. And so I'm like, oh crap, where else is this happening? And so part of my job every day was to go in and troubleshoot email rejections. And troubleshoot backup issues and make sure the backups have been going, oh no, because they had a backup network that was on its own subnet that on on or subnet was on its own VLAN to avoid collisions and void noise.
[00:35:20] And so I'm like, if he didn't know Saturday notation for proof point, where else is this causing problems on the network? And that's when Adam realized he wasn't going home anytime soon. Went to the backup network, found out, done the same thing there, except for he did not only not know what cider notation was, he did not know the difference between a VLAN and a subnet.
[00:35:43] So everything on this network is on a flat network. I mean, it's on one little like three com switch in the back, and he had just created different scopes. On the same vlan and our collision's happening left and right all night long. Like these backups are fit, these are disc space backups. So this, this little switch is barely keeping up in the first place, and it all happens to be on the same one, and it's just dying every night.
[00:36:07] We're getting corrupted backup, so I fix that, get another switch, put 'em on their own switch, put 'em on their own vlan, fix this ip. Scoping else and backups are going smoothly. I no longer have to do backups every morning. Emails going slowly. And then my final thing as a sys admin was to verify that the database upload to the website was happening.
[00:36:26] Took about 45 minutes to upload this database of activity to the website. And then if there was ever an error during upload, I had to figure out what the error was, work with the SQL administrator and get it submitted. And this was most of your day? This was like, you know, five hours of, of your day. Okay.
[00:36:45] Well, I realized that they only had a 1.5 meg connection. And for an extra 50 bucks, we could move up to a 10 meg connection, no brainer. Now my upload took about a minute and so we were able to troubleshoot and get the, get the SQL upload every day in about five minutes to 10 minutes after like figuring the errors out.
[00:37:05] The rest of the day, I had nothing to do. So my boss actually said, you got an Alienware laptop play. Wow. Just be ready when we're ready for you. And so this, this horror story of like all these things that kept unraveling because an overconfident, sys admin built a network without knowing what he was doing, with having just a little knowledge, just enough to be dangerous and had crafted a little like.
[00:37:34] Workload and by the time I was done there, I had about a half an hour of work every morning. I felt like office space. And the rest of the time I'm just kinda sitting there just trying to help out around the, the company where I can, and eventually they're like, why don't we just hire a consultant to do this?
[00:37:47] I save a ton of money, so eight months into my job, I got let go for having no work to do. And got a great eight week severance package with full benefits, with a huge apology and a glowing recommendation.
[00:38:01] Jonathan: I mean, I was gonna say, this is a true horror story. I mean, this is a thing that people worry about, especially with automation all the time.
[00:38:08] Are you going to do this so well that you, you work yourself out of a job? And that sounds like the true horror. But then, I don't know, the way you're phrasing this at the end sounds like this actually isn't the worst thing in the world.
[00:38:21] Adam: Yeah. I
[00:38:21] Jonathan: mean,
[00:38:21] Adam: it was, it was nice, but I loved that job. Yeah. I liked the people I was working with.
[00:38:25] Yeah. And the horror of it was, was like, you know, discovering that, like when I, my brain made the connection that he did not know what cider notation was if you didn't understand something. This basic, I'm pulling at this thread and I'm like, this is a big sweater, guys. And like, it's unraveling. It's unraveling, it's getting worse and worse and worse.
[00:38:45] And at the end, like I'm just, I started having to come with the assumption of anything I see that's built, I have to assume that somebody built it without un, who was smart, but not smart enough to do research and
[00:38:57] Jonathan: understand. It's all of a sudden you don't trust anything around you. I mean, it's, it's kind of like moving into a home and realizing that the, the prior occupants had known enough to do some handy work, and then you find out, oh, wait.
[00:39:11] Hm. They've done the, they've done all the wiring. They've gotten into the plumbing. What else can't you trust? What am I living in here?
[00:39:21] \ I think the
[00:39:21] Adam: horror is knowing that there's people like that out there, is that you as we're all in one network, people, every IT person is in one physical network, we're all connected.
[00:39:38] Your neighbor might be that overconfident, sys admin who doesn't understand what they're doing and causing problems for you. And so if your upstream routing is not working. You're gonna have to call them and correct them, like knowing that there are people out there now, the, I feel like today it's, it's such a much more, much more mature field, right?
[00:39:57] We can filter out those guys and those people that are, they don't really know what they're doing. But throughout my career, I kept finding them and I, I don't wanna make people feel like idiots, but I do want to enlighten them.
[00:40:09] Jonathan: so having, I mean the, the, the maturity of the space. Having more people, sharing more and, and, and, and talking more together. Do you think that's the check and balance against that or is there anything else?
[00:40:21] Yeah, being honest,
[00:40:23] Adam: be honest, be a part of a community where say, Hey, I built this thing anybody's see any issues with, or, Hey, I'm having problems like this back and forth of like, you don't have to be the person who knows everything. And when you're in a silo and you're the only sys admin or you're the only person in your role.
[00:40:39] You can get a bit of an ego going because there's no one to call you out and if you haven't screwed up in the past year, you're probably either one delusional or two you. You probably need to have some people around you to call you out. I love it. You just don't know you've screwed up yet. 'cause we in it.
[00:41:00] There's too many ways to mess up. And that's why we have the fail fast methodology.
[00:41:05] Jonathan: Adam Walter, co-founder of Humanized It. Thank you so much for taking time to share your story and your wisdom.
[00:41:13] Adam: What little of it there is.
[00:41:16] Jonathan: Adam, thank you again. Really appreciate your time and I can't wait to hear another one of those stories. And just like that, we've reached the end of this week's episode and the end of season one of it horror stories.
[00:41:30] If you're up to the task though, you can relive every terrifying tale from this season on Spotify Apple Podcast or@ninjaone.com slash it. Horror stories. We can't thank you enough for tuning in week after week and joining us through this chilling ride through all sides of it. But our journey doesn't end here.
[00:41:49] Got a tale of your own. Share it with us on LinkedIn or creep into our Discord community. You might just find yourself featured on our season two. Until then, lock your doors, patch your systems, and remember, whatever your horror story, you're never in it alone.
[00:42:05] We'll be back with more soon. In the meantime, stay safe out there.