Rendered at 19:43:40 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
amluto 21 hours ago [-]
I personally operate a fairly advanced closed-loop chlorination system, and this is beyond screwed up.
These systems can easily fail, and they tend to fail in a way that causes them to measure low or zero chlorine and thus overdispense chlorine. More below on why.
First, this means that an operator must always ensure safety, both against excessive chlorine and against insufficient chlorine. In a commercial context, that means that a real human (or maybe a totally different sensor design, e.g. a colorimetric sensor) need to measure the pool chlorine level, preferably with a high quality titration (drop test) every day before opening the pool. They should probably record the result in pen on water resistant paper, but this part is icing on the cake.
Second, hoping that an email gets noticed is ridiculous. If you do not have positive confirmation that the pool is safe to open, don’t open it.
Third, the systems should be sized so that they cannot overdose quickly. The maximum dispensing rate should be such that hours of continuous operation would be required to reach an uncomfortable level, let alone a dangerous level. (Hot tubs are sort of an exception here.)
Fourth, everyone should be using a reasonable amount of CYA in their pool. Even at a lowish level of 30ppm CYA it takes a lot of chlorine to become dangerous unless the operator has also lost control of pH. I realize that some codes are confused about this.
Fifth, chlorine is smelly. Someone should have noticed, even just casually. Admittedly, lots of swimmers may be conditioned to stinky pools, and checking for safety isn’t their job. But the staff should notice.
Sixth, the systems I’ve seen beep when there’s an alert. It’s easy to add flashing lights, too. (In fact, the system I’m familiar with automatically shuts down if it feeds for more than X minutes without reaching the setpoint, and you cannot turn this feature off. You can, however, defeat it by switching the power to the whole controller.)
—- sensors —-
Actual free chlorine sensors intended for pools use an amperometric probe immersed in iodide solution or gel, separated from the pool water by a fancy membrane. Free chlorine oxidizes iodide to iodine, and the sensor measures the iodine concentration. If the membrane fails (or general wear and tear or excessive pressure swings) cause the iodide to be lost, then the chlorine can’t react and the sensor reads zero (or low if there’s tiny bit of iodide left).
Interestingly, these sensors are rather expensive devices that appear to mostly come from France and Germany. The Chinese vendors seem to be rebranding European parts (!), and the major US vendor is very clearly reselling a French company’s product.
There are some vendors who sell a cheap (or absurdly marked up) potentiometric ORP-based closed-loop system. In my considered opinion, these are worse than useless.
jug 8 hours ago [-]
Yep, my wife worked on a hotel in the past and they always tested for chlorine before opening, in person with equipment. It was simply part of cleaning services etc. And that was no big deal of a hotel either in a fairly small city, indicative of how commonplace this _should_ be and how much of a screwup this is.
If I were to guess, this is not the only problem that hotel has as it speaks of systemic issues.
malfist 21 hours ago [-]
> Fifth, chlorine is smelly.
Isn't it the chloramines that are smelly? Or at least non free chlorine. A strong chlorine smell is often a sign of a poorly cared for pool or spa
amluto 21 hours ago [-]
Chlorine and chloramines have different smells. I find chlorine to have a cleanish smell and chloramines to smell nastier. (I don’t know precisely what compound I’m perceiving in the chloramine category. I think that monochloramine is likely almost odorless in the concentrations found in pools (this is one of the reasons it’s used in water distribution), so dichloramine and nitrogen trichloride are likely responsible for the stink.
Plain chlorine ought to be noticeable before it sends people to the hospital.
misswaterfairy 9 hours ago [-]
> Plain chlorine ought to be noticeable before it sends people to the hospital.
One would have thought so, the story itself doesn't add up to me[1], perhaps it might be a translation issue?
Gaseous chlorine[2] systems for places like pools and hotels aren't all that common (I couldn't think of anywhere outside civil water treatment or major industrial), liquid chlorine (sodium hypochlorite [3]) chlorination systems are a lot more common because they're supposed to be relatively safer to operate.
Sodium hypochlorite is certainly much safer to handle compared to straight chlorine gas[4], but it will still give off the characteristic chlorine odour and will still happily hurt anyone not respecting it.
I wonder whether it was an absence of mind thing for patrons if the pool was indoor, the smell of chlorine tends to be stronger than outdoor pools? (enclosed setting, higher dosing given water warmth, etc.)
[1] Hazmat trained firefighter (though I am not a chemist)
This is only my anecdotal data from managing my spa. Chlorine is so reactive that I’m not sure I can ever smell it without it having reacted with something.
LoganDark 21 hours ago [-]
Is this why people say you only actually get the chlorine smell once someone pisses in the pool?
LtdJorge 19 hours ago [-]
If you've ever smelled the chlorine used in the pools, you know this is not true.
boie0025 13 hours ago [-]
So long ago, in another life I worked at a pool company as a backstockist/deliverist. I brought chlorine tanks out to cars, delivered it to houses and dealt with all matters of chlorine. I am not a chemist, and that was about 28 years ago, but I can tell you that all of the things I handled smelled like pool. I realize that a lot of what I was smelling was spilled chemicals onto dirty stockroom floors, but the times I did have to open a fresh container -- I can tell you -- chlorine smells like chlorine. That said, I am talking about highly concentrated death liquid; but having gone to many a site to deliver this chemistry.. I guess my point is that we don't need piss for our noses to detect the presence of chlorine. In my experience.
EDIT: I want to add that if you breathe that crap long enough, it really makes everything else smell really weird/awful/chlorine-y. Until now I never thought to look it up, but the after effects of being around chlorine for a while leaves a distinct olfactory residue, and it is awful.
julian_t 6 hours ago [-]
Definite agreement here! We used to have liquid chlorine in a lab where I used to work, so I'm familiar with that chlorine smell. Unfortunately that was by no means the nastiest stuff we had to work with (HF, anyone?)
thaumasiotes 11 hours ago [-]
> EDIT: I want to add that if you breathe that crap long enough, it really makes everything else smell really weird/awful/chlorine-y.
If you breathe chlorine long enough... won't much worse things happen to you?
You reminded me of nothing so much as The ICI Song:
And it's go, boys, go
They'll time your every breath
And every day you're in this place you're two days nearer death
But you go
There's overtime and bonus opportunities galore
The young men like their money and they all come back for more
But soon you're knockin' on and you look older than you should
For every bob made on the job, you pay with flesh and blood
mitxela 19 hours ago [-]
That's why they say it yes, but it's not actually true as chlorine reacts with all kinds of dirt which are not piss, such as sweat and dead skin.
4/5
Ygg2 10 hours ago [-]
Any organic matter, most common being bacteria and skin fragments.
KellyCriterion 9 hours ago [-]
Thats one of those responses which are the reason why I love to come to HN: On nearly every topic someone enters and says like "Im doing X as job, let me explain..."
0_____0 2 hours ago [-]
The way they worded it, it could be their backyard swimming pool with a fancy chlorine automation system they built?
wlonkly 20 hours ago [-]
> reasonable amount of CYA in their pool. Even at a lowish level of 30ppm CYA
The pool in the article clearly needed more than 30 ppm, they definitely did not sufficiently cover their asses here.
amluto 18 hours ago [-]
lol
CYA stands for cyanuric acid, not cover your ass :) It’s added because it reversibly binds to chlorine. This causes the majority of the chlorine in the water to be in a mostly inert state in which it’s not irritating, doesn’t offgas and is protected from UV. You need very little active chlorine to control bacteria, viruses and algae, but you need considerably more chlorine to avoid the water becoming locally dechlorinated due to any sort of contamination that consumes chlorine. CYA helps fix this mismatch.
pseudohadamard 14 hours ago [-]
They were using Windows for OT. That's all you need to say. OT done with Windows. The horror. The horror.
perpetualpear 14 hours ago [-]
At least some systems used in OT are on windows for decades now.
pseudohadamard 8 hours ago [-]
I worked on one of them some years ago. The project lead had a group spend something like six months on the design phase, before writing any code, planning out how to make it resilient and fault-tolerant and error-resistant. The finished project did that very well, to the extent that they then got a bunch of additional work from companies who had been involved in some high-profile failures on their own projects. So it can be done, but it typically isn't.
N19PEDL2 1 days ago [-]
This is 100% fault of the hotel, not of Microsoft. It’s unacceptable that the communications regarding such a dangerous thing totally rely on email, without any backup or double check.
DanHulton 23 hours ago [-]
Decades of “root cause analysis” poisoning have ruined our ability to think about safety and reliability holistically.
There isn’t one root cause here or one person or entity at fault. Many things went wrong and bad choices were made by many people and entities to lead us to this point, and they ought to be all regarded seriously and addressed rigorously.
- Whomever at the hotel that approved this as the sole (or even primary) alert system is at fault.
- The lack of training and experience this hotel employee had when in a position to make such a dangerous choice was at fault.
- Ditto the alarm company - everyone involved in designing, approving, and implementing this system is at fault, as well as whatever process ensured that nobody had the experience or safety to be able to speak up about this dangerous design is at fault.
- The regulatory system that allowed this design is at fault.
- The health and safety inspection system that didn’t catch this risk is at fault.
I’m sure there’s more; I’m just spitballing here. But the point is that there’s no one person or thing to blame here, and no one fix. This should kick off a wave of personal and professional soul-searching and improvements.
Alive-in-2025 23 hours ago [-]
I don't think we ruined our ability to think about things, this was a failure to build a reasonable system. It's just basic planning and trying to consider possible faults. If the only way you will be informed of a potentially deadly situation, a sensor reading out of range - that affects human health - is it okay if it is only via email???
What if the person who reads the email was gone that day? What if they forget to read it? What if the email system is down? What if it was lost? This is just careless and bad design.
At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".
It's not some tricky thing where you had multiple ways to detect it, giving some redundancy. Same thing if it was a text message only or a phone call only.
This is a poor safety design with a single failure mode on the notification.
trollbridge 21 hours ago [-]
We have a general design principle that we assume important emails will not be delivered, will not be read, and will not be actioned on.
helterskelter2 13 hours ago [-]
I generally ignore emails until I get bugged about something 2-3 times or the subject line is all caps.
danudey 2 hours ago [-]
I don't open my e-mail until someone tells me on Slack that they sent me an e-mail.
abustamam 2 hours ago [-]
Honestly most of my emails are either spam/marketing emails I haven't yet gotten around to unsubscribing from, or calendar invites which I generally ignore since they sync to my gcal and thus my slack cal.
If someone sends me an email, it will not get noticed. I think this is the general culture of most places I've worked at.
But even the slack alert channels get ignored by me because they're too noisy and we haven't yet prioritized denoisifying it. We aren't dealing with critical health alerts though
croon 10 hours ago [-]
I can tell the future, somewhat: If you ever tell anyone at work this, all your received emails will be in all caps.
andrewaylett 8 hours ago [-]
I worked in one place where explicit company policy was that you should resend emails after a couple of days if you didn't get a response the first time.
People had started sending calendar invitations for important emails, on the basis that recipients would then reply to the email and decline the invitation.
II2II 21 hours ago [-]
> At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".
More properly said, you may cause panic. People genuinely don't know how to respond when they hear an alarm in an unfamiliar place.
And that's if you're lucky. I once had to deal with people who did not want to evacuate a building when a fire alarm went off. And when I got them out the door I had trouble convincing them to leave the building's deck. Some people seem to think that it is okay to ignore a fire alarm, even a probable false alarm, in a wood structure with a wood deck.
db48x 19 hours ago [-]
It wouldn’t need to be that loud. Unlike a fire that becomes dangerous in mere minutes, this type of accident takes days to unfold. Most automatic pool systems will have a light or a beeping noise to let the operator of a failure. The email is just a convenience. The hotel staff should have been checking the system directly every morning to see if there were errors before they opened the pool for the day.
latentsea 4 hours ago [-]
>More properly said, you may cause panic. People genuinely don't know how to respond when they hear an alarm in an unfamiliar place.
It's a well known irony that every system intended to prevent accidents eventually itself causes one.
wongarsu 11 hours ago [-]
If you only need employees to react you can replace the rotating red light and klaxon with a smaller flashing white light that plays a pleasant but slightly annoying "ding" sound every five seconds
It just has to be annoying enough that even the most junior employee would radio in their supervisor to get it to stop
techiferous 22 hours ago [-]
Agreed. This is just poor system design.
tosti 12 hours ago [-]
[flagged]
shiandow 21 hours ago [-]
You don't even need something exotic for this to fail. A simple power failure would disable this system and most backups.
We've been through this with smoke detectors, it's not a new thing.
trollbridge 21 hours ago [-]
The system should be designed to fail-safe during a power loss. If it can't, then it needs to have appropriate battery (and possibly generator) backup.
hluska 16 hours ago [-]
Your analogies aren’t equivalent because physical security and health and safety are quite hard.
The power failure would presumably shut down the pumps and end in the pool being closed. Then pools should have a power outage protocol they’d follow before opening the pool again. Checking chemistry would be high up on the list.
Smoke detectors need backups because a fire can cause a power outage before there’s enough smoke to trigger the smoke detector.
YokoZar 22 hours ago [-]
Just because the alert signal is an email at one point in its journey from sensor to human doesn't mean you're relying on someone using a computer to see it. Automated paging systems can read email too.
Alive-in-2025 21 hours ago [-]
Yes, but we have to remember email is not guaranteed to be delivered, it's not reliable. You cannot in general tell if someone even looked at it (people try to build systems on top to provide these things, but it's not universal).
Also, I have a personal domain, some sites decided its not real enough and I can't get email from them.
Email is just not reliable. It could be part of a combined system of other things that are reliable.
account42 7 hours ago [-]
EMail is not inherently any less reliable than any other Internet-based communications protocol. The unreliability myth mainly comes from systems and people actively deciding to drop your mail. That's not unreliability, it's an operator choice.
Bost 7 hours ago [-]
Exactly. We had a customer whom we regularly informed about important events by email. Two or three times a year, an email arrived a few days late, and nobody ever figured out why.
hdgvhicv 21 hours ago [-]
You’ve got a chain of multiple single points of failure. That’s unacceptable.
YokoZar 20 hours ago [-]
Of course. But it's bad for that reason, not because of a presumed requirement for a person looking at an outlook inbox like OP implied.
delta_p_delta_x 20 hours ago [-]
> Decades of “root cause analysis” poisoning have ruined our ability to think about safety and reliability holistically.
This is not the case in aviation. Even in the most egregiously obvious single-cause failures, there's always an analysis of issues in the wider system that allowed that single cause to become the single cause in the first place.
I agree with your points, though. In this case, Microsoft absolutely takes shared responsibility. It is only a matter of time until a similar outage in some other wider networked system with a so-called 'five/seven/nine nines uptime guarantee' goes out for a couple of days, and takes with it some critical life-sustaining system, and people die.
fransje26 9 hours ago [-]
> This is not the case in aviation.
And yet, as late as yesterday, the idiots in charge have announced that there is absolutely "No Need To Investigate the FAA Ground Stop".
thephyber 18 hours ago [-]
This seems wrong (obviously I don't have any more info than you).
Ford doesn't build all of the modules of their cars. They design specs and outsource manufacturing. But legally speaking, Ford is responsible for design, testing, and support / recalls. It's their responsibility to contract in the terms that both protect their customers' safety and Ford's liability due to spec / contract violations.
When your company is responsible for the health of staff and guests while dealing with deadly gasses, it's your company's responsibility to ensure that all of your vendors detail how their failure modes fail safe.
This is the same exact logic that has been migrating software companies to detail their risk by demanding their vendors all supply a "Software Bill of Materials" (which is very difficult in practice, but is necessary for the downstream users to protect themselves from upstream risk).
gregglain 23 hours ago [-]
First this is so sad and preventable. I would be interested in the "follow the money" on this.
notatoad 17 hours ago [-]
the point of root cause analysis isn't to find the one person or entity at fault. the point is to determine what you can fix, so this doesn't happen again next time.
anything that was "at fault" that you can't do anything about doesn't matter. and you can't do anything about a microsoft outage, all you can do is design around it.
Because the are too much alive. AI robots cannot get gas poisoning. Case closed!
thaumasiotes 10 hours ago [-]
"Five Whys" is never going to mean anything as a methodology. It works if competent people use it and it fails if incompetent people use it. But pretty much everything works if you're free to assume that everyone involved is competent.
The point of a process is to constrain incompetent people into doing something that will work despite their incompetence. Five Whys is entirely subjective and can never constrain anyone; it's just something that some people think sounds deep.
perching_aix 22 hours ago [-]
Pinning a fault to a single entity has great cultural tradition. The number of times I witness people being just completely unable to process the idea of shared responsibility, or the difference between having an ability to affect something and having culpability to affect that thing, is staggering.
ponector 21 hours ago [-]
In real life shared responsibility means no one is responsible for it.
perching_aix 21 hours ago [-]
Hah, fun that you bring that up. Just had quite the struggle articulating to management a few weeks ago that "shared ownership" runs contrary to the very idea of ownership, which I guess is basically the same topic just with different words. Either we take shared ownership seriously and actually organize to form consensus on things, or it's just bullshit. Naturally, did not really lead anywhere. So you definitely have a point there.
ponector 11 hours ago [-]
Shared ownership is perfect: if project succeeded you can claim you've contributed heavily to the success. If project failed - you did everything toward success, but other team members failed.
3 hours ago [-]
toofy 23 hours ago [-]
> This is 100% fault of the hotel, not of Microsoft.
yeah, these companies need to be told to stop relying on cloud services for everything.
its wild to me how many critical things that absolutely do not need the internet are relying on the internet.
much blame to go around here:
- whoever designed the stupid alert system.
- the marketing departments in countless companies ignoring engineers when they market to normie customers “connect everything”
- the hotel company for seemingly doing no dilligence at all. and just blindly listening to some salesman “connect everything”
- accountant types and their “we can save pennie’s if we cut all of these corners.” and the stupid corners they choose to cut.
- etc…
on a macro scale it’s amazing to me how many people are victims to slimey car salesman types leading to situations like this.
mitxela 19 hours ago [-]
It's unlikely that the hotel themselves manages the pool. They pay a pool company to keep their pool running.
5/5
18 hours ago [-]
globular-toast 13 hours ago [-]
It has nothing to do with the cloud. For safety critical systems you don't rely on any single thing, cloud or otherwise.
raxxorraxor 12 hours ago [-]
True. This is why the cloud should never be a requirement for anything safety critical. Or if you make that bad choice, design your system with a sensible offline operation mode.
Regardless of high availability promises of cloud services, a local internet outage will supersede the risk and is a risk factor in most places of the world (probably everywhere). Risk that nobody aside the local ISP could quantify either. So logically any system critically depending on the cloud and isn't designed with an offline fallback has unmitigated risk factors.
Sure, if you host some online service, this risk cannot be mitigated. But it can for everything else. And if you need an offline operation mode, in most cases the non-cloud design is simply superior. So cloud is synonymous with having a single point of failure?
This case is one where on risk analysis was done at all or this would have surfaced pretty quickly. There are problems too if a pool doesn't get chlorinated, but it is probably better than to poison your guests.
mrweasel 11 hours ago [-]
Also in this case, it's people completely misunderstanding email. Way to many basically view email as a messaging platform, but with a slight lag. I'm apparently old, because I remember reading that emails could take seven days to arrive and that would be within spec.
Knowing that your email could be delayed by days changes how you view it and the applications you deem it suitable for.
nullsanity 19 hours ago [-]
[dead]
ashleyn 24 hours ago [-]
This seems like the kind of thing which should have an audible alarm mounted on the wall in the pool room.
_trampeltier 22 hours ago [-]
And then the horn or the cable is broken when you need it.
So you need a redundant system for important things.
sparrish 21 hours ago [-]
Then you test the horn and cable daily (maybe even hourly if it level could become toxic that quickly) to make sure they are working.
Similar to doing regular restores of your data backups. If you never test a restore, you don't have a backup.
_trampeltier 14 hours ago [-]
Yes, things like cable loop test for such cases exist since ever. But what do you do if it fails? Send an email?
namibj 11 hours ago [-]
Use the hot spare cable loop to yell at people.
No occupancy without two working ones; this means you want a third so you can do maintenance without having to evict occupants first.
InterlooperX 11 hours ago [-]
That, and gauges telling you what's what and Dashboards mounted and so on and so forth.
I did an apprenticeship as an electrical engineer and have been in those catacombs underneath such water park type of things.
There should be such devices present, and of course personnel to operate them.
I mean, can you imagine your boss' face when you tell em the reason your entire system is down is you didnt get the email from checkmk that something is wrong?
Instead of, you know, checking the dashboard yourself periodically.
croes 24 hours ago [-]
[flagged]
s0rce 24 hours ago [-]
Facilities I worked in that had chlorine gas had audible alarms if leaks are detected because everyone could die.
windowsrookie 24 hours ago [-]
So what if people panic? People panicking because of an alarm is better than being poisoned...
ssl-3 22 hours ago [-]
Whenever chlorine gas is attacking people, whether it is deliberate or incidental: Yes, I want panic.
cowsandmilk 23 hours ago [-]
They literally had to call the fire department and send 4 people to the hospital. That is what causes panic.
mastax 24 hours ago [-]
How do you feel about fire alarms?
RetroTechie 22 hours ago [-]
Most o/t time they're just annoying. But if I'm in a building & hear one, I will ALWAYS check where it comes from, and whether further action (like evacuation) is needed.
It's easy & the 1 time you don't, can cost liveS.
croes 23 hours ago [-]
Had a fire alarm on my last vacation.
People ignored it.
A message to the fire department would have been the better idea
fc417fc802 23 hours ago [-]
People just panic and do ridiculous things like evacuating the apartment building at 2 am in their pajamas in the rain, blocking the road. Meanwhile 9 times out of 10 it's a false alarm. We should instead email the fire department while the residents keep sleeping. /s
asgraham 22 hours ago [-]
You’re being sarcastic but that’s a weirdly popular view.
I called 911 once because of two CO monitors going off in my top floor apartment of a mid-rise. Half a dozen engines responded including hazmat, everyone’s standing around outside the building. FD had to break down the door of someone who wasn’t home because there was no response. Somehow it got around that I had been the one to call me in. Some guy came up to me saying something like “some of us have jobs in the morning. I hope you’re happy, moron.”
FD found a leak in the basement furnace. Without the alarms, everyone in the building would’ve been dead by morning.
ssl-3 22 hours ago [-]
Good. When they're outside in their PJs standing in the rain and blocking the road like complete dullards, then: They're not in the building, and they are neither on fire nor breathing chlorine gas.
Dylan16807 19 hours ago [-]
Yes, that is the point of the comment you responded to.
ssl-3 17 hours ago [-]
It is.
Is there an obligation to always be disagreeable here?
Dylan16807 17 hours ago [-]
Agreeing is fine if you have something to add.
But a "Good." to that kind of sarcasm looks like a lack of understanding, and then you rephrased the joke as a non-joke with nothing extra. If that's the full intent, I'd say just upvote.
ssl-3 16 hours ago [-]
I see.
So what's the value of the contribution of your own comments in this thread?
Dylan16807 14 hours ago [-]
Letting you know.
It's not much value but if it leads to better outcomes in the future then that's a success.
ssl-3 2 hours ago [-]
Okay, chief.
garyfirestorm 23 hours ago [-]
You want a panic? That’s how you get a panic /s
shepherdjerred 23 hours ago [-]
That's exactly why I deleted my team's metrics, dashboards, and alerts
skeledrew 23 hours ago [-]
So it's better to have deaths rather than panic?
DougN7 23 hours ago [-]
Panic seems like exactly what you want in this situation.
greggsy 24 hours ago [-]
Exactly. Email is also a ‘zero guarantee’ service, and should never be relied upon for life or death solutions.
mikeryan 23 hours ago [-]
Not just that, in this case there was no way for the alert system to even know that the emails were delivered. I can’t speak to everyone but for my company emails were not bouncing, I just got them 12-24 hours late. At least SMS and the like know if the message reached the actual end recipients inbox.
InsidiousWombat 22 hours ago [-]
SMS is also zero guarantee service
_bernd 21 hours ago [-]
Please chill out. Chlor Level in a swimming pool is now where life and death or as the other user here noted, the user of the pool should be able to notice that something wrong by them self.
You still have think a little bit for your self to prevent self harm.
JsonDemWitOster 10 hours ago [-]
Usually with these things you notice that something's wrong because you've already been harmed. The only thing self-preservation does is _mitigate further harm_. Note: "harm" not "dead". The people injured may very well have gotten off the pool when they noticed.
Reformulated: If pool management's cue that something's wrong is because their guests are leaving the water, they are already liable for damage.
ssl-3 20 hours ago [-]
15 people had respiratory issues. 4 were sent to a hospital. 1 was kept there.
> the user of the pool should be able to notice that something wrong by them self.
Clearly, their motivations for autonomous self-preservation were insufficient.
_bernd 12 hours ago [-]
oh you sweet usofa boys.
Just because the ambulance brings you to the hospital for monitoring doesn't make your condition life threatening nor gives you bankruptcy.
I still hold the point. To much chlorine in the water should be noticable otherwise I would argue you shouldn't even be allowed or able to buy a ticket because your are unable to perform proper choises.
cbarnes99 1 days ago [-]
It's the fault of the company that designed the automatic alarm. Not the hotel. But it is accurate to say that the cause of the issue was the outage.
buran77 24 hours ago [-]
> It's the fault of the company that designed the automatic alarm. Not the hotel.
The hotel picked a monitoring solution which relies exclusively on email, or configured it like that. If anything they are absolutely both to blame but for different things. The company for designing a safety related system like this, the hotel for selecting an inadequate solution/configuration thus allowing their guests to be harmed.
> But it is accurate to say that the cause of the issue was the outage
The cause of the issue is the more immediate one - the configuration of the monitoring system. This avoids going up the causality chain to far - shitty coffee caused engineer to take down email system, and such.
zardo 24 hours ago [-]
Email going down is the proximate cause. A system configuration relying on a single unreliable communications route is further up the causal chain.
buran77 21 hours ago [-]
Let's not forget that we're talking about people getting hurt, that's the core topic here, yes? The most reasonable immediate cause for the people getting hurt is that there was too much chlorine in the water. Because the dosing system pumped too much chlorine AND the monitoring system failed to alert. Because it was set up to only use email AND the email failed. Because someone deployed a wrong config on MS's cloud infra. And so on.
Not one but two of the hotel's systems failed to operate properly. The chlorine pumping system pumped too much chlorine and the last resort alarm failed to alert.
As I said, there's plenty of blame to go around as a matter of general principle. But for what happened here the hotel bears the responsibility towards their guests, not the pumping or monitoring system developers, not Microsoft. The only reason MS in the title is because that failure is big news these days.
On a technical note, even if they had a backup email provider they'd still be screwed because the people getting the alert were probably employees on the single provider (back to SPOF). SMS could have been a good parallel option. Ideally also some physical control panel that triggers an alarm when any safety critical condition occurs. On the chlorine pumping system, that shouldn't be a dumb preprogrammed pump, it should come with it's own sensors and detection loop independent of any external system, something that proactively stops it before it breaches the safety limits.
stackskipton 23 hours ago [-]
Agreed, I'm not sure how it's acceptable to say "Email is exclusive option you can have" vs "We can email you when alarm is going off but light/siren will flash and it must be visible to those using the facility"
Dylan16807 19 hours ago [-]
It's not the alarm that was dumping in dangerous levels of chlorine. Microsoft is in the line of causes to the lack of warning, but not the actual poisonous conditions.
ktm5j 22 hours ago [-]
I feel like there are things we aren't being told here.. what kind of automatic chlorine dosing system both knows that there's too much chlorine and continues to add enough chlorine to the point of being harmful. Something is fishy here...
mmoll 13 hours ago [-]
The control loop probably used email for messaging.
ktm5j 5 hours ago [-]
Doubtful.
sokols 24 hours ago [-]
Well somebody has to sign off the solution, in this case the hotel.
ithkuil 23 hours ago [-]
It would be accurate to call it the proximate cause.
Using the unqualified "cause" is misleading click bait and in some places it could get you sued
spunker540 22 hours ago [-]
Isn’t the cause whatever sprung a leak in the first place? Ie a faulty pipe or valve?
01HNNWZ0MV43FF 24 hours ago [-]
We can assign fault, or we can ask which levers we can pull to make it better:
1. Can there not just be a light / siren right next to the gas sensors? Easy to fix, but it requires hardware changes on-site.
2. Could Microsoft have not had an outage? Hard to fix.
3. Could they use some better notifications systems like idk self-hosted ntfy.sh with a warning if the server goes down? Yes but nobody wants to pay for that.
croes 24 hours ago [-]
A light has to be seen, a siren has be heard.
I bet email has a better chance of alarming the right person.
And you have to check those lights and sirens that they still work.
Everybody knew email was down.
The real problem is nobody thought of the consequences regarding the alarm system
stackskipton 23 hours ago [-]
Generally, for sensitive safety systems like this, Light and Siren are visible to those using the facility. So if problem developed, all patrons would be going "Umm, you know your sauna has flashing red light and siren going off?" after they all left the facility.
Yes, they still have to work but first off, both of those are extremely reliable systems, alarm circuits have extremely low risk of failure. Second, testing them takes 5 minutes a week/month, press the button, look for flashing light and listen to wailing noise.
It's inexcusable to think that email/slack or whatever is acceptable as anything other than notification system so people not in facility can be notified that alarms are going off.
EDIT: You are defending this setup and I'm not sure why. This seems email only setup seem indefensible.
croes 23 hours ago [-]
Because we don’t live in an ideal world.
Testing requires staff and you can be sure those parks are understaffed.
Best thing is to avoid them, next best thing is a cheap solution with low maintenance by the staff to notify as much people as possible.
Email wins.
Not a good solution but the best you will get for the price they are willing to pay
ssl-3 22 hours ago [-]
Things do break, so all alarms need periodic testing, whether they consist of email and/or something else.
Whether it breaks because a relay has corroded due to years of service in a chlorine-enriched environment, or because O365 is down, or because O365 tightened up on DMARC requirements: It needs tested.
And it's not a binary choice. This isn't the World Cup, the Super Bowl, or Highlander. There can be more than one.
The alarm system can wear a belt, and suspenders. It can send emails, and it can flash a bright light, and it can also sound an audible alarm.
icehawk 19 hours ago [-]
> A light has to be seen, a siren has be heard.
Curious: what do you think has to be done with an email?
JsonDemWitOster 8 hours ago [-]
In the English language, the verbs usually associated when "email" is the direct object are, in no particular order: "read", "open", "write", "delete".
Sarcasm aside, in this case, the email should've triggered the literal lights and sirens. Preferably automated (tho in that case, why even go through email as middleman) but could also be an actual person flipping a switch.
You see this in software ops too: certain subject formats trigger a special ringtone or even a pager call. Emails are too subtle even with ALERT ALL CAPS SUBJECT LINES to be alarming.
al_borland 21 hours ago [-]
It's easy to say this in hindsight, but how many pools/hot tubs have employees monitoring an alert panel around the clock for water quality metrics of the water?
Countless systems are relying on email as a means to alert people to issues, for better or worse. Having a physical alarm for everything is rarely practical for a myriad of reasons.
I worked in a data center 20 years ago that had alarms on the wall for when the halon was preparing to be released for fire suppression. There are likely dozens of things that could be monitored, but it was only the catastrophic thing that once it started, there was no stopping it, that got lights on the wall.
Too many loud alarms make people start ignoring alarms. I would guess that there is a fairly large margin of error in pools before there is a health risk, so it likely wasn't deemed alarm worthy. As someone who's been in a pool before, I'd also imagine that people would notice and say something, or it would be unpleasant enough to not want to spend time in it. That's what I find most surprising... that people weren't immediately turned off just being around it, to the point of staying out.
bmitch3020 20 hours ago [-]
On every public pool I've ever been to, there was at least one lifeguard on duty. And part of that guard's duties included manually checking the chemical levels. Automation is great, but there still needs to be a human present that can verify those systems are working at the start of their shift, and to be somewhere they can hear or see any alarms if they go off.
swolios 14 hours ago [-]
almost all hotel pools have a sign " no lifeguard on duty " as no-non mega corp hotel (and even those dont) have lifeguards.
Maybe a waterpark resort, but hotels no. These days they barely afford a front desk employee and a cleaner.
francisofascii 3 hours ago [-]
Right. I'd rather have a "no lifeguard on duty" pool over no pool at all, due to liability issues.
macNchz 1 days ago [-]
Seems like if the system can send an email it should be able to, like, illuminate a red light somewhere or something, if what it’s doing can create an actual safety risk.
madaxe_again 23 hours ago [-]
This gives me an idea for installation art:
Title: 5 nines of uptime
A grand piano is suspended from the ceiling with an electromagnet.
The electromagnet works as long as there is power.
No special battery backups, anything.
Guests are invited to stand underneath it, after signing a waiver.
That’s it. That’s the art.
Dylan16807 19 hours ago [-]
Is the joke that the name is wrong? I'm pretty sure no power grids reach 5 nines of uptime.
madaxe_again 10 hours ago [-]
Of course. And also the TOS that nobody will read and will just accept.
Dylan16807 9 hours ago [-]
Eh, a waiver is perfectly reasonable for something that's slightly dangerous.
juujian 23 hours ago [-]
Please do it, I would spend ~4$ to do that.
crazygringo 22 hours ago [-]
Seriously, what's next -- hotels which decide that smoke detectors ought to e-mail the front desk rather than blaring a loud alarm?
I'm surprised an e-mail only alert is even legal for this. Or is chlorine level monitoring not something that is regulated?
trollbridge 21 hours ago [-]
100% correct. An email service is not to be used for critical alerting and communication. There are other systems that are up to the task - including infrastructure built on Azure which could have the required level of fault tolerance in it.
jrm4 23 hours ago [-]
The law is smarter than this, thankfully.
Your second sentence is 100% correct. Your first is questionable; Microsoft, a kabillion dollar company has, for years, held itself out to be reliable and safe. And it repeatedly has shown that it is not. That opens them up to some liability here; and frankly deserves to be better held to the standards it (either directly or indirectly) claims.
And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.
teraflop 20 hours ago [-]
Well, notwithstanding the fact that this incident happened in Germany...
> And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.
"Implied warranty of merchantability" is a term of art in United States law, and it applies only to tangible goods, not to services.
jrm4 6 hours ago [-]
That's why I said "e.g."
The greater concept is that the entire point of law is so that you CAN'T just wave away things in fake contracts.
Hizonner 19 hours ago [-]
What exposure I have to German law leads me to suspect that their equivalent concepts are not forgiving on this kind of issue.
damowangcy 15 hours ago [-]
Outside tech, the further you are away from Silicon Valley (hint: not physically but industry wise), the more common it is to see such design of systems. Why? Because technology is a cost and IT department is a cost center which doesn't increase revenue.
When an incident like this happened, they might add new "system" to improve the situation, probably in the lines of having someone to patrol and check off a list routinely.
sieste 11 hours ago [-]
"Sorry all those kids burned to death on the rollercoaster, the fire alarm must have gone to spam."
freitasm 1 days ago [-]
Came here to say this. A push notification to an app or Slack/Discord. Or even a light.
I know the S in IoT means safety/security, but there must be things that can be done for urgent notification.
temac 22 hours ago [-]
> A push notification to an app or Slack/Discord
Wtf?
I must have missed the memo where Slack Discord or “apps“ became suitable for safety critical task. Must be what USA has tried to use just before grounding all planes.
croes 24 hours ago [-]
I doubt they have Slack or Discord and I bet there is nobody who constantly watches a light.
You know what they all have? An email account
xyst 24 hours ago [-]
hindsight is 20/20.
I do agree with this sentiment. At the same time, it’s still unacceptable for a global service like MS365 to go down for extended periods of time. I wouldn’t be surprised if the hotel chain was sold on some MS guarantee that it would never become unavailable.
Although I would also believe a scenario where the hotel chain "IT” (aka the owners son or grandchild) just used default alert settings and didn’t bother to setup fallback alerting mechanisms.
In short, all parties suck here (MS and hotel)
Espressosaurus 24 hours ago [-]
Hindsight is 20/20? Bullshit. Safety critical systems should not rely on transport layers that are guaranteed to fail. Email especially! Physical alarms are the way here. The email should be for convenience and to inform higher-ups.
reaperducer 24 hours ago [-]
Safety critical systems should not rely on transport layers that are guaranteed to fail.
Go invent a transport layer that will never fail, and make yourself rich.
Because right now, there is no such thing.
Espressosaurus 24 hours ago [-]
Everything can fail even down to individual capacitors on the board but that doesn’t make the failure probabilities or failure mode equivalent to something like email that is best effort only and can be actively blocked by things like spam filters that aren’t even under your control
Rygian 23 hours ago [-]
"Loud noises and flash bright red lights" is an industry- and regulations-accepted transport layer for a life-or-death signal.
Email is not.
reaperducer 22 hours ago [-]
"Loud noises and flash bright red lights" is an industry- and regulations-accepted transport layer
And yet, still not "guaranteed," which was the criteria specified by the OP.
Rygian 3 hours ago [-]
If you really want something guaranteed, your only choices are "death" and "taxes", as they say.
cwillu 21 hours ago [-]
Please quote where the OP said “guaranteed”; all I see is a claim that email is guaranteed to fail.
cwillu 23 hours ago [-]
Audible alarms, failsafe heartbeats and scheduled networking are not new technologies.
reaperducer 22 hours ago [-]
Audible alarms, failsafe heartbeats and scheduled networking are not new technologies.
And yet, still not "guaranteed," which was the criteria specified by the OP.
sham1 13 hours ago [-]
I know this is HN and all, but you're being way overly pedantic about the "guaranteed to fail" phrasing of the person you originally replied to.
Yes it's true, no system is guaranteed to function, but for Pete's sake, it's clear to everyone that having only an email-based alarm system for chlorine levels is the wrong thing to do just because "well there's no alarm system that is guaranteed to alert people, so let's rely only on email, tee-hee".
Any redundancy would have been better than this. People's lives were at risk, and here we are, debating the meaning of the word "guaranteed". I'm going to be frank here, are you an MSFT shareholder or tied to the hotel in question?
bketelsen 23 hours ago [-]
I'm not even going to read the article. If your hotel depends on some cloud service to prevent chlorine gas poisoning you're doing something terribly wrong. Full stop. I don't know anything about the laws in Germany but this is dangerously close to libel in the US.
Disclaimer: I work in Azure, this is not a defense of the outage and I didn't read the submitted article.
itopaloglu83 22 hours ago [-]
I don’t understand why someone would design a pool system that can harm swimmers with chlorine and the only safety is an automated email notification, and then blame the email provider and not their own system design for it.
fasterik 22 hours ago [-]
Why is it libel? "Outage leads to poisoning" is a factual description of what happened, not an assignment of blame.
sgustard 20 hours ago [-]
The actual subhead translated from German reads "Due to a Microsoft outage on Friday, Phantasialand was unaware that there was too much chlorine in the pool." That seems like an accurate statement of events. Not that this was competent system design, but that as designed it depended on an email being delivered.
aziaziazi 25 minutes ago [-]
That’s fair, however the choice of (accurate) statement leads to an understanding or another:
> A cyclist died on the road. He wasn’t wearing a helmet.
> A car blew through a stop sign and ran over a cyclist.
verst 12 hours ago [-]
The only thing libel-like was committed by user DeepLogin who editorialized the headline too much. The WDR article does not insinuate blame.
aeonik 21 hours ago [-]
Because the outage didn't lead to the poisoning. Operator negligience and poor design did.
If I built a chlorine tank out of toothpicks and duct tape, and it leaks everywhere and kills people, you don't publish a headline that says DuctTape and Global Toothpicks Inc are responsible for chlorine deaths.
fasterik 21 hours ago [-]
"A leads to B" means that A resulted in B. It doesn't imply that A was the root cause of B or deserves blame for B. For example, "heavy rainfall leads to flooding" can be true as a factual statement, even if the blame goes to poor drainage infrastructure. Also, the headline is in German and translates as "15 guests injured in Phantasialand".
Steve16384 9 hours ago [-]
"Charles Babbage's invention of computer leads to chlorine gas poisoning". How far back do you take the connections?
Ukv 6 hours ago [-]
I don't think "leads to" itself has any time or causal chain length limit. You're free to say that a choice your ancestor made millions of years ago lead to you being here today, for instance.
For events that typically don't happen, I feel it's relevant to give the factor that changed leading to the event happening specifically now. Standard background information (the invention of computing, the earth continuing to orbit the sun) less so.
fasterik 2 hours ago [-]
It's a ridiculous example. Obviously, the usage depends on context. I don't understand why people are so hung up on this.
eluru 18 hours ago [-]
Being that the case, the tittle should be 'badly designed system leads to poisoning'
dominicrose 9 hours ago [-]
We rely too much on technology. It is ridiculous that freezers can be hacked to destabilize a country by defreezing its food.
A freezer is an obvious example of something that should be controlled directly by a human operating on it and not by some "smart" connectivity.
dmix 22 hours ago [-]
There's no details in the article so you didn't miss anything. TLDR
> So no one noticed that the maintenance safety system of the sauna landscape has long since sounded the alarm by mail. Because the automatic dosing system had pumped too much chlorine into the pool.
ipsum2 13 hours ago [-]
Yes, living in ignorance is the correct solution!
DANmode 23 hours ago [-]
> I work in Azure
I’m so sorry.
Havoc 19 hours ago [-]
Comments seem to be losing a fair bit in translation so rough translation:
* 13 guests complaining about irritation of airways (+2 later)
* 4 taken to hospital, 1 described as medium injury, rest discharged
* Dual failure - dosing system had a defect (dosing too high) and the email of the monitoring failed (M365). Article sounds like it's two distinct systems.
Overall the German text sounds a lot less dramatic than the hn title.
tosti 12 hours ago [-]
[flagged]
zahlman 8 hours ago [-]
The joke wasn't any good the other two times you made it in this thread, either.
tosti 7 hours ago [-]
Sorry, they practically wrote themselves.
discordance 1 days ago [-]
I know it’s fun to blame Microsoft for things, but that is a poor and dangerous design.
Anything that could result in that sort of harm should have a more resilient and redundant architecture.
> Due to the (Microsoft) disruption, no emails were received, as the company informed WDR on Saturday. As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool.
jamal-kumar 23 hours ago [-]
excuse my language but holy shit thats fucking stupid. email? for chlorine gas alerts?
_bernd 11 hours ago [-]
They probably have some industrial device which is mostly only capable of sending alerts via smtp.
And if the amusement park uses office 365 for their email accounts then we'll the stuff was not informed. That nobody checked it manually like it is done with toilets and bath rooms makes me wonder too but what surprises me that most dudes here have ne er seen normal industrial devices which are all shit and look like stuck in the 90s regarding their UI and UX and their technical features.
If you even get snmp besides smtp it's real luxury.
Most of theses devices do not even have an TCP IP based API. All you get is serial connections.
And if the device is 20 years old by now then expectations should be even lower.
Just aa few days ago in German media there was the story of an chicken egg producer and seller and his egg sorting machine still runs by a controller PC with win95. Nuff said.
Saris 23 hours ago [-]
I would have thought that would be a loud alarm and flashing lights or something, not an email!
user_7832 14 hours ago [-]
Reminds me of a particular email...
> Subject: Fire
Dear Sir/Madam,
Fire! Fire! Help me 123 Carrendon Road. Looking forward to hearing from you.
All the best,
Maurice Moss
labrador 23 hours ago [-]
Chlorine gas is no joke. In high school I was making it by running electricity through salt water and then diffusing the gas through pool water to chlorinate my parents pool. My cousin took a whiff when I was distracted and had to go the emergency room for breathing problems. Fortunately he recovered fully.
thor-rodrigues 22 hours ago [-]
Why?
Like, what leads a teenager to do such things?
_ink_ 21 hours ago [-]
Curiosity. I picked apart that component that made my shoes blink. It was very difficult to get it open, but the silver liquid in it was fun. Today I think that was mercury and that I probably shouldn't have done that.
spunker540 22 hours ago [-]
Are humans born with innate knowledge that some things are deadly to smell? Or are we born with an instinct to smell things to figure out if it’s good or bad?
matheusmoreira 21 hours ago [-]
Chemistry is literally the science of making deadly substances, smelling them and writing down one's impressions of the odor before the onset of death.
This is a funny question on a forum, that's called "hacker news"...
nsoam 22 hours ago [-]
What, you’ve never been a teenager?
marshray 21 hours ago [-]
I did plenty of experiments as a teenager, but I never attempted to chlorinate a freakin swimming pool.
voxelghost 20 hours ago [-]
> ... but I never attempted to chlorinate a freakin swimming pool.
Me neither, but that is probably attributable to the fact that I didnt have a swiming pool.
rapnie 21 hours ago [-]
Perhaps for the same reason a kid [0] built a nuclear reactor in their backyard.
> With one experiment, he created chloroform and as the book encouraged him to sniff the chemical, he did so and was unconscious for more than an hour, according to his recollection.
Back in the day, i tried to sniff NH3 because i've heard it smell like piss.
It's a lot of worse.
matheusmoreira 21 hours ago [-]
Volunteer in an hospital sometime. You will see some shit.
spaldingcactus 22 hours ago [-]
Maybe the information is being lost in translation, but I'm trying to understand where the "chlorine gas" is being talked about. The articles I found just say "too much chlorine", which is hazardous but not the same as chlorine gas. Can anyone shed some light here?
amluto 17 hours ago [-]
There are a couple of pathways by which free chlorine (hypochlorous acid / hypochlorite) can turn into chlorine gas. The kinetics are complicated, and I’m not sure a fantastic model exists. In general it seems to be faster in acidic conditions and also faster if air is bubbled through the water.
High chlorine levels tend to reduce chloramines, so gasses like nitrogen trichloride seem unlikely to be the problem. You need a nitrogen source to make any large amount.
You favorite AI will happily model it, and I wouldn’t trust it to get the right answer :)
zahlman 8 hours ago [-]
> You favorite AI will happily model it
Really? I would expect to get refused or downgraded for "bio".
throooooo 22 hours ago [-]
I don't think "chlorine gas" but rather chloramine. In pools when chlorine binds to contaminates, it forms chloramine, which is a gas that sits on the surface of the pool.
spaldingcactus 22 hours ago [-]
I was thinking that seems more plausible. Especially where pool patrons have bad pool etiquette. I didn't want to steer the conversation in the event I got it wrong.
ChrisRR 6 hours ago [-]
As someone who works in medical devices, if the risk was genuinely that high then the notification shouldn't have only relied on email. Even if the emails hadn't been working, then what if no-one had been reading their emails.
The indication needs to be proportionate to the risk, and so if there is a high risk of harm, then the monitoring device needs a auditory and/or visual alarm as well as the email notification
7 hours ago [-]
verst 12 hours ago [-]
That's a very editorialized headline here.
Actual headline:
"15 Gäste im Phantasialand verletzt" (15 guests injured in Phantasialand)
And the browser tab shows the following summary title
"Chlor: Phantasialand bekam wegen Outlook nichts mit" (Chlorine: Phantasialand didn't notice anything due to Outlook)
Animats 15 hours ago [-]
Phantasialand is a sizable theme park with a lot going on. They must have a control room or operations center to monitor all their systems.
There are companies which do that kind of integration for theme parks.[1]
Disney does their own integration.
Did someone not wire the pool system alarms back to the control room?
This is usually a problem for a hotel/motel too small to have a control room, but big enough to have escalators, elevators, HVAC, pools, laundry equipment, fire suppression, water heaters, and other gear that needs to be monitored. What usually happens is that some manager's office or security guard station has a bunch of unrelated alarm panels.
That's where medium-sized problems get lost.
The next time you walk by a fire alarm panel in an office building, look to see if it's displaying any faults.
> During the incident, some of the gas monitors installed within the Yenkin-Majestic facility detected the flammable solvent release approximately one minute after the release began and sent an email to an offsite employee, but the monitors did not sound audible alarms to warn employees of the hazard.
1 dead, 8 non-fatally injured.
sharts 14 hours ago [-]
Sounds like incompetence lead to gas poisoning
tosti 12 hours ago [-]
[flagged]
WalterBright 20 hours ago [-]
Dangerous systems should always fail-safe.
ctrlaltdileep 23 hours ago [-]
its interesting in the sense that emails have sort of become the de-facto standard for sending alerts and keeping paper trails. Even for systems like this. However, I agree with what other folks are saying in that email should never have been the ONLY way someone gets notified about an issue.
pitabas106 5 hours ago [-]
[flagged]
Arech 13 hours ago [-]
What? The only communication channel for such a critical alarm was an email?
They are dangerous idiots, and no wonder that's a gross incompetence. Even more so, since they stupidly blamed MSFT instead of themselves.
I wonder what else could an audit uncover.
gamblor956 21 hours ago [-]
Anyone else notice that the date given by the hotel doesn't match the date of the Office 365 outage?
It appears that the hotel is trying to duck liability by latching onto the outage as the root cause even though this happened before the outage.
0xbadcafebee 19 hours ago [-]
Some people are saying it's Microsoft's fault, some the hotel's fault, some the sensor manufacturer's fault. But even if it were just those individual actors at fault, fixing it there won't prevent it happening at another place, and another, and another.
It should be illegal to build a safety system that depends on someone reading an e-mail. But there's no regulation saying "no safety system should depend solely on someone reading an e-mail." There's no generic regulation covering software used for safety. A few select industries and specific cases have limited regulations; but mostly it's the wild west. So we end up with these tragic, preventable accidents.
Governments need to pass regulation about how software can be used in safety-critical applications. At minimum, software should pass a systems engineering safety review, use standard approved methods, and pass independent lab testing. Every other safety-critical device has to follow these requirements. Software should not get a free pass. You can see now what happens when they do.
This event is just one story. There are thousands more like this, buried in the news, or never reported at all. Nobody notices because software is everywhere. You see the symptom but not the cause. CrowdStrike's shitty software ended up forcing airlines to cancel 7,000 flights, causing $500M in damage. Two days ago, shitty software/systems (related to AI) grounded all flights in the USA. In the UK, the BPO Horizon Scandal caused deaths of dozens and ruined hundreds of lives. Regulation can put processes in place to prevent these things. Without it, the problems will continue; with AI involved, it's going to get worse.
People will continue to be harmed by badly designed, badly tested, unregulated software until we do something. Next time it might be your kids poisoned at a water park; by then it'll be too late to act.
metalman 21 hours ago [-]
screen by pool door and hotel closed circuit
pool is open or pool is closed
readout of current water conditions, temp, ph, chlorine, turbidity, whatever is possible.
simple logic to dissalow abrupt changes without an override.
anything goes to zero or infinity, closes the pool.
thr0waway001 23 hours ago [-]
C'mon can we really blame Microsoft for this? This is stupid.
anjel 23 hours ago [-]
@ Risks Digest
lazzurs 1 days ago [-]
For anyone that doesn't read the link they were hurt but seriously.
This was bad system design. Hopefully people learn that safety critical services can't rely on Microsoft which they also tell you not to do.
rambojohnson 22 hours ago [-]
I'm not microsoft shill but that is such a lazy dogshit title for an article about what can easily be seen as dogshit hotel infra management.
johnnyApplePRNG 1 days ago [-]
Hot take.
tahham 11 hours ago [-]
[flagged]
userbinator 24 hours ago [-]
Surprising this happened in Germany of all places; don't they have a huge emphasis on Linux and OSS?
justincormack 24 hours ago [-]
No, there are many Linux users there, and some small high profile government cases of trying to move of office, but like everywhere else it is hugely dominated by Windows, especially this type of smaller business area, hotels, OT systems etc.
trollbridge 21 hours ago [-]
In the EU there is a lot of noise about the need to not rely on American-controlled platforms, but precious little activity towards actually doing something about it. If anything, I'd say usage of Microsoft platforms is far more prevelant there (at least in the UK, France, Benelux, DACH region, as that's what I'm familiar with) than in the US.
CalRobert 23 hours ago [-]
Seems like all of Europe is more tied to Microsoft than the US, oddly.
moooo99 22 hours ago [-]
Ironically, if the US wanted to completely shut down the German economy, they'd just have to disable Excel - everything would grind to a halt.
hunterpayne 20 hours ago [-]
Ironically that wouldn't completely do it. You would need to knock out faxes too. Just shutting down Excel would probably work for the Netherlands though.
CalRobert 9 hours ago [-]
I can't help but wonder if the vendor decision disparity between the US and Europe is evidence of engineers deciding who to use vs management.
tchalla 23 hours ago [-]
We have news articles of emphasis on Linux. That’s enough.
ThrowawayTestr 23 hours ago [-]
I didn't know Torvalds ran his own email service
dangoodmanUT 20 hours ago [-]
A correct title would be "Hotel negligently manages chlorine gas on Microsoft 365, poisons 15 guests during Nov 13th(?) outage."
dlcarrier 24 hours ago [-]
Who needs a horn or a flashing light, when you can just send an email?
tchalla 23 hours ago [-]
Since this is Germany, of course there will be no learning because we can do no wrong!
Strange thing to drag our entire country in just because one hotel had a very stupidly designed alert system.
JoeAltmaier 19 hours ago [-]
Astonishes me that a PC is trusted with lives. My PC should not be trusted.
Story: early days of Windows, they invented a power-save mode. We updated our furnace controller PC with the lastest. The customer complained - it would stop regulating the furnace in the middle of the night shift, just sit there without moving the electrodes for hours.
Upon investigation we discovered (of course) that it went into power-save mode. That cost the customer, but more interestingly, considering a 30MW furnace ran for 8 hours without producing a result meant that it would have been better to have never invented the 'power saving' ecological feature that year - the savings for the nation was burned up in that 8 hours.
rawgabbit 19 hours ago [-]
When I had surgery ten years ago, I recovered in an expensive private room. There was three windows PCs attached to devices monitoring me. Every night precisely at 2AM they all rebooted at the same time. It was a good thing I was high on pain killers as normally something like that would have unnerved me.
leni536 24 hours ago [-]
I assume the backup pigeons didn't arrive in time either.
teekert 9 hours ago [-]
Someone wasn't educated on the principle of "fail closed". I am, and I see that my agents also thoroughly operate and build using this principle (always nice to see).
lwansbrough 24 hours ago [-]
I suppose if you really think about it, butterflies are responsible for everything.
LeonB 13 hours ago [-]
I configured my speedometer to send me an email when my speed exceeds the legal limit, but due to a Microsoft outage — entirely outside my control — I was indeed speeding, Officer.
rpigab 12 hours ago [-]
This is irresponsible, they should never have used email for this critical alerting system.
They should have use a highly available Cloud-based Kubernetes-cluster on Talos Linux!
ErroneousBosh 24 hours ago [-]
BRB, off to read the Therac-25 paper for the third time this month.
pidgeon_lover 7 hours ago [-]
German holiday resort chlorine gasses its guests, you say?
mococa 22 hours ago [-]
Sure, trust on a company that’s 10000% on AI, nothing bad will happen
__m 12 hours ago [-]
Surprised they didn't have a fax backup
creatonez 22 hours ago [-]
Yeah, and everyone who dies in a hurricane was killed by a particular pattern of wind
sylware 9 hours ago [-]
It is like using the public internet for food critical fridges or blood banks...
Who is in charge of all that???
londons_explore 1 days ago [-]
Employee finds pool water has dangerous amounts of chlorine, yet instead of locking the door to the pool they just send an email into the void?
ocdtrekkie 1 days ago [-]
This is an automated system almost certainly. It monitors the chemistry and sends alerts if things fall out of range.
dgoldstein0 1 days ago [-]
Whoever made email safety critical was not thinking... What a dumb design decision
throwaway85825 24 hours ago [-]
OK you design a new out of band notification system that somehow has better reliability than email but is also used frequently enough that people will notice when it stops working.
biot 22 hours ago [-]
This one’s really simple. The current system is a bad one because if the email password changes or internet connectivity gets interrupted or the system sending the email dies (eg: power supply failure), the system fails because on any lack of signal it’s assumed that everything’s good. That’s a terrible design. Here’s a better one:
- External system polls the monitor and records levels.
- Local system reaches out to external system and fetches status.
- Someone has the responsibility to do rounds and record the status indicated by the local system.
External system down or indicates unacceptable levels, monitor unreachable, etc.? Local system shows a failure. Investigate. Local system down? Investigate.
You can even add redundancy by having a manager double check that the responsible employee is checking off the status plus the manager can do a daily sweep as a fallback. Throw something like PagerDuty into the mix and have multi-channel notifications on any failure that escalate over time.
dzhiurgis 21 hours ago [-]
POV: you are creating a new role in an understaffed, VC fleeced hotel in an economically stagnant state
In reality such system would end up sending email notification because it's just too complex.
I'd hope we can do better tho.
popcornricecake 8 hours ago [-]
Luckily the cheapest way to "fix" the problem is probably to have the system always send an email at regular intervals, and have an alarm go off (pool be closed) at regular intervals unless an employee has seen & checked that email. Probably 1 checkbox in settings plus a cheap desk alarm clock will suffice, and when it happens next time, the hotel has a scapegoat.
CoastalCoder 24 hours ago [-]
How about an audible alarm and a flashing light at the dangerous location?
This seems like a solved problem, e.g. for smoke detectors.
krisoft 23 hours ago [-]
Do not design anything. Buy an already designed and approved horn with a flashing light.
simoncion 24 hours ago [-]
Phones are good. So are loud, annoying alarms as well as labeled, illuminated alert lights in the potentially-hazardous area.
The alarms and lights are easy to test, too. Make pressing the self-test button part of the nightly or weekly rounds.
latentsea 3 hours ago [-]
PagerDuty.
ssl-3 22 hours ago [-]
The notification does not need to be out-of-band.
It can be very much in-band, like a fire alarm primarily also is.
1 days ago [-]
Animats 1 days ago [-]
Apparently the alarm system sent its alarms via email.
"As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool."
We're in an era where alarms go out via email, major policy changes are announced on Truth Social, and war is declared on X/Twitter.
DANmode 23 hours ago [-]
We’re in an era where war is not declared.
lukan 21 hours ago [-]
1942 was the last time the US formally declared war.
hunterpayne 20 hours ago [-]
1941. December 8th to be exact. The Korean war was declared through the UN. It wasn't until the War Powers Act around the time of Vietnam where Congress gave up formally declaring war. In all fairness, declaring war used to involve reconfiguring the manufacturing base. We don't need to do that anymore so formal declarations of war went away with that. Also, Congress tends to not be trustworthy these days so involving them in military decisions tends to be something to be avoided. Want to be involved again? Try not leaking to the press so often.
DANmode 19 hours ago [-]
But…that’s not how rule of law works.
Maybe “we” do need to do that.
If only to be aware of expenses.
DANmode 6 hours ago [-]
> Also, Congress tends to not be trustworthy these days
Do you hear yourself?
joshu 20 hours ago [-]
Therac ‘26
13 hours ago [-]
mmaunder 19 hours ago [-]
Hotel dead man’s switch launches chemical weapons attack on guests. FTFY.
tjwebbnorfolk 18 hours ago [-]
by this logic, loss of internet access would have killed people here
jon128 22 hours ago [-]
This time someone absolutely should be fired for buying Microsoft.
MadrasThorn 21 hours ago [-]
Dumb as shit, sue the hotel
yukIttEft 23 hours ago [-]
[flagged]
refresh89347 21 hours ago [-]
It is 100% fault of New York Times. They've distracted everyone in the IT industry with their "reportage", when Outlook stuff started failing - engineers were reading the news - so no one at Microsoft paid attention. And that wiped out Outlook for everyone. Outlook wipeout caused "no email" and "no office" for everyone for 24-48h, I imagine that people were complaining to IT. So we've got attention spreading thin for the whole IT industry. The result - cascading failures.
Posting a bunch of semi-fake out of context images on New York Times should be investigated, really. Same goes for that "classified" leak they've reported. This wiped out billions from the market, disparaged multiple woman (who likely wanted nothing to do with the reporters), affected travel for tens of thousands, now this gas poisoning.
rawgabbit 22 hours ago [-]
MICROSOFT TERMS OF SERVICE
Use of Microsoft products and services confer no warranties. Written explicit warranties are for entertainment purposes only. Use at your risk. Use of Microsoft products are dangerous to your mental and emotional health. If you experience thoughts of deep frustration, call 1-800-MSCares. All inquiries will be promptly processed in the order they will be received or when the sole contractor can get to it.
zkmon 24 hours ago [-]
> This was because the automatic dosing system had pumped too much chlorine into the pool.
What caused what? There is a story in Mahabharatha where Arjuna was claiming that he achieved killing of Karna. Krishna, the guide, enlightens him about the causal chains and shows how Arjuna' contribution was minimal.
bendergarcia 23 hours ago [-]
Buncha Microsoft bootlickers in here.
Why does it matter that the service someone is paying for is an api or something else versus email? The email service failed. First and foremost Microsoft’s service failed. It doesn’t matter HOW it should be engineered.
cwillu 23 hours ago [-]
Email as the sole notification source for a safety critical situation is criminal, in the literal “the responsible engineer or coder masquerading as an engineer should be in jail” sense of the term.
ThrowawayTestr 23 hours ago [-]
More people need to get arrested for shitty engineering
tgv 23 hours ago [-]
We all know that services can fail, and consequently will fail at some moment. It is therefore primarily the responsibility of the designers of this system.
trollbridge 21 hours ago [-]
Because Office 365 is specifically marketed and contracted to customers as something not to be relied upon for critical real time controls when life or death matter.
Microsoft has many other offerings for such things, ranging from some services inside Azure to software (like operating systems) intended for real-time use on on-premises hardware.
luxuryballs 23 hours ago [-]
They should have closed the pool.
lifestyleguru 24 hours ago [-]
lol Germans and the internet. Of course it's the internet's fault, not theirs.
CalRobert 23 hours ago [-]
It’s a neuland for them after all.
kittikitti 22 hours ago [-]
Microsoft bullies enterprises into their products and targets tech workers who are against it. This is 100% Microsoft's fault and the amount of corporate goons on YCombinator is ridiculous. I don't think this will change at all when the designated response is gaslight and deny.
shnock 21 hours ago [-]
This is absolutely misguided and is itself the only gaslighting happening in this comment section
kittikitti 6 hours ago [-]
You don't know what gaslighting means at all. You are incredibly misguided.
These systems can easily fail, and they tend to fail in a way that causes them to measure low or zero chlorine and thus overdispense chlorine. More below on why.
First, this means that an operator must always ensure safety, both against excessive chlorine and against insufficient chlorine. In a commercial context, that means that a real human (or maybe a totally different sensor design, e.g. a colorimetric sensor) need to measure the pool chlorine level, preferably with a high quality titration (drop test) every day before opening the pool. They should probably record the result in pen on water resistant paper, but this part is icing on the cake.
Second, hoping that an email gets noticed is ridiculous. If you do not have positive confirmation that the pool is safe to open, don’t open it.
Third, the systems should be sized so that they cannot overdose quickly. The maximum dispensing rate should be such that hours of continuous operation would be required to reach an uncomfortable level, let alone a dangerous level. (Hot tubs are sort of an exception here.)
Fourth, everyone should be using a reasonable amount of CYA in their pool. Even at a lowish level of 30ppm CYA it takes a lot of chlorine to become dangerous unless the operator has also lost control of pH. I realize that some codes are confused about this.
Fifth, chlorine is smelly. Someone should have noticed, even just casually. Admittedly, lots of swimmers may be conditioned to stinky pools, and checking for safety isn’t their job. But the staff should notice.
Sixth, the systems I’ve seen beep when there’s an alert. It’s easy to add flashing lights, too. (In fact, the system I’m familiar with automatically shuts down if it feeds for more than X minutes without reaching the setpoint, and you cannot turn this feature off. You can, however, defeat it by switching the power to the whole controller.)
—- sensors —-
Actual free chlorine sensors intended for pools use an amperometric probe immersed in iodide solution or gel, separated from the pool water by a fancy membrane. Free chlorine oxidizes iodide to iodine, and the sensor measures the iodine concentration. If the membrane fails (or general wear and tear or excessive pressure swings) cause the iodide to be lost, then the chlorine can’t react and the sensor reads zero (or low if there’s tiny bit of iodide left).
Interestingly, these sensors are rather expensive devices that appear to mostly come from France and Germany. The Chinese vendors seem to be rebranding European parts (!), and the major US vendor is very clearly reselling a French company’s product.
There are some vendors who sell a cheap (or absurdly marked up) potentiometric ORP-based closed-loop system. In my considered opinion, these are worse than useless.
If I were to guess, this is not the only problem that hotel has as it speaks of systemic issues.
Isn't it the chloramines that are smelly? Or at least non free chlorine. A strong chlorine smell is often a sign of a poorly cared for pool or spa
Plain chlorine ought to be noticeable before it sends people to the hospital.
One would have thought so, the story itself doesn't add up to me[1], perhaps it might be a translation issue?
Gaseous chlorine[2] systems for places like pools and hotels aren't all that common (I couldn't think of anywhere outside civil water treatment or major industrial), liquid chlorine (sodium hypochlorite [3]) chlorination systems are a lot more common because they're supposed to be relatively safer to operate.
Sodium hypochlorite is certainly much safer to handle compared to straight chlorine gas[4], but it will still give off the characteristic chlorine odour and will still happily hurt anyone not respecting it.
I wonder whether it was an absence of mind thing for patrons if the pool was indoor, the smell of chlorine tends to be stronger than outdoor pools? (enclosed setting, higher dosing given water warmth, etc.)
[1] Hazmat trained firefighter (though I am not a chemist)
[2] https://docs.airliquide.com.au/msdsau/AL640.pdf
[3] https://hyclor.com.au/downloads/msds-sheets/HY-CLOR_Liquid_C...
[4] Both have similar response requirements (Hazchem codes 2X, vs. 2XE), though chlorine gas releases demand much more extensive public safety measures (evacuation). See UN 3310 and 1791: https://www.ntc.gov.au/sites/default/files/assets/files/Aust...
EDIT: I want to add that if you breathe that crap long enough, it really makes everything else smell really weird/awful/chlorine-y. Until now I never thought to look it up, but the after effects of being around chlorine for a while leaves a distinct olfactory residue, and it is awful.
If you breathe chlorine long enough... won't much worse things happen to you?
You reminded me of nothing so much as The ICI Song:
And it's go, boys, go
They'll time your every breath
And every day you're in this place you're two days nearer death
But you go
There's overtime and bonus opportunities galore
The young men like their money and they all come back for more
But soon you're knockin' on and you look older than you should
For every bob made on the job, you pay with flesh and blood
4/5
The pool in the article clearly needed more than 30 ppm, they definitely did not sufficiently cover their asses here.
CYA stands for cyanuric acid, not cover your ass :) It’s added because it reversibly binds to chlorine. This causes the majority of the chlorine in the water to be in a mostly inert state in which it’s not irritating, doesn’t offgas and is protected from UV. You need very little active chlorine to control bacteria, viruses and algae, but you need considerably more chlorine to avoid the water becoming locally dechlorinated due to any sort of contamination that consumes chlorine. CYA helps fix this mismatch.
There isn’t one root cause here or one person or entity at fault. Many things went wrong and bad choices were made by many people and entities to lead us to this point, and they ought to be all regarded seriously and addressed rigorously.
- Whomever at the hotel that approved this as the sole (or even primary) alert system is at fault. - The lack of training and experience this hotel employee had when in a position to make such a dangerous choice was at fault. - Ditto the alarm company - everyone involved in designing, approving, and implementing this system is at fault, as well as whatever process ensured that nobody had the experience or safety to be able to speak up about this dangerous design is at fault. - The regulatory system that allowed this design is at fault. - The health and safety inspection system that didn’t catch this risk is at fault.
I’m sure there’s more; I’m just spitballing here. But the point is that there’s no one person or thing to blame here, and no one fix. This should kick off a wave of personal and professional soul-searching and improvements.
What if the person who reads the email was gone that day? What if they forget to read it? What if the email system is down? What if it was lost? This is just careless and bad design.
At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".
It's not some tricky thing where you had multiple ways to detect it, giving some redundancy. Same thing if it was a text message only or a phone call only.
This is a poor safety design with a single failure mode on the notification.
If someone sends me an email, it will not get noticed. I think this is the general culture of most places I've worked at.
But even the slack alert channels get ignored by me because they're too noisy and we haven't yet prioritized denoisifying it. We aren't dealing with critical health alerts though
People had started sending calendar invitations for important emails, on the basis that recipients would then reply to the email and decline the invitation.
More properly said, you may cause panic. People genuinely don't know how to respond when they hear an alarm in an unfamiliar place.
And that's if you're lucky. I once had to deal with people who did not want to evacuate a building when a fire alarm went off. And when I got them out the door I had trouble convincing them to leave the building's deck. Some people seem to think that it is okay to ignore a fire alarm, even a probable false alarm, in a wood structure with a wood deck.
It's a well known irony that every system intended to prevent accidents eventually itself causes one.
It just has to be annoying enough that even the most junior employee would radio in their supervisor to get it to stop
We've been through this with smoke detectors, it's not a new thing.
The power failure would presumably shut down the pumps and end in the pool being closed. Then pools should have a power outage protocol they’d follow before opening the pool again. Checking chemistry would be high up on the list.
Smoke detectors need backups because a fire can cause a power outage before there’s enough smoke to trigger the smoke detector.
Also, I have a personal domain, some sites decided its not real enough and I can't get email from them.
Email is just not reliable. It could be part of a combined system of other things that are reliable.
This is not the case in aviation. Even in the most egregiously obvious single-cause failures, there's always an analysis of issues in the wider system that allowed that single cause to become the single cause in the first place.
I agree with your points, though. In this case, Microsoft absolutely takes shared responsibility. It is only a matter of time until a similar outage in some other wider networked system with a so-called 'five/seven/nine nines uptime guarantee' goes out for a couple of days, and takes with it some critical life-sustaining system, and people die.
And yet, as late as yesterday, the idiots in charge have announced that there is absolutely "No Need To Investigate the FAA Ground Stop".
Ford doesn't build all of the modules of their cars. They design specs and outsource manufacturing. But legally speaking, Ford is responsible for design, testing, and support / recalls. It's their responsibility to contract in the terms that both protect their customers' safety and Ford's liability due to spec / contract violations.
When your company is responsible for the health of staff and guests while dealing with deadly gasses, it's your company's responsibility to ensure that all of your vendors detail how their failure modes fail safe.
This is the same exact logic that has been migrating software companies to detail their risk by demanding their vendors all supply a "Software Bill of Materials" (which is very difficult in practice, but is necessary for the downstream users to protect themselves from upstream risk).
anything that was "at fault" that you can't do anything about doesn't matter. and you can't do anything about a microsoft outage, all you can do is design around it.
"Why did people get gas poisoning"
https://en.wikipedia.org/wiki/Five_whys
The point of a process is to constrain incompetent people into doing something that will work despite their incompetence. Five Whys is entirely subjective and can never constrain anyone; it's just something that some people think sounds deep.
yeah, these companies need to be told to stop relying on cloud services for everything.
its wild to me how many critical things that absolutely do not need the internet are relying on the internet.
much blame to go around here:
- whoever designed the stupid alert system.
- the marketing departments in countless companies ignoring engineers when they market to normie customers “connect everything”
- the hotel company for seemingly doing no dilligence at all. and just blindly listening to some salesman “connect everything”
- accountant types and their “we can save pennie’s if we cut all of these corners.” and the stupid corners they choose to cut.
- etc…
on a macro scale it’s amazing to me how many people are victims to slimey car salesman types leading to situations like this.
5/5
Regardless of high availability promises of cloud services, a local internet outage will supersede the risk and is a risk factor in most places of the world (probably everywhere). Risk that nobody aside the local ISP could quantify either. So logically any system critically depending on the cloud and isn't designed with an offline fallback has unmitigated risk factors.
Sure, if you host some online service, this risk cannot be mitigated. But it can for everything else. And if you need an offline operation mode, in most cases the non-cloud design is simply superior. So cloud is synonymous with having a single point of failure?
This case is one where on risk analysis was done at all or this would have surfaced pretty quickly. There are problems too if a pool doesn't get chlorinated, but it is probably better than to poison your guests.
Knowing that your email could be delayed by days changes how you view it and the applications you deem it suitable for.
Similar to doing regular restores of your data backups. If you never test a restore, you don't have a backup.
I did an apprenticeship as an electrical engineer and have been in those catacombs underneath such water park type of things. There should be such devices present, and of course personnel to operate them.
I mean, can you imagine your boss' face when you tell em the reason your entire system is down is you didnt get the email from checkmk that something is wrong? Instead of, you know, checking the dashboard yourself periodically.
It's easy & the 1 time you don't, can cost liveS.
People ignored it.
A message to the fire department would have been the better idea
I called 911 once because of two CO monitors going off in my top floor apartment of a mid-rise. Half a dozen engines responded including hazmat, everyone’s standing around outside the building. FD had to break down the door of someone who wasn’t home because there was no response. Somehow it got around that I had been the one to call me in. Some guy came up to me saying something like “some of us have jobs in the morning. I hope you’re happy, moron.”
FD found a leak in the basement furnace. Without the alarms, everyone in the building would’ve been dead by morning.
Is there an obligation to always be disagreeable here?
But a "Good." to that kind of sarcasm looks like a lack of understanding, and then you rephrased the joke as a non-joke with nothing extra. If that's the full intent, I'd say just upvote.
So what's the value of the contribution of your own comments in this thread?
It's not much value but if it leads to better outcomes in the future then that's a success.
Reformulated: If pool management's cue that something's wrong is because their guests are leaving the water, they are already liable for damage.
> the user of the pool should be able to notice that something wrong by them self.
Clearly, their motivations for autonomous self-preservation were insufficient.
The hotel picked a monitoring solution which relies exclusively on email, or configured it like that. If anything they are absolutely both to blame but for different things. The company for designing a safety related system like this, the hotel for selecting an inadequate solution/configuration thus allowing their guests to be harmed.
> But it is accurate to say that the cause of the issue was the outage
The cause of the issue is the more immediate one - the configuration of the monitoring system. This avoids going up the causality chain to far - shitty coffee caused engineer to take down email system, and such.
Not one but two of the hotel's systems failed to operate properly. The chlorine pumping system pumped too much chlorine and the last resort alarm failed to alert.
As I said, there's plenty of blame to go around as a matter of general principle. But for what happened here the hotel bears the responsibility towards their guests, not the pumping or monitoring system developers, not Microsoft. The only reason MS in the title is because that failure is big news these days.
On a technical note, even if they had a backup email provider they'd still be screwed because the people getting the alert were probably employees on the single provider (back to SPOF). SMS could have been a good parallel option. Ideally also some physical control panel that triggers an alarm when any safety critical condition occurs. On the chlorine pumping system, that shouldn't be a dumb preprogrammed pump, it should come with it's own sensors and detection loop independent of any external system, something that proactively stops it before it breaches the safety limits.
Using the unqualified "cause" is misleading click bait and in some places it could get you sued
1. Can there not just be a light / siren right next to the gas sensors? Easy to fix, but it requires hardware changes on-site.
2. Could Microsoft have not had an outage? Hard to fix.
3. Could they use some better notifications systems like idk self-hosted ntfy.sh with a warning if the server goes down? Yes but nobody wants to pay for that.
And you have to check those lights and sirens that they still work.
Everybody knew email was down.
The real problem is nobody thought of the consequences regarding the alarm system
Yes, they still have to work but first off, both of those are extremely reliable systems, alarm circuits have extremely low risk of failure. Second, testing them takes 5 minutes a week/month, press the button, look for flashing light and listen to wailing noise.
It's inexcusable to think that email/slack or whatever is acceptable as anything other than notification system so people not in facility can be notified that alarms are going off.
EDIT: You are defending this setup and I'm not sure why. This seems email only setup seem indefensible.
Testing requires staff and you can be sure those parks are understaffed.
Best thing is to avoid them, next best thing is a cheap solution with low maintenance by the staff to notify as much people as possible.
Email wins.
Not a good solution but the best you will get for the price they are willing to pay
Whether it breaks because a relay has corroded due to years of service in a chlorine-enriched environment, or because O365 is down, or because O365 tightened up on DMARC requirements: It needs tested.
And it's not a binary choice. This isn't the World Cup, the Super Bowl, or Highlander. There can be more than one.
The alarm system can wear a belt, and suspenders. It can send emails, and it can flash a bright light, and it can also sound an audible alarm.
Curious: what do you think has to be done with an email?
Sarcasm aside, in this case, the email should've triggered the literal lights and sirens. Preferably automated (tho in that case, why even go through email as middleman) but could also be an actual person flipping a switch.
You see this in software ops too: certain subject formats trigger a special ringtone or even a pager call. Emails are too subtle even with ALERT ALL CAPS SUBJECT LINES to be alarming.
Countless systems are relying on email as a means to alert people to issues, for better or worse. Having a physical alarm for everything is rarely practical for a myriad of reasons.
I worked in a data center 20 years ago that had alarms on the wall for when the halon was preparing to be released for fire suppression. There are likely dozens of things that could be monitored, but it was only the catastrophic thing that once it started, there was no stopping it, that got lights on the wall.
Too many loud alarms make people start ignoring alarms. I would guess that there is a fairly large margin of error in pools before there is a health risk, so it likely wasn't deemed alarm worthy. As someone who's been in a pool before, I'd also imagine that people would notice and say something, or it would be unpleasant enough to not want to spend time in it. That's what I find most surprising... that people weren't immediately turned off just being around it, to the point of staying out.
Maybe a waterpark resort, but hotels no. These days they barely afford a front desk employee and a cleaner.
Title: 5 nines of uptime
A grand piano is suspended from the ceiling with an electromagnet.
The electromagnet works as long as there is power.
No special battery backups, anything.
Guests are invited to stand underneath it, after signing a waiver.
That’s it. That’s the art.
I'm surprised an e-mail only alert is even legal for this. Or is chlorine level monitoring not something that is regulated?
Your second sentence is 100% correct. Your first is questionable; Microsoft, a kabillion dollar company has, for years, held itself out to be reliable and safe. And it repeatedly has shown that it is not. That opens them up to some liability here; and frankly deserves to be better held to the standards it (either directly or indirectly) claims.
And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.
> And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.
"Implied warranty of merchantability" is a term of art in United States law, and it applies only to tangible goods, not to services.
The greater concept is that the entire point of law is so that you CAN'T just wave away things in fake contracts.
When an incident like this happened, they might add new "system" to improve the situation, probably in the lines of having someone to patrol and check off a list routinely.
I know the S in IoT means safety/security, but there must be things that can be done for urgent notification.
Wtf?
I must have missed the memo where Slack Discord or “apps“ became suitable for safety critical task. Must be what USA has tried to use just before grounding all planes.
You know what they all have? An email account
I do agree with this sentiment. At the same time, it’s still unacceptable for a global service like MS365 to go down for extended periods of time. I wouldn’t be surprised if the hotel chain was sold on some MS guarantee that it would never become unavailable.
Although I would also believe a scenario where the hotel chain "IT” (aka the owners son or grandchild) just used default alert settings and didn’t bother to setup fallback alerting mechanisms.
In short, all parties suck here (MS and hotel)
Go invent a transport layer that will never fail, and make yourself rich.
Because right now, there is no such thing.
Email is not.
And yet, still not "guaranteed," which was the criteria specified by the OP.
And yet, still not "guaranteed," which was the criteria specified by the OP.
Yes it's true, no system is guaranteed to function, but for Pete's sake, it's clear to everyone that having only an email-based alarm system for chlorine levels is the wrong thing to do just because "well there's no alarm system that is guaranteed to alert people, so let's rely only on email, tee-hee".
Any redundancy would have been better than this. People's lives were at risk, and here we are, debating the meaning of the word "guaranteed". I'm going to be frank here, are you an MSFT shareholder or tied to the hotel in question?
Disclaimer: I work in Azure, this is not a defense of the outage and I didn't read the submitted article.
> A cyclist died on the road. He wasn’t wearing a helmet.
> A car blew through a stop sign and ran over a cyclist.
If I built a chlorine tank out of toothpicks and duct tape, and it leaks everywhere and kills people, you don't publish a headline that says DuctTape and Global Toothpicks Inc are responsible for chlorine deaths.
For events that typically don't happen, I feel it's relevant to give the factor that changed leading to the event happening specifically now. Standard background information (the invention of computing, the earth continuing to orbit the sun) less so.
A freezer is an obvious example of something that should be controlled directly by a human operating on it and not by some "smart" connectivity.
> So no one noticed that the maintenance safety system of the sauna landscape has long since sounded the alarm by mail. Because the automatic dosing system had pumped too much chlorine into the pool.
I’m so sorry.
* 13 guests complaining about irritation of airways (+2 later)
* 4 taken to hospital, 1 described as medium injury, rest discharged
* Dual failure - dosing system had a defect (dosing too high) and the email of the monitoring failed (M365). Article sounds like it's two distinct systems.
Overall the German text sounds a lot less dramatic than the hn title.
Anything that could result in that sort of harm should have a more resilient and redundant architecture.
> 15 guests injured at Phantasialand
> Due to the (Microsoft) disruption, no emails were received, as the company informed WDR on Saturday. As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool.
> Subject: Fire
Dear Sir/Madam,
Fire! Fire! Help me 123 Carrendon Road. Looking forward to hearing from you.
All the best, Maurice Moss
Like, what leads a teenager to do such things?
https://www.science.org/content/blog-post/things-i-won-t-wor...
https://www.science.org/content/blog-post/things-i-won-t-wor...
https://www.science.org/content/blog-post/things-i-won-t-wor...
https://www.science.org/content/blog-post/things-i-won-t-wor...
Me neither, but that is probably attributable to the fact that I didnt have a swiming pool.
> With one experiment, he created chloroform and as the book encouraged him to sniff the chemical, he did so and was unconscious for more than an hour, according to his recollection.
[0] https://en.wikipedia.org/wiki/David_Hahn
Back in the day, i tried to sniff NH3 because i've heard it smell like piss.
It's a lot of worse.
High chlorine levels tend to reduce chloramines, so gasses like nitrogen trichloride seem unlikely to be the problem. You need a nitrogen source to make any large amount.
You favorite AI will happily model it, and I wouldn’t trust it to get the right answer :)
Really? I would expect to get refused or downgraded for "bio".
The indication needs to be proportionate to the risk, and so if there is a high risk of harm, then the monitoring device needs a auditory and/or visual alarm as well as the email notification
Actual headline: "15 Gäste im Phantasialand verletzt" (15 guests injured in Phantasialand)
And the browser tab shows the following summary title "Chlor: Phantasialand bekam wegen Outlook nichts mit" (Chlorine: Phantasialand didn't notice anything due to Outlook)
This is usually a problem for a hotel/motel too small to have a control room, but big enough to have escalators, elevators, HVAC, pools, laundry equipment, fire suppression, water heaters, and other gear that needs to be monitored. What usually happens is that some manager's office or security guard station has a bunch of unrelated alarm panels. That's where medium-sized problems get lost.
The next time you walk by a fire alarm panel in an office building, look to see if it's displaying any faults.
[1] https://pacificblueengineering.com/advanced-control-system-a...
> During the incident, some of the gas monitors installed within the Yenkin-Majestic facility detected the flammable solvent release approximately one minute after the release began and sent an email to an offsite employee, but the monitors did not sound audible alarms to warn employees of the hazard.
1 dead, 8 non-fatally injured.
They are dangerous idiots, and no wonder that's a gross incompetence. Even more so, since they stupidly blamed MSFT instead of themselves.
I wonder what else could an audit uncover.
It appears that the hotel is trying to duck liability by latching onto the outage as the root cause even though this happened before the outage.
It should be illegal to build a safety system that depends on someone reading an e-mail. But there's no regulation saying "no safety system should depend solely on someone reading an e-mail." There's no generic regulation covering software used for safety. A few select industries and specific cases have limited regulations; but mostly it's the wild west. So we end up with these tragic, preventable accidents.
Governments need to pass regulation about how software can be used in safety-critical applications. At minimum, software should pass a systems engineering safety review, use standard approved methods, and pass independent lab testing. Every other safety-critical device has to follow these requirements. Software should not get a free pass. You can see now what happens when they do.
This event is just one story. There are thousands more like this, buried in the news, or never reported at all. Nobody notices because software is everywhere. You see the symptom but not the cause. CrowdStrike's shitty software ended up forcing airlines to cancel 7,000 flights, causing $500M in damage. Two days ago, shitty software/systems (related to AI) grounded all flights in the USA. In the UK, the BPO Horizon Scandal caused deaths of dozens and ruined hundreds of lives. Regulation can put processes in place to prevent these things. Without it, the problems will continue; with AI involved, it's going to get worse.
People will continue to be harmed by badly designed, badly tested, unregulated software until we do something. Next time it might be your kids poisoned at a water park; by then it'll be too late to act.
This was bad system design. Hopefully people learn that safety critical services can't rely on Microsoft which they also tell you not to do.
https://how.complexsystems.fail/
Story: early days of Windows, they invented a power-save mode. We updated our furnace controller PC with the lastest. The customer complained - it would stop regulating the furnace in the middle of the night shift, just sit there without moving the electrodes for hours.
Upon investigation we discovered (of course) that it went into power-save mode. That cost the customer, but more interestingly, considering a 30MW furnace ran for 8 hours without producing a result meant that it would have been better to have never invented the 'power saving' ecological feature that year - the savings for the nation was burned up in that 8 hours.
They should have use a highly available Cloud-based Kubernetes-cluster on Talos Linux!
Who is in charge of all that???
- External system polls the monitor and records levels.
- Local system reaches out to external system and fetches status.
- Someone has the responsibility to do rounds and record the status indicated by the local system.
External system down or indicates unacceptable levels, monitor unreachable, etc.? Local system shows a failure. Investigate. Local system down? Investigate.
You can even add redundancy by having a manager double check that the responsible employee is checking off the status plus the manager can do a daily sweep as a fallback. Throw something like PagerDuty into the mix and have multi-channel notifications on any failure that escalate over time.
In reality such system would end up sending email notification because it's just too complex.
I'd hope we can do better tho.
This seems like a solved problem, e.g. for smoke detectors.
The alarms and lights are easy to test, too. Make pressing the self-test button part of the nightly or weekly rounds.
It can be very much in-band, like a fire alarm primarily also is.
"As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool."
We're in an era where alarms go out via email, major policy changes are announced on Truth Social, and war is declared on X/Twitter.
Maybe “we” do need to do that.
If only to be aware of expenses.
Do you hear yourself?
Posting a bunch of semi-fake out of context images on New York Times should be investigated, really. Same goes for that "classified" leak they've reported. This wiped out billions from the market, disparaged multiple woman (who likely wanted nothing to do with the reporters), affected travel for tens of thousands, now this gas poisoning.
Use of Microsoft products and services confer no warranties. Written explicit warranties are for entertainment purposes only. Use at your risk. Use of Microsoft products are dangerous to your mental and emotional health. If you experience thoughts of deep frustration, call 1-800-MSCares. All inquiries will be promptly processed in the order they will be received or when the sole contractor can get to it.
What caused what? There is a story in Mahabharatha where Arjuna was claiming that he achieved killing of Karna. Krishna, the guide, enlightens him about the causal chains and shows how Arjuna' contribution was minimal.
Why does it matter that the service someone is paying for is an api or something else versus email? The email service failed. First and foremost Microsoft’s service failed. It doesn’t matter HOW it should be engineered.
Microsoft has many other offerings for such things, ranging from some services inside Azure to software (like operating systems) intended for real-time use on on-premises hardware.