Meet our Infrastructure
As a sysadmin at Panter, you’re also responsible for the infrastructure of Impact Hub Zürich, a vibrant coworking space in the heart of Zürich. On 3 floors and a total of roughly 2000 square meters we’re making sure that all the members fit into our WiFi network. As always in such a crowded place there’s not much margin for errors. And because our frequencies are almost as crowded with rogue WiFis like the death star’s having holes for thermonuclear missiles, this can be quite challenging and sometimes quite frustrating.
In this blog post, we’d like to give you a little bit of insight into our configuration and how we’re troubleshooting our network.
Our WiFi runs exclusively on Mikrotik devices because this gives us a lot of flexibility. Their devices come at a fair price and we’re not forced to pay fees to use their products.
When we first came to the new Colab, we had nothing but an empty rack, an old DIY router and some broken cables. Step by step we built up the infrastructure, adjusting it to our needs.
The dawn of WiFi
We started with an emergency setup (2 access points) because we had to move on faster than expected. There was not much of a concept, but it worked for us Panters who were pretty much alone at that time. While we had our office on a construction site which was quite dusty at the time, we even managed to nearly kill our router because it suffocated in the dust (the whole rack was accidentally covered in dust) but it was nothing a little bit of compressed air couldn’t solve.
Working on a construction site has its downsides, but it also gives you the freedom to plan your LAN outlets and whatnot, or so we thought, but everything got built quite differently as planned, so those outlets are still a vision. We had to reverse engineer every single one of a bunch of cables because documentation of the existing network was nonexistent.
When we moved to the Colab, our rack was neat and clean and there were almost no cables. We also had to install the cables by ourselves because our top floor was a dusty attic which looked like it hadn’t been touched for the last half of the decade. Since then all visible and non-visible LAN outlets were patched. So if you spot an outlet at our Colab, feel free to use it.
With the help of our network crack (Andreas König), we got a solid setup and started to use Mikrotiks CAPsMAN, which is a bit of a pain to get into and a little bit of a hassle to wrap your head around it, but it works like a charm after the initial setup (and 10 factory resets later).
There’s always a gap between sysadmins and users when it comes to quality. Us sysadmins are pretty much the only fans of cabled connections and we love the convenience of cables. And since we had a lot of LAN ports in this building, where a school was located back in the days, we had this sweet vision of small access points and switch combos on every table so people who preferred WiFi could get it from the ether and the others could use cables as they desire. But well, that idea got chased out of town pretty fast (Except the makerZspace, they love cables and so they have them).
We decided to aim for total 5g coverage and slowly starve out 2.4g because it’s way faster and it allows you to put up a higher density of access points. And if fewer people use one access point everyone will have a faster connection. And since we were living in the year 2015 where every device is capable of 5g there shouldn’t be an issue with partially dropping 2.4g, right? TOTALLY WRONG
As it turns out, the 5g capable devices aren’t consistently great with 5g connectivity. Most of those devices only support the lower bands 5-036 and 5-044 and hollow glass bricks are apparently the death of every 5g connection. There are also quite some differences between mikrotik’s different products. Here’s a list of a few which we went through and the ones we’re currently using.
- Netmetal - A lot of DIY, so it’s time consuming and thus not quite cheap, but nearly indestructible and perfect for outdoors
- Netbox - Also DIY, pretty huge and not quite cheap, also not the most aesthetic ones
- cAP2n - Plastic toy, not recommended
- wAP - Aesthetic pretty good dualband access points with a good range
- SXT SA5 ac - With an angle of 90° perfect for the corners in our coworking space
- wAP ac - very fast, performant AP
- hAP ac - like the wAP ac but with a built-in switch
We were also pretty harsh when it comes down to kicking people out of our access points because if you’ve got a bad connection, you’re giving a bad time to all the other people using that access point. When we lowered the connectivity threshold to our sweet spot of 65dBm, we had an awesome connection, but somehow all the other users were having a bad time. This is the point you have to realize that you obviously have another perception of “things going well” (although you’re right). At the moment we’re having 20 AP’s on 3 floors, serving over 300 clients during peak hours.
With all those different devices and total freedom for our coworkers, there are always some people having connectivity issues nonetheless, so the bigger we get, the more important it is that we know how to monitor our network and how to distribute the load that’s hammering on our infrastructure.
As a sysadmin - responsible for something which quite a few people rely on every day, you should be able to tell someone where the problem comes from. So it’s quite important to monitor your radio waves, so you can start investigating rising issues before a lot of people are affected by it and to troubleshoot specific clients’ issues.
Having some schools and offices around us, we’re sometimes experiencing a peak in traffic every time there’s a bunch of school boys and girls in front of the office watching YouTube and whatnot over our free WiFi. Because that free WiFi is actually intended for the visitors of the coworking space, it’s important to drop clients that are too far away (across the street inside the school building). That’s why we’re constantly monitoring our frequencies and repositioning our AP’s to supply our guests with a good connection (that’s also where our fancy graphs come from).
We’re also periodically mapping out our space to check for coverage issues, noise sources, interferences and overlapping channels.
A really cool tool is our self developed self-service portal (only reachable from within our network) where you can send a rating to our company chat as well as chasing the issue yourself. It’s an open source project and you can get it here and tinker with it however you like.
And finally we’ve gone so far as to even supply our guests with an intranet dashboard where they can check their WiFi performance using Prometheus to save the timeseries and use Grafana to visualize them.