December 6, 2019

Horror Stories: Dropbox Driven Deployments

It was quite an exciting time: I’d just spent the last 6 months building a new engineering team from the ground-up, mentoring existing members of staff, and undertaking a complete overhaul of technical processes at a relatively large enterprise. I was keen to get to grips with a similar challenge, and began my first day with a wide-eyed smile and a large dose of optimism.

First order of the day: a one-on-one with the CEO and a chat about the future of the company.

“Don’t trust them, they’re bad energy and they’re going out the door”

Armed with my trusty coloured-pens and a fresh notepad, I began making notes on how the business worked: what technologies were in use, what the short term objectives were, which staff members had which responsibilities. Having spent the last couple of years in larger companies, it was refreshing to know I’d be working with a handful of people with direct communications across the company: awesome.

Then came the first bombshell: in the next few weeks we’d be moving office, now to be located on the other side of London. Mentally I tried to work out my new commute, it would likely be double the time and involve quite a lengthy journey on the tube - one of the things I’d actually taken this role to avoid. “Oh well, I’m sure that slipped his mind during the interview process.

As the meeting came to it’s conclusion I closed my notepad, and shook the CEOs hand before walking out. “Oh, Fergus - before you go I need another word with you — it’s about the other two engineers.” It was at this point I realised that there was minimal involvement with them during the recruitment process, and the greetings on my first morning were.. well, subdued to say the least.

If I were you, I’d not trust them. I really want to minimise your involvement with them initially if I’m honest. They’re a lot of negativity and bad energy amongst them, and being French - I know that a lot of what they’re saying in their native tongue is quite questionable. For the moment, they’re on their way out and I don’t want them to put you off of the company.

A tour of the tech

I’ve always been sceptical of management making judgements on technical staff - especially when they’re so negative. I also realised that this explained the absence of technical involvement during the interview process. Still, how bad can it actually be? And surely I can try and turn this around if I’m being given the autonomy to build a new team and revitalise their tech?

I left the meeting room and proceeded to park myself down beside one of the engineers and — despite my initial scepticism about the greeting I received — I was able to quickly build up a rapport about every day things. From engineer-to-engineer, I could tell their was a sense of frustration on his behalf though.

We began the onboarding process by logging in to their AWS account and viewing the inventory of the services they used; I was immediately struck by the fact that they weren’t utilising many of the AWS features, and were mainly using EC2 instances as normal VPSs. Sure, by not leveraging more of the AWS ecosystem they were likely paying more than was needed, not to mention it was indicative of some questionable IAM practices, ever-the-optimist though, I simply explained it away as “good, these guys worry about vendor lock-in!”.

Then came the next bombshell: despite having SSH access, no one knew what half of these boxes were actually doing… and that meant they were scared to turn them off.

In what was very likely more idiocy than optimism, I still felt fine: “Well shit, I’ve just had to deal with that at my last company - it’s not ideal, but I can certainly deal with this! I’ll make a spreadsheet, monitor network connections, list what services are running, and we can reverse engineer this cluster. No biggie!” I thought to myself.

I asked the engineer to log in to one of the unknown boxes, and run ps aux - just so we could begin the inventory immediately, and try and gain some knowledge about what exactly the machine was responsible for - if anything.

“Huh, it’s running dropbox.”

The output from ps aux was quite simple; relatively few services, with the only ones standing out being Dropbox and Apache. Given that Apache was running I thought it was a good shout not to turn the box off. As for Dropbox? Surely it was just a very primitive way of backing up - whilst not ideal, not a complete show-stopper by any means.

I asked the engineer to check the Apache configuration files, then we could find out whether it was simply the default configuration and serving a static page, or whether it actually constituted part of the application. That’s when I saw it:

  DocumentRoot "/root/Dropbox"
  ErrorLog "/root/Dropbox/logs"
  TransferLog "/root/Dropbox/logs"

The service was being deployed from an ex-member of staffs personal Dropbox account… oh, and the logs were also served from this same account. The engineer looked at me with a disappointed look — devoid of any surprise.

Sensing my dismay, the CEO quickly called me over and told me that the product owner and marketing lead would like to go for a coffee with me; he knew I wanted to get an understanding of the entire company, and that it could be valuable.

I scribbled the plans for a CI pipeline down on a bit of paper, and wrote some action steps for drafting an inventory and taking control of the current infrastructure, before going for my coffee.

The “3 hour job”

I’d arrived at 10am, and it was only 12:30 by the time we reached the coffee-shop. I made friendly chit-chat with the two employees, and tried to swerve any talk of the company itself; my mind had been made up, I couldn’t work here. The commute, the engineering team, the management style, and the technical challenges… all combined they were too much.

By 1pm I had finished my coffee and told the other employees that I needed to run an errand, and in quite possibly the most unprofessional moment of my career, I sat in another coffee-shop and sent an email to the CEO - I wouldn’t be returning this afternoon, and would have to politely decline the position.

Much to my surprise I received an email shortly after, whilst it asked me to reconsider, it also contained the rather brilliant line of “I know that DevOps practices may be quite new to you, I’m sure we can get you up to speed in no time!”… I didn’t reply.

© Fergus In London 2019

Powered by Hugo & Kiss. Source available on Github.