AuthorJim Plush

Bonus AntiPatterns – How to set the wrong incentives

Entering my second decade in software development and moving up in various leadership roles, I’ve been exposed to many different incentive plans to help “boost morale and increase performance”.  In my experiences, people were trying to do the right thing and give people more of the carrots and less of the sticks. However, most times you wind up establishing the wrong incentives that could in fact backfire so greatly it changes the cultural fabric of your company. Below I outline what I’ve found to be anti-patterns in bonus structures for engineering teams and what a better approach could look like.

To make sure we level set on what I’m talking about…  This article is referring to any compensation package that may involve a bonus structure whereby meeting certain criteria you get additional money paid out to you.

These are based on my experiences, feedback from teams I’ve run and my own personal beliefs on the subject. These are based on personal observations that any devised system will ultimately be gamed. This also applies more towards engineering teams since sales team goals are a different beast.

Anti-Pattern #1

“Your bonus is based on hitting goals you commit to”

This one is pretty common in “agile” type shops where you’re committing to hitting targets or OKRs (objectives and key results) every 2-4 weeks. While good, in theory, it goes against everything it means to be agile unless when a change in the landscape occurs your bonus targets for that period also change. For example: “I promised Project A but if I spend a couple days helping our sales person get around a bug, they could land a huge deal for the company. If I help our sales person I’ll get a pat on the back, but my team will miss our bonus. I’ll help sales later.”

Perceived Incentive: “Pad as much as possible to not have a chance of missing the date, commit to as little as possible and make your boss challenge everything you say, Avoid helping across teams”

Result:  A culture of mediocrity due to no one pushing hard to meet deadlines because they’re padded so much and everything takes 50-80% longer to get delivered. If I think I might miss my dates I ship anyway even if I know it will fail in production. Hey, I shipped! This anti-pattern promotes silos and limits cross-team communication.

Anti-Pattern #2

“Your bonus is partly based on hitting cost cutting or COGS metrics” 

This one is based on keeping costs down. COGS = Cost of Goods Sold. Otherwise known as cost control measures. In a fast growing tech company, this might be the most dangerous one. There are so many factors at play that would require a highly detailed model of the world. For example, if you said costs can’t grow past 20% this quarter, but we get a sudden boon of customers, or we have to scale for a big sales event or promotion, how do you tie that back to the original commitments? It’s highly tedious and error prone. Other examples would be not developing a feature unless it has a certain expected dollar amount of return. The analysis of that alone will cripple you.

Incentive: “I will take as little risk as possible, I will overload as many servers as I can and not run anything over capacity and will accept much high latencies to protect our bonuses. I will not try out new ideas because I don’t want my teammates to be punished if I have to spend money temporarily to do that until next year.”

Result: You have a culture that incentivizes complacency and dissuades innovation. New features will never get approved because COGS have to get adjusted every QTR to account for new projects. Those adjustments will be wildly inaccurate and you now need a COGS review board for new initiatives. Don’t let finance dictate growth in a growth phase. COGS come into play most effectively when you have a solid business in the exploit phase.

Anti-Pattern #3

“Your bonus is partly based on production status”

This one is based on the idea that service uptime, the number of production bugs found or other service level metrics tie into your bonus structure. While at a company level this makes sense at the employee level it falls apart pretty fast. Many things are out of your control, such as someone cutting your data center’s primary and redundant fiber pipes while doing routine maintenance with a shovel  (it happens!). Did I do a bad job or were other incentives in place for to not consider a more fault tolerant architecture like COGS?

Incentive: “Never push code unless mandated by your boss, do not take risks, pad your estimates another 200% to factor in longer times in testing and QA cycles, push off acceptance to some other group to get blamed for uptime if possible”

Result: You definitely do not have a ship-it culture, you have a culture of fear of change. Deployments slow down, new projects stagnate, no one wants the responsibility to touch production.

Let’s tie all those anti-patterns together with the data center pipes being cut example from above and you can see how these start to de-incentivize employees.

Yes, we had a production outage so my bonus will get dinged because I didn’t hit our uptime criteria. I was hitting your cost-cutting goals by not running multiple data centers and I didn’t push out the fallback code because I was too scared to touch production and potentially cause issues and lose that part of my bonus. I did notice I could have moved a backup server to the other datacenter but I would have missed our deadline for Project A and we would have missed our OKR bonus.”

So now you can’t win because you set up a system where we will fail at some tier.

Whenever a bonus structure is proposed you have to look at what the actual incentives  are. As many have learned from the book “Freakonomics”, decisions can have odd repercussions that the original authors did not intend, “The Cobra effect “

Some of the better approaches I’ve seen that didn’t have the incentive downsides:

Profit sharing / sales-based bonuses

When the company does well everyone does well. It can get everyone moving in the same direction to help attract and retain customers by getting everyone caring about success. Some have mixed feelings on that one but I’ve seen it work more effectively. It’s not about creating one-off features just to land a deal,  it’s about helping the sales team feel like engineering is a partner and everyone is rowing in the same direction.

Put it into salary

When recruiting is all said and done and it’s time to make an offer the #1 factor is usually base salary. A bonus is seen as just that, something that may or may not be paid out based on some criteria. If you want to increase your chances of winning against competing offers put that bonus into base salary.

Thought based bonuses.

If this is all too complex for you, perhaps get rid of bonuses altogether, raise salaries and offer more spontaneous rewards. The greatest bonus I ever received was after a successful launch of a project after working a good deal of hours on it. My boss came to me and said: “Take 3 nights at the hotel of your choice with the family and we’ll cover everything.” Now this wasn’t the biggest monetary bonus I had ever gotten but it stuck with me the most. There was actual thought into how that launch must have affected my family life and how I needed some face to face with them again. He could have just given me a check and be done with it, but he put some thought into it. Even if it was a little thought, it was something. (thanks, JB!)

Another nice bonus is autonomy. If someone has been a consistent performer give them a chunk of time where they get to work on whatever project they feel interests them most. Do they want to improve a query? Refactor something? Hackathon up a new product? Go for it, time is your bonus.

I’ve also seen gestures like letting employees expense nights out with their spouses, babysitting included if they have kids, blocks of days off, gift cards for going above and beyond, and other various perks. Did I do a great job at something? Send a note to the CTO for a hive five.  At the end of the day it’s about recognition.

Bottom Line

Your top performers would be your top performers regardless of bonus structure. Top performers are internally incentivized. Increase their pay so they don’t have to worry about money issues and focus on retaining them.

You need to incentivize risk and stretch goals. No one will stretch if they are hit with a stick if they don’t make it a full 100% of their commits, 100% of the time. When you look at those incentives there are no upsides for stretching and taking a risk. Why make stretch goals when if you don’t hit that 90-100% goal you only get a stick?

You’re telling people to set mediocre goals because there’s only downside to taking risks. What about those who don’t strive to be bold and innovate, no matter what system you set up? Don’t fear letting people go. If you have a consistent low performer it’s cheaper for the company to let them go rather then keep shifting them around to other projects. The larger the company the more time you can invest in improving someone but small companies don’t have that luxury and each headcount needs to have a significant impact.

As John Doerr who brought the OKR model to Google says:

“Don’t tie the OKR goals to bonus payments, except for sales quotas. We want to build a bold, risk-taking culture.”

Some references that also cover this topic:

Joel On Software – Fog Creek Compensation

http://www.joelonsoftware.com/articles/fog0000000038.html

John Doerr on OKRs

http://blog.betterworks.com/keys-okr-success-qa-john-doerr/

The Surprising Truth on What Motivates Us (only 10 mins)

 

Ted Talk  – Dan Pink The puzzle of motivation

The Engineering Assistant

I made a snarky comment in our chat channel the other week about having an executive assistant concept for engineers and what was initially a joke became an interesting idea to me. I haven’t researched it so it’s probably already been talked about but my thoughts on it are outlined below.

Executives have long had the concept of an EA or Executive Assistant. A big part of their job is to offload some of the non-critical tasks that need to be done, but don’t warrant the hourly rate or responsibilities of a CEO, CTO or VP. For example, an EA will help with scheduling meetings and working around attendees schedules, organizing logistics for events, and triage correspondence like email, etc… you get the point. A CEO’s role is to drive the company forward and provide vision and leadership; anything that takes away from that focus should be offloaded. If your CEO is looking up people’s availability for a meeting and spends 10 minutes on that task, the company is worse off. I believe this same concept can apply to Engineering teams. Companies spend huge sums of money optimizing manufacturing processes but most spend relatively fewer resources on optimizing engineering productivity.

I’ve been in software for 20 years across a number of companies and organizations from small startups to large enterprises. They all have a similar technical structure. You have tiers of technical abilities. You have an upper tier with the technical visionaries, architects (hopefully not ivory tower) and principal engineers. Then you have senior engineers, mid level and jr developers or interns. Engineers in the uppermost tier tend to have a multiplying effect in the organization.  A big part of their role is to provide technical leadership, drive the vision at a technical level, mentor and tackle some of the harder problems in the org. A lot of times this group is paving the way for the core IP of the company. They also try to look ahead and make sure the company is in a good spot for the future.

Some days are spent writing code, software abstractions,  debugging many of the critical issues that come up. There are also weeks when they have to perform due diligence on a potential acquisition, or do detailed evaluations of potential new partners and/or vendors. They may be frequently asked to de-brief C-level execs on architecture, future planning, or potential product features.

Like a CEO, their time is sparse and valuable. In any typical day for these individuals there are pockets of time spent on mundane tasks that could be offloaded. Unlike an internship where someone is temporary and perhaps one day full time, the new EA (engineering assistant from now on) is a full time hire that would be embedded with their engineer. When your engineering teams are critical to your business you want your best and brightest minds focused intensely on solving the hard problems, technical challenges and communication around technical directions with the teams. Just as a CEO, spending 10 minutes on scheduling a meeting and finding free slots for 20 people isn’t a good use of their training either. This isn’t about making time for people to play more golf, it’s about matching roles with problems.

Line Fitting

As an example below, we have an engineer with a blue line representing the expected impact they have in the organization. The first graph shows tasks as red dots. We see we’re fitting that line pretty well. We have some tasks that make a big impact and tasks that fall just short or about as expected with that person’s performance but generally it’s a well fitting line.

Screenshot 2016-04-17 10.53.29

In the graph below we see an engineer working on tasks that are well below their expected impact levels. We have a mismatch of expectation and reality. We need to make sure we’re fitting our lines properly.

Screenshot 2016-04-17 10.54.09

“The last source of major competitive advantage is the selection and placement of people” – Drucker

Just as startup CEOs don’t usually have an EA, the Engineering EA would typically apply to mid-size and large organizations where differing geography and increased number of internal teams teams are likely.  As a company gets larger and more spread out, there’s more connection points that require better communication. It’s all about opportunity cost. Did your company miss out on shipping a major release because your top engineers spent 10% of their time on non-critical tasks? If you had paid an EA $50,000 a year, could that EA’s engineer have shipped faster with higher quality and made the company $3 million from hitting the marketing deadline? That’s what I mean by opportunity cost, and it happens all the time in software.

When I look back at previous jobs and I try and think about the people I’ve worked with in those roles, I started to compile a list of some of the things that having an EA would have helped with. When I think of an EA I think it’s also a great opportunity to unlock potential talent. Most engineers in software nowadays have to have comp sci degrees and training in data structures, algorithms, etc.. You could hire an EA with a much lower barrier to entry and cost. A smart, hard working person with a desire to break into the field but doesn’t yet have the experience or education could be a great fit for this role.  Someone that could be trained and mentored by the orgs best talent. I’ve seen a quite a few folks like that shine in organizations in the past. You would build a relationship with your EA and they would eventually learn how you like to work and the relationship becomes more efficient over time when things can be predictably achieved. A true test of success would be an EA transitioning to an engineering role within the organization and in turn spreading the great lessons they were taught.

Here are some example tasks that are part of many more senior engineers’ daily lives. Some of these are more realistic depending on how the devops/dev owns qa interactions are in your company.

  • Meetings
    • Jennifer is having a meeting on Friday about our new “foo” that I’m really interested in. I’m pretty swamped getting this release out. Could you sit in that meeting, take notes and message me if there’s anything I should be in the loop about.
    • Can you schedule a meeting with our database vendor when it works for both our schedules? I need more details around that issue we hit last week. Make sure to invite Aaron from the site reliability team as well.
    • I need to fly to the Calgary office for a vendor meeting. While I’m there, I should probably do some architecture reviews with the teams there. Can you schedule  reviews for the 2nd and 3rd day with the Foo and Bar teams? If that works then msg me over a copy of the design docs from their latest revision that I can review on the flight.
  • Communication
    • We settled on a new abstraction for accessing our datastores. I want to give a quick tech talk to the team and show them where to find all the docs and answer their questions. Can you set up a town hall with all the engineers two weeks from now? Make sure we have the conference room booked so the local folks can be in the same room. Have that place that everyone loves cater in some food as well.
  • Email
    • Could you triage my email in the mornings. If anything from support, production notices or questions from executive staff come up ping me right away, otherwise triage and label any anything else and schedule something on my calendar to make sure I respond to those items.
    • We’re really in crunch time, can you monitor my email and chat messages. I’m going dark for two days with the team to help the team ship this feature for our next release. Ping me if any emergencies come up.
  • New Application / big change released
    • I just released that new application into production, can you monitor the logs first thing in the morning for the next week and report any  anomalies to me while I build out the first version of the monitoring dashboard.
    • Can you build out a dashboard for the logs from that new application and set up the alerts to tie into our alerting systems. I want to also be paged any time database latency goes over 100ms for the next 2 weeks.
    • Can you set up the monitoring dashboards for the new App, I labeled all the meters I want in there with the suffix KPI.
    • While I’m setting up the test automation, can you run through the app for the next week and do these 5 things, then check the logs and metrics and make sure everything is still running smooth.
    • Can you work with the QA team and get external monitoring setup for these 6 HTTP endpoints for this new service I just launched.
    • I wrote a few basic integration tests so far, can you finish up that project and make sure we get better coverage on how foo interacts with bar?
  • Support issues
    • I just got a call from the support team. One of our large customers is complaining about the App the team just released and they need me to help triage. Can you do me a favor and grab the logs of some sample hosts from our logging system while I get my test environment set up? Filter out any of the info statements for me as a first pass.
    • I just got a call from the support team. One of our large customers is complaining about latency in the App the team just released can you check out the metrics dashboard for the past 2 weeks and see if there are any spikes on latencies anywhere, then ping ops to see if there were any network related issues in the past 2 weeks we should be aware of.
  • Pre-Deployment
    • I need to have a meeting with our operations team on the requirements for this new role in production. Can you put together the list of services we need to talk to, the environment variables and settings that need to be made, then schedule a call with the ops team.
    • We’re going to need a new role for this project. Can you set up the role template, get all the directories and setup in place. Work with ops or our internal tools and get a running server up in the dev environment we can start iterating on.
  • Simple scripts / automation as they progress in skills….
    • Can you write a quick command line script that does A, B and C then get that out to the team? I think that will save everyone a few minutes of their day.
    • Sarah sent me the new ElasticSearch schema for users. Can you do a quick once over and make sure it matches up with our data model in the code? Let me know if there’s any differences in structure. I’m planning to look at it tomorrow but it will save some time if you check for low hanging fruit.

 

As you can see those tasks above are usually part of everyday life but you’re not paying those people some of the highest salaries in the company to focus on those things. Having an EA that can be trained and mentored with an engineer can be a valuable competitive advantage and you need to look to optimize your organization at all levels.

So what do you think? Crazy idea? Worth exploring?  Hit me up on Twitter and let’s talk about it https://twitter.com/jimplush

Moving a team from Scala to Golang

Scala has long been part of the CrowdStrike stack, the primary language in fact. I helped lead the adoption of Scala as we first started to develop our applications back in 2012. In fact it was one of pros for my decision making process of wanting to come to CrowdStrike. Several of the early developers were interested in adopting it as well so it seemed to be a nice fit.

I had come from a company called Gravity, who were also heavy Scala users. It was the primary language there. I was used to it, enjoyed it, saw the power of it and thought I could prevent some of the issues I saw with Scala as CrowdStrike grew. We were doing high scale analytics, batch jobs over Hadoop and our Chief Architect (hi Bissel!) was doing lambda architecture before it was what the cool kids were doing.

A recent quote from one of our senior engineers prompted me to finally write this post describing why we’re transitioning most of our stack to Go and why new services default to Go by developer choice.

Instead of waiting until the end of the this post I should clarify that Scala will not be leaving our stack completely. In fact it will complement where Go does not shine. Scala is a big part of our machine learning / analytics stack. It’s interop with java projects we use, and its ability to provide a nice DSL that our analysts can use still make Scala a solid choice. It’s becoming more of a specialized tool vs the core development language.

I’m going to take you through this from the lens of a Technical Director. A lens where you need to scale a company from the early days of 5 engineers to 200+ engineers as the business grows. It’s about having a maintainable code base where you can have people cross projects easily and get new hires up to speed rapidly.

I remember when I first saw the potential issues of scaling Scala at Gravity back in 2009/10ish. It was close to the end of the day when we had a major issue reported in production that was affecting our large customers. Several of us started investigating and were able to track the source of the issue. The only problem was we had no idea what the code was doing at first. We came across a strange symbol we hadn’t seen in our projects before. The spaceship operator <|*|> . Someone said out loud “what the hell is that?”.  There was some other implicit magic going on that wasn’t immediately apparent. A CMD+B to traverse into the method yielded nothing in our IDE as it couldn’t find the symbol (IDE’s have improved here since). A quick googling of “<|*|>” yielded nothing as well. We were stumped and didn’t have sources pulled down. [1]

Screenshot 2015-11-21 11.20.00

The developer who wrote the code was unreachable on vacation so we had to figure it out. We noticed a new library was being included called scalaz, hours later we had tracked down the mystery symbol and grok’d what was going on, made the fix and life was good again. That blip turned a fix that should have taken minutes into a fix that took hours. That was the point I started seeing the split in our engineering team.

Scala is a powerful language, it comes from academic roots and gives enough flexibility that you can easily start writing “write-once” type code.  Scala developers typically travel down two paths: You have the “it’s a better java ” camp you have the “I (heart) Applicative Functors” camp.

Screenshot 2015-11-21 12.07.04

The “it’s a better java” camp like the terseness of Scala, and  the standard features that make Scala generally more enjoyable than Java. There’s functional programming in their new Scala code but it’s not the main focus. The “I (heart) Applicative Functors” camp really takes to their new functional world and begin to expand their knowledge deeper down that path or they bring their already functional backgrounds from places like Haskell.

Based on my experiences you start to split down these camps and have excellent programmers in each so you can’t say one side is superior to the other. On the semi-functional side you potentially have more generalists working across languages, or those who may not want to learn lambda calculus theories to work with an API server.

As an example, this is a code sample from a project we had one of our more advanced Scala developers start:

Some of you may look at that and say awesome, but some will say WTF is that? There were thousands more lines like above. This was to be something the whole team could work on but half the team didn’t want anything to do with it. The developer who wrote it is a brilliant person but the fact it divided half the team was a problem. Luckily this was caught in code review and rejected based on our internal guidelines. It never made it out to production.

As you’re scaling an engineering team this split becomes more apparent when trying to get new hires up to speed. Scala has a lot of rough edges already around getting a build environment, SBT pain, IDE environment pain , release upgrade pain, slow build times, add on top of that a heavy dose of functional concepts required for proficiency and the ramp up time grows and dev output slows down. Now there’s more upfront training from existing developers required which slows things down as well. SBT is also a real sore spot. There’s always that one person who actually knows what the heck SBT is doing and can debug everyone’s issues. I’m aware that Scala is not scalaz and they can be mutually exclusive, but I’ve seen issues with or without it.

It’s also not a question of Scala being “too hard”. I’ve never had someone not be able to learn the language. It’s about the investment. You make an investment in something with the hope of it paying off somehow down the line. Whether that’s faster time to market, higher performance/lower cost,  or increased stability. There are specialized places I’ve seen that happen with Scala but not in the general case. We discussed as we were about to scale up if we wanted to invest in more docs, guidelines, example projects, etc… but the reality was we didn’t think that investment would come back vs Go.

This isn’t unique to companies I’ve worked for. Twitter has gone through the same growing pains, as well as other companies I’ve talked to at conferences and people I know working with Scala. It’s a common theme, in fact. While you can have very high performing small teams going with Scala, trying to grow and engineering organization > 50 is an uphill battle. If you’re already invested in the JVM, Java8 is a solid choice that borrows some of the concepts of Scala to make Java easier to work with.

Other references from larger teams seeing issues:

Yammer   http://codahale.com/downloads/email-to-donald.txt

Is LinkedIn Moving off of Scala? https://www.quora.com/Is-LinkedIn-getting-rid-of-Scala

Former Twitter Platform VP heard saying Java8 may have been a better choice if available when they made the choice of Scala for similar reasons https://www.quora.com/Is-Twitter-getting-rid-of-Scala

You can also see a trend at Twitter where the latest OSS projects released are in Java (Heron, DistributedLog, etc..) In fact a telling line in the Hero release reads: “It is written in industry-standard languages (Java/C++/Python) for efficiency, maintainability, and easier community adoption.

Is Scala on it’s way out? (article):

https://www.linkedin.com/pulse/scala-way-out-owen-rubel

 

and a funny Tweet about the subject:

Screenshot 2016-02-17 07.52.50

That’s where Go enters the picture. One of Go’s reasons for existence is to make developers more productive, limit the number of ways you can do something and have a very opinionated view of the world at the compiler level. I pushed back on Go adoption internally for a while. I worried about splintering even more, having another language for people to learn as we already had quite a few technologies in play. We have an internal policy about looking at adopting new technologies when you have at least 3 people willing to support it at 3am if there’s a production issue. After much prod’ing (thanks Sean Berry) and getting past that 3 number, I dug in to the Go world and saw it solved a lot of issues I had with Scala at the organizational scaling level.

Fast build times, small binaries, one file, built in formatting, great tooling, built in test framework, race detector, visual profilers, a nice concurrency model? Wow, sold! We did a sample project in Go that was successful, then another, then another, expanded out the number of developers we had on Go and it started to become the language of choice people wanted to write in. You can jump into any Go project and know immediately what it’s doing. Do I miss immutable types and some of the great features of Scala? Sure do, but I think the maintainability side of the story is too great to overlook with Go. We’ve seen faster ship times, better stability and better test coverage being written.

One of the other benefits of Go was widening the pool of backgrounds we can hire. We can take someone from any language background and have them ramped up on Go in weeks. With the Scala side there’s the JVM learning curve, the Java world of containers, black magic of JVM tuning, profiling tools,  etc…

New developers we’ve hired are ramped up in weeks vs months (we have lots of services that operate at extremely high scale, across several divisions).  We now have the majority of our services written in Go and one of the last holdouts to move to Go just wrote his first project and said to me afterward “Wow, I read through that library once and I knew exactly what it was doing , I’ve read the Scala version of that library four times and I still have no idea what it does, I can see why you guys like it so much”. That was one of our senior engineers who’s previously worked for one of the largest web properties in the world. This process was a complete bottoms up initiative from our development team who pushed for the move to Go.

We now process hundreds of thousands of messages per second and Terabytes of data per day with our GoLang services. Some try to equate Go’s simplicity with weakness, I’ve seen the opposite. You can do some pretty powerful stuff in Go. There is power in simplicity. The error handling, while seemingly annoying at first, actually has lead to more robust error handling and stability in application code. You can’t just throw something and hope it gets caught somewhere.

Google is also a major investor and having that technical alignment opens up access to additional resources that we can leverage.

I’m not here to bash Scala, or ScalaZ (I love ValidationNel! ) but more to give some real world context of Scala in a production environment over 7 years with two companies. I still use Scala and love hacking in Scalding.  Some of our more ambitious projects coming up will most likely be Scala based but I’m just not as sold on it anymore as the core language when trying to scale a fast growing engineering team. There are always exceptions and if your team really loves Scala you can make it work, and some companies are.

Go sits in a place that happens often… when you need small’ish, high performance services. Where you’re doing light transformations, shuffling data around and putting APIs in front of data or supporting systems.

Go just makes it too easy.

 

If you’re interested in hearing more or chatting, follow me on Twitter https://twitter.com/jimplush

…and we’re hiring 🙂

 

Updates:

** Marius, one of the twitter Scala gurus, also seems to be getting keen on Go.

screenshot-2016-09-24-08-58-56 screenshot-2016-09-24-08-58-42

 

 

68747470733a2f2f7261772e6769746875622e636f6d2f676f6c616e672d73616d706c65732f676f706865722d766563746f722f6d61737465722f676f706865722e706e67

——

[1] Code Reviews would have solved this issue as someone else would have stumbled on that symbol and asked for more clarity and understanding. That would have help introduced the library to the team easier than a surprise. Unfortunately, at Gravity we didn’t have required code reviews in place. We do have code reviews in place at CrowdStrike. However, given not every person is on every code review there’s no guarantees.

 

 

 

Path to Productivity: No Meeting Thursday

At small startups there’s usually very few meetings initially and the number of people you have to communicate with on a daily basis is typically quite manageable. You feel productive and you’re cranking out code, designs or other tangible deliverables. Life is good! Your social graph at work looks similar to the graph below.

small-social-graph

Hopefully, your company is successful and you start hiring. As you hire, your social graph at work also grows. Perhaps not on every hire but as the company expands the number of people you need to interact with will go up as well. As you get even bigger your social graph looks like the new graph below. Now things aren’t as fast. You need more co-ordination, more people have to understand what’s going on. You start to have more meetings, and if you’re distributed, potentially even more.

large-social-graph

 

You start to become sad that you’re not as productive as you once were but understand it’s part of growing. Then you get to a point where you feel like you’re in too many meetings, they’re scattered throughout the day, you’re working in bits and pieces but can’t really tackle anything of substance. You need a long stretch of time for understanding a problem, building solutions and algorithms in your head. You find yourself having to work after-hours or early in the day just to be able to chew on meaty projects and feel somewhat productive again. If you’re in a big company this process can happen even faster.

More than half the engineers I interview cite “too many meetings” and “not feeling productive” as one of the key drivers as to why they’re looking to leave their current company.

After hearing this at my previous company during interviews I wanted to instill a guard against that early on at CrowdStrike. Very early on we put in place the concept of “No Meeting Thursday”.  Meetings were not a huge problem early in our life, our social graph was small. However, I knew as we grew and expanded there would be more interrupts throughout the day for engineers. I also know if you bake things like that into the culture early on, it’s easier to fight to keep it, than to fight to instill it.

So with that, I put a calendar invite on everyone’s calendar that blocked off the whole day for No Meeting Thursday. The goal of No Meeting Thursday is not to have a “don’t talk to me” day. It’s about not having anything scheduled or re-occurring to give engineers (and other groups) the chance to own their own schedule. You’re an adult, do the right thing.  If they need to be alone for 8 or more hours to really dig into a problem, great that’s what the day is for. If someone is working on a team and that team wants a whole day of uninterrupted time to focus on design, testing or anything else related to their project and everyone is on board then great, that’s what the day is for. Obviously, production issues still come first.

As new executives come in there is a training period of “What’s this No Meeting Thursday thing about and why did you deny my meeting invite.”  I wrote this post to reduce the number of times I have to explain the concept so here it is….

Managers and Engineers are usually on different schedules. Paul Graham famously calls it “Manager vs Maker schedule“.  Managers are used to being interrupted and work on broken chunks of the day. They work in blocks of time and usually have calendars that will make you cry. Managers can go from meeting to meeting with minimal context switching. Engineers on the other hand work in long stretches and need hours to get into the zone where they’ve built a problem in their head enough to the point then can work directly out of mental RAM. That’s where the productivity sweet spot is.

I like to think of meetings this way for engineers… You’re an engineer, you’re building  a mental chandelier in a big, tall entryway. You have a high ladder where you climbed to the top and you’re starting to put together your big, beautiful chandelier in your head. You added the base, you start adding crystals, things are going well. You’ve got a great mental model of your problem and some corner cases.

chandelier-ladder

 

Then a calendar reminder pops up, reminding you that you have a meeting in the middle of the day. You head to the meeting to talk about something completely off topic and Boom!!, Crash!! Your mental chandelier starts to wobble and fall, crashing on the floor. Not all the crystals are busted but you definitely are starting to swap out that mental RAM for this new topic in the meeting. Meanwhile you’re actually still thinking about where you left off in  your other task so you’re only somewhat paying attention to this new topic. So you’re losing context on your previous project and this new subject is getting half your attention.

chandelier-ladder-broken

The meeting ends and you have to clean up to start rebuilding your chandelier. Hrm, where was I?  You may get back to your desk and you think you’re already interrupted so why not just check email real quick. You read an email and start to think about that and now you’re two levels removed from your actual project. Ok time to refocus… This could take anywhere from minutes to hours. If it’s towards the end of the day you might just say forget, you’ll start again tomorrow and work on little tasks the remainder of the day. I tried to find the source for where I read that chandelier analogy before but couldn’t find it. If someone comes across the source feel free to drop a comment so I can attribute it properly.

As someone who’s been in both the management and and engineering side I can validate for myself and many others I’ve talked to this is the way it goes. When I’m in a managers schedule I know I’ll be interrupted all day so things involving deep thought are left for early in the morning, or after office hours. In my opinion a good manager needs to understand the affect random meetings have on engineers and the time wasted going in and out of context.

The phrase of the day nowadays is “more productivity”. To put it as bluntly as possible. Would you want to work at a company that couldn’t even give you 8 hours of uninterrupted work time? How can someone be productive if you never give them long stretches of time? It sounds silly when you say it out loud but that’s what a lot of companies do by default. With so many connections people could be interrupting each other left and right.

Scheduling random meetings in the afternoons? The worst. If you are going to schedule meetings for engineering teams, keep them in blocks. If I’m already interrupted at least keep it going so I can get a long stretch of time in the afternoon or in the morning. Good managers keep tabs on meeting workload and fight to reduce it. They take meetings for the team and distribute information asynchronously or ask that meetings be changed to discussions in a chat tool, or via a ticket as done in the open source community.

I’ve heard of other companies doing no meeting afternoons, however, for distributed teams in many time zones that makes it hard to establish when that time will be. Having a full day evens it out more across the globe.

Meetings can be fruitful but can also be highly disruptive. For engineering teams please be conscious of the effect meetings are having. Because if you’re not, you may find yourself having to hire replacements for a situation that could have been avoided.

We’re 3 years into No Meeting Thursday. I put out an email to the team recently on if we should preserve NMT and if people thought it was still effective. The responses were an overwhelming “please keep NMT, fight for it and let me work”.

 

If a culture like that interests you, ping me… we’re hiring 🙂

or follow me on Twitter: https://twitter.com/jimplush

 

thanks for the reviews from:

John Kurkowski

Roger Clermont

1 Million Writes on 60 nodes with Cassandra and AWS EBS Service

The videos are now online from the 2015 Amazon Re:Invent Conference where I had the privilege to speak on behalf of CrowdStrike on how we were able to achieve 1 million writes per second on 60 nodes with Amazon EBS.

We put a good 3+ months into testing and benchmarking Cassandra with EBS and getting the current EBS story with their team  to understand what improvements were made in regards to availability and performance since the outages of the 2011/2012 timeframe.

I’m happy to say that because of our work with EBS and DataStax, that DataStax has now given the green-light to run EBS GP2 in production with Cassandra. Get ready to save some $$!

Below is a link to the presentation

Microservices allow for localized tech debt

At my current company we employ a microservice style cloud architecture. One of the side benefits of this approach has been our ability to localize our tech debt into size-able chunks. When you’re a startup you need to be able to build things fast and fail fast if you need to pivot. Sometimes being perfect is the enemy of building a company, and sometimes it is not. This is where microservices can help mitigate long term issues and allow you to apply perfection where it could block you from growth and allow tech debt to pool in places where you can address it in a reasonable manner in the future,  essentially allowing tech debt to truly be like financial debt. It’s much easier mentally to tackle $10,000 of debt across 4 credit cards at $2500 each than 1 card at the full $10,000. It allows you to apply a sharper, more targeted focus that gives a more immediate gratification. Let’s level set on the phrase tech debt for the purposes of this article. “You want to make a “quick change” to your software of a kind you mean to be able to, and it isn’t quick.  Whatever made that happen, that’s tech debt.” – Dave Diehl.

When trying to build a microservice architecture at a startup you need to identify what are the core services you need designed and well structured to allow for future growth that if you get wrong will hurt your chances of being able to move at a fast, safe pace.  At first glance this might seem easy. We’re going to be a huge consumer play and will have a billion users so you go off and build a complex database sharding layer from scratch, because hey, you’re gonna grow too big for a single master DB.

This however will be highly dangerous. Startups are fluid organisms. Assumptions made on day 1 can quickly become invalidated in practice. In a previous startup where the previous founding team was from an extremely large scale web property and scalability was baked into their DNA, when it came to decide on how we’d store user information for our new product it quickly delved into a data center order of half a dozen DB class master boxes with replicas while the dev team started on a user sharding layer that took weeks of effort. In the end the product didn’t take off, that code was ripped out in multi night hackathons and the DB boxes were re-purposed for other projects leaving a teeny little box to handle the load. That was missed opportunity cost to work on features the users wanted by speculative investing. Perhaps instead just stubb’ing out the interfaces and APIs in such a way that all of the sharding/routing logic was localized to set of client methods and in the future those methods could be fully implemented or backed by a remote service.

Proper technical debt assessment is making known tradeoffs. If you don’t even know the tradeoffs you’re making that’s not so much technical debt as poor engineering and planning. As long as you focus on getting your APIs and interfaces modeled out you should be able to refactor the details behind those if the service turns out to be something that becomes core to your platform. Creating smaller, standalone microservices allows you to iteratively build scalable, well tested, easily digested services. At CrowdStrike we wrote some of our initial services in Python for quick prototyping, but focused on the API interactions with those services and less on the scalability of those particular services. For one particular service as we ramped up and had 10 of those Python boxes running with questionable stability we refactored them to GO (golang) and reduced down to 2 boxes. We actually only needed 1 box but used 2 for redundancy. The API did not change whatsoever and clients happily connected and were serviced as before.

We took on tech debt to test our hypothesis that those services would be critical down the road. It turned out they were and we were able to swap in highly performant, stable, tested code in place and paid off our technical tebt. $2500 less debt! Now to time to get that AMEX bill 🙂

If those sound like fun problems to work on, join the mission at CrowdStrike: http://www.crowdstrike.com/senior-software-engineer/

 

techdebt-blogpost

 

Cassandra’s DateTieredCompaction Strategy does not work as billed

UPDATE: While at first DTCS seemed like a workable solution, after running it under real world workloads, while needing to scale up the cluster, DTCS broke down completely. I recommend looking at TWCS by Jeff Jirsa of CrowdStrike. It’s what DTCS should have been. Reference: https://issues.apache.org/jira/browse/CASSANDRA-9666

 

I’ve been looking at throwing Cassandra at a use case I had come up. Storing billions of items a day but only needing to keep that data for a couple weeks on a rolling window. I was obviously nervous about having to TTL out that large of a data volume on a rolling window. Basically write(2TB), delete(2TB) in the same day. I started on the hunt on the latest C* docs and came across a gem that was recently released in the 2.0.11 release contributed by Spotify: DateTieredCompactionStrategy.  This strategy is great if you’re looking to have a table that is just on a rolling time window. Use cases like time series and analytics where every write has the same ttl and it comes in a forward only manner, meaning you’re not backfilling data later. The feature that really interested me was that it can look at an SSTable to determine if the whole table is out of expiry vs having to spend CPU cycles on merging and dropping data. rm *foo-Data.db. which should make large scale TTL’ing much less I/O and CPU intensive. I wanted to see if it lived up to the hype so I set up a cluster of 3 m3.2xlarge machines and created brand new keyspaces and a new table defined later in the article. I fired up another loader machine to act as a writer and tuned it to write 10,000 events per second to the new cluster on a continuous stream for 12-24 hours. Here’s what I set up

  • 3 node dse 4.60 test cluster with Cassandra 2.0.11
  • created a new table with the DateTieredCompactionStrategy
  • inserter in golang using gocql writing 10,000 inserts per second causing a load of 3 on an 8 core box
  • each insert had a 30 minute TTL with “USING TTL 1800”

I started the test hoping to see a new saw tooth pattern of SSTables on disk and disk space consumed by that column family. Instead to my shock it just kept going up and up all day. I scoured the docs but came up empty. Datastax was kind enough to point out it may be related to the “tombstone_compaction_interval” which defaults to 1 day. That basically means an SSTable won’t be a candidate for deletion until after that date. Once I changed that setting to 1 everything seemed to work like a champ. This was before I set the interval for testing

Screenshot 2015-01-30 16.01.30

after!

Screenshot 2015-01-30 16.02.13

You can also see with DTCS it results in a nice sawtooth on the SSTable count.

Screenshot 2015-01-30 16.04.02

 

For a comparison, below is the exact same test with LeveledCompaction instead of DateTiered. You can see the data volumes just continue to grow.

Screenshot 2015-02-03 08.05.18

Screenshot 2015-02-03 08.08.26

After 24 hours the leveledcompaction table has 10x the data on disk on a single node. Sadness.

Screenshot 2015-02-03 22.13.32

 

**UPDATE**

30 hours later the cluster running the LeveledCompaction 30 minute TTL inserts died a horrible death. It filled up all available disk space on a couple nodes and started crashing the Cassandra process. </endofcluster>

Screenshot 2015-02-04 09.05.01

If you want to come work on high-scale distributed systems, we’re hiring!

http://www.crowdstrike.com/senior-software-engineer/

 

here is a create table script if you’re interested in a test on your own hardware.

  CREATE TABLE IF NOT EXISTS ttl_test (
	id text,
	metatype text,
	event_time timestamp,
	rawbytes blob,
	PRIMARY KEY ((id, metatype), event_time)
) WITH
  CLUSTERING ORDER BY (event_time DESC) AND
  gc_grace_seconds = 0 AND
  compaction={'class': 'DateTieredCompactionStrategy','tombstone_compaction_interval': '1', 'tombstone_threshold': '.01', 'timestamp_resolution':'MICROSECONDS', 'base_time_seconds':'3600', 'max_sstable_age_days':'365'} AND
  compression={'sstable_compression': 'LZ4Compressor'};
session.Query(`INSERT INTO ttl_leveled (id, metatype, event_time, rawbytes) VALUES (?, ?, ?, ?) USING TTL 1800`,
		randomId, metaType, eventTime, rawBytes).Exec()

Leveled Compaction Table Create:

CREATE TABLE ttl_leveled (
  id text,
  metatype text,
  event_time timestamp,
  rawbytes blob,
  PRIMARY KEY ((id, metatype), event_time)
) WITH CLUSTERING ORDER BY (event_time DESC) AND
  bloom_filter_fp_chance=0.100000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.100000 AND
  gc_grace_seconds=0 AND
  read_repair_chance=0.000000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  compaction={'tombstone_threshold': '.01', 'tombstone_compaction_interval': '1', 'class': 'LeveledCompactionStrategy'} AND
  compression={'sstable_compression': 'LZ4Compressor'};

Your guide to the 2014 Amazon AWS re:Invent Conference

I’ll be attending my 3rd Amazon AWS re:Invent conference this year (as a speaker this time!) so I thought I’d put together some of the tips and tricks I’ve found over the last two years.

Screenshot 2014-11-08 14.23.34

I’m going to take the view of the advanced AWS user. If you’re new to AWS congrats, everything will be new and great. If not then you’ll want to think about some of the points listed below. Overall re:Invent is a fantastic conference and it’s huge.  Amazon spares no expense. Rumor is they put up 6 figures to the Venetian to bump up their wifi capabilities in the first year to support a growing number of techies attending. The parties are pretty awesome and everyone on staff is amazingly helpful. The first year was great but sometimes painful… over packed rooms and all the good talks were mostly unavailable. The netflix talks had lines around the casinos and you had to wait in line a full hour earlier and miss a talk to get into some of the advanced sessions. bleh. Last year they took notice and expanded all of the rooms and I had no issues with waiting or missing a talk I really wanted to attend. I have no reason to believe this year will be different, should be plenty of space for all.

(Photo of one of the lines for a Cockcroft talk the first year)2012-11-28 16.16.03

Tips

One thing I’m not super excited about is the sheer number of simultaneous talks. At one point there are 19 simultaneous talks going on. That’s way too many IMO. You usually want to hit 2 or 3 that are happening at the same time  so it’s sort of a bummer. They should have fewer, higher caliber talks. Last year there was a good deal of “salesy” type talks where after 10 minutes you’re asking yourself why you just skipped that other talk you wanted to attend.  Generally avoid any 100 or 200 level talk from an enterprise vendor or large company. Tweets on those last year were pretty bad, more sales pitch than useful information, and I walked out of a couple myself to catch the rest of another talk. Example: “We utilized AWS to streamline our customer product delivery system, improve our response times and increase our sales output by 30% , we won’t tell you how, we’ll just boast about high level shit we did and how you can integrate with us now” :/

Tip1

If you’re there to learn some focused information attend as many 400 tracks as you can. Amazon actually puts on some good advanced topics such as the internals of EMR map/reduce, EBS performance deep dives, etc… If you’re looking to learn, that’ll get it done usually. There aren’t a lot of them, they usually sell out and are packed. Get there very early so you’re not sitting on the floor.

Tip2

Go New! Go to talks you know nothing about but may have heard or may have some interest in. If you already know the ins and outs of AWS then most talks will leave you wanting for more. You may walk away with one or two nuggets but with only 45 minutes, 20 mins of that is usually intro/setup and 25 mins for meat. That doesn’t leave a lot of room for deep inspection of topics. Instead let that 45 minutes be completely new. You’ll get exposed to more things and actually learn more in that 45 minutes. For the topics you have experience in, wait for the videos to come out to see if you missed anything. When in doubt, flip a dice (although with so many tracks you’ll need 3 dice)

Tip 3

Power!!! you’ll eventually need power to charge your laptop and phone. In a 2,000 seat room, the only power plugs are usually against just one of the walls, since most walls are moveable. You’ll want to get to a talk early to grab a power seat. They go fast. Better pro tip would be to bring a power strip with you so if it’s full you can unplug everyone, plug in your power strip, reconnect everyone and add yourself. You might even get a thank you that someone else can now plugin next to you 🙂

Tip 4

This annoys some people but is the only way you’re going to get to the next talk in time to get that sweet power… sit in the outside seats in your row. If you’re in the middle, after a talk it’s about 10 mins to empty out the talk so you’ll be waiting for some turtles ahead of you. If you’re on the outside you can break out right as the talk ends and get a nice spot in the next talk. Be somewhat courteous and make yourself small so people can get by easier.

Tip 5

Ready for dining? Public House  is the spot where the cloud gods hangout. Here you’ll find your Cockcroft, Vogels, and Berkholz’s of the cloud world. It’s a great spot for beers and people watching. You’ll get to interact with other techies and talk shop over some great craft brews. For breakfast, the best pancakes I’ve had in my life are at the Grand Lux Cafe . That’s where you’ll find me every morning 🙂 I usually gain a couple pounds just from breakfast but it’s worth it.

Tip 6

Get outside! The first year of the conference I realized on day 3 I hadn’t seen the outside world in 3 days straight. The Venetian has everything you need and more. Kind of scary actually… make it a point to stretch the legs and get some sun on your face.

Tip 7

Go to the parties! AWS spares no expense with their parties either. Deadmaus was pretty awesome last year, you could walk right up to the stage. They had laser limbo, mini helicopter landing games, old school arcade games and more. It was a good time, throw your anti-social caution to the wind and just go hang out for an hour.

2013-11-14 22.31.40

Tip 8

The shirts! I still wear my re:Invent shirt from 3 years ago. They last forever and are high quality. I wear my re:Invent shirts every week still. Make sure to get yours at the party.

Tip 9

Go to plenty of Netflix talks. They are some of the high scale AWS pioneers and have paved the way for you. Hear some of their problems/solutions and it will help you with your architecture. They also have valuable content. All of their talks are good, they’re all about the bass.

Tip 10

Vendor booths…. Lots of vendors to see and talk to at the conference. They all give shirts away so get there early if you want swag, they run out pretty fast. Most vendors were pretty knowledgeable and the companies sent their best reps out for this big conference. Be forewarned, if you let vendors scan your badge for a shirt you will be spammed for weeks afterwards. Fair trade I think.

Tip 11

Use twitter heavily. Follow the hashtag #reinvent to keep up to date on good talks, people trying to get together to discuss topics, and locations of beer meet-ups. If you’re looking to network that’s the best way to keep up with the pulse of the conference. I met a great engineer over twitter using the #reinvent hashtag who wanted to talk shop and have a beer at the public house. We gabbed for a few hours that night. Shout out to Martin Cozzi 🙂

Tip 12

Utilize AWS Solutions Architects. In the vendor area Amazon will have solution architects available for every service they have. Bring your problems and they will get on a whiteboard with you and get into the details. Wanna know the best Dynamo schema or write capacity heuristic? Wanna know if your EMR workflow is sound or stupid? Someone will literally sit with you and get you a solid solution. They’re there, free and they have their most senior people attending. Take advantage of that knowledge sharing.

Tip 13

Werner Vogels is cool as shit. You may not meet another C-level exec as cool as Vogels with an Amazon size market cap. He does his presentations in Nirvana shirts and converse. I was lucky enough to meet him at the party last year and he couldn’t have been more excited about the event and friendly to talk to. Don’t be afraid to go up and say hello. You might even catch a nugget of knowledge.

download_20131115_000016

Me and Vogels

Wrapping up

So that’s some tips off the top of my head for attending this year. If you want to see an advanced talk Mr Sean Berry and I are going to be representing CrowdStrike and doing a talk on how we do our blue/green deployment architecture.  APP307 – Leverage the Cloud with a Blue/Green Deployment Architecture We’ll be getting into the nitty, gritty of how we utilize a blue/green deployment model at scale with terabytes of daily data ingested. Kafka, zookeeper, load balancers and more!

Follow me on twitter for frequent updates and if you want to grab a beer while you’re there. I want to have some deep tech discussions while I’m there so let’s nerd out. https://twitter.com/jimplush

If I missed any good tips, leave a comment.

Speaking at AWS re:Invent 2014 conference

Screenshot 2014-11-02 08.57.27

I was lucky enough to be selected to speak on behalf of CrowdStrike at this years Amazon Web Services re:Invent conference on the subject of blue/green deployments. If you’ve been looking to grok blue/green deployments or get to the stage past the high level “just switch your load balancer” type talks then this talk is for you. Leverage the Cloud with a Blue/Green Architecture

We will get into how blue/green works with the data plane including how to use it with a stream processing data system where Message Queues and/or Kafka are involved.

 

Screenshot 2014-11-02 09.00.46

I’ll be co-presenting with Sean Berry from CrowdStrike and we’ll be happy to talk tech over beers after the talk. See you in Vegas 🙂

Talk Overview:
APP307 – Leverage the Cloud with a Blue/Green Deployment Architecture

Minimizing customer impact is a key feature in successfully rolling out frequent code updates. Learn how to leverage the AWS cloud so you can minimize bug impacts, test your services in isolation with canary data, and easily roll back changes. Learn to love deployments, not fear them, with a blue/green architecture model. This talk walks you through the reasons it works for us and how we set up our AWS infrastructure, including package repositories, Elastic Load Balancing load balancers, Auto Scaling groups, internal tools, and more to help orchestrate the process. Learn to view thousands of servers as resources at your command to help improve your engineering environment, take bigger risks, and not spend weekends firefighting bad deployments.

Profiling GOLANG applications using kcachegrind and SVG

One of the things I missed when starting to use golang was departing from profiling tools like visualvm and yourkit for the JVM. Golang has a few options and one of them I’ll be documenting here is profiling your running application and outputting a callgrind file that you can use to understand your running golang app.

 

Screenshot 2014-10-15 12.45.02

 

or via SVG!

Screenshot 2014-10-15 13.01.24

 

To enable  this kind of profiling, follow the instructions for adding pprof to your application: http://golang.org/pkg/net/http/pprof/

Once your app is back up and running with pprof and you’re putting some stress on it then it’s time to check out a profile.

Installing kcachegrind on ubuntu:

 sudo apt-get install kcachegrind

launch it:

kcachegrind

start debugging an application


jim@localhost: ~
➺ go tool pprof http://127.0.0.1:8001/debug/pprof/profile
Read http://127.0.0.1:8001/debug/pprof/symbol
Gathering CPU profile from http://127.0.0.1:8001/debug/pprof/profile?seconds=30 for 30 seconds to
/var/folders/mg/x8pzmrlj6msgkkvvcbpybd580000gn/T/TjhdgALH3Z
Be patient...
Wrote profile to /var/folders/mg/x8pzmrlj6msgkkvvcbpybd580000gn/T/TjhdgALH3Z
Welcome to pprof! For help, type 'help'.
(pprof) callgrind my-cool-grindfile

now just click OPEN on kcachegrind and you should be able to visualize your running application from it’s snapshot.

If you want to view the SVG version then just type in “web” from the pprof interactive shell instead of “callgrind my-cool-grindfile”

 

Notes:
If you get the error:
Failed to POST to http://127.0.0.1:8001/debug/pprof/symbol

edit the following file:
jim@ubuntu:~$ vim /usr/local/go/pkg/tool/linux_amd64/pprof

and change:
if ($@) {
to:
if (1) {

© 2017 Jim Plush: Blog

Theme by Anders NorenUp ↑