Can we stop losing?

To whomever this may concern,

This is a personal post in so much that I need to clear out that I am not speaking for my employer or any other organization that I may be affiliated with. It is also personal on a level where I will not go into discussions in comments, on twitter or any other forum than face to face meetings. However, I think it is important enough to put this out there.

I have lost acquaintances, colleagues, friends. I have not lost them because of fate. I have lost them for stupid reasons. I have lost peers for things that I, we, could have prevented or at least I think I, we, could have. I am DONE with losing people I love for the wrong reasons. I hope WE are done with losing people for the wrong reasons.

My experience with losing started at a rather young age. I was working at my first employer when we were all gathered in a meeting room when we came in on a fateful morning. Some of us were sure that they were going to be fired, some of us were excited because maybe we would get a raise. Nobody was prepared to hear that one of our colleagues has lost his life when he crashed his car against a lighting pole very early in the morning. I was shocked, not surprised. A few weeks prior to his death we had been on the road for an engagement in a remote part of the country. While I was driving, this guy started to roll a joint and it was his clear intention to light it. As I was the lead on this engagement I told him not to but he laughed at me and lit it nonetheless. I kindly asked him to put it out or I would have to remove him from the car. He obviously thought it was funny enough, until I stopped the car and told him to get out. After he got out I returned to the office, told my boss what hat happened and somebody went to pick him up. He wasn't fired, just reprimanded. To this day I believe I did the right thing. It's obvious that a joint didn't kill this person, not enough persons being available to him to guide him about his substance abuse did. I'm also pretty sure that it wasn't just weed. It doesn't really matter WHAT it was, I lost a friend and an awesome colleague.

We have lost enough people for stupid reasons. We have lost enough people because I, we, fail to notice subtle changes in behavior. We have lost people because they felt lost. I believe everybody has at least this one person they can think of right now and say "I wish he or she was still here. I wish I could've helped". I want this to stop.

Over the years I have also witnessed substance abuse in our community and the stupid things people do to "fit in". It doesn't get more pathetic than seeing a dude fake snorting a line of coke to make sure he remained "one of them". I've seen alcohol, weed, cocaine and heroin do things to people you wouldn't wish to your worst enemies. I've seen a lack of support and understanding do the same.

I'm the first to admit that I like to get my drink on. Damn, I got more than enough stories to fill a pretty funny book with. However, I can remember the times that I was absolutely and utterly shitfaced on one hand. In almost 22 years that I'm legally allowed to drink.

So, from here on going forward I won't stop being the guy that wants to have fun. But I will be that guy that asks if you are ok. I will also be that guy that takes you aside and says "I think you had enough". Fuck, I will probably ruin your night but only because I don't want to see you ruin your life. 

I will definitely be that guy that you can call 24/7/365 because I don't want to lose anymore. I hate losing, I want us all to win. 

The enemy within

I happened to find myself in the couch this evening. Somehow I managed to get hold of the remote control and leisurely zapped through the available channels (there's a crapload of them, unbelievable).   I don't think I was really paying attention until I heard someone speak Chinese. While rusty, my Chinese is good enough to understand a normal conversation so my interest peaked. 

The documentary I ended up watching was about the (actually very recent) history of how companies have mushroomed in the GuangDong province, especially in the ShenZhen area. Their business model has always been as simple as it was genius. With the availability of huge numbers of unskilled workers, these companies were able to deliver quality work at prices that were unparallelled. 

As rumor got out that money was to be made, even more workers travelled from the rural areas to make money. Or rather, to make a living. The money they made was used to support their families back home. Thousands of kilometers away. It was used to feed and, more importantly, educate their kids. And with that, something interesting happened. The next generation, still willing to work but more than ever aware of their value and empowered by an improved education, in their turn traveled to the south. They did not have a family back home and their interest was to make an income that supports their desires. As labor became more expensive, companies in the Guangdong province struggled to stay competitive (in the end they all do much of the same) and inventive entrepreneurs found a new Guangdong in Cambodja. Cheap labor, not many questions asked, willingness to work.

Now this doesn't really go anywhere specifically related to infosec but it hit me like a brick that the people featured in this documentary were not unlike you and me. They were normal people, with families, with responsibilities, with ambitions and with dreams. Whether you are in the US, in Europe, in Africa or in Asia doesn't really matter. As a human being (with some outliers) you want the best for yourself, for your family.

This is most probably the biggest problem I have today with the whole "ZOMG China is hacking us and stealing our sekritz"  movement (you know who you are Tao).

Over the past centuries various regimes across the world have passed blame for their failure to innovate on to larger or smaller populations. This has happened in Europe, Africa, Asia, ... It is not new and it has never worked. Many people have died there, often under gruelsome circumstances. 

The fact that companies, individuals and politicians engage in the culpabilization of more than 1/6th of the world population is -to me- a disgusting realization. While I'm sure that there is little doubt that Chinese entities have used hacking methods to gather intelligence and knowledge (or a competitive edge), I don't see a reason why the other players in this game have not done exactly the same. 

We have become our own worst enemies but instead of admitting it, we look for the next country that we agree to demonize.

It utterly sucks.

Caveat emptor 101

I try to read as much as I can. Whether it's articles, books, journals, blogposts doesn't really matter. If it is infosec related I'll soak it up and if it's any good I'll probably blend it into a corpus of knowledge I've gathered over the year.

In the past half decade our industry and community has been finding out what it takes to be relevant. A lot of relevancy is to be customer focused. Note that "customer" does not always imply that you have a commercial relation with the group or person you categorize as that. Naming the entities you deliver for "customers" helps you realize that it is your sole obligation to deliver value instead of firewall rules or anti-virus or secure code.

Caveat emptor is a very old principle (that's why it's written in Latin right?) that boils down to "buyer beware". It basically means that whenever you are in a buying position, you have to be careful that the thing you are being sold is what you asked for and not something else or worse ... something incomplete.

With that said, I introduce you to this post on the HP Enterprise Security blog.

The post is constructed around the failed interpretation of an RFP process, that in the author's experience apparently goes a little something like this :

1. Customer issues RFP
2. Customer makes purchase decision
3. Customer buys just the widget
4. Customer attempts to implement the widget themselves
5. Customer fails to leverage full capability of widget, project falls to the wayside, considered a failure or abandoned.

From this he continues to pass blame for everything that goes wrong in corporate security programs to the customer ... obviously preparing us for the next ground-breaking offering that will wipe out all of our collective sins of the past. We shall be whole again.

I take issue with may things in that premise. I'll try to address them seperately.

1. This RFP process is overly simplified for the purpose of your post. It disregards the role that both vendors and integrators/resellers play in forming the opinion of a customer, often shoving their concerns under the rug with an extra discount or a shiny presentation. 
2. It's easy to put blame on the customer and their "buying" behaviour. In various roles at various organisations in the past I have spent time on all sides of the table in RFP processes. I've written them, I've answered them and I've evaluated them. If there is one thing vendors/resellers can do it is not jumping on any opportunity for the sole monetary benefit it will get them. Honesty goes a long way, even in business. 
3. If you have any experience in running such projects at scale you will recognize the occurences where scope was suddenly interpreted differently because the reseller found out they would not deliver on time if the real scope needed to be covered. In this case it is often the customer that doesn't grasp what is happening to them and they're actively being ripped off. No worry, the project was a success and nobody noticed. 
4. Lastly there is the issue of knowledge transfer. I haven't seen a single RFP where knowledge transfer was not an item. The answers at the beginning of the process promise a unique approach enabling the customer resources to develop themselves into experts. The reality a few months later is bleak, knowledge transfer exists of a 2 page admin manual and a stock preso about the awesome features of widgetX. Box checked, project delivered, let's get drunk!

The post further elaborates on specific experience from the author. The first example (Project ADD) is an example of failure that is rooted rather in a failed project management approach than a failed infosec program approach. This is not something a "reboot of your infosec program" can necessarily fix.  

Now back to "caveat emptor". 

As old as it is, the warning still stands. If you are looking to buy a widget to protect your infrastructure or a consultancy service "to reboot your infosec program", be suspicious of the promises made. 

"Data Honesty" and why IOCs are not (yet)

In the past half decade I've been working in incident response and data analysis extensively, working on projects that helped monitor security-related data on very large networks and helping to building Incident Response capabilities that were empowered rather than paralyzed by that data. It is no secret that I have always been a big proponent of data sharing among peers to improve data quality and more importantly to build what I have started to dub "data honesty". A term I can only hope will find some level of adoption sooner than later.

"Data honesty" is, in its simplest form, the level of maturity where anybody with access to a dataset and the methodology used to derive information from said dataset will arrive at the same conclusions as you did. It is this level where you are confident to publish both your dataset and the methodology used.

There is no doubt that the Verizon Risk Team as a whole has pushed the envelope in this field for our industry. The work they have done in developing VERIS ("Vocabulary for Event Recording and Incident Sharing") is a tremendous effort that I feel our community has largely ignored. It helps to understand that the V in the acronym stood for Verizon in a previous incarnation in order to realize that this is exactly the reason why widespread adoption of VERIS among Incident Response service providers has not occurred. As with any standard, commercial organizations are only interested in the standard if there is an opportunity to OWN the standard (and thus an opportunity to exploit it for profit). It is, unfortunately, a problem that almost any standard in our industry has suffered from since forever.

All this has resulted in report after report based on, undoubtedly, real data but without any structured methodology. This doesn't mean the reports were constructed in bad faith but one always wonders how the data conveniently correlated towards a need for more services/product/... conveniently provided by the publisher of the report (or the one sponsoring it). 

All jokes aside, there has never been more interest in security-related data. To a point where industries are waiting for any data that can help make them informed risk decisions. The latest data points being thrown at us are 'Indicators of Compromise' or IOCs. VERIS covers IOCs here:http://www.veriscommunity.net/doku.php?id=iocs and frankly, I love them. They're little nuggets of information that, when related to properly analyzed incidents can turn an incident response effort upside down. But caveat emptor!

IOCs lose a lot of their value when not related to properly analyzed incidents. There probably lies their biggest weakness. Anybody can publish IOCs these days without the need to link them to any active (or terminated) targeted attack campaign and attribute them to "Wim Remes' bad-ass Belgian Hacking Team extra-ordinaire". It is no longer the thorough methodology that supports the credibility of the IOCs but the commercial power and (perceived) familiarity with the "threat du jour" of the publisher.

When respected researchers and entrepreneurs start calling for "IOC Wednesday" I would urge you to take a step back and look at VERIS first. If you can analyze and categorize your incidents based on the data YOU can gather, you won't need to wait for others to be compromised to protect yourself better. 

IOCs are, at this moment, not by definition "honest data" but you can use them to streamline the processes you build based on your own "honest data".

For those companies with a stake in Incident Response services, now is the time to set aside your egotistical reasons for not using VERIS in your analysis or reports. Any methodology will always have weaknesses but this is the one we have RIGHT NOW and for a flawed methodology, it's a pretty darn good starting point if you ask me. The RISK team deserves that credit and the industry deserves that standard.

2 million downloads and nobody cares ...

As I'm enjoying a little bit of holidays before I start my new job in 2013, I'm also having the privilege of setting up my new work machine. Some tools are must haves for anybody doing infosec work, one of them undoubtedly is 'The Social Engineering Toolkit' (SET) written by Dave Kennedy of TrustedSec.

Dave is awesome and not only for writing SET. This post is not intended to criticize Dave in any way, rather it is written to point out something I've noticed in this community for quite a while now. Everybody cheers when another free tool is released. Metasploit, SET, NMap, Wireshark, OllyDbg, ... you name it, we use it and we throw our hands in the air like we just don't ... seriously, we really don't care.

These tools are written by people, smart people who have limited bandwidth for these efforts. They spend time away from their families to give this community tools that we use to do our jobs ... the least we can do is give something back.

Now, I remember sitting in a talk by HD Moore in 2007 at FOSDEM, a conference in Brussels (Belgium) where he explained why he put so much effort in making sure that Metasploit worked on Windows. A lot of people had been commenting that pentesters shouldn't use Windows and enabling Metasploit on Windows wasn't that much of a priority. HD subscribed to another logic, if 90% of the computer using population couldn't run Metasploit, how then could it become more awesome than it already was? (It was something along those lines but maybe not literally, it's 5 years ago and HD already spoke at 500 words per minute...).

This brings me to today, where I'm setting up my new work machine, a MacBook Pro, and thus getting to the point where I check-out SET and go run "./setup.py install" only to be greeted by this message :

"Installer not finished for this type of Linux distro."

Now, we may not be the ones who invent and write the spiffy tools but if we have any sense of community we CAN be the ones who enable them to run (easily) on as many platforms as possible.

Python 2.7.2 is installed by default on OS X and so is easy_install. The modules required for SET are the following :

• pexpect
• beautifulsoup
• pycrypto
• pyopenssl
• pefile

The original setup.py file lists them under the names as they are known in aptitude so finding out those names was the hardest part of modding the setup.py script to work on Mac as well.

add the following elif to the script and it will smoothly install all dependencies:


elif os.path.isfile("/mach_kernel"):
            subprocess.Popen("easy_install pexpect beautifulsoup pycrypto pyopenssl pefile", shell=True).wait()

It uses "/mach_kernel" to identify the host as an OS X machine and then proceeds to install all dependencies using easy_install. If you paste the elif statement at line 35 of the existing script, you're done.

Update -- as I was saying...contribute:

The Grugq was nice enough to point out the platform module:

add the following line to the top of the script ( where all the other imports happen ) :

import platform

then modify my previous contribution to :


elif platform.system()==Darwin:
            subprocess.Popen("easy_install pexpect beautifulsoup pycrypto pyopenssl pefile", shell=True).wait()


Using a more standard setup.py would make life a little easier but I understand Dave for rolling his own.
End Update

Have fun with it and next time you find something that doesn't work as smoothly as you would expect and you have some time to fix it, do it yourself instead of shooting the developer an email.

Peace out.

P.S. : yes, the update was sent to Dave as well ... this post does not have the intention to document an update, it is meant to point out how all of us can work together to make things better,

high-rolling hot shot executive (m/f) wanted - a perspective

Ok, no, I'm not looking to hire someone. This blogpost is triggered by a question asked by @hackerhuntress earlier today :

"if you're passed over for a job, do you mind being notified via voicemail or email?"

My answer is plain and simple : "neither, call them!"

My opinion on this one is very strong and for several (I think) very good reasons. Allow me to explain :

You are the hiring manager. you're the person who the new hire will report to. You're probably also the one who will lean back, balancing your chair and sipping from your nice glass of Shiraz while you say, I quote, "I'm the one that hires and fires." (blank stare goes here).

By delegating bringing bad news to a direct report or, worse, someone in HR you have showed that you are only willing to be responsible for your hiring decisions and not your firing decisions. At the same time you have passed on the chance to take ownership of this decision. You have reduced your candidates to resumes that you pick from and not the actual humans that will bring value to your team. By not committing to stand behind your choice, you actually do yourself, your team and your organisation a disservice.

Now, I do understand you are a busy person so the first argument I will hear when discussing the practice is "I don't have the time to call everyone". I call bullshit.

If you're THAT busy you will have your hiring process geared to your agenda. You will be choosing from 2 to 5 candidates that you will have personally seen.  Those persons that didn't make it will take less than a minute. The chance that they are willing to strike a lengthy conversation is zero. Even though they won't realize it, they will appreciate this gesture in the long run.  To the extent that, if a similar position opens up they may even be enthusiastic enough to consider working for you after all.

In the end you're looking at less than 10 minutes of your time to call ALL candidates (including the one you do chose to hire).

Note that I've been on both the dealing and the receiving end of this practice. I know that it's not easy to tell someone they're not hired. I also know there is no good way to receive that information, especially if you've thrown yourself passionately at a job opportunity. It fucking hurts.

As a manager, telling someone they're not hired is peanuts. You just gather yourself and tell it. Look at it this way : at least it prepares you for the really hard decisions. If you can't even tell someone they're not hired, how are you ever going to handle firing someone?

Now grow some cojones and stand behind your decisions. That or don't call yourself a leader while I'm near you ;-)

Hire great infosec people (and keep them) !

Earlier this week I had an interesting exchange with several people on Twitter after a statement by Mary Ann Davidson (Oracle CSO) gave at the -apparently awesome- ISSA Conference. It was paraphrased by George Hulme :

"entry level people have to work to convince orgs they are worth giving a break for a position."

 Roughly translated this would mean that if you are planning on joining an organisation to make a living from the skills you have built in college, university and/or through diligent selfstudy, you shouldn't expect any favors (or even a wage?) from that organisation. They will evaluate you against their set of values to see if you're 'a match' and, maybe, then you could expect being respected and could look forward to making progress.

The problems I see here are legion. I'll try to address the most important ones as I see them and specifically for information security profiles.

1. I admit that there is a certain professional etiquette but I'm also convinced that this is covered by a decent upbringing provided by parents. You don't want a potty-mouth cursing at your users through the telephone and you want somebody that behaves normally. A smart person knows when he has to dress up and when he can wear a t-shirt. I think we're way past the era where 3 piece suits were mandatory uniform. Dreadlocks or long hair? Facial hair? tattoos? If you still care about that, don't expect to find the creative, intelligent, driven individuals you are looking for. If you open your mind, your ass will follow. (thanks En Vogue...)
2. You will indeed need to 'test' people beyond your initial HR filter way after you hired them if you are convinced that a conventional hiring process still works. 'Way back when' job ads were used because normal ads were way more expensive and all that we wanted was to have our brand in the papers. Now that Google ads is even cheaper than job ads, what is the use for them? Imagine that you are fishing at sea: you can throw in a line to catch a random fish that you may or may not be able to eat. Another choice is to tune your lines, depth and bait to the particular species you want to serve your friends tonight. Job ads are your average bait, luring in ALL THE PEOPLE and thus requiring a very coarse filter to keep the edible. For infosec, you want very specific people so you're going to have to adapt your hiring process. In short I would advise to get 'out there'. Visit (local) conferences, hackerspaces, linux groups meetups, DEFCON chapter meetups, ISSA|ISC2|OWASP meetings, etc. etc. etc. You'll get to meet the exact kind of people you are looking for and you didn't even need your HR person to filter resumes by degree or certificate letter soup. You are winning! Throw in a CTF (Capture The Flag) style skills assessment maybe sprinkled with business-specific challenges and I don't doubt you'll fill in the empty spots on your team.
3. "People have to prove themselves..." Lord please .... The exact people that you are looking for (driven, intelligent, skillful, ...) already have proven themselves and they probably have a job that is paying their bread and butter. Assuming we're still stuck in 'classic' hiring practices the solution would be to throw more money at them. You'd be in for an interesting experience :-) I would call it the axiom of infosec hiring :
   
   "For each org acquiring or retaining talent solely by pay raises, there's at least two orgs with more money to spend."
   
   The people you need obviously expect to make a decent living. They have spent hours and hours honing their skills and while, in many cases, their professional activities overlap with their hobbies they are not to be exploited. From personal experience however I can tell you that money is rarely the primary motivator. These individuals are to be challenged and to some extent allowed to roam free. It can be as easy as organizing regular internal hackatons focused on business problems but don't expect them to get tied to a chair performing repetitive tasks. Depending on the market circumstances it will take between 3 and 24 months before you've lost them.

I was actually not going to spend the time on writing down my opinion on the whole 'finding the right people' discussion were it not for Rafal Los coming with another article on the subject. The article is kind of all over the place and most could be answered with what I've written here, I want to specifically counter some of the four recommendations that he offers.

• Partner with a good recruiting agency that can weed out the talent from the talent.(sic)
• WRONG! Get out there yourself. Get a mandate from HR to play a more active role in the hiring process. Send out your team members to cons and local meets. They will learn at the same time as they are meeting their future colleagues. That's full of win. Recruiting agencies are awesome to find people to 'fill' your organisations, your 'foundation' type of people you generally don't find through recruiting agencies.
• Offload as much of the non-business criticical work to a partner organization allowing you to keep your business-smart security folks for critical LOB tasks.
• WRONG! Note that I'm not saying not to outsource stuff that's not critical. But there is some gross negligence in this recommendation. Firstly, outsourcing at first means an increased workload for your internal people. Processes that are 'informal' currently need to be fleshed out and formalized. What your internal people take for granted isn't so obvious for the third party. As your organization evolves, outsourcing may even be more high-maintainance than insourcing. I would never recommend it as a blanket solution. Secondly, this recommendation is made to specifically focus your business-savvy security folks on critical LOB tasks. The problem is that while performing LOB tasks they will need data to support the points they're making to the business. By outsourcing (for instance) SIEM and IDS, perimeter device management, IAM management, etc. etc. you are further removed from the data you actually need and especially the business context melts away at a rapid pace (getting locked in PDF's containing service reports and such). Remember that the primary concern of that third party is their business, not yours.
• Offer incentives to keep people in working for you
• If incentives are considered to be solely of the monetary kind, please consider what I've written under point 3.
• Train existing human resources as you may already have the 'right' people working for you in roles where they're under-utilized.
• I would add some perspective here. Just training resources because someone right may surface that you could internally hire for your team is a giant waste of money. You are looking for people with a very specific skillset that you could 'test' for. One could, as an example, run Incident Response exercises involving staff from different departments. Apart from just looking how well your processes work, keep an eye out for individuals that perform extremely well.  Those are the raw gems you want to polish through training and mentoring, not *everybody*.

I think I could write a book about infosec hiring but I don't think there is a big market for it and I don't seem to have a problem finding the right people for the job myself ;-) For some of you it could sound like I'm talking from my butthole and I accept your criticism, YMMV ;-)

***UPDATE***
A reader anonymously sent me the following message :

"I turned down an offer that was twice the offer I took because I felt I was more understood as an auto-didact at the organisation of which I accepted the offer."

Money matters but only up to a certain degree. If a person can eat, sleep under a fixed roof and maintain his drinking habit (j/k) other things start to count. Be very much aware of that.