I actually remember the first time I saw Lincoln Logs. It wasn’t at my house. I had some great stuff as a kid, but some toys were just too expensive. I’ve never been able to understand why the most expensive toys are the ones that make YOU do the most work to enjoy them. Suffice it to say I was not impressed. I just couldn’t understand it. A bunch of tiny logs that you had to put together yourself with the hopes of making something meaningful. Even if you were a Lincoln Log master, would your finished product really be that great? Would it actually look like what you had in mind, or would you just end up with something that causes your parents to say “ohhhhhh…I see it now, very…ummm…interesting son…how much did we pay for those again?”

Don’t believe me? Take a look at the picture featured in the Wikipedia article, do you know what it is?

It’s a sawmill….do you see it now? 😏

I should take a moment to mention that the “We’ll do all of this, you just do that” products are entirely different from Lincoln Logs. If you pick up a pizza from Papa Murphy’s they help you choose the ingredients, put it together, box it up, and you just take it home and bake it yourself. I don’t have a problem with that. But if I walk into your pizza place and you sit me at a table next to an oven with a bunch of ingredients to make dough, sauce, etc., tell me to use my imagination, and then charge me more than any pizza is worth…well…I’m going fight you. No other way to put it.

 I thought I could put Lincoln Logs behind me, but I still deal with them almost every day. Well, something close anyways, today I deal with Lincoln Log Files (software logs). Software logs and Lincoln Logs have quite a bit in common. You open a box and you find a bunch of disconnected pieces, some similar some identical, and you’re expected to figure out how to put them together to make something useful. Usually people don’t read software logs for fun. More often, we do it because something has gone wrong (sometimes oh so very wrong) and we need help figuring out “what”. Instead of getting an answer to that question we usually end up faced with another one. A question that starts with “what the…” (you can use your imagination for the rest). I’m not going to waste time (mine or yours) by offering examples, citing references, or otherwise trying to prove this is usually the case. You’ve been there, we’ve all been there. Unfortunately, I seriously doubt this short article is going to persuade software vendors to change their ways, but I hope it will at least change you (and me). I can’t guarantee it though, as the old saying goes “we can give advice but not conduct”. I guess we’ll see. 

To the cyber defender / IT professional
You’re on your own. Don’t expect software vendors to help answer questions when you need it most. Questions like: “Who used the VPN in the last 30 days and from where?”, “Who accessed the spreadsheet last night?”, or “What is the IP address of the device that is locking out these accounts?”. Oh don’t get me wrong, the software vendors have logs. They give you everything. Some of it will be useful, some not so useful, some will only be useful to them, and some will be totally useless to literally everyone on the planet. The one thing all of these logs have in common is that the information you need is going to be mixed in with information you don’t, they don’t, and no one else will likely ever need. Ok, so that’s on them. Now let’s talk about you. 

If you’ve been in IT for long at all, even 3 months, you know that log files are basically a sad (and sick) joke that software vendors never get tired of telling. They just give you everything and let you sort it out. The problem with you is that you’re going to wait to sort it out until you really need to. That means in the midst of needing to answer some very important questions (like “which user downloaded 10.2 GB from the file server over the VPN last night?”) you’re going to be downloading, filtering, piping, grepping, and googling your sorry butt all the way to something that kind of looks like an answer. Even then, you still won’t be sure it’s the right answer. When the crisis is over…you’ll have learned nothing. You’ll break down your little “sawmill”, put the logs back in the box, and do it all again next time. That’s on you, but it’s no longer going to be you. From now on you’re going to be proactive. You’re going to do this:

• Write down a list of important questions that you’ve needed to answer in the past, things like:
  • which users have accessed the VPN in the last 30 days?
  • are any accounts showing signs of suspicious login activity?
  • which accounts accessed the file server over the last 30 days?
  • did Frank regain access to his old account after we let him go?

• Write down a list of important questions that you anticipate needing to answer in the future, things like:
  • what accounts have been locked out lately and from where?
  • which users are currently receiving the most phishing Emails?
  • have there been any unusual database queries lately?

• Find your best AVAILABLE sources of information to answer these questions
  • you’ve probably guessed by now that those are likely going to be Lincoln Log Files, so it’s up to you to determine which ones

• Figure out how to filter and extract only the information needed to answer those specific questions, and save that knowledge for future use. Often, this process looks something like this:
  • The software vendor provides a graphical interface to search and review log files, customize columns, and export the results, but not a single USEFUL canned query. What do you do? You figure out what to search for, only choose the columns you actually need, and then SAVE THE QUERY for FUTURE USE
  • The software vendor basically gives you bulk text or JSON files to parse. Sometimes the information you need will be buried in nested arrays multiple layers deep.  What do you do? You do whatever it takes, grep, filter, pipe, cut, repeat until you get what you need. Then SAVE THAT AS A SCRIPT OR ONELINER for FUTURE USE.
  • The software vendor has graciously provided you with with yet another query language to learn but not a single USEFUL canned query. What do you do? You learn it, write queries for what you need (and only what you need), and SAVE THEM for FUTURE USE

Now when questions inevitably arise you’ll be able to answer them quickly. You might even decide to ride the “proactive bandwagon” a little further and start actually periodically reviewing log files like you know you’re supposed to. I doubt it, but maybe.

At this point you might object. You might tell me this isn’t necessary because you have a shiny SIEM. Well, I object to your objection. In my experience, the problem with most SIEMs is that they’re basically a big box for all of the other software vendors to dump their Lincoln Logs into right along with their own. Now it’s the SIEM that provides you with a nice new query language to learn, but not a single USEFUL canned query. You still end up needing to do what I just described above. Be proactive. Learn the language, write the queries, and save them for when you need them.

Before you set out on this adventure I have two last pieces of advice:

• Check to see if someone else has already written the queries, scripts, or oneliners that get what you need and use that. You may even find stuff that you didn’t think of or didn’t know you could get.
• If you write some useful queries, scripts, or oneliners share them. They don’t need to be pretty, just pretty useful. You’ll literally be making the world a better place.  

To the software vendor
I almost didn’t write this section but not because it isn’t needed. It’s because I don’t want to talk to you. I don’t like you. If I ever get to see a pie chart of the way I spent my time on this Earth, the purple piece of pie that represents me downloading, filtering, piping, grepping, and googling my sorry butt through your log files is going to be too big. It’s already too big. I’m not confident you’re going to change. But it doesn’t seem right that I should accuse you of wrongdoing (or even dumbdoing), and not point out what exactly you’re doing wrong / dumb. So here is some advice just for you:

• I don’t care if you want to log everything, storage is cheap, but don’t mix it with the stuff that I actually need. If you don’t know what I actually need ask me…I’ll tell you. 
• Giving me a new query language to learn is awfully nice of you. Let me use it, but don’t make me have to. Store some USEFUL queries for me, and let me write some custom ones if I need to…and I shouldn’t need to.
• Don’t mix information that only you can interpret with the information I need. I don’t care what the query GUID is. I don’t care what the InternalVendorSessionIdTokenEvent is. You can log that in your own logs. Don’t give it to me as yet another useless column in my spreadsheet. On average my spreadsheets need 5 to 7 columns of information to answer most of my questions. When you give me a bunch of columns that only you (possibly no one) needs or understands it divides the information that I actually need. And that makes me mad.
• If I have to Google what an entry in your log file means you’ve failed. Admit it. Own it. Fix it. I have a home security system. I know when my front door opens. I know when someone disables the alarm at the keypad. If someone decides to break my basement window, the notification I’m going to get on my phone isn’t going to be:
  
  
  
  Do you know why it isn’t going to be that? Because it isn’t you. It doesn’t JUST make software. For some reason vendors that sell products that can use software seem to get this right, whereas vendors that only sell software don’t have a clue. That’s you. You don’t have a clue. You should get a clue.
• Don’t you ever, and I mean ever, stuff a JSON payload (with the information I actually need) inside a single column of my CSV export and then provide a “helpful” article telling me how to use some Excel wizardry to almost get what I need out of it. Maybe I can cat and cut that one column from the command line and then jq my way through it to get what I need…is that right? Is that what you were thinking? This is why you don’t get invited to the “cool parties”.

Bottom line, you’re acting like a total…I’m not going to say it because this is a family blog. Let’s try this though, based on a psychological theory called “reflective atomic self-ascribed association” (don’t look it up because I just made it up) whatever word you just thought I was going to say is how your subconscious says you are acting. So stop it. If you don’t, then I can only assume you like acting this way. You might even be going out of your way to act this way. If that’s the case then please, I humbly request, open a pizza place. Because I want to fight you 😠