1. The quality and quantity of our data - the breadth and depth of telemetry to which we have access (network traffic logs, runtime detection alerts, file samples, cloud control plane logs, SaaS platform logs, etc.), our sources of metadata enrichment (such as VirusTotal or Shodan), and our corpus of past incidents.

2. Our conceptual framework - our assumptions about how attackers operate, their motivations, their goals, their tooling, and how their behavior might express itself in our data.

In other words, our perception of any given malicious activity is always indirect and subjective, as it depends on the parameters of our working environment: our collection capabilities, our analysis tools, our methodology; our clientele, their sector, their geography, their platforms, their hardware, their software; the actors targeting them, and many other variables.

Every vendor, government agency, research group, and independent researcher might view the same malicious activities through their own porthole, analyzing a unique cross-section of cyberspace. In the words of Kurt Vonnegut:

[Billy was] strapped to a steel lattice which was bolted to a flatcar on rails, and there was no way he could turn his head or touch the pipe. The far end of the pipe rested on a bi-pod which was also bolted to the flatcar. All Billy could see was the little dot at the end of the pipe. He didn’t know he was on a flatcar, didn’t even know there was anything peculiar about his situation. The flatcar sometimes crept, sometimes went extremely fast, often stopped - went uphill, downhill, around curves, along straightaways. Whatever poor Billy saw through the pipe, he had no choice but to say to himself, “That’s life”.

As for threat actors, we must always recall that they’re real people - albeit working within constraints defined by organizations, bureaucracies and social norms - meaning they exist across all aspects of reality. As they traverse the various media of cyberspace, they leave traces that we might detect in the course of our monitoring and investigation.

A hacker pressing a key on their keyboard in a cubicle somewhere in Moscow or Beijing may eventually and indirectly cause a new row to be recorded in one of our telemetry databases, but that can hardly be said to be the full picture. Our task is to interpret such signals as more than mere shadow puppetry, and to deduce what other signals may exist beyond our line of sight.

Our ability to observe any given type of evidence depends on the sophistication and configuration of our instrumentation, and collecting that evidence requires pointing our devices at different “layers” of our subject matter. This is analogous to using a combination of infrared, X-ray and radio telescopes in astronomy, while also operating robotic probes to retrieve exotic materials from asteroids and employing chemical analysis techniques such as gas chromatography to determine their composition.

Much like flatlanders, at our weakest and most solitary, threat researchers are akin to two-dimensional creatures striving to make sense of the chaotic goings-on of multi-dimensional space. Moreover, we find ourselves contending with malevolent higher beings that appear to span the unknowable infinitude of cyberspace.

For example, a resourceful attacker targeting our organization as a whole is likely to elude our comprehension if we’re overly focused on network traffic analysis while ignoring their activity on endpoint and mobile devices or across cloud and SaaS platforms. Moreso if we wrongly assume that network traffic is all there is.

But all is not lost - we can invest in intelligence to guide our efforts (and product roadmap), join forces with others working towards the same goal (each bringing their own unique perspective), expand our telemetry collection apparatus, and point it in the most promising directions. So long as we know more or less what we’re looking for and have a well-trained nose for novelty, we can jointly overcome our individual sensorial limitations and more accurately perceive the motion of our adversaries on the wire.

I’ve found that methodically monitoring for apparent “plot holes” is an effective method of revealing missing signals (false negatives), indicating hidden aspects of malicious activity that we’re not perceiving to the fullest, taking place in higher dimensions, so to speak.

For instance, if we detect a machine in our environment in communication with a known C2 server but without detecting any files that would otherwise indicate the presence of malware, we can immediately deduce that we’re missing something.

Similarly, a compromised server with no prior risk detections (assuming we’re scanning for such things) might indicate a potential 0-day vulnerability affecting the software installed on it. This technique can be automated, and is applicable to both threat hunting and detection engineering.

The next step in such cases is to determine whether we need to adjust existing detection rules, add new ones, or perhaps collect a brand new type of telemetry, essentially expanding the scope of our research into yet another dimension of cyberspace (fun!).

Working with other people entails effective communication. In most modern organizations, this almost certainly involves meetings and some form of Email, Slack, Discord, Teams, or an equivalent messaging platform. In research (especially in academia), a more indirect variant of this communication also occurs within circles of researchers working on overlapping subjects, by way of publishing and reading each other’s papers or blogposts.

As I’m sure many people do, in my current and past roles I’ve often found myself questioning how much time I should be spending every day on reading and writing messages, just as I wonder how much time I should be devoting to meetings. There are entire books on topics like this, and the required dosage changes depending on whether you’re a manager or an individual contributor, the size of your team and the sort of work you’re doing. Regardless, being on top of things is a seemingly universal necessity in all aspects of life.

Whenever I feel like maybe I’ve been spending too much time reading, writing, or replying to messages, I’m reminded of classic real time strategy games (RTS), such as StarCraft, Warcraft, Dune II, or Age of Empires.

Real-time strategy games

In RTS games, the player typically needs to gather certain types of resources in order to build buildings, summon units, perform upgrades, or advance through the tech tree, with the ultimate goal being to develop their offensive and defensive capabilities as fast as possible and thereby defeat the other players or reach an objective somewhere on the map (depending on whether you’re in multi-player or single-player mode).

Part of the strategy in strategy games comes down to which resources to gather, how much of your existing resources to spend on additional resource gathering efforts (such as building additional buildings and units tasked with resource discovery and retrieval), and when and how to spend resources on things that more directly influence the outcome of the game, like creating and upgrading fighting units and, more importantly, sending them into battle.

In most RTS games I’ve played, devoting all your time and resources to gathering resources is not a winning strategy.

Making an impact

Returning to the analogy, gathering resources in the professional sense – things like finishing books, watching other people’s talks, reading other people’s articles, and working through other people’s messages – can sometimes feel rewarding and seem like an accomplishment (especially when the unread count gets uncomfortably high). However, if that’s all I end up doing before lunch, it feels rather hollow (much like constructing pylons).

To me, impactful work means making tangible things happen (or actively preventing them from happening, depending on the nature of impact). So it’s important to differentiate between activities that ultimately only lead to more resources being gathered – developing skills or acquiring knowledge of what’s going on in the world and within the organization – and activities that utilize those resources to achieve measurable and impactful goals, like enabling my team members to succeed and finishing projects on time.

In individual contributor roles, I find that it’s been slightly more straight-forward for me to find the right balance between gathering resources and making things (writing, building, teaching, etc.), whereas in managerial roles, knowing what’s going on is at once both imperative and tricky to do without getting absorbed in events beyond your own scope of responsibilities.

So whenever I think I might be overdoing it, I think back to strategy games: why am I reading this lengthy Slack message? What is this resource for? Will I be able to use the information it contains to upgrade something important on my team’s tech tree or expand our base with additional turrets or walls (so to speak)? Will writing a well-thought out response to this message help achieve one of our objectives, or perhaps it would be less resource-intensive to simply call this person or organize a quick meeting?

Or maybe it would be best to just skip it altogether and even mute the Slack channel so I can focus on other things?

A means to an end

It’s tempting to invest a lot of time in resource gathering; it’s often low-effort work that can be done even when I’m tired or preoccupied, but I need to remind myself that it’s a means rather than a goal. While it can give a sense of short-term accomplishment, if I’m not careful I can end up doing it "forever" (more generally, I must be wary of potentially infinite tasks; they indicate that something has gone very wrong with my time management).

Under this paradigm, work can be divided into a few different “modes”, each of which is mutually dependent on the others:

1. Gathering new resources, or investing them into developing more efficient methods for gathering resources.

2. Investing existing resources into things of value (building things, developing skills, etc.)

3. Using those things to accomplish your goals (whether this means protecting your base from an enemy’s attack, destroying their base, or meeting your quarterly KPIs)

While the latter mode is obviously the most important, in practice knowing how best to invest my resources is an ongoing dilemma. I don’t always know how to solve it, but an important principle can be adopted from other multiplayer cooperative efforts: working with other people means that it’s important to prioritize resolving dependencies and divert resources to places where I’ve become a bottleneck of sorts.

In other words, avoid becoming a blocker to other people's work, and prioritize doing things that only you can do. In managerial roles, this usually involves recruitment, training, setting goals, answering questions, making decisions that solve other people’s dilemmas, reviewing other people’s work, providing them with feedback, connecting people with others who can help them, and helping people manage their own time.

The Pyramid of Pain is well known cyber threat intelligence concept, originally conceived by David J. Bianco in 2013. For those unfamiliar, the lower parts of the pyramid list things that are widely available to threat actors and they can therefore more easily move on from when they’re “burned”, by simply shaking it off and switching things up at minimal cost, whereas the higher parts of the pyramid include things that will be painful for threat actors to replace, and therefore have greater impact if we work to make them unavailable and thereby impose greater cost.

Note that “denying” doesn't just mean discovering something or making it public. Instead, it means (ideally) preventing the threat actor from using something that they rely on in their cyber operations, or at the very least, making it much harder or prohibitively expensive to continue using it. As in all things related to threat intel, this involves making our discoveries truly actionable for defenders.

As a side-note, every threat actor has different things they might find painful, depending on what’s important to them and how they operate. For example, an actor using bespoke self-developed tooling and relying on exploitation of 0-day vulnerabilities or supply chain vectors would be severely impacted just by having them exposed, but other actors might not be bothered at all if we merely figure out what tools they’re using – this is especially true if they’re not bringing anything with them in the first place (i.e., Living off the Land); if they’re using open-source or commercial offensive security software; or if they gain access to their targets via n-days, misconfigurations, or compromised credentials. Some particularly resourceful threat actors might not even be especially hindered by the loss of a few 0day vulnerabilities, treating it as the cost of doing business, and will only truly be impacted by robust mitigations that deny whole bug classes or even entire attack surfaces.

Anyway, while the Pyramid of Pain is an incredibly useful concept, impact to adversaries is only one metric to consider when gathering threat intelligence and maintaining our threat detection strategy. There are other scales by which one should measure the various qualities of threat detection artifacts. Some good prior examples are Steve Miller’s “Detectrum” approach, which shows how different types of artifacts lend themselves to different types of threat intelligence and detection work, and Jared Atkinson’s slightly similarly named Detection Spectrum, which places detections themselves on a scale that moves between precise (with low false positives but high false negatives) and broad (with high false positives but low false negatives).

An interesting aspect of the Pyramid of Pain is that the impact to threat actors when a certain type of artifact is burned often positively correlates with other completely different qualities of that artifact. For instance, the more painful the artifact is to lose – say, a bespoke tool – the more reliably we can use (a sample of) it as a source for robust indicators of compromise, while making it harder for the adversary to avoid detection through mere polymorphism. To this end, we can write YARA rules to detect common denominators between variants of the same tool and even employ code similarity techniques to deny an adversary of opportunities to use related tools against their potential targets.

However, there are some points at which this correlation between qualities isn’t necessarily true – for example, while the level of pain for losing access to tools corresponds to our ability to attribute activity involving that tool to a specific actor, the same isn’t true of TTPs, which are more likely to be shared by multiple actors and are therefore less attributable. This discrepancy leads me to the conclusion that we probably need at least a few more pyramids than the one we already have, to rank detection artifacts according to different metrics, and shed light on how they differ from one another.

At this point you might suspect that I am in fact an Egyptian pharaoh, or perhaps you have surmised that this blogpost is all a ruse to convince you to invest in my pyramid-themed pyramid scheme, but I hope you’ll disregard these suspicions and continue reading regardless. In fact, let’s drop the pyramid thing altogether and focus on other stackable structures instead.

So, without further ado, at the bottom of this post you will find an expanded list of pyramid and pyramid-like concepts you might consider incorporating in your threat detection program, to be used as axes (the mathematical kind, not the wood-chopping kind) by which to evaluate the various qualities of threat detection artifacts.

Some disclaimers (I was lying before about the lack of ado):

1. There are almost certainly other metrics I have failed to include here.

2. The rankings in each metric are arguable as they are a bit handwavey and based entirely on my own judgement, so I invite the reader to focus mostly on the metrics themselves, and you are more than welcome to rank things differently.

3. While not part of the original Pyramid of Pain, each of the following artifacts can be very painful to lose as well: people (hackers, tool developers, vulnerability researchers, etc.) or their personal information; 0-day vulnerabilities or exploits for them; and targets. I’ve included these as part of an expanded version of the OG pyramid.

4. The significance of losing access to a target network differs rather dramatically between threat actors – for example, an opportunistic cybercrime actor might be unphased by our denying them access to a single target (as there are many more cyber-fish in the cyber-sea), whereas for a state-sponsored espionage actor, long-term loss of access to a specific target might be devastating, depending on the identity of the target.

5. Individual targets do not have much long-term value as detection artifacts, in the sense that once an operation is blown and forensic analysis is complete, there isn’t much else to glean from the target environment unless the threat actor returns to the scene of the crime in the future. However, sets of targets do remain highly valuable as a detection artifact, since they teach us about an actor’s victimology and their preferred sectors of operation, allowing better risk management and supporting future attribution (h/t to Merav for this late addition).

Given recent events, I’ve been thinking a lot about intelligence failure. Among other things, I’ve been considering how it might relate to threat intelligence and detection engineering, where failure could result in threat actors successfully compromising an otherwise well-defended target and achieving their goals.

This post started off as a way to organize my thoughts, but developed over a period of several weeks into a slightly rambling (and somewhat metaphoric) treatise on dealing with false negatives.

Introduction: estimating adversary movement

Defenders are expected to continuously gather information about (1) what threat actors want, and (2) how they go about getting what they want. We’re meant to make use of this knowledge to implement effective mitigations and detections to stop bad things from happening, react to them when it still matters, or at the very least, make the outcome less bad. In practical terms, this involves installing preventative and detective security controls at key choke points throughout the defended network, and ideally architecting the network itself to be more robust to malicious cyber activity.

One way of thinking about this is imagining the various “moves” (as in Chess) that a threat actor might make through each stage of the ATT&CK matrix (or cyber kill-chain, or targeted attack lifecycle), and applying impediments to either reduce the likelihood of success of any given move or eliminate the possibility of performing it entirely (which isn’t always an option, and might come at the expense of required functionality).

For example, one impediment might take the form of a firewall rule to block initial access to Internet-facing devices; while another might segment the network and limit access to a particularly sensitive subnet to only a specific set of users in order to prevent lateral movement between environments; and another might block outgoing connections to prevent traditional means of exfiltration. If any of these impediments should fail, or if a threat actor finds a way to bypass them, we must be able detect that a “move” has occurred, preferably as soon as possible after it happens, so that we can take responsive action such as disabling external access to a compromised user or limiting their privileges within the network.

Planning effectively for such scenarios requires that defenders have a strong grasp of both space and time, meaning that they understand the architecture of the defended network and the timeline of a potential security incident. Network architecture is a relatively solid, albeit complex and dynamic thing, but from a defender’s perspective, an offensive cyber operation is a fuzzy, branching, not-quite-deterministic series of events playing out against the background noise of many other normal benign events.

The implications of branching timelines

In the context of detection engineering, the embedded uncertainty of threat analysis can lead to both false positives and false negatives – each of which are equivalent to intelligence failure (more on this later). There are in fact many ways defenders can fail in the context of both threat intelligence and threat detection:

1. Failure modes related to threat detection:

   1. Failure of observability – e.g., we didn’t realize that a certain device is installed in our network, or we weren’t collecting any logs from it.

   2. Failure of utilization – e.g., we were collecting logs but didn't have detection rules in place to take advantage of them; alternatively, our rules were faulty (leading to false negatives / positives).

   3. Failure of analysis – e.g., our rules were successfully triggered by malicious activity but we didn’t evaluate the results correctly, or we didn’t understand their significance.

   4. Failure of action on analysis – e.g., our rules were triggered by malicious activity and we even realized what was happening, but our response was too slow.

2. Failure modes related to threat intelligence:

   1. Failure of source development – e.g., a threat actor was planning to compromise our network, but we didn’t have any relevant sources that could shed light on their activity, nor could we determine what tools or techniques they were employing.

   2. Failure of utilization and/or analysis – e.g., we gleaned information from our sources about the threat actor’s activity, but failed to deduce indicators of compromise that could have alerted us to the actor’s presence in our network.

   3. Failure of action on analysis – e.g., we knew that a threat actor was actively exploiting a vulnerability that affects a certain product in our network, but for whatever reason we didn't patch it in time.

Beyond threat intelligence and detection engineering, Chris Sanders touches on the subject of branching timelines in relation to forensic analysis as well – once a potentially malicious event has already been observed via detection, we should conduct investigative forecasting. As Chris explains, this involves building a hypothesis by asking (1) what actions might have led to the observed event and (2) what actions might follow them. Then, we perform further forensic analysis, aiming to either confirm or refute our hypothesis and thereby figure out if an incident has truly occurred (i.e., a true positive), and if so, how the adversary gained access to the compromised system and what else they did from that point onwards.

Returning to detection engineering, the same principles can be applied here as well, but at a much larger scale – every potential initial access event could branch into one or more privilege escalation events, branching in turn into any of several lateral movement events, and so on. We can build detection rules by recognizing causality and correlation relationships between potential events (while differentiating between necessity and sufficiency): if A leads to B which then leads to C, then detecting B could be equivalent to detecting both A and C.

For example, let’s say our network contains an account with access to a database, and the account is protected by a strong password. This means that in order to gain access to the database, one must first authenticate to the account, which requires knowing the password in advance (let’s assume that the password is strong enough that brute-forcing it would be unlikely). Therefore, if we observe a threat actor successfully authenticating to such an account, at that point we can confidently deduce that the password is likely to have been compromised, and that we should therefore lock down the account, investigate where the password might have been stolen from (since the source is likely to have been compromised as well), and check if the account has recently accessed the database.

However, in reality things are rarely so simple – multiple accounts might have access to multiple databases in the network, and the password for each account might be stored in multiple locations. Furthermore, the database might be hosted on a machine affected by a vulnerability, allowing an attacker to bypass authentication altogether. As a result, just because we observe a suspicious event related to a certain account doesn’t necessarily mean we can clearly deduce what actually happened prior to the event or what will occur after it.

The trade-offs of time travel

The further back in time we begin our analysis, the less certain our forecasts become, leading to a greater chance of a false positive. We can’t warn against every single eventuality, because our organization only has so many resources to invest in monitoring efforts. We also cannot adequately prepare to effectively respond to every single potential occurrence, whether or not we’re capable of detecting that it’s about to happen (or has already happened). Conversely, if we begin our analysis further forward in time, we can reach greater certainty and higher levels of detail, but we will also have less time to prepare, thereby minimizing the actionability of our analysis and our impact on decision-making.

For instance, learning about a ransomware attack when the attacker has already left a ransom note is obviously far too late; identifying a breach within minutes and understanding the significance of what we’re observing allows time for an effective response but might not always prevent the threat actor from achieving their goals; identifying reconnaissance activity against an Internet-facing network device allows time for hardening it to prevent follow-on activity; and realizing ahead of time that the device isn’t properly secured would probably be the best outcome of all.

However, we can’t always know in advance that any specific device we decide to harden would have otherwise been compromised, just as we can’t necessarily know which of many such devices threat actors will eventually choose to compromise (perhaps all of them, or maybe only those included in the adversary’s evident reconnaissance activity). We also can’t know how threat actors will choose to respond to our precautionary measures if they do prove effective at hindering their malicious activity – adversaries might just try to pursue a different attack vector, such as phishing an employee or researching a 0day vulnerability in the network device.

Regardless, by mapping eventualities, we can better identify choke points or bottlenecks for proper placement of mitigation and detection mechanisms. For instance, if most paths an adversary can take all lead through the same device within the network, or require them to perform a specific action, we can treat these as a sort of singularity in which an attacker could potentially be “trapped”. Moreover, we can try to identify series of bottlenecks wherein an adversary’s movement will be predictable, bordering on deterministic.

Even if we do manage to map as many potential movements as possible, no matter how familiar we are with the layout of our network and threat actor profiles, there will always be some unknown-unknowns manifesting as “fog of war”. As a result, the actual graph of actor “moves” (in red) and their complementary detection/prevention opportunities (in blue) might look something like this, enveloped in fog to some degree:

On the subject of bottlenecks, John Lambert has discussed a similar idea in relation to detection, which is that defenders can “time travel” through logs in search of bottlenecks in the kill chain – these are cases where a given technique has relatively few modes of expression, meaning that there are only so many procedures by which a threat actor could go about performing the action.

As a side-note, one could interpret the very existence of such bottlenecks in attacker behavior as a sign of “convergent evolution” of different threat actors in certain areas of operation. In such areas, we might be able to identify these commonalities and use them to our advantage (so long as they exist) when building detective and preventive controls and in the course of investigations. In Daniel Dennett’s book on evolution, “Darwin’s Dangerous Idea”, he refers to such commonalities as “forced moves”:

In chess, when there is only one way of staving off disaster, it is called a forced move. Such a move is not forced by the rules of chess, and certainly not by the laws of physics (you can always kick the table over and run away), but by what Hume might call a "dictate of reason." It is simply dead obvious that there is one and only one solution, as anybody with an ounce of wit can plainly see. Any alternatives are immediately suicidal.

[…] So at least some "biological necessities" may be recast as obvious solutions to most general problems, as forced moves in Design Space. These are cases in which, for one reason or another, there is only one way things can be done.

Prioritization, self-reflection & compensation

Threat intelligence allows us to understand how threat actors operate, which enables us to assign greater likelihood to some sequences of potential events over others. This ultimately lets us focus on what matters most and invest resources in detection and prevention efforts that actually work. In other words, threat intelligence enables better prioritization. However, we must always remember that prioritization carries risk, and risk must be compensated for.

Returning to the subject of intelligence failure, false negatives (such as failing to warn of a surprise attack) are understandably perceived as worse than any false positive could ever be, since misdetections are so tightly coupled in both space and time with their effects on reality, whether they result in disaster or a mere near-miss. There is almost certainly an emotional component to this as well, as expressed by the defender’s sense of guilt.

However, surprises are never the result of simply ignoring a source of data, but rather prioritizing some sources and paradigms over others. In other words, surprises are caused by investing too much of our attention (to a lesser or greater degree) in the wrong place at the wrong time, and anticipating the wrong outcome. But that doesn’t mean that our attention was entirely wasted, or that we should have ignored the eventualities that we did decide to prioritize. In all likelihood, whatever preventative and detective measures we did have in place were successful until they ceased to be so, and for as long as were effective, they most likely forced our adversaries to adapt.

Conversely, the direct and indirect effects of a false positive are often harder to determine, but they always manifest in the way we choose to react to them. Learning the wrong lesson from a false positive and shifting too much of our focus from one area to another could end up being the very thing that precipitates an eventual false negative. This is because our reaction to a false positive, perhaps due to alert fatigue, will often involve a change in our level of trust in the source (i.e., a change in our attitude towards the instrument, person, or organization that has cried wolf one too many times). In the context of adversarial relationships, this is a dynamic that threat actors can abuse to our disadvantage via deception efforts.

Unlikely occurrences, even if they have been justifiably deprioritized, can still be incredibly impactful, and they therefore cannot be entirely ignored. At first this might seem contradictory – isn’t the whole point of intelligence to grant us insight into adversarial activity and prioritize our defenses against the most likely of scenarios? How can we possibly deal with every single risky eventuality? The solution to this dissonance begins with self-reflection and must culminate in some sort of compensation, as mentioned above.

To compensate for the risks entailed by (unavoidable!) prioritization, defenders must routinely ask themselves a critical, difficult, self-reflective question in regard to disastrous eventualities that are considered unlikely under the reigning paradigm: “Would we know if this event was about to happen?”.

If the answer is yes, we should consider how we might know about it (i.e., through what source of information), and how early we would know (and particularly, if this would be early enough to react effectively). If the answer is no, we must compensate through one or more of the following measures:

1. Reprioritize in order to change our collection posture (at the expense of other efforts which might be equally important), in such a way as to either:

   1. Reduce the likelihood of a false negative (i.e., not be surprised).

   2. Understand enough about how the scenario will play out in order to prevent it from occurring (via preventative security controls).

2. Make sure that we’re sufficiently prepared to deal with the scenario in real time even if we’re surprised (via detective security controls and break-glass procedures).

Properly balancing our investment of resources in every identified eventuality is very difficult to pull off, and it requires a great deal of bravery on the part of the individuals involved. Small mistakes will happen (both false negatives and false positives), but they should be treated as precious learning opportunities and precursors to larger avoidable mistakes that might occur if we continue on our current path without making any changes. Perform blameless post-mortems, compensate but don’t overcompensate, and insist on asking hard questions like the one posed above.

Background

The common currency of threat detection is made up of IOCs and TTPs, short for “Indicators of Compromise” and “Tactics, Techniques and Procedures”, respectively. The first mainly deals in unique artifacts that can be observed on compromised machines and are therefore indicative of compromise (hence their name), while the second concerns the malicious behavior of threat actors viewed through the lens of the methods they choose to employ in order to achieve their goals, such as gaining initial access to a target environment, moving laterally within a network or exfiltrating sensitive data.

The theory that makes IOCs and TTPs useful in practice is essentially that these artifacts and behaviors can be defined in such a way as to be distinct enough from legitimate activity in a given environment that malicious actions (or at least suspicious ones) stand out in logs, and can therefore be identified through database queries or with the help of tools like YARA and Sigma. The application of this theory is an important facet of detection engineering.

Sometimes this idea is applied in the service of widescale detection, such as when implemented in an Endpoint Detection & Response product (EDR), which involves searching through a large dataset of logs collected from many machines with the goal of surfacing potential incidents. Other times it’s used for forensic analysis, to confirm suspected compromise or to better understand what occurred within a known-compromised environment. Moreover, if a threat actor is using a specific enough set of tools while exhibiting a specific enough set of behaviors, then in some cases this practice can also support attribution.

However, in the context of threat hunting – especially when said hunting extends beyond the perimeter of a defended network and crosses into the wider Internet – we begin to encounter certain types of indicators that don’t necessarily fit neatly into the strict definition of “IOC”. This is because they either aren’t indicative enough (they’re simply too noisy to serve as detection rules without additional context) or because they don’t strictly indicate compromise. I’d like to dig into the latter case a bit.

Lumps of indicators

Probably the most popular framework by far for discussing TTPs is MITRE ATT&CK, which delineates threat actor behavior across the various phases of a cyber operation, from reconnaissance through initial access all the way to exfiltration and impact. I dare say that ATT&CK is a mature and well-defined framework, with new techniques describing novel malicious behavior regularly added to MITRE’s database.

However, I think this state of affairs isn’t quite the case for IOCs, which have been somewhat neglected in terms of practical categorization. ATT&CK and equivalent frameworks can be useful for defining which techniques a given IOC relates to, but they don’t necessarily help someone understand where and how any given IOC should be put to work.

Furthermore, rigorous formats do exist for sharing IOCs along with descriptive metadata (such as STIX), but when it comes to public threat intel reporting, as an industry we seem to have developed a tendency to just lump all types of indicators together in the appendix of a report without much distinction between them. In my experience, this makes it harder for threat intelligence consumers to make the best use of indicators, and is a form of passing the buck.

Searching in the right context

Something that has been on my mind recently is the common use of the term IOC to describe observables which, although related to malicious activity, don’t indicate compromise per se. These indicators simply aren’t meant to be observed anywhere within a target environment, but rather elsewhere in cyberspace, such as on attacker-controlled servers.

As such, they probably shouldn’t be called “IOCs” at all. In my view, this isn’t just an issue of terminology, and I don’t think I’m being unfairly pedantic, since this confusion probably ends up wasting time across many organizations.

As an example, let’s say that a researcher discovers an open directory on a web server used by a threat actor for staging their activity (this happens more often than one might think). Let’s say that this server happens to contain various logs as well as tools for crafting and sending phishing messages to potential victims.

There are several things of value that could be extracted while reviewing these files, some of which might also be worth sharing externally:

1. The IP address of the server.

2. The method by which the researcher discovered the server in the first place, shared in the form of a JA3S or HHHash fingerprint.

3. IP addresses listed in the access logs, which might represent the threat actor’s workstation (or jump box).

4. Samples and SHA1 hashes of a bespoke phishing toolkit developed by the threat actor.

5. Additionally, perhaps the researcher succeeded in reverse engineering the toolkit and discerned certain unique artifacts such as a string that is erroneously included in every email message generated by this toolkit.

Operationalizing observables

Once all this information is disseminated through a threat intel report (publicly or privately), recipient organizations are now tasked with deciding if the activity described in the report is relevant to their threat model. If so, then their next step would be making use of the IOCs to search their environments for evidence of attempted or successful compromise (the threat report might include TTPs as well, but those are outside the scope of this post).

This process entails translating the various IOCs into detection rules, and then checking network logs for evidence of communication with the server IP address and scanning email servers for any messages containing the aforementioned string. Any of these might indicate that an organization was targeted by this threat actor, perhaps successfully.

One could argue that the server fingerprint also has value to defenders as an IOC, since organizations could theoretically conduct their own scans of any servers communicating with their network and check if they match the fingerprint, or look them up in a scan result database such as Censys or Shodan.

The IOC value of IP addresses suspected to be associated with the threat actor’s workstation (or jump box) would depend on the attacker’s modus operandi; if they never connect directly to target networks, then potential victims couldn’t effectively utilize these addresses for detection. Having said that, the attacker might make a mistake at some point, so it would be best to treat these addresses as IOCs anyway.

However, unlike all the above indicators, the SHA1 hashes of the toolkit itself are not especially useful for most organizations, since they are only likely to be observed outside a target environment, namely on the attacker’s infrastructure (e.g., other staging servers or the threat actor’s workstation), unless they just happen to work for an organization scanning their networks for these IOCs.

The only organizations that are likely to have any sort of access to non-victim environments are other security researchers (and perhaps intelligence agencies tasked with counter-cyber operations), and they can certainly make use of this information to advance their research efforts. However, public sharing of non-IOC observables is problematic, since in some cases their publication might end up being more helpful to threat actors than to defenders, especially if the former can make simple adaptations to avoid future detection (like fixing the bug that caused the unique string to be included in every phishing email).

In short, producers of threat intel reporting can make their reports easier to consume by differentiating between IOC and non-IOC observables. This makes reports more properly actionable for the audience, who can prioritize the most relevant indicators for threat detection.

Exothrunting!

In ATT&CK terminology, non-IOC observables often relate to techniques included in the PRE Matrix, which covers reconnaissance and resource development. However, some “pre-attack” observables can be IOCs, such as a user agent utilized by a threat actor during recon when scanning a target network’s public IP ranges for vulnerabilities and misconfigurations; or a directory in the threat actor’s development environment which might show up as a PDB string in a malware executable. If an observable has a chance to show up somewhere in an organization’s logs, then it should certainly be considered an IOC.

This brings me back to my earlier point, which is that existing frameworks might be good for describing the malicious behavior that an observable indicates (e.g., scanning for vulnerabilities, generating phishing messages, or developing malware), but not necessarily where and how that observable is likely to be observed by defenders, if at all.

I don’t especially want to suggest yet another framework to solve this problem (since I might be the only person bothered by it), but I am sometimes inclined to invent jargon that no one but me will ever use, so I will say that the best solution I’ve come up with so far is to simply distinguish between observables fit for internally-focused threat detection (IOCs) and those applicable to externally focused threat hunting, which I’ll call exothrunting, thereby making these non-IOC observables exothruntables (but this term is incredibly ridiculous and I only mention it here because I find it amusing).

Anyway, another way of viewing the distinction I’m trying to make here is through the lens of a client-server model – some observables are “client-side” (only likely to be observed in the victim’s environment, and therefore considered IOCs) while others are “server-side” (in the sense that they’ll only ever be found in the threat actor’s own environment). However, this distinction can fall apart if the threat actor happens to make a useful mistake, such as uploading their tools to VirusTotal for testing purposes.

Finally, I’ve included the following table with additional examples of observable types where this distinction might make sense, and I hope others find it useful. Note that there’s further nuance to be explored here in terms of which specific types of logs in a target network are relevant for detecting each type of observable (e.g., a C2 domain might show up in DNS logs; a malware hash would be detected through an endpoint disk or memory scan; an attacker’s cloud account ID can appear in cloud logs such as CloudTrail, etc.).

Share Rhythms of Research

Subscribe now

As a threat researcher (and a maintainer of cloudvulndb.org), I get to read, share and discuss a lot of technical blogposts and reports on a daily basis, some describing cloud vulnerabilities, others about threat actors and their activities.

But not all blogposts are easy or enjoyable to peruse. So here are a few things I hope to see whenever I start reading one, since they tend to make it easier to understand what’s going on:

1. A meaningful TL;DR - Let me know what I’m about to get into if I keep reading, and give me an honest chance to turn back. I might stop reading if I decide that the subject or content doesn’t concern me as much as the title made me think it would (which is perfectly fine). In some cases reading a good TL;DR might be enough for me to grasp enough of what the post is about so I can share a link with someone else on my team who might find it more useful than I would. I added “meaningful” here since sometimes TL;DRs don’t actually do their job.

2. Tables - If you’re comparing between things (or should be doing so), then tables make it much easier to understand the differences between these things, and the parameters by which you are comparing them. Tables give me straightforward answers to questions that arise while reading, and they are very useful references for later use.

3. Diagrams - If you’re describing how an event unfolded, or how a certain technique plays out, then you should definitely be using a diagram. Personally, these days I’m using a mix of Excalidraw and PowerPoint (which is Turing-complete, I’ll have you know). If you have the resources to do so, invest wisely in animated GIFs, which can make complicated diagrams easier to parse (you should not be surprised to learn that I often use PowerPoint for this as well).

4. Screenshots - These can be very useful as visualizations and to prove your point, but don’t overdo them, since they might make the blogpost seem longer and therefore more daunting.

5. Bullet points - If you’re writing about a vulnerability or misconfiguration, it is absolutely essential that you clearly define prerequisites for exploitation in the form of a list.

I’ve previously discussed the use of outlines to help organize (and re-organize) a given research product and ensure that its structure facilitates the intended message we’re trying to get across to our audience. I’ve also spoken about the importance that researchers should place on narrative in the presentation of their conclusions, and how to take advantage of typical storytelling structures to make research products resonate with decision-makers. However, it can be difficult to choose the right organizing principle for our content, which determines the sequence in which we present things to the reader.

Let us start with a short example of a well-formed presentation of research findings, courtesy of Eric Hutchins - the first paragraph of this executive summary is written in a style I shall refer to herein as logical foundation (which I shall promptly explain):

Generally speaking, I can think of three primary orders of presentation in research products, ranked from most to least intuitive, each appropriate for different audiences and achieving different modes of understanding:

1. Chronology of discovery

2. Chronology of occurrence

3. Logical foundation

Chronology of Discovery

This order is relatively easy to put into practice, as it’s highly aligned with the flow of our day-to-day experience - we simply lay out the evidence, facts, breakthroughs and conclusions in the same order we arrived at them in the course of our work. This style of writing highlights how we made our discoveries, the tools we utilized and the tricks we implemented (we might also choose to detail our various blunders, setbacks and dead-ends).

For example, we could begin our presentation with explaining the rationale behind our research project and how the idea came about, then move on to describe how we went about it (databases we queried, experiments we conducted, people we consulted, etc.), and then talk about our theorizing process and how it led us to our final conclusions.

We can see this style of writing in action in this next (hypothetical) example from Eric’s Twitter thread:

While this is probably the most straightforward method of presentation, it does not lend itself very well to explaining our hypotheses or recommendations to outside audiences, who might feel alienated by the technical aspects or justifiably perceive much of this to be “inside baseball”, preferring that we get to the point as quickly as possible. Moreover, we would be wasting their time and attention on many details that are most likely irrelevant to the decision-making process, no matter how essential we might personally consider them to be.

Making ourselves the heroes of our narrative also steals the spotlight away from our bottom line, and therefore risks lessening the impact of our work. This particular style does, however, lend us credibility among our professional peers, and is therefore appropriate for methodological settings such as training and meta discussions on how we do what we do, in order to share the lessons we learned along the way.

Chronology of Occurrence

This order is also quite simple to execute - we essentially tell a story about the subject of our research, starting at the beginning and finishing at the end (not to be confused with the order in which our research itself took place, as that would be chronology of discovery). This style of presentation mainly applies to describing processes or outlining a historical analysis of events.

To this end, we lay out all our gathered facts and conjecture on a timeline, choose which events need to be told in more detail than others, and decide which parts of the narrative are irrelevant to our hypotheses and can be put aside altogether. These last two bits are challenging. While it can be tempting to simply dump every piece of data on the reader unfiltered and hope for the best, it’s worthwhile to invest time and effort in editing to trim and focus the narrative.

Logical Foundation

This is undoubtedly the most difficult style of presentation to apply, and if used incorrectly we run the risk of confusing our audience. Moreover, this style’s implementation must vary greatly depending on subject matter, making it even more challenging to utilize. Essentially, what I mean by logical foundation is to organize facts and conclusions in the order that makes them easiest to understand, regardless of chronology. This might include “slicing” content thematically or topically (i.e., dividing it into themes or topics), and ordering the slices from simplest to most complex.

The basic mindset behind this method of arrangement entails recognizing that there could very well be little correlation between the order in which we discovered things, or the sequence in which they occurred in reality, and the best order in which to explain them to outside audiences.

For instance, a piece of evidence we happened to discover first might best be presented dead last in the telling, depending on how exactly it fits into our puzzle. Similarly, in some cases the reader must first understand someone’s motive for taking action before they can understand why and how they acted as they did, which requires us to explain effect prior to analyzing cause.

One of my favorite examples of the use of logical foundation is the way that Newton’s second law of motion is usually taught in modern classrooms: by presenting the formula F = m·a (“force equals mass times acceleration”). Compare this to how Newton originally formulated the second law: without the use of a formula, and in terms of momentum rather than velocity or acceleration.

However, since momentum is a less intuitive concept for those on the first steps of their academic journey through physics, and the second law is usually proven mathematically and then immediately utilized as such in various equations, the modern form is rightfully favored over the historical one, thereby eschewing Newton’s chronology of discovery.

Choosing the Right Order

The main question you should be asking yourself in the planning and editing phases of your research products is: what piece of information do I need to present before others, so that the rest can be better understood? For example, for this blogpost I chose to explain chronology of discovery first because this style seemed to be the easiest to grasp.

It usually makes sense to mix and match between each of the three styles throughout your presentation, but remember to maintain internal consistency and clearly differentiate sections using different styles (e.g., avoid suddenly shifting between them within the same paragraph).

Perhaps you could start with chronology of discovery (to explain why your research is important) and then shif to chronology of occurrence or logical foundation. This can be a good way to pull the reader in, by giving them an interesting hook to latch onto. It is also beneficial to give some background on your research effort, especially if the subject matter is somewhat esoteric, and if you assume that your readers will wonder how on earth you found yourself investigating it in the first place.

My advice is to play with different variations on sequencing to determine what has the best “flow”, and to use the peer review process to request feedback about the order in which things are presented. You should also take note of comments containing questions which are answered later in the text - these usually indicate that the reader expected to receive the answer sooner than you provided it, and unlike in prose (where such mysteries are often intentional), you should consider shifting the structure of your product to accommodate this.

Research necessitates bravery, in the sense that the researcher must overcome their fears in order to succeed in their role. This post will discuss what makes a research product useful, examine how a product can be “too good for its own good”, and attempt to explain why bravery is therefore essential for research.

What Makes a Product Useful?

The utility of a research product can generally be said to depend upon it’s accuracy and consequentiality (i.e., it’s ability to affect reality). In academia for example, one might measure the latter quality in terms of an article’s citation impact.

In intelligence, a research product’s consequentiality hinges on its actionability. An actionable product must be able to convey the researcher’s estimations and recommendations to decision-makers on time, when they need to make a well-informed decision (whether they are waiting on the researcher’s insight or not).

There are several factors that determine a researcher’s ability to influence reality, including their technical ability to make clear, convincing and practical recommendations; their understanding of their organization’s culture; their ability to involve themselves in decision-making processes; as well as their personal reputation and that of the team with which they are associated.

Each of these subjects is deserving of its own post, but today I’d like to discuss two significant limiting factors which are in constant competition throughout a product’s lifetime:

1. Timeliness (i.e., releasing the product on time)

2. Analytic confidence (i.e., how certain we are of our results, and how easily we can rule out competing hypotheses)

The Tradeoff

The contest between these two factors can be exemplified as follows: the price we often pay for taking the time to construct a “perfect” research product is to lose the opportunity to influence reality by contributing to decision-making. If we take too long to gather more data to build our arguments or further articulate them, we run the risk of simply being too late — the decisions will have already been made without our involvement, and our product will therefore justifiably be deemed irrelevant.

Conversely, delivering an “undercooked” product might have disastrous effects if decision-makers choose to act on our premature assessments, or we might be ignored for lacking meaningful contribution to the discussion other than “fuzzy” and unfounded hypotheses.

Therefore, we must aim to find the right balance between relevance and rigor (for lack of a better term). As the window of opportunity begins to close for making a decision, the potential relevance of our product decreases, and while our knowledge may deepen as a function of our work, we will most likely begin to see diminishing returns on our investment as time goes by. Moreover, excessive time spent on one project is less time spent on another.

At a certain point in the course of producing a research product, perfect becomes the enemy of good, and it is no longer worthwhile to strive for rigor, or else we risk the following undesired outcomes:

• Succumbing to inertia, by continuing to work without clearly defined objectives or parameters of success, or in pursuit of an unattainable goal.

• Being overly-thorough, wherein our product suffers from “bloat” in the form of extraneous and irrelevant data.

• Scope creep, which manifests in iteratively moving the goalposts of our product, adding more features and attempting to answer new (less relevant) questions, all the while straying from our original target.

In the long term, we can compensate for tight deadlines and improve our chances of maintaining relevancy by routinely building our sources and knowledge base in the right areas (or growing it in the right directions). Thus, by having a good starting point on hand in terms of both collection and analysis, we can answer research questions more easily and at a faster pace, resulting in products that are both timely and well-founded.

The trick is in choosing where to focus our attention at any given time, and in anticipating what questions we might be required to answer. Generally speaking, this can be achieved by identifying potential convergence points between our organization and reality. Moreover, we must make sure that at least some of our analysis and collection capacity is devoted to exploratory and future-facing efforts, regardless of whatever questions we are confronted with by decision-makers at present.

A Place for Bravery

It should be apparent at this point why bravery plays such an important role — when tasked with delivering an assessment or recommendation (particularly on a short schedule), researchers will almost assuredly encounter the following challenges:

• There is no straightforward or trivial analytic solution to the problem (if such a thing existed, the research would be unnecessary), and we often lack for data (analysis nearly always involves extrapolation to a certain extent).

• We must take ownership of our assessments, and if we successfully convince decision-makers to act on our advice, we will impact reality, for better or for worse.

These facets of research can be highly intimidating, as we may find ourselves doubting our own analytic and rhetorical capabilities; we might dread the possibility of being ignored and the fruits of our labor shown to be uninteresting or unconvincing; or of being proven wrong, our assessment turning out to be woefully incorrect.

Faced with these fears, it would therefore be tempting to prefer rigor over relevance. But bravery entails identifying the proper point at which to finalize our work, killing our darlings if necessary, and ultimately taking responsibility for the consequences of our product, whether our not it proves consequential.

Optimizing for Utility

The questions we must answer are therefore how to determine when to put down our pen and cease our efforts, where we should compromise and where we should not. This can be especially difficult if we’ve been working on something for so long that we’re no longer able to view it objectively.

An important idea to keep in mind is the Pareto principle, which encourages us to maintain focus on the aspects of our work which are most impactful, while carefully compromising on the quality of others. In the context of intelligence research, this focus should manifest in working to improve the confidence levels (or foundations) of our assessments, whereas impact is equivalent to criticality. This parameter can be determined by our estimation of whether or not a given issue deals with phenomena in reality posing a significant danger or opportunity.

In other words, the standard we set for analytic confidence should correlate positively with the criticality of the issue at hand. To this end, we should learn to differentiate between issues of higher and lower criticality, and between assessments of higher and lower maturity (in terms of fulfilling their confidence potential).

We can use this concept as a sort of compass, to guide our collection and analysis efforts throughout the entire research process, and to help us make difficult decisions when push comes to shove — as our deadline fast approaches (whether external or self-imposed to avoid losing relevancy), by applying “differential treatment” to different parts of our product, we can make the following choice on how best to invest our remaining time (and thereby avoid bike-shedding):

• The most critical parts of our product are the most deserving of further refinement (assuming they require it), in terms of both data (foundation) and clarity (phrasing).

• Less-founded parts of the product which happen to deal with less critical issues should be preempted by low confidence descriptors, reduced in scope (and word count), or perhaps removed entirely.

The above principle should also be adhered to for another important reason: critical issues are generally more deserving of critical review, and we will often need to work harder than usual to convince both our peers and decision-makers of the veracity of our conclusions regarding them (especially if our results contradict commonly held views on the subject).

Like most things, our estimation of criticality becomes more accurate as we accrue research experience and gain a better handle on the bigger picture, but we can compensate for lack of experience by relying on the advice of our peers. Additionally, our research team’s collective ability to determine criticality can be honed by self-investigating the successes and failures of past products.

Beyond sustaining focus on critical issues, we can further optimize for utility by working progressively, and thus ensuring continuous capability to rapidly deliver a serviceable product if the need arises, one that is at least somewhat actionable (much like the wings of bats served various purposes throughout their evolution, initially allowing gliding and eventually enabling true flight). This style of work has the added benefit of allowing us to maintain an accurate “snapshot” of our progress throughout the entire lifetime of the product, which can make it easier to expose ourselves to the criticism of our peers (itself an act requiring a great deal of bravery).

A Coda on Failure

If our research product ends up being inconsequential or our recommendations have negative consequences, try to remember that every product (and process) has room for improvement, and research is a long-term endeavor. We should therefore seek, create and seize opportunities to learn from our mistakes (both analytical and operational), by understanding what went wrong and how we can avoid such errors in the future. Additionally, it is vital that we admit to our mistakes and correct them (tapping into our bravery once more).

1. Research products are made up of “layers”, namely:

   1. Assertion (what are you saying?)

   2. Evidence (how do you know?)

   3. Citation (how do you know what you know?)

2. You should clearly differentiate between the various layers throughout your entire product, so that your readers can distinguish fact from conjecture or assumption.

3. It’s important that you qualify your statements with confidence and probability descriptors.

There is a lot to say about writing good research products, and I intend to eventually cover this topic in greater detail, including a discussion on the basic building blocks of a research-oriented paragraph and how to construct one effectively according to a general blueprint. However, today I’ll focus on a more narrow and elementary subject: the informational “layers” of a paragraph.

Every research product, long or short, is made up of paragraphs, which are usually organized into sections, chapters, etc. Each paragraph in a research product plays a very particular role, which is to convey a specific assertion to the reader. This might be about the existence of some phenomenon (“Jupiter has the largest anticyclonic storm in the Solar System”) or about a connection between multiple phenomena, such as correlation (“Taller people are usually heavier”), causation (“Smoking increases the risk of developing lung cancer”) or lack thereof.

Cohesive and well-structured paragraphs usually contain only one assertion, but sometimes it’s best to take a single large or bold assertion (e.g., “Bright clothing is a liability during a stealth mission”) and break it down into a “chain” of smaller assertions while adding explanations as necessary (“Bright colors are easier for most people to make out against a dark background, so wearing bright clothing at night could cause us to be seen by the enemy, which would be detrimental to the success of a stealth mission”). The level of detail here depends on how knowledgeable we expect our audience to be concerning the subject matter of the product.

An unfounded assertion is easily toppled, so the topmost assertion layer is supported by a middle layer of evidence / facts, which is supported in turn by a bottom layer of citations (usually residing in footnotes, hyperlinks or parenthesized references). Essentially, each layer holds the answers to the following questions, separated by conjunctions (e.g., “and”, “but”, “therefore”):

1. Assertion / Claim / Conjecture — What are you saying? What is your point?

2. Evidence / Data / Facts — Why do you say so? How do you know?

3. Citation — How do you know what you know? (and how can someone independently verify this?)

Note that evidence could either be primary (as in raw material, such as an interview we conducted, a manuscript we analyzed or the results of some experiment we performed) or secondary (an existing prior work, such as an article or book).

While the above composition is a good fit for most paragraphs you’ll encounter in the wild, there are some notable exceptions, such as a Conclusions section, which brings forth previous assertions as evidence and presents a higher-level analysis / thesis as an assertion (it would be improper to include a citation layer in this section, as that would mean entail presenting new evidence that should have been included much earlier in the product). Likewise, a Recommendations section shows prior analysis as evidence, whereas the assertion contains a call to action.

Furthermore, some short-form products have the recommendation appear in the same paragraph as the assertion, taking up a fourth layer, and some very short-form products even have five-layer paragraphs, presenting the reader with citations, recommendations and everything else in between.

Regardless, it’s critical that the reader be able to easily distinguish between the various layers throughout the entire product, so that they can understand which statements are factual as opposed to those which are conjecture (and thus arguable to some degree) or assumptions (i.e., assertions explicilty unbacked by evidence). Additionally, statements should be qualified with confidence and probability descriptors, so that the reader knows how certain we are of the veracity of the facts we have gathered, and how well-founded we perceive our conclusions to be.

To summarize, here’s a color-coded stratigraphic diagram of a very simple research-oriented paragraph (one might even call this “stratiparagraphy”):

Further reading:

1. Communicating with Intelligence

2. The Craft of Research

3. Analytic Thinking and Presentation for Intelligence Producers

4. Analytic Writing Guide

5. DNI Analytic Standards

I like to use the concept of a Pokédex (a portmanteau of “Pokémon” and “index”) to explain how best to approach information management in the context of research. For those not familiar with the term, I promise I’ll only dedicate a few paragraphs to the subject – suffice to say that in the world of Pokémon, trainers use a sort of electronic encyclopedia called a Pokédex to scan any Pokémon they encounter in the wild, during Pokémon battles or in the course of capturing and training them. This serves two main purposes:

1. The trainer can immediately learn whatever information about the Pokémon has been stored in the Pokédex’s database prior to an encounter.

2. The Pokédex is updated with whatever new data it can collect during the encounter (such as unrecognized behaviors displayed by the Pokémon), much like an automatic logbook.

Thus, trainers can refer to the Pokédex to study their opponents' Pokémon during a battle or prior to it, and they can also use it to check how best to care for their own Pokémon and discover what abilities they possess.

The interactive nature of the Pokédex allows its dual use as both a reference guide (essentially a knowledge output device) and logbook (a knowledge input device), and it is because of this electrically-enabled duality that I view it as a near-ideal model for an information management system (or IMS).

IMS in the service of research

Moving on from the Pokédex example, I should explain the value of good documentation while researching, especially when working as part of a group.

In research, we should strive to cultivate a knowledge base and make it as comprehensive and useful as possible. This entails maintaining some sort of documentation platform that facilitates the following:

1. Mutual learning with our current peers

2. Drawing on the shared experience of past researchers

3. Enabling future recollection by our future selves and peers

The last point is especially important, as we often cannot anticipate the operative value a piece of information will have until a later date when it becomes relevant. In the long term, documenting our knowledge can be somewhat likened to an ant leaving a pheromone trail – both are actions undertaken by an individual in the present that can have a delayed effect on ourselves or on others, and both serve as examples of stigmergy:

stigmergy (noun)

a mechanism of indirect coordination in which a trace left by an action in a medium stimulates a subsequent action.

Therefore, a documentation platform can be said to require efficient functionality across both space and time. Succeeding in this grand endeavor depends on our collective ability to effectively contribute and organize our own findings (input and processing), and easily extract existing knowledge (output).

In a sense, using a well-maintained documentation platform allows us to “outsource” our individual and collective long-term memory to something with greater storage capacity and less inclination to embellish facts over time. If you're doing it really well, you should be able to:

1. Confidently proclaim that the database contains everything important that we know about every subject we've ever looked into in the past, and some good starting points for anything else we might research in the future.

2. Simply search and copy-paste from the database the next time you're asked a question about something you’ve looked into before (with some minimal editing to account for real-time context).

Conversely, in the absence of good documentation:

1. The researcher forgets things, and has no technical means of compensating for forgetfulness. Moreover, they waste time attempting to recover forgotten information.

2. It is much more difficult to train new researchers, as they are entirely dependent on their more experienced peers to learn new information, or they waste time “re-inventing wheels”.

3. Research is discontinuous, in the sense that every time a researcher leaves the team, the team itself “forgets” a great deal of knowledge (compounded by the unavoidable loss of experience).

I don’t have much practical guidance to offer about how to actually utilize the documentation platform (though I’ll a have a bit more to say on this subject in a future post), other than suggesting you make two habits:

1. Use the platform to look up names and terms you encounter “in the wild” (this should come quite naturally).

2. Don’t hesitate to essentially dump references, quotes and your own initial musings into existing entries, or even create new stubs to edit and expand on sometime later (a stub can be useful even if it’s nearly empty, as it signals to your peers and to your future self that the subject has yet to be properly researched).

The ideal IMS

I’d like to describe the characteristics I believe an IMS should have in order to best facilitate research, and I’ll mention in square brackets [like this] a few examples of systems which offer at least some of these features (though please let me know about additional options I may have missed and I’ll add them as suggestions here — I’m still searching for the best system).

Note that no matter what type of documentation platform you use, how you and your research team make use of it (and how much) is usually the most important aspect. However, having a comfortable feature-rich platform will make you more inclined to interact with the platform on a regular basis, and certain advanced features might even help you reach new conclusions and fast-track your analysis.

Essential features

1. Formatting — the editor should support WYSIWYG [Confluence / MediaWiki] (and preferably Markdown behind the scenes) [Notion / Obsidian]. It should also have excellent support for multiple languages in the same text, via use of complex text layout (CTL) and right-to-left (RTL).

2. References / footnotes as both data and metadata — it should be easy to add references to specific pieces of information, and not just by manually adding numbers in superscript (Microsoft Word’s footnote support is a good standard in my opinion), and references should also be parsed and indexed as metadata [Mendeley / MediaWiki], to allow dynamic organizing (see below).

3. Search — this may seem obvious, but queries need to be blazing fast and highly reliable (meaning that I should be able to assume that if I’ve searched for something but couldn’t find it, that means it definitely doesn’t exist in the database). Search can be augmented by auto-suggest; search term highlighting in results; and optionally allowing for variations, such as synonyms, hypernyms, morphology and error correction (“did you mean”).

4. Tags / labels — I should be prompted to add tags at some point during the editing process [Notion / Obsidian / Confluence] (It’s incredibly frustrating to know that you once inputted an item but be unable to find it because you’re not sure exactly how to search for it — tags make this scenario a bit less likely to occur).

5. Structure — the database should support the organization of items in such a way as to allow expression of relationships or hierarchies (such as a tree or graph) [Notion / Obsidian].

6. Interconnection — I should be able to link between different items in the database, and examine links and backlinks (i.e., what items link to the current item) [Notion / Obsidian / MediaWiki].

7. Retroactive linking - as new items are added, mentions of them in existing items should be converted to links.

8. Versioning / version control — records should be kept of previous versions of every item [Notion / Confluence / MediaWiki], both as a backup in case of accidental edits and to allow reflection on past paradigms and conceptions (or for nostalgia’s sake, or to simply appreciate how far our knowledge has come).

9. Comfort / ease of use (this one is self-explanatory, but hard to implement)

10. Scalability (both in terms of the number of items the database can contain without affecting search speed, and the number of users)

Optional features

1. Dynamic organizing principle — it would be useful to be able to change the way the data structure is presented on the fly (for example, if multiple articles in the database rely on the same references, this data could be presented as either a collection of articles or as a collection of references, depending on the user’s preference) [Mendeley, to some degree].

2. Tooltips — hovering over a linked item should pop up a short description of it, to make reading more efficient [MediaWiki] (this feature can greatly benefit from auto-synopsis - see below).

3. Semantic search — search should account for semantic information that exists within the database itself when (optionally) allowing for hypernyms in queries. For example, if the database happens to contain an entry about a king named George VI, then the results for “king” (a hypernym of George VI) should include every item that mentions George VI (along with other kings and any mention of the word “king”), whether or not he is referred to by his title in the texts.

4. Hyponym-tagging — the database should allow users to manually input indexes that match certain keywords with categories, so that searching for “ruler” in a history research database, for example, should return results about various kings, emperors, sultans and presidents (all hyponyms of “ruler”), whether or not their names are preceded by their title in the texts (this used to be possible in Microsoft Word via manual “smart-tags”, but the feature has since been deprecated). Note that manual hyponym-tagging can be made redundant by semantic search.

5. Auto-synopsis / auto-summarization

6. Auto-analysis — a machine learning model could theoretically be trained on the contents of the database, such that asking a question or beginning to write a sentence should produce a passable (basic) analysis.

7. Auto-complete — words should be suggested as I type in the editor, and the suggestion vocabulary should be based on words in the database itself rather than just including common phrases.

8. Auto-tagging — tags should be suggested based on an automatic analysis of content and detection of its subject matter.

9. OCR and attachment indexing — documents attached to database entries should be scanned and indexed to allow concurrent searching through both knowledge and information.

10. Collect-as-you-browse — this feature isn’t so much about knowledge curation than it is about reliable recollection of raw information encountered in the course of research, by essentially indexing everything the researcher sees [hunch.ly].

11. Annotate-as-you-browse — an extension of the previous feature, this basically means that markup and notes on raw information are indexed [hunch.ly] and shared among the rest of the research team.

TL;DR - Thinking about our knowledge in terms of tables can help us acquire the following important research habits:

1. Cultivate and re-use good questions.

2. Use the very structure of our knowledge to confront us with gaps and steer our research, by learning what we should be looking for and how we can find it.

3. Progress from data representation to ideation (idea generation) by relying on what we already know (and don’t know) to come up with (and sometimes answer) new questions.

4. Represent and synthesize the same information in different ways.

5. Use what we already know as a guide to help us determine how we should handle anomalies: by correcting our data measurements, changing our interpretation of the data, or adapting our model of reality.

Today I’d like to discuss the role of tables in research, and how thinking in tabular terms – by approaching both raw information and knowledge as things to be arranged in tables – can help us make better decisions and come up with new ideas. For this post, you don’t need to know anything about spreadsheet editing software (e.g., Excel), but I’ll assume you’re at least somewhat familiar with high-school level chemistry (and vegetables).

Tables are vital tools for organizing our accumulated information and knowledge in the course of research. While collecting data, we might use several tables (preferably in the form of a database) to properly document and keep track of everything we find, allowing us to support future analysis. Each row in our collection tables should represent an item (such as an article) and each column should represent a field, parameter or question to be answered about the item (e.g., “who wrote this article?”, “when was it written?”, “what is it about?”, etc.).

However, beyond supporting our collection efforts, tables are incredibly useful for analysis as well. Unlike our collection table, each row of our analysis table might represent a subject or aspect of our research (such as a person, place, group, event, phenomenon or thing), rather than a piece of raw information. Our analysis table (or some version of it) could also end up being useful as a means of presenting our conclusions in our final research products.

In purely hypothesis-driven research (wherein we devise a hypothesis and set out to falsify or support it), our columns are usually preset according to our initial research questions, and change little throughout collection and analysis. Conversely, in exploratory research, each new piece of information we discover might teach us to ask new questions about our subject matter – thus evolving our set of columns – as we learn to pay more (or less) attention to certain aspects of reality.

Over time, as we tackle other research problems, it makes sense to re-use good questions, meaning those that have generated answers that have proven useful. This involves relying on our collective experience to incorporate in our paradigms certain parameters we have deemed important, thus developing (unavoidable) biases that determine how we approach research itself. For example, in the course of our cucumber-related research, we might observe that cucumber cultivars each have a different number of seeds, so it would be a good idea to keep this in mind for our approach to tomato-related research as well.

Tabular mindset #1 – Cultivation and re-use of good questions, manifesting in the form of columns (or checklists, which deserve their own post).

Tables have a few mathematically-inclined cousins from which they could probably learn a thing or two, namely arrays, matrices (the plural for matrix, which means wildly different things in different contexts, but you might be familiar with some well-known examples of the type I have in mind) and tensors (which are basically multi-dimensional matrices).

I’d rather avoid the strict definitions of these words altogether, and so instead I’ll blasphemously refer to them as fancy tables. One of the things that makes them fancier than regular tables is their directionality, meaning that columns (and sometimes rows as well) serve much like the axes of a graph, representing some changing quantifiable property such as the progression of time. Moreover, we can also identify trends in other quantifiable properties as we move throughout the table. In other words, there exists a relationship between a cell’s content and its relative position.

For example, consider a geographic map with a coordinate system or grid: we should expect places in the same physical vicinity to be represented by neighboring cells on the map. Additionally, if cells are close to one another, they should exhibit similar climate properties, and local temperature should generally decrease as we move toward the poles.

An especially fancy table I’d like to talk about is the periodic table of chemical elements, which organizes elements according to their chemical group and period (represented by columns and rows, respectively).

The periodic table’s fanciness is particularly apparent in its trends – as we move in various directions throughout the table, we can identify changes in certain quantifiable elemental properties (such as atomic radius, electronegativity and metallic character).

One of the utilities of the periodic table as a scientific tool has historically been its power of prediction as a model of reality. In other words, if we accept the table’s premise and validity, then it follows that a gap in the table must indicate an undiscovered element. Moreover, we can predict the properties of this missing element, meaning that we will know what to look for – which might also teach us how to find it – and it will be that much easier to recognize it when we encounter it.

Similarly, as we develop our own analysis table, we will probably need to leave empty spaces for our known unknowns (the things we know that we don’t know). However, the structure and trends of the table itself, along with the content of non-empty cells, may be able to teach us something new about the gaps in our knowledge (much like deducing the contents of a missing square in a Sudoku grid).

Furthermore, we may encounter unknown unknowns as well (surprising gaps in our knowledge that we were previously unaware existed) which is another way our analysis table can teach us important new things about our research domain. This is similar to laying out events on a timeline to better identify missing links or “plot holes” in the narrative (though timelines are more graph-like than table-like, and they also deserve their own post).

Tabular mindset #2 – Using the very structure of our knowledge to confront us with gaps and steer our research, by learning what we should be looking for and how we can find it.

For example, say we have written an analysis table summarizing our knowledge of various vehicles according to their number of wheels, wheel layout and whether or not they’re motorized.

By fancifying our table and correlating number of wheels (a quantifiable property) with wheel layout (a qualifiable property), we can introduce structure and trends that confront us with gaps (known or unknown) in our accumulated knowledge of this particular subject. These can teach us to ask new questions, such as “why haven’t we been able to identify any vehicles with more than two wheels in a sequential layout?” (the answer to which probably has something to do with lateral instability).

Tabular mindset #3 – Progressing from data representation to ideation (idea generation) by relying on what we already know (and don’t know) to come up with (and sometimes answer) new questions.

Now, if you’ll allow me another anecdote about the periodic table: there in fact exist multiple alternative methods of tabulating the elements – all scientifically sound – each set up according to a different organizing principle meant to emphasize different chemical or physical phenomena.

In the same way, our analysis table might take different forms over the lifetime of our research, its structure evolving according to our current data set and whatever research questions we are in the midst of tackling. It’s also quite possible to have multiple equivalent analysis tables, each relying on the same data set. We might also find it helpful to split larger tables into several smaller ones, each focused on a different specific facet of our research.

Tabular mindset #4 – Representing and synthesizing the same information in different ways.

Speaking of models, what if our data doesn’t fit? Well, we should usually take this to mean that it’s faulty (perhaps our instruments require recalibration), or that we’re misinterpreting it somehow. But it could also mean that our model is partially or wholly incorrect, and must change in order to accommodate our strange new data. In tabular terms, this could mean adding new columns, changing existing ones, splitting them up, or restructuring entirely.

Understandably, this last option is considered less likely in the case of well-established models (and such claims will undoubtedly encounter resistance), but revolutions have happened before, and we must accept their possibility, especially during the stages of early hypothesizing.

Tabular mindset #5 – Using what we already know as a guide to help us determine how we should handle anomalies: by correcting our data measurements, changing our interpretation of the data, or adapting our model of reality.

In conclusion, the use of tables and one’s ability to fancify them are critical skills at nearly every phase of research: tables streamline our collection process, greatly improve our analysis, and make our products easier to understand. However, actually using a table isn’t always necessary for analysis – the point is to acquire the habits of tabular thinking.

Research and analysis involve quite a great deal of searching for information. In practical terms, this entails querying various databases and cross-referencing with our own ever-expanding research corpus. These databases usually contain some form or another of collected information (whether raw, pre-processed or enriched), and our goal in this is usually to find specific data, answer our research questions and reach new conclusions.

The most familiar example of this process is probably the use of a general web search engine (e.g. Google), but more specialized research scenarios often involve domain-specific tools (for example, a cyber security researcher might query malware samples stored in VirusTotal by utilizing YARA, a domain-specific querying language). In any case, we can identify a common strategy - or at the very least, a common set of tactics - for querying data in the course of research.

Obviously, we can improve our chances of success with this general strategy by augmenting it with orientation (by which I mean familiarizing ourselves with the characteristics and quirks of our databases), intuition (as in accumulated experience, or perhaps research fluency) as well as some degree of creativity and common sense. Furthermore, we can streamline our queries by investing in proper organization and enrichment of our data.

The most basic element of our general querying strategy is pivoting, by which we continually proceed from one query to the next, adapting both our queries (how we search) and our goals (what we’re searching for) based on everything that we have discovered so far, and considering which avenues of research still have potential as opposed to those which have been apparently exhausted.

pivot (noun)

1. a shaft or pin on which something turns

2. the center point of a rotational system

pivot (verb)

1. to turn on or as if on a pivot

2. to adapt or improve by adjusting or modifying something

It’s best to approach this strategy without putting too much thought into where we should begin (the chicken or the egg), but rather just picking up wherever we happen to be (in medias res, so to speak). It doesn’t really matter if our actual starting point was an idea or some previous query (which may have been the result of an even earlier query, and so on) - what matters is what we know right now, and how we should use that knowledge to make the best decision on our next steps.

Our short-term goal at any given point in the process is to learn something interesting, as in information that is both new and relevant to our research. In other words, the information we’re looking for should engage somehow with our current understanding of the issues at hand - perhaps it serves to confirm something we already suspected, challenges our assumptions or surprises us. We should use each such piece of information to do one or more of the following:

• Answer our research questions, by supporting or refuting a claim.

• Properly record the information in our corpus for future use.

• Continue our search by pivoting on the information, whether we hope to find more of the same or locate something else entirely.

Sometimes we have a specific item or piece of information in mind, or even know exactly what we’re searching for (perhaps we’re looking for a website, article or quote we’ve read before). At other times, we may only have a general idea of what we’re searching for, or haven’t the slightest clue where the information might be. Either way, it’s important to imagine how the information could be presented and visualize the form it might take in the database. Even if we end up being wrong about it, the thought exercise will often supply us with a good starting point to iterate on, and we might get it right on our next try.

Say we’ve managed to find an item of significance with interesting information (such as a webpage, document or database entry). At this point, our pivoting thought process should take into account two aspects of the item of interest:

• Content:

  • What is it about?

  • What names does it mention? (e.g. people, places, things)

    • How exactly does it refer to them? (note the use of nicknames or technical terms)

  • What unique identifiers does it contain? (e.g. catalog numbers, CVEs)

• Metadata:

  • What is it?

  • What is it called?

  • How is it structured?

  • Who made it?

  • When was it created?

  • Where did we find it?

  • Where did it originally come from?

Each of the answers to the above questions can be used to finetune or revamp our next queries, by either broadening, narrowing or shifting the scope of our search, depending on what we hope to find. At each juncture, we should keep in mind two complementary measures of quality for our query, which can help us decide if our scope is too broad, too narrow or completely off the mark:

• Recall - how many of the sought-after items are included in the results?

• Precision - how many of the items included in the results are sought-after?

Our assessment of these factors need not be scientific or exact. Instead, its usually best to just get a feel for the results and make a snap judgement on how to proceed. For example, if our query returns too many results for our liking, or if the first few results seem irrelevant, we can simply assume that the precision is too low and adjust our query accordingly. Conversely, if our query returns too few results, none of which seem relevant (except perhaps for items we’ve already located), then we can surmise that the recall is too low and that we should try something else.

If we’re looking for more of the same (such as additional articles written by the same person, or documents in the same format), we must identify the item’s unique characteristics (from both a content and metadata perspective). This entails recognizing what makes the item special or rare in comparison to most other items in our databases, and what properties it probably shares with similar items (i.e. in what way we expect them to be similar; such as tables sharing the same field names but containing different sets of data). Then, we can intersect these attributes in our queries (with AND operators) to more easily locate other items of the same type. Conversly, if we wish to remove similar items from our quety results (such as documents in a particular format that aren’t of interest to our current research), we can exclude these attributes in our queries.

Otherwise, if we intend to find different types of items that somehow relate to our current one - such as others touching on the same subject - then we need to extract leads that we can follow up on, or “threads” we can pull, in our next queries.

Either way, our general strategy at any given point in the process will involve:

1. Extracting leads or attributes from the interesting item, in the form of both metadata and content, and adding them to our existing query “vocabulary”.

2. Building a new query, or finetuning a previous one, by:

   1. Including promising leads.

   2. Including certain “helper” components (see below).

   3. Excluding noise indicators (with NOT operators), such as words that often show up in items that we’d like to filter out of our query results (these indicators can often be just as valuable as good leads).

3. Executing the query:

   1. If the precision seems satisfactory, browsing through the results until we find an item of interest, and [go to step 1].

   2. Otherwise, [go to step 2].

The halting conditions in the algorithm outlined above are somewhat difficult to precisely define, but we should generally move on once we’ve (optimistically) discovered enough interesting information or (more realistically) once we feel that we’ve run out of ideas or exhausted our databases. Often, it’s best to simply try again at a later date, once more data becomes available or we come up with new ideas.

As mentioned above, we can make excellent use of “helper” components in our queries in order to improve recall and/or precision, namely:

• Synonyms of the words we’re looking for (preferably while using an OR operator)

• Words in the same semantic field (e.g. “driver” or “garage” in relation to “car”)

• Hypernyms (generalizations, e.g. “vehicle” in relation to “car”)

• Hyponyms (specializations, e.g. “Toyota” in relation to “car”)

• Co-hyponyms (e.g. “tractor” in relation to “car”, as both are types of ‘‘vehicle’’)

• Meronyms (smaller parts, e.g. “tire” in relation to “car”)

• Holonyms (greater wholes, e.g. “fleet” in relation to “car”)

• Collocates (words that often appear together, e.g. “driver” in relation to “car”)

For a given word, we can use a thesaurus or tools such as WordNet to help us discover other words in the same semantic field, including synonyms, hypernyms and hyponyms.

Including these “helper” components (sparingly) in our queries allows us to detect items that would be otherwise hard to spot, due to their potentially unexpected inclusion of variations on certain words (in some cases, our unaugmented queries would miss these items entirely).

Collocates in particular are very useful for narrowing the scope of our queries to specific domains in order to improve precision (for example, we could differentiate between “virus” in the medical sense and the computer science sense by including either medical collocates such as “antibody” and “prescription” or computer science collocates such as “vulnerability” and “executable”). Alternatively, we could (carefully) exclude collocates related to irrelevant domains in order to denoise our query results.

In closing, a few words of advice:

• Pivot between different databases, and avoid fixating on a single database.

• Adjust your querying strategy according to the characteristics of each database.

• Consider that others may have recorded and organized the information you seek in ways that seem counterintuitive.

• As a rule, quantity is more important than quality (an unpopular opinion, to be sure):

  • It’s okay to compromise on low precision and sift through many results, as long as we’re quite confident in our recall (otherwise we’re at greater risk of wasting our time).

  • While we should hope to find “holy grails” in our data (as in a single piece of information that answers our research questions), it’s more feasible to discover lots of smaller bits of information that we can piece together to reach an important conclusion - we must not let this discourage us.

• Some items we encounter are valuable for the information they contain that pertains directly to our research efforts, whereas some items are primarily useful for pivoting purposes (i.e. growing our query “vocabulary”, for either inclusion or exclusion), or for honing our orientation by teaching us about the data itself.

• Record (or rather, hoard) the information you discover by adding it to your corpus while including the exact source (so you can return to reexamine it later and also use it as a reference in your research products), as well as any notes on ideas or possible connections to other pieces of information you’ve encountered so far. Additionally, you should tag the information so that you can more easily retrieve it from your corpus at a later point in time.

Our minds make stories, and stories make our minds. Each culture's Make-a-Human kit is built from stories, and maintained by stories. A story can be a rule for living according to one's culture, a useful survival trick, a clue to the grandeur of the universe, or a mental hypothesis about what might happen if we pursue a particular course.

— The Science of Discworld, Terry Pratchett

The use of narratives in research, communication and organization is one of my favorite topics, and I hope to expand on this in future posts. This is a popular subject, and most of the concepts I introduce here are based on the works of others — if you find this overview interesting, please be sure to check out the links in the Further Reading section.

The Researcher’s Role

My personal experience in research began with the misguided impression that my role was merely to create useful knowledge that would hopefully be consumed by others at some point. However, I later realized that in doing so, I was passing the buck and not properly fulfilling my role within my organization. My responsibility as a researcher is in fact much broader — I must be proactive in ensuring that my knowledge is well-utilized (imparting it on the right person at the right time). Taking this a step further, this often means that my role entails both creating knowledge and utilizing it myself — I need to be my own best customer.

The researcher should play an active role of sense-maker and perception-shaper — using their knowledge and understanding of reality to induce change (or impermeability) in their customers’ posture or policy. This is usually initiated in response to significant changes in reality, whether external circumstances or internal aspects relating to the customers themselves (by which I mean decisionmakers that the researcher supports or reports to, inside or outside their own organization, depending on context).

To this end, the researcher should weave narratives and act as a kind of story-teller. But how can we make our research accessible and relevant to others? How should we best distill the complexity of our research so that our products are meaningful to our audience, such that our conclusions resonate with them in ways that matter?

You might be familiar with the following scenario: say you recently discovered something of interest in the course of your research, decided to present your findings to your customers, and assumed they would deduce the implications on their own. However, you found yourself caught off-guard by follow-up questions such as “so what?” or “what should we do about this?” — these are in fact the most important questions a researcher must answer.

Furthermore, we shouldn’t settle for merely generating research products, but rather strive to provide an end-to-end service — what might even be best-described as an experience — to decision-makers. Our goal is to help them develop a better grasp on reality in order to make better judgements on how to deal with current circumstances.

Once we reach an assessment in our research, we must put it to good use:

1. Direct attention of decision-makers and steer discourse within the organization towards specific occurrences in reality, by answering the question “what is happening?” (or “what has changed?”)

2. Help our customers understand the value of these occurrences, by answering the question “so what?”

3. Guide the decision-making process towards specific courses of action, by answering the question “what should we do (and how)?”

Our answers should take into account several crucial parameters:

1. What aspects of reality are relevant to the organization’s function?

2. What do our customers hope to achieve? What are they afraid of? What do they have to gain or lose?

3. How might our customers be affected by current trends? How could these changes support their goals, undermine them or put them at risk?

4. What tools do we have at our disposal to adapt to current circumstances or take advantage of them? What can our shared experience teach us about how best to utilize these tools for this purpose?

In many ways, our role as researchers is to tell a story about the intersection between the organization and phenomena in reality (in other words: making sure we understand what we’re facing, ourselves, and our context). The story that we tell about a given research subject is how we organize our assesemments about it. To make use of our story to influence decision-making (or inform future research), we must proactively take a part in the process — our customers and colleagues cannot afford to wait for our input indefinitely (even if they are inclined to do so), and the discussion will take place regardless, with or without us. We need to be in the room where it happens, and provide answers to the above questions on time, while it still matters.

But how does one learn to influence decision-making? Much like any other skill, we should try to imitate past efforts (both in terms of process and final product); request feedback from peers about our interactions with customers; make the most of the internal review process by learning from others; and cultivate our experience by analyzing our own achievements and mistakes.

Telling Stories

Stories have a typical structure that our audience is almost certainly highly familiar with, generally composed of multiple stages: exposition, conflict, climax and resolution. By viewing the researcher’s role as that of a (reliable and factual) story-teller, we can take advantage of this deep-rooted familiarity and structure our products in ways that “plug-and-play” into our customer’s story port, so to speak. If you’ll allow me a biological analogy, we should fold our narrative proteins such that they bind to our audience’s story receptors.

We can make our talking points easier to understand by highlighting narrative elements such as causality, directionality and contrast. By regarding our customers as “protagonists”, and meeting their hardwired expectations of what constitutes a “good” story about their encounter with the phenomenon at hand, we can provide them with an effective sense-making experience.

A story, in this sense, is a structure we use to explain why the information it contains is meaningful to our audience. It serves this purpose both in the context of a single product and in our continuous interaction with our customers (which offers a shared learning experience that informs and refines our narrative over time). It exists in our head in some initial shape from the very start of the research process, but undergoes many changes throughout. Finally, the narrative ceases to serve its purpose when it doesn’t match up with reality anymore, and it is vitally important that we recognize this in time.

Complex narratives tend to “compress” into simpler forms that are easier to digest and communicate, such as visual representations, metaphors and figures of speech (if I may digress from the subject of research for a moment — some historical examples of controversial narratives that were intentionally “constricted” in this manner include slogans and catchphrases such as “Make America Great Again”, “It’s the Economy, Stupid”, and “Pro-Life” — each of these tells a story about the intersection between Americans and their reality). Rather than resisting this tendency, we should recognize its inevitability and carefully guide the process of simplification by use of illustration and thoughtful choice of wording.

Something else to consider is that narratives are often in competition (this is most obviously true in the above political examples) — the larger our organization or research community, the more likely we are to encounter differing grasps on reality — influencing decision-makers corresponds to propagating a narrative of our choosing.

However, the point of contention in such cases is not necessarily the veracity of one fact or another, but rather how to interpret what we know, such as claiming causation as opposed to correlation, or assigning relative importance to certain data points. Thus, competing narratives are not always contradictory, but rather emphasize different elements of reality. As researchers, we must be knowledgeable of competing narratives, and assume that our customers are well aware of them (or even actively promoting one or another). Furthermore, we should ensure that our own products acknowledge and converse with other points of view, and either support or refute them explicitly (counterintuitively, the lower the quality of competing narratives, the more challenging and time-consuming this becomes).

By this point, you might be concerned that adopting such an approach to research communication is deceitful, but story-telling is merely a tool, meaning one must learn to wield it effectively; it can be used for both good and evil (narratives don’t tell lies — people use narratives to tell lies); and there are stories of varying quality (independent of their basis in fact or fiction) — a truthful narrative is only as good as the data and analysis on which it is based. I’m not advocating that we cherry-pick facts or fabricate causality, but rather highlight what already exists within our data.

Narrative Training Wheels

In closing, I’d like to recommend two methods I’ve found useful to promote a narrative mindset in research communication:

1. ABT (short for “And But Therefore”) — Look for opportunities in your talking points to replace a neutral “and” with a more appropriate and narrative-conducing conjunction, such as “but” or “therefore”. This serves to emphasize contrast (such as differences between prior expectations and how things turned out; what we hope to achieve and what stands in our way, etc.), and also allows us to draw and highlight connections (such as relationships of cause and effect, conflict and resolution, basis and recommendation, etc.). By aiming to structure our products according to this format, we can more easily compress our messaging into a concise and compelling elevator pitch. This story-telling device was originally conceived by South Park creators Trey Parker and Matt Stone, and further developed by marine-scientist-turned-filmmaker Randy Olson.

2. Message Box — Generally speaking, it’s a good idea to plan (and re-plan) our products, and effective communication of our research findings requires honing our talking points before and throughout the drafting process. The Message Box framework, developed by Nancy Baron of COMPASS, offers us guidelines to develop our messaging by way of the following questions:

   • Who is our audience?

   • What big-picture issue are we addressing?

   • What specific problems within the larger issue are we focusing on?

   • Why should our audience care? How does it impact them? (“So what?”)

   • What solutions are we proposing? How do we suggest dealing with the problems at hand?

   • What are the benefits of addressing these problems?

Both of these techniques are worth giving a try, either habitually or when faced with a particularly challenging or important product requirement.

Further Reading

1. What Would You Say You Do Here? Redefining the Role of Intelligence in the Information Age, by Zachary Tyson Brown (War on the Rocks)

2. The Art of Judgement, by Sir Geoffrey Vickers

3. Theory and Practice, by Gregory F. Treverton

4. Sensemaking in Organizations, by Karl E. Weick

5. Houston, We Have a Narrative: Why Science Needs Story, by Randy Olson

6. From Aristotle to South Park, by Randy Olson

7. The Message Box, by Nancy Baron of COMPASS

“Questions were disorder awaiting organization. The more you understood, the more the world aligned. The more the chaos made sense, as all things should.”

— Rhythm of War, Brandon Sanderson

Individual research experience should ideally lead us from competency to fluency (to borrow the linguistic term). But what does it mean to be fluent in researching, and how does one reach research fluency?

Defining Research Fluency

First, we should point out two types of fluency:

1. Domain-specific fluency, which we can expect to be a function of personal “time-on-target” in a certain research domain.

2. General fluency, which can be assumed to be a function of overall personal experience in various domains of research over time, and may promote greater initial proficiency in new research ventures.

In my view, being research fluent (either domain-specific or in general) means being capable of the following:

• Identifying questions in need of answers, both independently and by relying on the input of others (such as requests or dilemmas put forward by customers).

• Choosing the best methods and tools for the job (as a prerequisite, we should be familiar with those available and highly competent with those most useful).

• Recognizing novelty and patterns worthy of our attention (more on this later).

• Efficiently searching (as in datamining, sifting and performing lookups in our raw material), “gisting” (a good guide can be found here) and curating on the go (by this I mean continuously re-organizing all our various collected reference material and notes in a sort of “Pokédex” - I’ll expand on this concept in a later post).

• Generalizing and abstracting as necessary, while adapting our wording to the target audience.

• Structuring outlines that best fit the intended messages of the product.

• Iterating on drafts while prioritizing aspects of most importance (otherwise we might find ourselves iterating indefinitely in search of perfection, and thus missing the opportunity to make an impact - avoiding this requires bravery of sorts, in order to choose our battles and know when to stop).

How to Get There

Research must be treated as a discipline, meaning that we can learn and improve by studying theory, imitating the work of others, and cultivating our experience; there is room for trial and error, and therefore reflection; and we must adapt our methods to different domains and refine our toolkit as new technologies emerge.

However, research can also be treated as an art form, meaning that there are different ways to achieve the same goal; intuition develops over time and plays an important role in our decision-making; and the craft requires a certain headspace or state of mind.

Taking this into consideration, I think the tl;dr of achieving research fluency is picking up the following habits (I’ll elaborate on each of these in turn).

1. Prepare

2. Submerge

3. Flow

4. Practice

1. Prepare

This means equipping yourself by having a vast knowledge-base on hand, including previous related works (your own and those of others). This is meant to best support “flow”, by allowing the fluent researcher to continuously (bordering on obsessively) cross-reference via this knowledge-base, curate it (as described above) and utilize it for re-orientation (i.e. familiarizing ourselves with our requirements, sources, etc.).

Additionally, preparation refers to planning - but only as much as necessary. For long-term research projects, we should begin by clarifying our goals, mapping relevant sources of information and past research products (to be used either as references or exemplars) and identifying subject-matter experts we should consult with.

2. Submerge

Basically, submerging means starting quickly and boldly, by reflexively looking up information as soon as we have the slightest idea what to search for, and by drafting as early as possible (in terms of content as well as design).

The fluent researcher confidently gets to work almost immediately, and just tries doing look-ups for whatever; asking around to utilize the expertise of peers; and putting something (anything!) on paper, without fear of making a mess (and messes will surely be made).

Submerging entails adopting a (conservative) “fail fast” mentality, by testing ideas in the field, quickly getting feedback and continuously iterating. To paraphrase Cunningham’s Law, the simplest way to get the conversation and iteration process going is to take initiative and just start writing.

However, the fluent researcher retains responsibility and ownership over their product, and never sends a draft to review without meticulous proofreading and marking specific sections and issues for consultation during the review process.

3. Flow

The fluent researcher works progressively, so that each draft iteration can serve as a relatively useful product on its own. An analogy for this style of work can be found in the progressive JPEG compression format, wherein the image is completely visible albeit blurred during the entire loading process (as opposed to baseline format, wherein the image remains partially obscured throughout):

We can point to another analogy in the intermediate levels of organ complexity observed throughout evolution by natural selection, à la ”The Blind Watchmaker”:

Practically speaking, this means quickly documenting our findings by gisting the material and moving on to the next item of data; continuously drafting on all “layers” of the product in parallel (reference, fact, assessment and conjecture); and of course, curating our knowledge-base as we go.

To flow is to keep pulling threads, by “pushing through” the raw material, imagining or visualizing what we’re looking for and pivoting accordingly, and cross-referencing and re-orienting via our knowledge-base (as explained above). It also means having a solid research plan but valiantly revising it as we learn new things, and iterating on our product outline as necessary.

As we flow, our attention should be focused on solving important puzzles, by evaluating novelty and recognizing and dealing with contradictions (between various items within the material, or between the material and our current assessment). In other words, the fluent researcher is attuned to novelty and contradiction, and devotes much of their time and energy to resolving contradictory discoveries.

While examining raw material, we must strive to collect useful and valuable information, such as information that can lead us to more information (via pivoting); information we can use as evidence to support an argument (this usually translates to references in footnotes); or information that might prove useful only at a later point in our research (to support analysis of competing hypotheses).

4. Practice

The fluent researcher is constantly iterating (as made clear above), and spends time rereading and utilizing their own past works and discourse in order to cultivate their experience.

They also spend a great deal of time interacting with their peers, by reading others’ work critically and sharing their thoughts with them; exposing their own work to critique, feedback and ideas; taking part in discourse, finding something distinct to say and making their voice heard; showing (off) their work early, while sharing raw findings and initial insights (but clearly indicating them as such in order to avoid confusion).

As we practice research as a discipline, we should periodically go meta, by thinking about research professionally and methodically; reflecting on our work and learning from past triumphs and mistakes - these should be developed into checklists, to be used during proofreading, for example.

Common Research Mistakes

• Parroting technical terms (one should never use a word without understanding its meaning, at least abstractly).

• Straying (too far, or without justification) from the local conventional style and wording.

• Mixing “layers” (reference, fact, assessment and conjecture).

• Mixing chronological vs. logical sequence (i.e. hopping between different types of story-telling within the same tale).

• Misrepresenting evidence.

• Misusing probability (likelihood) and confidence descriptors, or avoiding them entirely.

• Working too linearly (as opposed to exploring diverse avenues in parallel).

• Analyzing in a vacuum:

  • Ignoring biases (the Streetlight effect)

  • Missing (or ignoring) the state-of-the-art:

    • Not “conversing” with the reigning paradigm.

    • Not referencing earlier work.

  • Not utilizing personal or organizational experience in other domains.

Closing Thoughts

In conclusion, to attain research fluency, we should work on acquiring the habits mentioned above (prepare, submerge, flow, and practice).

This takes time and effort, but there are also unique advantages to being inexperienced (generally speaking or in a specific domain): a novice has the privilege of a fresh perspective; it can be easier to spot new types of patterns; they are less constrained to reigning paradigms; they have less emotional baggage related to past failures; and they are more open to doing things differently (as well as giving failed methods another chance). Beginners should (cautiously) make the most of these traits while they last, for fluency is sometimes accompanied by a certain rigidity of thought…

The following is an outline I wrote (and rewrote several times) to best prescribe (and later describe) the structure of this post:

1. What is an outline?

   1. How best to describe an outline?

      1. Somewhat like a table of contents

      2. A multi-level list, where each section poses a different question to be answered

      3. Sub-sections compose sequences (such as lists, logical arguments, or chronologies)

   2. What is an outline meant to accomplish?

2. How do we make an outline?

   1. Where to begin?

      1. Use research questions

      2. Just start writing

   2. What are some signs of a good outline?

      1. Linearity (prerequisites first)

      2. Contains most important messages

      3. Minimalist

      4. Simple and straightforward

      5. Good “flow”, or a compelling narrative

One of the most useful tools for research-oriented writing is the outline, a sort of table of contents that maps out the research product, meant to be used by the writer as a blueprint or “recipe” during the writing process.

When considering the purpose of the outline, we can point to three main points:

1. We can use it as a “compass” to keep us focused and help us separate the wheat from the chaff, as well as stick to the planned scope of the product and thus avoid feature-creep.

2. The act of preparing the outline in and of itself encourages us to contemplate how best to organize the content, and there is implicit value in putting our initial thoughts on paper and workshopping until we can express them with high specificity.

3. When working as part of a team, the outline should be discussed by everyone who is expected to later be involved in the writing and editing process. This discussion should promote a consensus on the product’s structure, as well as its goals, audience, subject matter and main messages. This is meant to ensure proper coordination of expectations between all participants.

An outline is made up of a multi-level list, into which the writer is tasked with “infusing” content. Each section contains a question that the writer must answer in the corresponding place within the product, or a headline that accurately prescribes (and eventually describes) the intended content. We can use sub-sections to compose sequences, such as the various components of a logical argument, a list of subjects to be expanded upon, or a chronological series of events.

Given the above-mentioned purpose of the outline, unlike an actual table of contents, we should prepare it at an early stage of the writing process, more or less once we have reached a point in our research at which we feel primed to begin writing (better sooner than later).

However, an outline can and should be iterated upon during the lifetime of the product, and we should expect to refine it as we discover new findings and reach new conclusions in our research. Furthermore, regardless of new findings or conclusions, sometimes our original outline may simply prove irrelevant as the product materializes - when the words cease to be just a theoretical idea in our heads and begin to take shape on paper, they may “behave” differently than we initially imagined, and in this case we should “play” with the outline (reorganize or substitute sections as needed) until we arrive at a flow that feels more coherent and less arbitrary. Generally speaking, our final outline may look drastically different than our initial one.

A good starting point for an outline is to take our research questions and lay them out as a list of sections, then shift them around - the order of sections should be determined by which question should or must be answered before others (prerequisites go first). Then, we can convert some of our questions to sub-sections where appropriate - for example, if we can answer a more specific question as a means to answer a broader one, then the first should be a sub-section of the second.

To determine if our outline is up to standard, we should check for the following characteristics (note that these are quite similar to the characteristics of a good research product, but should be adhered to in a stricter fashion):

1. Linearity (the reader should be able to understand each section of the product based on their understanding of the previous sections).

2. Contains the most important messages (as agreed upon during the planning process).

3. Minimalist (no “fluff”).

4. Simple and straightforward language (meant to eliminate ambiguity and thus ensure that our intent remains clear both to ourselves and to our team members during the entire writing process).

5. Good “flow”, or a compelling narrative (this is more difficult to assess objectively, but trust your instincts).