What Is Open Source Intelligence and How to Conduct OSINT Investigations

If you are just getting started in infosec or cybersecurity, you have probably heard of the term OSINT. OSINT or Open-Source Intelligence plays a crucial role in all sorts of investigations. If investigating means to move from a question to an answer, OSINT can often be employed to find the answer.

In this article, we will provide a brief overview of OSINT and how investigators use OSINT in a practical case. We will also share guidelines to keep in mind when practicing OSINT, and must-know OSINT resources to enrich your approach and sharpen your skills.

What is Open-Source Intelligence? 🔗︎

OSINT is what is known as one of the intelligence sources, or intelligence collection disciplines. Even though ”open-source intelligence” as a term has been in use for over three decades, to this day, there is no standardized definition for it. We define OSINT as intelligence produced from publicly available information that is collected, analyzed, and shared for the purpose of aiding a specific investigation.

In other words, OSINT is intelligence gathered by collecting and analyzing publicly available information and data for investigative purposes. OSINT data sources collected from the internet can encompass pretty much anything you can find online, from an IP address to public governmental records. In a broad sense, OSINT gathering can even encompass performing a Google search or reading through a public forum about learning how to fix a leaking pipe.

While OSINT collected from the internet is currently used across multiple industries – driving its popularization, there are more “traditional” sources of OSINT such as printed books, newspapers and journals, TV and radio broadcasts, photographs, and many more.

Differentiating Between OSINT and WEBINT 🔗︎

WEBINT, or web intelligence, is a term that has gained popularity in the past couple of years. Contrary to what is suggested by its name, WEBINT is not considered part of the intelligence collection disciplines or intelligence sources. So, what exactly is WEBINT?

The term “WEBINT” was initially coined by Ning Zhong, Jiming Liu Yao and Y.Y. Ohsuga in a paper they wrote for the Computer Software and Applications Conference of 2000. According to this first definition, WEBINT as a whole refers to the research and development fields which seeks to explore the different roles and the practical impacts of artificial intelligence (AI) on the different webs of wisdom (web of people, data, things, trust, agents, and health and smart living in the 5G Era), and advanced information technology on web-empowered products, systems, services, and activities.

But, if OSINT is an intelligence collection discipline, and WEBINT is a research and development discipline, why are these terms often confused?

One of the main sources of confusion has been the increase in the use of “WEBINT” as a marketing term, which has been primarily driven by technology vendors following a differentiation strategy within the OSINT landscape. This differentiation is needed because WEBINT technology can be used to improve the analysis of the data available via internet-based OSINT sources.

For further clarification, we need to look at this issue from the data perspective. OSINT is most valuable when the data collected for its production is properly exploited, analyzed, and verified. However, the intelligence cycle—collection, filtering, analysis, production— tends to be very time-intensive since the raw data is often unstructured. This is where WEBINT comes in.

Maltego’s data partner Recorded Future, describes WEBINT as the means to “efficiently identify the intelligence available via OSINT” by normalizing or providing structure to the unstructured information that is collected from the internet. Because of its use of artificial intelligence (AI), WEBINT can improve analysis time and effectiveness by providing features such as sentiment identification across social media posts, identification of trends over time, and predictive analysis for identification of future events.

How is Open-Source Intelligence Used? 🔗︎

OSINT techniques are practiced by all sorts of investigators and analysts across a range of industries—cybersecurity operations analysts, law enforcement agencies, fraud investigators, threat hunters, researchers, investigative journalists, and many more.

This means that OSINT can be used for a wide range of use cases, especially those relying on, or involving internet-based research.

Let’s take a look at the most common use cases we observe involving OSINT.

1. Cybersecurity Operations 🔗︎

Security professionals and ethical hackers use OSINT to identify potential weaknesses in a network, vulnerabilities, and external threats. From finding unpatched software or unsecured IoT vulnerabilities to monitoring underground criminal chatter, OSINT provides a wealth of data and information to aid incident response, damage remediation, threat intelligence, brand protection, penetration testing, network footprinting, and other investigations.

2. Social Media Investigations 🔗︎

According to Statista, over 3.6 billion people worldwide were social media users in 2020 and another billion will become social media users by 2025. The large amount of content and information created and shared on social media platforms has naturally become a form of open-source intelligence:

• Social activities like posts, photos, location pins, comments, and event RSVPs
• Personal information such as names, aliases, birthday, email addresses, education and career history, and more
• Interpersonal connections like friends, families, followers, and followings
• Theme-based groups, communities, and memberships

Investigators can comb through a cybercriminal’s social media footprint to identify potential accomplices and map out the related cybercriminal network. They can also trace a missing person’s social media activities to make our the last-seen location, possible suspects, and the victim’s lifestyle.

3. Risk Management and Fraud Investigations 🔗︎

Fraud investigators and risk management professionals can leverage OSINT to manage and identify infringements to intellectual property (IP), as well as fraudulent, counterfeit, and diverted products sold via the internet. They can also leverage OSINT to reduce their organizational risk exposure, identify potential loss incidents, and assist in loss recovery efforts.

The range and depth of OSINT is almost borderless in that it can also be used for other use cases like phishing, cryptocurrency laundering, due diligence, deep and dark web investigations, and more. It all depends on what OSINT tools and data sources you keep in your pocket and how you interpret and connect the information you have in hand.

Open-Source Intelligence Techniques: How to Use OSINT for Your Work 🔗︎

Since the limit of OSINT ends where the internet has no reach, it is as easy to acquire a ton of data as it is to get lost in the rabbit hole—or in some cases, become overwhelmed by the sea of “noise” and the sheer volume of information out there.

The following techniques should help you steer away from that risk and pivot onto the right path.

Keeping the OSINT Mindset: Always Start with A Question 🔗︎

For starters, you must keep in mind that practicing OSINT is not about finding anything and everything that could be useful, but instead, finding the answers to your questions—for this, you need to have a question to start with.

Before your investigation, you should define a goal you wish to achieve and outline all the questions you must answer to achieve that goal. With the questions in hand, you can then formulate a clear strategy and select the right OSINT tools or framework to search for the answer—the key information.

There are two types of approaches when it comes to OSINT information gathering: Active collection and passive collection. As the terms suggest, the difference between these approaches lies in whether contact is made with the target during the investigation.

Active OSINT Collection 🔗︎

Active OSINT collection makes contact with the target in order to gather real-time or more accurate data. However, this also exposes the investigators to a higher risk of detection by the target. Once the target is alerted and becomes aware that someone is trying to gather intelligence on them, they would very likely shut down external access to the data or try to trace the source to the investigators for counter attacks.

An example of offensive OSINT would be scanning a target website.

Passive OSINT Collection 🔗︎

Passive OSINT approach is a comparatively safer practice that gathers historical data or data hosted by third-party sources. Although the data retrieved might not be 100% up-to-date, investigators remain distant from the target and thus have a lower risk of being detected. Historical data also comes in handy when real-time data is not available anymore—such as after the malicious actors remove the website.

An example of passive OSINT would be looking up historical DNS records.

As it is often time-and energy-intensive to manually collect, comb through, and consolidate available OSINT information, many OSINT practitioners, experts, communities, and companies have developed various tools and frameworks to automate or simplify the data collection and analysis processes, or generate curated, attributed intelligence reports and databases.

It is almost safe to say that for every type of OSINT data, there is a tool or website you can use. And this is shown by the OSINT Framework.

The OSINT Framework 🔗︎

As you start exploring OSINT practices in your investigations, you will start hearing more about the term “OSINT Framework.” OSINT Framework is the most extensive repository of tools for OSINT information gathering and investigations. It sorts the resources into 32 categories based on the type of data investigators are looking for: Public records, social networks, images, videos, digital currency, dark web, archives, and more. In each category, you can find free and paid tools that can be used to locate the data in question.

Search Engines 🔗︎

Some of the most commonly used open-source intelligence tools—unsurprisingly—are search engines, since web-based research is a large part of OSINT.

Google and Bing are probably the most used search engines and advanced OSINT practitioners know to use advanced search functions called “Google Dorks” to narrow search results to specific file types, websites, strings, and IP addresses. There are also similar search operators, “Bing Dorks,” for the Bing search engine.

However, the realm of search engines does not stop there. Investigators also use less popular search engines like Yandex, DuckDuckGo, and Baidu to explore more results. There are also search engines designed for specific domains, such as:

• Metasearch Engines: For example, AnswerThePublic.com
• Code Search Engines: For example, GitHub or PublicWWW
• Academic and Publication Search Engines
• News Search Engines

…and many more.

The Awesome OSINT list curates a gigantic list of the various useful search engines for open-source intelligence investigations.

Link Analysis Tools 🔗︎

Link analysis tools are great for OSINT investigations as they are designed to automatically mine and map data relationships visually.

While some OSINT data sources like VirusTotal and CipherTrace also offer a data visualization feature, link analysis tools go a step further. They often integrate with multiple data sources and can merge different data onto the same graph, thus taking away the trouble of switching between tabs and tools.

There are countless other tools that can support OSINT investigations and it is almost impossible to list them all. Check out our top recommendations for OSINT tools here!

Learning to use OSINT: How Should I Get Started? 🔗︎

The true advice to learn how to OSINT effectively is to simply start by doing. However, there are many blogs, websites, podcasts, and books that you can turn to for the latest tips, information, and pointers.

Our group of experts put together a master collection that includes some of the best resources out there: More than 100 websites, blogs, podcasts, YouTube channels, books, and even more! Download the Master Collection of OSINT Resources now.

We hope you found this article useful for gaining a deeper understanding of what OSINT is and how to use OSINT for your investigations. Feel free to download the infographic here to share it with your industry connections!

Make sure you follow us on Twitter and LinkedIn and subscribe to our email newsletter to stay updated on new use cases, tutorials, and product development.

Happy investigating!