Difference between revisions of "Open-Source Intelligence - OSINT"

From
Jump to: navigation, search
m (→‎Primer.ai)
m
 
(38 intermediate revisions by the same user not shown)
Line 2: Line 2:
 
|title=PRIMO.ai
 
|title=PRIMO.ai
 
|titlemode=append
 
|titlemode=append
−
|keywords=artificial, intelligence, machine, learning, models, algorithms, data, singularity, moonshot, TensorFlow, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Meta, Facebook
+
|keywords=ChatGPT, artificial, intelligence, machine, learning, GPT-4, GPT-5, NLP, NLG, NLC, NLU, models, data, singularity, moonshot, Sentience, AGI, Emergence, Moonshot, Explainable, TensorFlow, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Hugging Face, OpenAI, Tensorflow, OpenAI, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Meta, LLM, metaverse, assistants, agents, digital twin, IoT, Transhumanism, Immersive Reality, Generative AI, Conversational AI, Perplexity, Bing, You, Bard, Ernie, prompt Engineering LangChain, Video/Image, Vision, End-to-End Speech, Synthesize Speech, Speech Recognition, Stanford, MIT |description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools
−
|description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools  
+
 
 +
<!-- Google tag (gtag.js) -->
 +
<script async src="https://www.googletagmanager.com/gtag/js?id=G-4GCWLBVJ7T"></script>
 +
<script>
 +
  window.dataLayer = window.dataLayer || [];
 +
  function gtag(){dataLayer.push(arguments);}
 +
  gtag('js', new Date());
 +
 
 +
  gtag('config', 'G-4GCWLBVJ7T');
 +
</script>
 
}}
 
}}
 
[https://www.youtube.com/results?search_query=ai+Open+Source+intelligence+OSINT YouTube]
 
[https://www.youtube.com/results?search_query=ai+Open+Source+intelligence+OSINT YouTube]
Line 13: Line 22:
 
* [[Cybersecurity]] ... [[Open-Source Intelligence - OSINT |OSINT]] ... [[Cybersecurity Frameworks, Architectures & Roadmaps | Frameworks]] ... [[Cybersecurity References|References]] ... [[Offense - Adversarial Threats/Attacks| Offense]] ... [[National Institute of Standards and Technology (NIST)|NIST]] ... [[U.S. Department of Homeland Security (DHS)| DHS]] ... [[Screening; Passenger, Luggage, & Cargo|Screening]] ... [[Law Enforcement]] ... [[Government Services|Government]] ... [[Defense]] ... [[Joint Capabilities Integration and Development System (JCIDS)#Cybersecurity & Acquisition Lifecycle Integration| Lifecycle Integration]] ... [[Cybersecurity Companies/Products|Products]] ... [[Cybersecurity: Evaluating & Selling|Evaluating]]
 
* [[Cybersecurity]] ... [[Open-Source Intelligence - OSINT |OSINT]] ... [[Cybersecurity Frameworks, Architectures & Roadmaps | Frameworks]] ... [[Cybersecurity References|References]] ... [[Offense - Adversarial Threats/Attacks| Offense]] ... [[National Institute of Standards and Technology (NIST)|NIST]] ... [[U.S. Department of Homeland Security (DHS)| DHS]] ... [[Screening; Passenger, Luggage, & Cargo|Screening]] ... [[Law Enforcement]] ... [[Government Services|Government]] ... [[Defense]] ... [[Joint Capabilities Integration and Development System (JCIDS)#Cybersecurity & Acquisition Lifecycle Integration| Lifecycle Integration]] ... [[Cybersecurity Companies/Products|Products]] ... [[Cybersecurity: Evaluating & Selling|Evaluating]]
 
* [[Large Language Model (LLM)]] ... [[Natural Language Processing (NLP)]] ... [[Natural Language Generation (NLG)|Generation]] ... [[Natural Language Classification (NLC)|Classification]] ... [[Natural Language Processing (NLP)#Natural Language Understanding (NLU)|Understanding]] ... [[Language Translation|Translation]] ...  [[Natural Language Tools & Services|Tools & Services]]
 
* [[Large Language Model (LLM)]] ... [[Natural Language Processing (NLP)]] ... [[Natural Language Generation (NLG)|Generation]] ... [[Natural Language Classification (NLC)|Classification]] ... [[Natural Language Processing (NLP)#Natural Language Understanding (NLU)|Understanding]] ... [[Language Translation|Translation]] ...  [[Natural Language Tools & Services|Tools & Services]]
−
* [https://osintframework.com/ OSINT Framework | Justin Nordine]
+
* [[Video/Image]] ... [[Vision]] ... [[Enhancement]] ... [[Fake]] ... [[Reconstruction]] ... [[Colorize]] ... [[Occlusions]] ... [[Predict image]] ... [[Image/Video Transfer Learning]]
 +
* [[End-to-End Speech]] ... [[Synthesize Speech]] ... [[Speech Recognition]] ... [[Music]]
 +
* [https://en.wikipedia.org/wiki/List_of_intelligence_gathering_disciplines List of intelligence gathering disciplines | Wikipedia]
 +
* [https://osintframework.com/ OSINT Framework | Justin Nordine] ... offers a web-based interface that breaks down different topic areas of interest to OSINT researchers and connects you to the tools that can help you sniff out the info you need.
 
** [https://github.com/lockfale/osint-framework OSINT Framework GitHub]
 
** [https://github.com/lockfale/osint-framework OSINT Framework GitHub]
 
* [https://www.economist.com/ukraine-osint-pod How has open-source intelligence influenced the war in Ukraine? | The Economist]
 
* [https://www.economist.com/ukraine-osint-pod How has open-source intelligence influenced the war in Ukraine? | The Economist]
Line 25: Line 37:
 
* [https://www.secjuice.com/artificial-intelligence-ai-and-osint AI + OSINT : Part One]
 
* [https://www.secjuice.com/artificial-intelligence-ai-and-osint AI + OSINT : Part One]
 
* [https://espysys.com/blog/why-ai-and-osint-made-private-investigators-work-easier-the-insiders-views Why AI and OSINT Made Private Investigators' Work Easier?]
 
* [https://espysys.com/blog/why-ai-and-osint-made-private-investigators-work-easier-the-insiders-views Why AI and OSINT Made Private Investigators' Work Easier?]
 +
* [https://www.sentinelone.com/cybersecurity-101/open-source-intelligence-osint What is Open Source Intelligence (OSINT)? | SentinelOne]
 +
* [https://www.secjuice.com/introduction-to-open-source-intelligence-osint/ An Introduction To Open Source Intelligence (OSINT) Gathering | Secjuice]
 +
* [https://usnwc.libguides.com/c.php?g=494120&p=3381426 Types of Intelligence Collection | Intelligence Studies - LibGuides]
 +
* [https://www.sciencedirect.com/topics/computer-science/intelligence-gathering Intelligence Gathering - an overview | ScienceDirect Topics]
 +
* [https://www.csoonline.com/article/564313/what-is-the-dark-web-how-to-access-it-and-what-youll-find.html What is the dark web? How to access it and what you’ll find | Darren Guccione - Foundry]
 +
 +
 +
Open Source Intelligence (OSINT) is the collection, analysis, and dissemination of information that is publicly available and legally accessible. OSINT operations, whether practiced by IT security pros, malicious hackers, or state-sanctioned intelligence operatives, use advanced techniques to search through the vast haystack of visible data to find the needles they’re looking for to achieve their goals—and learn information that many don’t realize is public.  OSINT operations use advanced techniques to search through visible data to find information that many don’t realize is public. OSINT differs from other types of intelligence gathering in several ways, OSINT:
  
−
Open Source Intelligence (OSINT) is the collection, analysis, and dissemination of information that is publicly available and legally accessible. It is used by organizations such as governments, businesses, and non-governmental organizations. OSINT operations use advanced techniques to search through visible data to find information that many don’t realize is public.
+
* is focused on publicly available and legally obtainable information, whereas other forms of intelligence gathering may involve confidential or classified sources.
 +
* uses various sources, including social media, news articles, public records, and government reports. In contrast, other forms of intelligence gathering may focus on a specific source type, such as human, signals, imagery, or geospatial.
 +
* is accessible and affordable to anyone with an internet connection and some basic skills, whereas other forms of intelligence gathering may require specialized equipment, training, or authorization.
 +
* is dynamic and diverse, as it reflects the changing and heterogeneous nature of the open source environment. However, this also means that OSINT may be inaccurate, incomplete, or unreliable, as it depends on the quality and credibility of the sources. Therefore, OSINT needs to be verified and validated with other forms of intelligence or evidence.
  
 
AI can be used to turn the challenge of large data volumes into an advantage and ultimately create a symbiotic relationship with OSINT. AI, machine learning and automation tools are able to scan massive amounts of data and conversations, proficiently identifying connections and risks that warrant further investigation. AI has dramatically increased the volume of data that can be analyzed in real-time. Today’s AI-enabled platforms empower intelligence analysts to leverage OSINT as the foundation that can help uncover hidden threats, corroborate classified reporting, and pinpoint the targets that warrant resource-intensive, traditional intelligence gathering.   
 
AI can be used to turn the challenge of large data volumes into an advantage and ultimately create a symbiotic relationship with OSINT. AI, machine learning and automation tools are able to scan massive amounts of data and conversations, proficiently identifying connections and risks that warrant further investigation. AI has dramatically increased the volume of data that can be analyzed in real-time. Today’s AI-enabled platforms empower intelligence analysts to leverage OSINT as the foundation that can help uncover hidden threats, corroborate classified reporting, and pinpoint the targets that warrant resource-intensive, traditional intelligence gathering.   
 +
 +
OSINT is in many ways the mirror image of operational security (OPSEC), which is the security process by which organizations protect public data about themselves that could, if properly analyzed, reveal damaging truths. IT security departments are increasingly tasked with performing OSINT operations on their own organizations to shore up operational security. - [https://www.csoonline.com/article/567859/what-is-osint-top-open-source-intelligence-tools.html 15 top open-source intelligence tools | J. Fruhlinger, A. Sharma & J. Breeden - Foundry]
  
  
Line 76: Line 101:
 
|}<!-- B -->
 
|}<!-- B -->
  
−
= Companies using AI for OSINT =
+
= OSINT Companies/Tools =
 +
* [https://osintframework.com/ OSINT Framework | Justin Nordine] ... offers a web-based interface that breaks down different topic areas of interest to OSINT researchers and connects you to the tools that can help you sniff out the info you need.
 +
** [https://github.com/lockfale/osint-framework OSINT Framework GitHub]
 +
 
 +
Several companies/products are using AI for OSINT. Here are some of them:
 +
 
 +
* <b>[https://www.babelstreet.com/ Babel X - Babel Street]</b> ... a multilingual search tool for the public internet, including blogs, social media, message boards and news sites. It also searches the dark web, including Onion sites, and some deep web content that Babel X can access through agreements or licensing from the content owners. The product is able to geo-locate the source of information it finds, and it can perform text analysis to identify relevant results.
 +
* <b>[https://builtwith.com/ BuiltWith]</b> ... lets you find what popular websites are built with. Different tech stacks and platforms power different sites. BuiltWith can, for example, detect whether a website is using WordPress, Joomla, or Drupal as its CMS and provide further details.
 +
* <b>[https://www.cellebrite.com/ Cellebrite]</b> ... provides digital intelligence solutions for law enforcement, military, intelligence, and enterprise customers. It enables users to extract, decode, and analyze data from various digital sources, including mobile devices, cloud services, computers, and IoT devices.
 +
* <b>[https://cobwebs.com/ Cobwebs Technologies]</b> ... provides mission-critical AI-powered OSINT solutions for law enforcement, government, financial institutions, and corporations. It enables users to search and analyze the web's big data using machine learning algorithms, and generate real-time intelligence reports.
 +
* <b>[https://www.dassaultsystemes.com/ Dassault Systemes]</b> ... provides 3D design, engineering, simulation, and data intelligence solutions for various industries. It enables users to create and manage digital twins of products, processes, and systems, and leverage AI to optimize performance and innovation.
 +
* <b>[https://www.digitalclues.io/ Digital Clues]</b> ... provides AI-powered OSINT solutions for law enforcement and intelligence agencies. It enables users to collect, analyze, and visualize data from various online sources, including social media, dark web, blogs, forums, and news sites.
 +
* <b>[https://espysys.com/ ESPY]</b> ... provides AI-powered OSINT solutions for security and crisis management. It enables users to automate the process of collecting, analyzing, and synthesizing information from various open sources, such as web scraping, geolocation, [[Sentiment Analysis]], and image recognition.
 +
* <b>[https://expert.ai/ Expert.AI]</b> ... provides natural language understanding solutions for various industries and applications. It enables users to extract insights and knowledge from unstructured text data using AI and natural language processing techniques.
 +
* <b>[https://grep.app/search?q=open%20source%20intelligence Grep.app]</b> ... search across half million git repos 
 +
* <b>[https://www.hensoldt.net/ Hensoldt]</b> ... provides sensor solutions for defense and security applications. It enables users to collect, process, and analyze data from various sources, such as radar, optical, electronic warfare, and cyber systems.
 +
* <b>[https://intelx.io/ Intelligence X]</b> ... preserves not only historic versions of web pages but also entire leaked data sets that are otherwise removed from the web due to the objectionable nature of content or legal reasons.
 +
* <b>[https://lampyre.io/ Lampyre]</b> ... provides a platform for conducting OSINT investigations. It enables users to collect data from various sources using queries or scripts, analyze data using filters or graphs, and export data in various formats.
 +
* <b>[https://www.maltego.com/ Maltego]</b> ... provides a visual tool for conducting OSINT investigations. It enables users to gather data from various sources using transforms or integrations, analyze data using link analysis or clustering techniques, and share data using reports or collaboration features.
 +
* <b>[https://github.com/laramies/metagoofil Metagoofil]</b> ... is optimized to extract metadata from public documents. Metagoofil can investigate almost any kind of document that it can reach through public channels including .pfd, .doc, .ppt, .xls and many others. Searches return things like the usernames associated with discovered documents, as well as real names if available. It also maps the paths of how to get to those documents, which in turn would provide things like server names, shared resources and directory tree information about the host organization.
 +
* <b>[https://chrome.google.com/webstore/detail/mitaka/bfjbejmeoibbdpfdbmbacmefcbannnbg Mitaka]</b> ... Available as a Chrome extension and Firefox add-on, search over six dozen search engines for IP addresses, domains, URLs, hashes, ASNs, Bitcoin wallet addresses, and various indicators of compromise (IOCs) from web browser.
 +
* <b>[https://nexvision.com/ NexVision]</b> ... provides vision solutions and optronic systems for various applications, such as defense, security, cinema, and industry. It specializes in electronic reference design, especially in complex vision processing systems. It uses AI and computer vision to extract insights and metadata from video data, such as object detection, activity recognition, and scene understanding
 +
* <b>[https://owlint.fr/ Owlint]</b> ... centralizes the digital footprint from the Clear Web, Deep Web, and Dark Web. Its proprietary AI OwlAnalytics assesses a level of risk on each detected element.
 +
* <b>[https://github.com/lanmaster53/recon-ng Recon-ng]</b> ... provides a modular framework for conducting OSINT investigations. It enables users to gather data from various sources using built-in or custom modules, store data in a local database, and visualize data using interactive reports.
 +
* <b>[https://searchcode.com/ searchcode]</b> ... is a highly specialized search engine that looks for useful intelligence inside source code.
 +
* <b>[https://www.shodan.io/ Shodan]</b> ... can monitor and search as part of an OSINT effort is impressive. It’s one of the few engines capable of examining operational technology (OT) such as the kind used in industrial control systems at places like power plants and manufacturing facilities. Any OSINT gathering effort in industries that deploy both information technology and OT would miss a huge chunk of that infrastructure without a tool like Shodan. In addition to IoT devices like cameras, building sensors and security devices, Shodan can also be turned to look at things like databases to see if any information is publicly accessible through paths other than the main interface. It can even work with videogames, discovering things like [[Minecraft]] or Counter-Strike: Global Offensive servers hiding on corporate networks where they should not be, and what vulnerabilities they generate.
 +
* <b>[https://www.signifyd.com/ Signifyd]</b> ... provides fraud protection and chargeback prevention solutions for e-commerce businesses. It enables users to leverage AI and machine learning to analyze customer behavior, transaction data, and network intelligence to detect and prevent fraud.
 +
* <b>[https://www.spiderfoot.net/attack-surface-monitoring/ Spiderfoot]</b> ... a free OSINT reconnaissance tool that integrates with multiple data sources to gather and analyze IP addresses, CIDR ranges, domains and subdomains, ASNs, email addresses, phone numbers, names and usernames, BTC addresses, etc. Available on GitHub, Spiderfoot comes with both a command-line interface and an embedded web-server for providing an intuitive web-based GUI. The application itself comes with over 200 modules
 +
* <b>[https://github.com/spyse Spyse]</b> ... Relied on by projects like OWASP, IntelligenceX, and Spiderfoot, Spyse collects publicly available data on websites, their owners, associated servers, and IoT devices. This data is then analyzed by the Spyse engine to spot any security risks in and connections between these different entities.
 +
* <b>[https://github.com/laramies/theHarvester theHarvester]</b> ... uses include popular search engines like Bing and Google, as well as lesser known ones like dogpile, DNSdumpster and the Exalead meta data engine. It also uses Netcraft Data Mining and the AlienVault Open Threat Exchange. It can even tap the Shodan search engine to discover open ports on discovered hosts. In general, theHarvester tool gathers emails, names, subdomains, IPs and URLs.
 +
* <b>[https://www.verint.com/ Verint]</b> ... provides customer engagement and cyber intelligence solutions for various sectors and organizations. It enables users to use AI and analytics to optimize customer interactions, enhance workforce performance, and protect from cyber threats.
 +
* <b>[https://voxel51.com/ Voxel51]</b> ... provides video analytics solutions for various industries and applications. It enables users to use AI and computer vision to extract insights and metadata from video data, such as object detection, activity recognition, and scene understanding.
 +
 
  
−
Several companies are using AI for OSINT. Here are some of them:
 
−
* Digital Clues
 
−
* Alfresco Software Inc.
 
−
* Octogence Tech Solutions Pvt. Ltd.
 
−
* Maltego
 
−
* SEON
 
−
* Lampyre
 
−
* Recon-ng
 
−
* SpiderFoot
 
−
* NexVision
 
−
* Verint
 
−
* Recorded Future
 
−
* Palantir Technologies
 
−
* Cellebrite
 
−
* Dassault Systemes
 
−
* Expert.AI
 
−
* Hensoldt
 
  
 
== <span id="Primer.ai"></span>Primer.ai ==
 
== <span id="Primer.ai"></span>Primer.ai ==
 
* [https://primer.ai Primer.ai]  
 
* [https://primer.ai Primer.ai]  
−
* [https://primer.ai/business-solutions/introducing-primer-delta-our-next-gen-ai-native-platform-to-transform-information-overload-into-decision-advantage/ Primer Delta]
+
* [https://primer.ai/products/primer-command/ Primer Command]
  
−
Using [[Machine Learning (ML)]] and [[Natural Language Processing (NLP)]] [[transformer]]s focused on software for DoD/IC customers with real-time news and social media insights, situational reporting, intelligence gathering, and faction analysis. Lists its customers: U.S. Air Force, Army, and U.S. Special Operations Command, and Fortune 500 companies. - [https://venturebeat.com/ai/military-contractor-primer-announces-69m-round-to-build-ai-for-those-who-support-and-defend-our-democracy/ Military contractor Primer announces $69M to build AI for ‘those who support and defend our democracy’ | Carl Franzen - VentureBeat]   
+
Using [[Machine Learning (ML)]], [[Vision|computer vision]] and [[Natural Language Processing (NLP)]] [[transformer]]s focused on software for DoD/IC customers with real-time news and social media insights, situational reporting, intelligence gathering, and faction analysis. Lists its customers: U.S. Air Force, Army, and U.S. Special Operations Command, and Fortune 500 companies. - [https://venturebeat.com/ai/military-contractor-primer-announces-69m-round-to-build-ai-for-those-who-support-and-defend-our-democracy/ Military contractor Primer announces $69M to build AI for ‘those who support and defend our democracy’ | Carl Franzen - VentureBeat]   
  
 
Some of [https://primer.ai Primer.ai] services include:
 
Some of [https://primer.ai Primer.ai] services include:
Line 107: Line 147:
 
* Semantic Search: Securely and fluently search, summarize, and explore data at scale with greater precision
 
* Semantic Search: Securely and fluently search, summarize, and explore data at scale with greater precision
  
−
Primer Delta Platform, a natural language document sorting and analysis tool that can sort through millions of documents in seconds. Primer Delta can “reliably extract entities, locations and topics of interest, surface hard to find insights, and generate comprehensive summaries,” and it can be deployed on local machines or on the cloud. A lighter version of Primer Delta is available on the [[Amazon |AWS]] Marketplace for commercial clients with a smaller trove of documents, from 100 to 1000 files, that they can use to “explore your unstructured text data within the comfort and security of your environment without needing to enter into a contract.”
+
Primer Command - By leveraging high-scale streaming data, you can detect kinetic and cyber threats up to several hours earlier than traditional methods and respond quickly. Primer Command is a real-time threat detection tool that analyzes tens of thousands of sources in 100 languages, including news and social media. With Primer Command, teams can monitor, respond, and maintain situational awareness of fast-breaking global events, such as kinetic and cyber threats.  
 +
Primer Command can:
  
−
Primer Command - By leveraging high-scale streaming data, you can detect kinetic and cyber threats up to several hours earlier than traditional methods and respond quickly.
+
* detect and summarize the "Who, What, When, Where, and Why" of every scenario, providing contextual intelligence and actionable insights.
 +
* generate situation reports that can be downloaded, edited, and shared, saving time and facilitating collaboration.
 +
* perform location-based searches and computer vision capabilities, such as identifying faces, objects, logos, and text in images.
 +
* filter and flag items of importance, and delegate responsibilities among team members.
 +
* integrate with [[Large Language Model (LLM)]].
 +
* operate in a secure and compliant manner, respecting data privacy and sovereignty.
  
−
https://primer.ai/wp-content/uploads/2022/03/command-01.png
 
  
 +
<youtube>e0PcOfP3P3A</youtube>
 
<youtube>C0XPkToznxU</youtube>
 
<youtube>C0XPkToznxU</youtube>
 +
 +
= Tools for Analyzing Social Media =
 +
* [https://sproutsocial.com/insights/social-media-analytics-tools/ 12 of the best social media analytics tools for marketers | Chloe West - SproutSocial]
 +
* [https://blog.hootsuite.com/social-media-analytics/ 10 Social Media Analytics Tools That Do the Math For You 2023 | Shannon Tien - Hootsuite]
 +
* [https://www.trustradius.com/social-media-analytics List of Top Social Media Analytics Tools 2023 | TrustRadius]
 +
* [https://buffer.com/library/social-media-analytics-tools/ 24 Free and Paid Social Media Analytics for Marketers | Ash Read - Buffer]
 +
* [https://www.socialpilot.co/social-media-analytics-tools Best 12 Paid and Free Social Media Analytics Tools for 2023 | Manvi Agarwal - SocialPilot]
 +
 +
Some of the most popular tools:
 +
 +
* <b>[https://blog.hootsuite.com/social-media-analytics/ Hootsuite Analytics]</b>: This tool allows you to track performance data from every social network in one place, with easy-to-understand reports. You can measure metrics like reach, followers, engagement, clicks, and conversions. You can also compare results across multiple networks and create custom reports based on your KPIs.
 +
** Pros: Comprehensive, user-friendly, and customizable.
 +
** Cons: Paid tool, requires a Hootsuite account, and may not support some niche networks.
 +
* <b>[https://buzzsumo.com/ BuzzSumo]</b>: This tool helps you discover the most engaging content and influencers on social media. You can analyze topics, keywords, domains, and authors to see what resonates with your audience. You can also monitor mentions, trends, and competitors to stay ahead of the curve.
 +
** Pros: Powerful, insightful, and easy to use.
 +
** Cons: Paid tool, limited free version, and may not cover all social platforms.
 +
* <b>[https://analytics.google.com/ Google Analytics]</b>: This tool is a cloud-based web analytics service that delivers different tools to optimize digital marketing. You can use it to track social media traffic, conversions, and campaigns. You can also integrate it with other Google products like Ads, Search Console, and Data Studio.
 +
** Pros: Free, reliable, and versatile.
 +
** Cons: Complex, requires technical skills, and may not provide all social media metrics.
 +
* <b>[https://keyhole.co/ Keyhole]</b>: This tool is a hashtag analytics and social media monitoring platform. You can use it to track keywords, hashtags, accounts, and URLs across Twitter, Instagram, Facebook, YouTube, and TikTok. You can also measure sentiment, engagement, reach, impressions, and share of voice.
 +
** Pros: Accurate, real-time, and comprehensive.
 +
** Cons: Paid tool, limited free trial, and may not support some niche networks.
 +
* <b>[https://izea.com/ IZEA]</b>: helps you track your social presence.
 +
** Pros: Its VizSearch feature, which matches visual content to keyword searches for influencer discovery and for your content library, and its private network of influencers, which means competitors won’t find them when you get them to sign onto the platform and work with you
 +
* <b>[https://www.quintly.com/ quintly]</b>: This tool is a social media analytics and benchmarking platform. You can use it to analyze your own and your competitors' performance on Facebook, Twitter, Instagram, LinkedIn, YouTube, Pinterest, and Snapchat. You can also create custom dashboards and reports with over 400 metrics.
 +
** Pros: Flexible, customizable, and scalable.
 +
** Cons: Paid tool, no free version available (only a demo), and may not cover all social platforms.

Latest revision as of 09:20, 18 August 2023

YouTube ... Quora ...Google search ...Google News ...Bing News


Open Source Intelligence (OSINT) is the collection, analysis, and dissemination of information that is publicly available and legally accessible. OSINT operations, whether practiced by IT security pros, malicious hackers, or state-sanctioned intelligence operatives, use advanced techniques to search through the vast haystack of visible data to find the needles they’re looking for to achieve their goals—and learn information that many don’t realize is public. OSINT operations use advanced techniques to search through visible data to find information that many don’t realize is public. OSINT differs from other types of intelligence gathering in several ways, OSINT:

  • is focused on publicly available and legally obtainable information, whereas other forms of intelligence gathering may involve confidential or classified sources.
  • uses various sources, including social media, news articles, public records, and government reports. In contrast, other forms of intelligence gathering may focus on a specific source type, such as human, signals, imagery, or geospatial.
  • is accessible and affordable to anyone with an internet connection and some basic skills, whereas other forms of intelligence gathering may require specialized equipment, training, or authorization.
  • is dynamic and diverse, as it reflects the changing and heterogeneous nature of the open source environment. However, this also means that OSINT may be inaccurate, incomplete, or unreliable, as it depends on the quality and credibility of the sources. Therefore, OSINT needs to be verified and validated with other forms of intelligence or evidence.

AI can be used to turn the challenge of large data volumes into an advantage and ultimately create a symbiotic relationship with OSINT. AI, machine learning and automation tools are able to scan massive amounts of data and conversations, proficiently identifying connections and risks that warrant further investigation. AI has dramatically increased the volume of data that can be analyzed in real-time. Today’s AI-enabled platforms empower intelligence analysts to leverage OSINT as the foundation that can help uncover hidden threats, corroborate classified reporting, and pinpoint the targets that warrant resource-intensive, traditional intelligence gathering.

OSINT is in many ways the mirror image of operational security (OPSEC), which is the security process by which organizations protect public data about themselves that could, if properly analyzed, reveal damaging truths. IT security departments are increasingly tasked with performing OSINT operations on their own organizations to shore up operational security. - 15 top open-source intelligence tools | J. Fruhlinger, A. Sharma & J. Breeden - Foundry


The most common OSINT resources are as follows;

  • Metadata search
  • Search Engines
  • Social Media
  • Online Communities
  • Email Addresses
  • Usernames
  • People Search Engines
  • Telephone Numbers
  • Online Maps
  • Code search
  • Documents
  • Images
  • Videos
  • Domain Names
  • IP Addresses
  • Government & Business Records
  • Virtual Currencies
  • Advanced Linux Tools
  • Geospatial Research
  • Data Breaches & Leaks


Open-Source Intelligence (OSINT) in 5 Hours - Full Course - Learn OSINT!
This course focuses on Open Source Intelligence (OSINT) tactics and techniques designed to help you improve your investigative game.

OSINT: You can't hide // Your privacy is dead // Best resources to get started
You cannot hide. Your privacy is over. Want to learn OSINT? Want to learn how easy it is to find information online? Time to learn Open Source Intelligence from the best. - OSINT Curious

OSINT Companies/Tools

Several companies/products are using AI for OSINT. Here are some of them:

  • Babel X - Babel Street ... a multilingual search tool for the public internet, including blogs, social media, message boards and news sites. It also searches the dark web, including Onion sites, and some deep web content that Babel X can access through agreements or licensing from the content owners. The product is able to geo-locate the source of information it finds, and it can perform text analysis to identify relevant results.
  • BuiltWith ... lets you find what popular websites are built with. Different tech stacks and platforms power different sites. BuiltWith can, for example, detect whether a website is using WordPress, Joomla, or Drupal as its CMS and provide further details.
  • Cellebrite ... provides digital intelligence solutions for law enforcement, military, intelligence, and enterprise customers. It enables users to extract, decode, and analyze data from various digital sources, including mobile devices, cloud services, computers, and IoT devices.
  • Cobwebs Technologies ... provides mission-critical AI-powered OSINT solutions for law enforcement, government, financial institutions, and corporations. It enables users to search and analyze the web's big data using machine learning algorithms, and generate real-time intelligence reports.
  • Dassault Systemes ... provides 3D design, engineering, simulation, and data intelligence solutions for various industries. It enables users to create and manage digital twins of products, processes, and systems, and leverage AI to optimize performance and innovation.
  • Digital Clues ... provides AI-powered OSINT solutions for law enforcement and intelligence agencies. It enables users to collect, analyze, and visualize data from various online sources, including social media, dark web, blogs, forums, and news sites.
  • ESPY ... provides AI-powered OSINT solutions for security and crisis management. It enables users to automate the process of collecting, analyzing, and synthesizing information from various open sources, such as web scraping, geolocation, Sentiment Analysis, and image recognition.
  • Expert.AI ... provides natural language understanding solutions for various industries and applications. It enables users to extract insights and knowledge from unstructured text data using AI and natural language processing techniques.
  • Grep.app ... search across half million git repos
  • Hensoldt ... provides sensor solutions for defense and security applications. It enables users to collect, process, and analyze data from various sources, such as radar, optical, electronic warfare, and cyber systems.
  • Intelligence X ... preserves not only historic versions of web pages but also entire leaked data sets that are otherwise removed from the web due to the objectionable nature of content or legal reasons.
  • Lampyre ... provides a platform for conducting OSINT investigations. It enables users to collect data from various sources using queries or scripts, analyze data using filters or graphs, and export data in various formats.
  • Maltego ... provides a visual tool for conducting OSINT investigations. It enables users to gather data from various sources using transforms or integrations, analyze data using link analysis or clustering techniques, and share data using reports or collaboration features.
  • Metagoofil ... is optimized to extract metadata from public documents. Metagoofil can investigate almost any kind of document that it can reach through public channels including .pfd, .doc, .ppt, .xls and many others. Searches return things like the usernames associated with discovered documents, as well as real names if available. It also maps the paths of how to get to those documents, which in turn would provide things like server names, shared resources and directory tree information about the host organization.
  • Mitaka ... Available as a Chrome extension and Firefox add-on, search over six dozen search engines for IP addresses, domains, URLs, hashes, ASNs, Bitcoin wallet addresses, and various indicators of compromise (IOCs) from web browser.
  • NexVision ... provides vision solutions and optronic systems for various applications, such as defense, security, cinema, and industry. It specializes in electronic reference design, especially in complex vision processing systems. It uses AI and computer vision to extract insights and metadata from video data, such as object detection, activity recognition, and scene understanding
  • Owlint ... centralizes the digital footprint from the Clear Web, Deep Web, and Dark Web. Its proprietary AI OwlAnalytics assesses a level of risk on each detected element.
  • Recon-ng ... provides a modular framework for conducting OSINT investigations. It enables users to gather data from various sources using built-in or custom modules, store data in a local database, and visualize data using interactive reports.
  • searchcode ... is a highly specialized search engine that looks for useful intelligence inside source code.
  • Shodan ... can monitor and search as part of an OSINT effort is impressive. It’s one of the few engines capable of examining operational technology (OT) such as the kind used in industrial control systems at places like power plants and manufacturing facilities. Any OSINT gathering effort in industries that deploy both information technology and OT would miss a huge chunk of that infrastructure without a tool like Shodan. In addition to IoT devices like cameras, building sensors and security devices, Shodan can also be turned to look at things like databases to see if any information is publicly accessible through paths other than the main interface. It can even work with videogames, discovering things like Minecraft or Counter-Strike: Global Offensive servers hiding on corporate networks where they should not be, and what vulnerabilities they generate.
  • Signifyd ... provides fraud protection and chargeback prevention solutions for e-commerce businesses. It enables users to leverage AI and machine learning to analyze customer behavior, transaction data, and network intelligence to detect and prevent fraud.
  • Spiderfoot ... a free OSINT reconnaissance tool that integrates with multiple data sources to gather and analyze IP addresses, CIDR ranges, domains and subdomains, ASNs, email addresses, phone numbers, names and usernames, BTC addresses, etc. Available on GitHub, Spiderfoot comes with both a command-line interface and an embedded web-server for providing an intuitive web-based GUI. The application itself comes with over 200 modules
  • Spyse ... Relied on by projects like OWASP, IntelligenceX, and Spiderfoot, Spyse collects publicly available data on websites, their owners, associated servers, and IoT devices. This data is then analyzed by the Spyse engine to spot any security risks in and connections between these different entities.
  • theHarvester ... uses include popular search engines like Bing and Google, as well as lesser known ones like dogpile, DNSdumpster and the Exalead meta data engine. It also uses Netcraft Data Mining and the AlienVault Open Threat Exchange. It can even tap the Shodan search engine to discover open ports on discovered hosts. In general, theHarvester tool gathers emails, names, subdomains, IPs and URLs.
  • Verint ... provides customer engagement and cyber intelligence solutions for various sectors and organizations. It enables users to use AI and analytics to optimize customer interactions, enhance workforce performance, and protect from cyber threats.
  • Voxel51 ... provides video analytics solutions for various industries and applications. It enables users to use AI and computer vision to extract insights and metadata from video data, such as object detection, activity recognition, and scene understanding.


Primer.ai

Using Machine Learning (ML), computer vision and Natural Language Processing (NLP) transformers focused on software for DoD/IC customers with real-time news and social media insights, situational reporting, intelligence gathering, and faction analysis. Lists its customers: U.S. Air Force, Army, and U.S. Special Operations Command, and Fortune 500 companies. - Military contractor Primer announces $69M to build AI for ‘those who support and defend our democracy’ | Carl Franzen - VentureBeat

Some of Primer.ai services include:

  • Threat Detection: Early detection of kinetic and cyber threats from social and news media to improve response time
  • Strategic Analysis: Structuring messy data for exploration to reduce time to insight
  • Semantic Search: Securely and fluently search, summarize, and explore data at scale with greater precision

Primer Command - By leveraging high-scale streaming data, you can detect kinetic and cyber threats up to several hours earlier than traditional methods and respond quickly. Primer Command is a real-time threat detection tool that analyzes tens of thousands of sources in 100 languages, including news and social media. With Primer Command, teams can monitor, respond, and maintain situational awareness of fast-breaking global events, such as kinetic and cyber threats. Primer Command can:

  • detect and summarize the "Who, What, When, Where, and Why" of every scenario, providing contextual intelligence and actionable insights.
  • generate situation reports that can be downloaded, edited, and shared, saving time and facilitating collaboration.
  • perform location-based searches and computer vision capabilities, such as identifying faces, objects, logos, and text in images.
  • filter and flag items of importance, and delegate responsibilities among team members.
  • integrate with Large Language Model (LLM).
  • operate in a secure and compliant manner, respecting data privacy and sovereignty.


Tools for Analyzing Social Media

Some of the most popular tools:

  • Hootsuite Analytics: This tool allows you to track performance data from every social network in one place, with easy-to-understand reports. You can measure metrics like reach, followers, engagement, clicks, and conversions. You can also compare results across multiple networks and create custom reports based on your KPIs.
    • Pros: Comprehensive, user-friendly, and customizable.
    • Cons: Paid tool, requires a Hootsuite account, and may not support some niche networks.
  • BuzzSumo: This tool helps you discover the most engaging content and influencers on social media. You can analyze topics, keywords, domains, and authors to see what resonates with your audience. You can also monitor mentions, trends, and competitors to stay ahead of the curve.
    • Pros: Powerful, insightful, and easy to use.
    • Cons: Paid tool, limited free version, and may not cover all social platforms.
  • Google Analytics: This tool is a cloud-based web analytics service that delivers different tools to optimize digital marketing. You can use it to track social media traffic, conversions, and campaigns. You can also integrate it with other Google products like Ads, Search Console, and Data Studio.
    • Pros: Free, reliable, and versatile.
    • Cons: Complex, requires technical skills, and may not provide all social media metrics.
  • Keyhole: This tool is a hashtag analytics and social media monitoring platform. You can use it to track keywords, hashtags, accounts, and URLs across Twitter, Instagram, Facebook, YouTube, and TikTok. You can also measure sentiment, engagement, reach, impressions, and share of voice.
    • Pros: Accurate, real-time, and comprehensive.
    • Cons: Paid tool, limited free trial, and may not support some niche networks.
  • IZEA: helps you track your social presence.
    • Pros: Its VizSearch feature, which matches visual content to keyword searches for influencer discovery and for your content library, and its private network of influencers, which means competitors won’t find them when you get them to sign onto the platform and work with you
  • quintly: This tool is a social media analytics and benchmarking platform. You can use it to analyze your own and your competitors' performance on Facebook, Twitter, Instagram, LinkedIn, YouTube, Pinterest, and Snapchat. You can also create custom dashboards and reports with over 400 metrics.
    • Pros: Flexible, customizable, and scalable.
    • Cons: Paid tool, no free version available (only a demo), and may not cover all social platforms.
Retrieved from "https://primo.ai/index.php?title=Open-Source_Intelligence_-_OSINT&oldid=33365"