Difference between revisions of "Finding Paul Revere"

Latest revision as of 13:26, 30 June 2023

Math for Intelligence ... Finding Paul Revere ... Social Network Analysis (SNA) ... Dot Product ... Kernel Trick
Using Metadata to Find Paul Revere | Kieran Healy
The data come from an appendix to David Hackett Fischer's Paul Revere's Ride (Oxford University Press, 1995)
The Other Ride of Paul Revere: The Brokerage Role In The Making of The American Revolution | Shin-Kap Han

London, 1772.

I have been asked by my superiors to give a brief demonstration of the surprising effectiveness of even the simplest techniques of the new-fangled Social Networke Analysis in the pursuit of those who would seek to undermine the liberty enjoyed by His Majesty’s subjects. This is in connection with the discussion of the role of “metadata” in certain recent events and the assurances of various respectable parties that the government was merely “sifting through this so-called metadata” and that the “information acquired does not include the content of any communications”. I will show how we can use this “metadata” to find key persons involved in terrorist groups operating within the Colonies at the present time. I shall also endeavour to show how these methods work in what might be called a relational manner...

To play...

1. Download from Kieran Healy files | Github for reference

2. Download from Mark Bernico's files | GitHub]; Boston - 1776.ipynb Jupyter notebook and PaulRevereAppD.csv data file

3. Upload Mark Bernico's files to your Google Drive

Notebook Setup

4. Obtain a Colaboratory account if you don't already have one

5. Setup Colab app in your Google Drive account by watching 1:45 minutes into the following video. Optionally starting at 2:35 setting up a GPU; menu Edit | Notebook settings

6. How to upload from a local file on your computer to Colab by watching the following video, and from this Colab's Help Notebook 'copy and paste' into new cells per the video

7. Note: As you step though the notebook, you will need to put parenthesis around two print statements; print(people_x_groups.shape) & print(groups_x_people.shape)

8. Save your notebook; File | Save <or> CNTRL+S Note: To open your recent files in Colab click here

9. Extra credit: Edit the notebook to graph organizations as shown in Healy's blog by switching the order of the matrix multiplication; groups_adj = groups_x_people.dot(people_x_groups) and replicating the code for groups_adj.

Functional Explanations

NumPy for Matrices are collections of elements into rows and columns. Matrix multiplication relies on Dot Product to multiply various combinations of rows and columns. A thing about multiplying matrices is that the order matters - multiply one way to see group relationships and multiply opposite way to view people relationships.
NetworkX for creation, manipulation, and study of the structure, dynamics, and functions of complex networks
Matplotlib for generating plots, histograms, power spectra, bar charts, errorcharts, scatterplots, etc
In 1974's The Duality of Persons and Groups, Ronald Breiger laid out a technique for obtaining a persons x persons adjacency matrix and a groups x groups adjacency matrix from a persons x groups affiliation matrix.
Adjacency Matrix - is a 2D array of size V x V where V is the number of vertices in a graph. The matrix is often used in network analysis to represent the adjacency of each person to each other person in a network. An adjacency matrix is a square person-by-person matrix

Eigenvector Centrality - a measure of the influence of a node in a network

Knowledge Graphs are very useful ways of presenting information about social networks.

Google Drive Option: How to upload from a local file on your computer to Colab by watching 6:30 minutes into the Notebook Setup video. From this Colab's Help Notebook 'copy and paste' per the video

Get Started: 3 Ways to Load CSV files into Colab | A Apte - Towards Data Science

@@ Line 2: / Line 2: @@
 |title=PRIMO.ai
 |titlemode=append
-|keywords=artificial, intelligence, machine, learning, models, algorithms, data, singularity, moonshot, Tensorflow, Google, Nvidia, Microsoft, Azure, Amazon, AWS
+|keywords=ChatGPT, artificial, intelligence, machine, learning, GPT-4, GPT-5, NLP, NLG, NLC, NLU, models, data, singularity, moonshot, Sentience, AGI, Emergence, Moonshot, Explainable, TensorFlow, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Hugging Face, OpenAI, Tensorflow, OpenAI, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Meta, LLM, metaverse, assistants, agents, digital twin, IoT, Transhumanism, Immersive Reality, Generative AI, Conversational AI, Perplexity, Bing, You, Bard, Ernie, prompt Engineering LangChain, Video/Image, Vision, End-to-End Speech, Synthesize Speech, Speech Recognition, Stanford, MIT |description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools
-|description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools
+<!-- Google tag (gtag.js) -->
+<script async src="https://www.googletagmanager.com/gtag/js?id=G-4GCWLBVJ7T"></script>
+<script>
+  window.dataLayer = window.dataLayer || [];
+  function gtag(){dataLayer.push(arguments);}
+  gtag('js', new Date());
+  gtag('config', 'G-4GCWLBVJ7T');
+</script>
 }}
-[http://www.youtube.com/results?search_query=Using+Metadata+to+Find+Paul+Revere YouTube search...]
+[https://www.youtube.com/results?search_query=Using+Metadata+to+Find+Paul+Revere YouTube search...]
-[http://www.google.com/search?q=Using+Metadata+to+Find+Paul+Revere ...Google search]
+[https://www.google.com/search?q=Using+Metadata+to+Find+Paul+Revere ...Google search]
-* [http://kieranhealy.org/blog/archives/2013/06/09/using-metadata-to-find-paul-revere/ Using Metadata to Find Paul Revere | Kieran Healy]
+* [[Math for Intelligence]] ... [[Finding Paul Revere]] ... [[Social Network Analysis (SNA)]] ... [[Dot Product]] ... [[Kernel Trick]]
+* [https://kieranhealy.org/blog/archives/2013/06/09/using-metadata-to-find-paul-revere/ Using Metadata to Find Paul Revere | Kieran Healy]
 * The data come from an appendix to [https://www.amazon.com/Paul-Reveres-David-Hackett-Fischer/dp/0195098315/ref=sr_1_1?keywords=David+Hackett+Fischer%27s+Paul+Revere%27s+Ride&qid=1563276971&s=gateway&sr=8-1 David Hackett Fischer's Paul Revere's Ride (Oxford University Press, 1995)]
-* [http://www.sscnet.ucla.edu/polisci/faculty/chwe/ps269/han.pdf  The Other Ride of Paul Revere: The Brokerage Role In The Making of The American Revolution | Shin-Kap Han]
+* [https://www.sscnet.ucla.edu/polisci/faculty/chwe/ps269/han.pdf  The Other Ride of Paul Revere: The Brokerage Role In The Making of The American Revolution | Shin-Kap Han]
-* [[Math for Intelligence]]
-* [[Social Network Analysis (SNA)]]
@@ Line 20: / Line 28: @@
 I have been asked by my superiors to give a brief demonstration of the surprising effectiveness of even the simplest techniques of the new-fangled Social Networke Analysis in the pursuit of those who would seek to undermine the liberty enjoyed by His Majesty’s subjects. This is in connection with the discussion of the role of “metadata” in certain recent events and the assurances of various respectable parties that the government was merely “sifting through this so-called metadata” and that the “information acquired does not include the content of any communications”. I will show how we can use this “metadata” to find key persons involved in terrorist groups operating within the Colonies at the present time. I shall also endeavour to show how these methods work in what might be called a relational manner...
-https://kieranhealy.org/files/misc/revere-closeup.png
 <youtube>qOVz_dVbTrE</youtube>
@@ Line 47: / Line 54: @@
 . Note: As you step though the notebook, you will need to put parenthesis around two print statements; <i>print(people_x_groups.shape)</i> & <i>print(groups_x_people.shape)</i>
-. Save your notebook; File | Save   <or> <i>CNTRL+S</i>    Note: [http://colab.research.google.com/notebooks/welcome.ipynb#recent=true To open your recent files in Colab click here]
+. Save your notebook; File | Save   <or> <i>CNTRL+S</i>    Note: [https://colab.research.google.com/notebooks/welcome.ipynb#recent=true To open your recent files in Colab click here]
 . Extra credit: Edit the notebook to graph organizations as shown in Healy's blog by switching the order of the matrix multiplication; <i>groups_adj =  groups_x_people.dot(people_x_groups)</i> and replicating the code for <i>groups_adj</i>.
@@ Line 53: / Line 60: @@
 === Functional Explanations ===
-* [[Python#NumPy |NumPy]] for [http://en.wikipedia.org/wiki/Matrix_(mathematics) Matrices] are collections of elements into rows and columns. [http://en.wikipedia.org/wiki/Matrix_multiplication Matrix multiplication] relies on [[Dot Product]] to multiply various combinations of rows and columns. A thing about multiplying matrices is that the order matters - multiply one way to see group relationships and multiply opposite way to view people relationships.
+* [[Python#NumPy |NumPy]] for [https://en.wikipedia.org/wiki/Matrix_(mathematics) Matrices] are collections of elements into rows and columns. [https://en.wikipedia.org/wiki/Matrix_multiplication Matrix multiplication] relies on [[Dot Product]] to multiply various combinations of rows and columns. A thing about multiplying matrices is that the order matters - multiply one way to see group relationships and multiply opposite way to view people relationships.
 * [[Python#NetworkX |NetworkX]] for creation, manipulation, and study of the structure, dynamics, and functions of complex networks
 * [[Python#Matplotlib |Matplotlib]] for generating plots, histograms, power spectra, bar charts, errorcharts, scatterplots, etc
-* In 1974's [http://www.umasocialmedia.com/socialnetworks/glossary/duality-of-persons-and-groups/ The Duality of Persons and Groups], Ronald Breiger laid out a technique for obtaining a persons x persons adjacency matrix and a groups x groups adjacency matrix from a persons x groups affiliation matrix.
+* In 1974's [https://www.umasocialmedia.com/socialnetworks/glossary/duality-of-persons-and-groups/ The Duality of Persons and Groups], Ronald Breiger laid out a technique for obtaining a persons x persons adjacency matrix and a groups x groups adjacency matrix from a persons x groups affiliation matrix.
-* [http://en.wikipedia.org/wiki/Adjacency_matrix Adjacency Matrix] - is a 2D array of size V x V where V is the number of vertices in a graph. The matrix is often used in network analysis to represent the adjacency of each person to each other person in a network. An adjacency matrix is a square person-by-person matrix
+* [https://en.wikipedia.org/wiki/Adjacency_matrix Adjacency Matrix] - is a 2D array of size V x V where V is the number of vertices in a graph. The matrix is often used in network analysis to represent the adjacency of each person to each other person in a network. An adjacency matrix is a square person-by-person matrix
 <youtube>HtTd548OCl0</youtube>
-* [http://en.wikipedia.org/wiki/Eigenvector_centrality Eigenvector Centrality] -  a measure of the influence of a node in a network
+* [https://en.wikipedia.org/wiki/Eigenvector_centrality Eigenvector Centrality] -  a measure of the influence of a node in a network
 <youtube>DGVvm-j-NG4</youtube>
@@ Line 67: / Line 74: @@
 * [[Knowledge Graphs]] are very useful ways of presenting information about social networks.
@@ Line 81: / Line 82: @@
 [https://towardsdatascience.com/3-ways-to-load-csv-files-into-colab-7c14fcbdcb92 Get Started: 3 Ways to Load CSV files into Colab | A Apte - Towards Data Science]
-= Network Pattern =
-[http://www.youtube.com/results?search_query=Knowledge+Graph+AI Youtube search...]
-[http://www.google.com/search?q=Knowledge+Graph+AI ...Google search]
-* [[Graph]]
-* [[Computer Networks]]
-* [http://barabasi.com/book/linked Linked: How Everything Is Connected to Everything Else and What It Means for Business, Science, and Everyday Life | Albert-László Barabási]
-* [http://www.uvm.edu/pdodds/files/papers/others/1969/travers1969.pdf An Experimental Study of the Small World Problem | Jeffrey Travers  and Stanley Milgram]
-* [http://www.penguinrandomhouse.com/books/549846/the-square-and-the-tower-by-niall-ferguson/ The Square and the Tower | Niall Ferguson]
-* [http://oracleofbacon.org/ The Oracle of Bacon] computes the Bacon number of any actor or actress from Wikipedia data. A previous implementation used IMDB data.   ...[http://en.wikipedia.org/wiki/Six_Degrees_of_Kevin_Bacon Six Degrees of Kevin Bacon | Wikipedia]
-Network science is an academic field which studies complex networks such as telecommunication networks, computer networks, biological networks, cognitive and semantic networks, and social networks, considering distinct elements or actors represented by nodes (or vertices) and the connections between the elements or actors as links (or edges). The field draws on theories and methods including graph theory from mathematics, statistical mechanics from physics, data mining and information visualization from computer science, inferential modeling from statistics, and social structure from sociology. The United States National Research Council defines network science as "the study of network representations of physical, biological, and social phenomena leading to predictive models of these phenomena." [http://en.wikipedia.org/wiki/Network_science#:~:text=Network%20science%20is%20an%20academic,connections%20between%20the%20elements%20or Network science | Wikipedia]
-{|<!-- T -->
-| valign="top" |
-{| class="wikitable" style="width: 550px;"
-||
-<youtube>rduH9mKvnjM</youtube>
-<b>Authors in Conversation: Niall Ferguson and Albert-Laszlo Barabasi
-</b><br>Harvard Club of Boston's Author Series hosted a special evening conversation with renowned historian Niall Ferguson and the nation's foremost network science expert Albert-Laszlo Barabasi. In his most recent book, The Square and the Tower, Ferguson applies lessons from Barabasi's pioneering work in network science to the domain of historical analysis, drawing insights from a wide range of fascinating examples across past decades and centuries, with important implications for current affairs. As Barabasi has demonstrated both in his academic work and in his popular writing (Linked), network science research has led to meaningful discoveries in areas ranging from biology and medicine, to institutional analysis and social networks.
-|}
-|<!-- M -->
-| valign="top" |
-{| class="wikitable" style="width: 550px;"
-||
-<youtube>4cADSlk5CHU</youtube>
-<b>Niall Ferguson on History’s Hidden Networks
-</b><br>Have historians misunderstood everything? Have they missed the single greatest idea that best explains the past?  Niall Ferguson is the preeminent historian of the ideas that define our time. He has challenged how we think about money, power, civilisation and empires. Now he wants to reimagine history itself.  In October 2017, Ferguson came to the Intelligence Squared stage to unveil his new book, 'The Square and The Tower'. Historians have always focused on hierarchies, he argues – on the elites that wield power. Economists have concentrated on the marketplace – on the economic forces that shape change. These twin structures are symbolised for Ferguson by Siena’s market square, and its civic tower looming above. But beneath both square and tower runs something more deeply significant: the hidden networks of relationships, ideas and influence.  Networks are the key to history. The greatest innovators have been ‘superhubs’ of connections. The most powerful states, empires and companies have been those with the most densely networked structures. And the most transformative ideas – from the printing presses that launched the Reformation to the Freemasonry that inspired the American Revolution – have gone viral precisely because of the networks within which they spread.  ‘When we understand these core insights of network science,’ says Ferg
-|}
-|}<!-- B -->
-{|<!-- T -->
-| valign="top" |
-{| class="wikitable" style="width: 550px;"
-||
-<youtube>7YFNf1ix_yY</youtube>
-<b>BURSTS: The Hidden Pattern Behind Everything We Do | Albert László Barabási | Talks at Google
-</b><br>The Authors@Google program welcomed Albert László Barabási to Google's New York office to discuss his book, "BURSTS: The Hidden Pattern Behind Everything We Do" "In BURSTS (April 2010), Barabasi, Director of the Center for Network Science at Northeastern University, shatters one of the most fundamental assumptions in modern science and technology regarding human behavior.  Barabasi argues that, rather than being random, humans actually act in predictable patterns.  We go along for long periods of quiet routine followed suddenly by loud bursts of activity. Barabasi demonstrates that these breaks in routine, or "bursts," are present in all aspects of our existence— in the way we write emails, spend our money, manage our health, form ideas.  Barabasi has even found "burstiness" in our webpage clicking activity and the online news cycle."
-|}
-|<!-- M -->
-| valign="top" |
-{| class="wikitable" style="width: 550px;"
-||
-<youtube>Lq5hlsJAOfc</youtube>
-<b>The Pattern in Nature's Networks
-</b><br>cience shows it's a small world after all—and nature's networks follow a similar pattern.  NOVA Facebook: https://www.facebook.com/NOVAonline   NOVA Twitter: https://www.twitter.com/novapbs  Follow Mark Zastrow on Twitter: @MarkZastrow   PRODUCTION CREDITS   Writer, Producer, and Narrator  Mark Zastrow   Music by Mark Zastrow
-Editorial Help from Anna Rothschild  Original Footage © WGBH Educational Foundation 2014   MEDIA CREDITS  Mississippi River watershed  National Park Service  Diffusion tensor images Human Connectome Project, NIH, Massachusetts General Hospital, Meredith Reid (University of Alabama--Birmingham)  Neurons, In Vitro Color!   Flickr /thelunch_box (CC BY-NC 2.0)  Small world neural network  based on figure from van den Heuvel and Sporns (2011) / The Journal of Neuroscience 31(44):15775--15786   Autism spectrum disorder networks  Barttfeld et al. (2011) / Neuropsychologia 49 (2011) 254--263  The Formation of Stars and Brown Dwarfs and the Truncation of Protoplanetary Discs in a Star Cluster  Matthew R. Bate, Ian A. Bonnell, and Volker Bromm, UK Astrophysical Fluids Facility   Floral Art  Flickr / Louise Docker (CC BY 2.0)  Dark matter filaments  Ralf Kaehler, Oliver Hahn and Tom Abel, Kavli Institute for Particle Astrophysics and Cosmology (Stanford)  Millennium Simulation flythroughs
-Springel et al. (2005)  © WGBH Educational Foundation 2014
-|}
-|}<!-- B -->
-{|<!-- T -->
-| valign="top" |
-{| class="wikitable" style="width: 550px;"
-||
-<youtube>TcxZSmzPw8k</youtube>
-<b>The Science of Six Degrees of Separation
-</b><br>Are all people on Earth really connected through just six steps?
-There's much more science in this than I initially expected. It turns out ordered networks with a small degree of randomness become small-work networks. This is why your acquaintances turn out to be more important in job searches and finding new opportunities than close friends.  Animations in this video by The Lyosacks: http://www.youtube.com/user/TheLyosacks   There are some great books on this topic:  Duncan Watts, Six Degrees: The Science of a Connected Age  Albert-Laszlo Barabasi, Linkds: How Everything is Connected to Everything Else  And here are articles I referred to:  [http://www.uvm.edu/pdodds/files/papers/others/1969/travers1969.pdf Milgram's small world experiment]  and [http://sociology.stanford.edu/sites/g/files/sbiybj9501/f/publications/the_strength_of_weak_ties_and_exch_w-gans.pdf Strength of Weak Ties |  Mark S. Granovetter]
-|}
-|<!-- M -->
-| valign="top" |
-{| class="wikitable" style="width: 550px;"
-||
-<youtube>ID2</youtube>
-<b>HH2
-</b><br>BB2
-|}
-|}<!-- B -->

Difference between revisions of "Finding Paul Revere"

Latest revision as of 13:26, 30 June 2023

To play...

Notebook Setup

Functional Explanations

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

Tools