Merzbau: Castle of Exotic Mathery

Managed to get a basic interactive network setup with some hover and onclick phenomena (play with it!). I intend to expand the effects to include alteration of size (links and nodes), substitution of an image for the circle representation of a node, and the appearance of text nearby. Because nodes and links are obviously sets of very similar objects, I will create them as groups in d3, but then allow selections of individual items. These selections should trigger phenomena that will be specifically associated with that particular node. This is just a quick entry; that’s all I have to say right now.

Update:
Node hover: Node expands (will eventually be pic of offender), background changes to another color via gradient. TO-DO: Tooltip with name of node (perpetrator)? Do I want that blocking part of the picture?
Node click: Node outlinks, node changes color permanently.
Edge hover: Edge changes color and thickness. Tooltip (dummy data, will be name of document linked to).
Edge click: Edge outlinks (dummy link, will be cited document that generated the link).

  See the Pen 
  Document-Outlinked Citation Network Proof of Concept by Eli Zarrindast (@zarrindast)
  on CodePen.

Stuff I’m still figuring out:
Upon node hover, bring up (or link out to) a little dataframe profile of the offender, maybe have it slide onscreen at the bottom. Ideally any data that overlaps between offenders (e.g. weapon type) could eventually be clickable to bring up a list of persons relevant to that data value for that column– so, maybe have a pandas dataframe full of the all of the data, then trigger filtering methods for certain values by clicking on those values and show the subset frame on the side of the whole thing. No idea how I would connect Python and JS/CSS/HTML (it’s probably well-known, but not by me).

Upon node hover, highlight nodes of degree 1.

Slider bar that visually grows the network over time, as you slide through each year when an event occurred. I tried to do this in Gephi and while if I recall correctly a little video animation was possible, it wasn’t possible to let the user physically move the slider around themself. I saw a demo of some new Tableau add-ons that seemed to serve this exact purpose so I’m going to take a look at that– not too familiar with Tableau and will have to see how well it can be integrated.

Continue reading “Community Clusters (supplement to “Influential Nodes in…”)” →

I’m going to skip ahead a bit in this series to show you guys where I’m at so far in my visualization of the graph of rampage killers (upgraded from “rampage shooters” because of my inclination to include Alek Minassian, whose recently released police interview revealed his idolizations of Elliot Rodger and Chris Harper-Mercer– he even claims he was in communication with them, but this is unconfirmed. I have not looked into it, but the podcast Incel claims that the information [e.g. dates] doesn’t line up).

As you can see, this isn’t exactly glorious yet. On top of filling in missing data (I’m conflicted on what to do with Adam Lanza’s references, since he kept a spreadsheet of virtually every mass killer before him period), I want to lay the graph out chronologically– left-to-right by date of event– as most killers do not make reference to anyone who came after them (it does happen sometimes, as still-living killers may make statements in speech or writing during court proceedings, on websites, etc).

In the last post we had gotten through obtaining a list of files (web addresses and descriptions) to download & extract data from. Let’s take a look at one of these.

Here we have an image of Pekka-Eric Auvinen’s former YouTube profile. Recall that we are seeking mentions of other shooters and their crimes.

While there is no overt, formal naming of another shooter, someone familiar with school shootings will recognize “Natural Selection” as a catchphrase of Eric Harris, the Columbine school shooter. However the actual phrase “Natural Selection” does not appear– instead, we see that it would be preferable in some cases to use a more general keyword such as “natural + [possible space] + select” (since it could be selectION, selectOR, etc). This generalization will be ideal in several instances. We may also notice the term “godlike” which is another Harris/Columbine meme. Fortunately for us, there are a limited number of shooters with such a substantial “fan following” among other shooters that their speech patterns are reproduced in endless combinations and permutations as in-jokes or references; most of these come from Columbine. In short, we should take note to include non-name references in the name search list as though they were nicknames for the shooters.

We may also notice some socio-political buzzwords that may be of interest later, when we are looking into ideological similarities between shooters other than direct influence. I see “sturm”/”storm” (can be a white nationalist metaphor), “atheist”, “humanist”, etc., as well as loaded descriptions of the shooting action, such as “eliminate” and “my cause” (shooters often reference their “justice day”, “wrath”, “final action”, or whatever). We should start a list of interesting terms to consider later.

We may also note that this is a pdf image and as such we cannot yet ask our computer to search it as text. We will need to obtain searchable text from our PDFs. I didn’t know about this before this project, but apparently there are a few basic types of PDFs:

1) “Digitally created PDFs” / “True PDFS”: PDFs created with software that gives them internal meta-information designating the locations of text, images, and so on: in a sense, the computer can “see” what the naked eye can see due to this internal structure, and there is preexisting, GUI-having, user-friendly software to navigate these documents with ease manually one at a time (or of course you could automate their navigation yourself in such a manner as you desire; I’m just saying that they’re broken down to the point that they’re already cracked open and layed out like a multicourse meal for a layperson to consume– if they have the appropriate tools onhand).

2) PDFs that are just images, such as a raw scan of an original document of any kind, or of a picture, or of handwriting, etc. Just any old thing: a picture of it. There is no internal structure to help your computer navigate this, it’s “dead” in its current state. This is the worst case (aka the fun case for us; you’ll see!).

3) PDFs that have been made searchable through application of (e.g.) OCR to an image PDF, yielding text that is then incorporated into the original document as an underlayer. This text can be selected, copied, etc. This is a good situation. In our case, when we OCR image files, we’ll just go ahead and save the text in a file by itself (one per source image PDF) rather than creating searchable PDFs– because that’s all we need!

This is a case of #2– just a plain ol’ screenshot that someone took of this YouTube profile.

Now, this is a relatively small document in terms of the text contained within it– if I had a transcription of this text, it wouldn’t kill me to just read through it and see for myself if anything notable is contained therein. However, besides the sheer number of documents, a lot of the documents we’re going to be dealing with are these really interminable, deadly-dry court documents or FBI compendium files that are just hundreds and in some cases (cough, Sandy Hook shooting FBI report) thousands of pages long– fine reading for a rainy day or when I’m sick in bed or something, but not something I want to suffer through on my first attempt to get a high-level glance at who’s talking about whom.

(Seriously, some of the content of these things– there’s stuff like the cops doing the interrogations squabbling about when they’re going on their lunch break and “Wilson, stop fiddling with the tape while I’m interviewing!” and people going back and forth about whether they just properly translated the question to the witness or whether they just asked her about a pig’s shoes by mistake, etc.– and Valery Fabrikant representing himself in trial– merciful God! I’m going to have to do a whole separate post on the comic relief I encountered while going through these, both in terms of actually funny content and in terms of stuff that my computer parsed in a comically bogus way, such as when someone’s love of Count *Cho*cula gave me a false positive for a reference to Seung-Hui *Cho*.) Point being, I’m not gonna do this, I’m gonna make my computer do it. So that’s gonna be half the battle, namely, the second half.

First half of the battle is going to be getting the text out of the PDF. Enter optical character recognition (OCR). OCR is, in short, your computer reading. So let’s back it up– when you’re looking at text in some program on your computer, you’re looking at what’s really a manifestation of an underlying numerical representation of that character (meaning your computer knows two different “A”s are both capital As in the sense that they both “mean” [such-and-such number]). It’s not trying to figure it out from the arrangement of pixels of the character in that font every single time. (Honestly I don’t feel I have enough experience in this area to judge an appropriate summary of the appropriate “main” topics, so I’m just going to link out to someone else and you can read more if you like.)

But when the computer is looking at a picture of someone’s handwriting, or a picture of printed-out text from another computer, it’s only seeing the geometric arrangement of the pixels; it doesn’t yet know where the letters stop and start, or which number to associate a written character with once it is isolated. So what would you do if someone in a part of the world that used a totally unfamiliar alphabet slid you their number scrawled on a napkin (which they’d set their drink on, leaving a big wet inky ring)? First you’d try to mentally eliminate everything that’s not even part of the number– any food, dirt, wet spots, etc. Then you’d try to separate out groupings of marks that constitute separate numbers (like how the word “in” has the line part and the dot part that together make up the letter i– that’s one group– and then the little hump part that constitutes the n– that’s the second group). Then you’d zoom in on the grouping that you had decided made up one numerical digit, and you’d look back and forth between that and a list of all the one-digit numbers in that language in some nice formal print from a book that you are satisfied is standard and “official enough”. So you’d start with the first number in the list, and compare it to your number. Now you could go about this comparison a couple of ways.

1) Draw a little pound sign / graph over your number and over the printed number (imagine a # over the number 5). Compare the bottom left box of your number to the bottom left box of the printed number. Plausibly the same or not at all? Then compare the bottom right box. Etc. Apply some standard of how many boxes have to be similar to decide it’s a match, and when you find a match in the list of printed numbers, stop (or, do the comparison algorithm for each entry in the printed list, and whichever printed number has the most similar boxes to your written number is picked as the answer).

There are some problems with this though, namely that things might be tilted, different sizes, written with little flourishes on the ends of the glyphs, etc. in such a way that on a micro level, the similarities are disguised: think of the “a” character as typed in this blog vs. the written “a” character as is taught in US primary school (basically looks like an “o” with the line part of an “i” stuck on the right-hand side). The written, elementary-school-“a” would likely be determined to be an “o” under the # system. Not good. This is called matrix matching.

2) Attempt to identify the major parts, or “features”, of the character. (For example, we will consider the line and the dot in an “i” to be separate features because they’re separated in space, or the lines in an “x” as individual features as they can be drawn in one stroke, or whatever.) For the “a” we have a round part, and on its left a vertical line of some sort. Okay, now we’re talking generally enough that the two “a”s described above sound pretty similar. This is called feature recognition. (As you can imagine, it gets pretty complicated to get a computer to decide how to look for features and what’s a feature.)

So, that’s the game. There are several “engines” / programs / packages / etc for performing this task, and I used Tesseract. It’s pretty great at reading all kinds of *typed* characters, but you have to train any engine to read handwriting (one handwriting at a time, by slowly feeding it samples of that writing so it can learn to recognize it). I had so many different people’s handwritings, and so few handwritten documents PER handwriting, that this didn’t seem like the project to get into that on. I’m definitely going to get back to that for the purposes of transcribing my own handwriting, as I write poetry and prose poetry longhand and have a huge backlog of notebooks to type up (securing all that data is one of my main outstanding life tasks, in fact– there’s really no excuse at this point to endanger all of my writing by leaving it in unbacked-up physical copies).

This post is getting a little long so I’m going to go ahead and put it up and get into the technical stuff in the next post. Peace!

As discussed in my previous post, I’ve been mass-downloading and automatedly searching rampage shooters’ manifestos, court documents, etc. for mentions of the other shooters (by name, event, associated symbols, and so on). For this I used a Python library called Beautiful Soup, and I’d like to say a few words about how the process goes.

What is Beautiful Soup?

• 

We can pull out parts of the (prettified) soup by specifying tags, for example:

soup.prettify
all_links = soup.find_all("a")

• 

Pulling out all the “href” elements within these “<a>” tags…

templist = list()
for link in all_links:
templist.append(link.get("href"))

One can work with nested tags as well by iterating the same procedure.

• 

Unfortunately this is still not my desired list of links, but a simple script of the type below can filter for the appropriate strings and create a file listing them.

f = open('linkslist.txt','w+')
for item in templist:
if "http" in item:
f.write(item + "\n")

yields a document such as below. Included are a couple other samples of basic ways one could have fished-out data written to documents. Please note that more documents were added to the database after I began this project, so e.g. the William Atchison files are not included here even though they appear in the “soup” pictures. I’ll synchronize the images later.

So now we’ve managed to extract some data that would have taken much longer to do by hand! Next on the agenda will be mass-downloading my desired files (and avoiding undesired ones) and crawling them for cross-referencing– while avoiding booby traps! See you in the next post.

I’ve been working on my latest project for a long time now, but, having underestimated how complex it would get, was waiting until I had “a draft finished” to post something– ha! Yeah, right! There may never be a true “end” to this project, so waiting for a “complete” draft might mean waiting forever. I’m going to start posting periodic updates as I pass through stages of the project instead. So, without further ago, introducing:

Citation Networks in Rampage Shootings.

Background: In my experience, media coverage and anecdotal discussion of mass shooting events typical portray them as essentially unconnected natural disasters. But as it turns out, “climatology” is a very complex science…

In the last few years the US public has started to realize that there is an element of media influence involved in the motivation for perpetration, and academics have begun to analyze the characteristics of perpetrators (examples: race, age, mental health diagnoses on record, year shooting committed) and even the frequency of some contents in their writings; for example this Peter Langman paper (link does not directly open the large document). We are able to do this because of the great deal of shooter/event information available to the public via the Freedom of Information Act (FOIA)

• 

• 

However there is not a concentration on rampage shootings as a covert political movement of sorts, and/or a type of abstract terrorist network operating in single-cell units.  I posit that this is a useful conception, as expressed by a network (acyclic directed graph) of “citations” between rampage shooters. The existing methodology of citation analysis can provide a framework or guide for this expression. For now, I am concentrating on creating the most thorough network I reasonably can while maintaining relevance to my interests and an appropriately tight scope (I’ll elaborate).

My vague “starting goal”: Obtain and organize documents (primarily manifestos) associated with rampage shooters, and datamine them for cross-references. Create a visual network.

My concretized, “actionable” goal: Scrape the web for documents written directly, or otherwise generated directly (e.g. FBI report list of websites visited by shooter as found on his computer), by a restricted population of mass shooters (starting with those on Langman’s schoolshooters.info website). [✔️] Process them all into searchable text (as some are PDF, handwritten, etc). [✔️] Create a list of shooter-associated names and terms (names, nicknames, schools attacked, and unambiguous referents) [✔️], and directly search the documents for these names/terms, creating a list of citations apparent between shooters (which will need to be cleaned for redundancy and ongoingly manually checked to see if the results are making sense). Go through and throw out false positives, attempt to identify false negatives from personal knowledge of the documents (which I viewed individually when classifying during OCR pre-processing). Create a visualized network of citations [❌] where the graph is an acyclic directed graph, the nodes are shooters, and the edges are citations. Ideally, the nodes will be physically laid out along a timeline. [❌]

Where I’m at now:
✔️ Outline my goals.
🆕  Blog about same.

✔️ Define the population to be included (restricted set of shooters). Start with shooters represented in Langman’s original documents database, and work outward from there.
✔️ Scrape the web for documents using BeautifulSoup.
🔜 Blog about the web scraping.

✔️ Learn what OCR options are out there. Use OCR and manual/voice transcription to convert all document types to searchable .txt files as follows. First classify all documents according to their status: useless document, desirable and already usable txt, desirable but needs OCR, desirable but handwritten / otherwise illegible to conventional prefab OCR. Use tesseract and other packages until either every typed document is successfully hit or I conclude that that is not going to happen anytime soon. For everything else, decide whether it’s worth training AI to decode the writing/printing style or whether it’s better to just grind it out by transcribing manually or reading aloud into a voice-to-text processor.
🔜 Blog about the document processing.

✔️ Create list of terms and names associated with each shooter.
✔️ For each shooter, use Python to mine associated list of outgoing citations (of other shooters in this set) from documents.
✔️ Review identified citations, scrutinize sensitivity and specificity.
❌ Repeat until satisfied.
🔜 Blog about the comedic mishaps encountered along the way (no spoilers, but let’s just say I had some very confusing results for awhile).

❌ Decide which language(s) to create visuals in.
❌ Create formal nodelist (easy modification of shooter list) and edgelist (slightly harder but still pretty easy translation of citations list/log once created).
❌ Plot and prettify visual graph.
❌ Partially order nodes in time, place physically along a timeline.

💭 Node expansion goals: flesh out the network with all “iconic figures” mentioned, starting with other mass shooters (not from Langman), then enlarging to other mass killers, then killers in general (e.g. Hitler) as well as media (e.g. The Basketball Diaries). Needless to say many of these would not need to be searched for most outward citations (e.g. The Basketball Diaries is likely not citing any modern shooters).
💭 Edge expansion goals: grow the network into a set of hypergraphs with edge criteria such as: same type of weapon used (may need consultation as to what is similar enough to constitute probable mimesis), similar poses in images released, same type of manifesto (video, etc) released to the news intentionally, and so on. Each criterion could be represented by a different color, for example, if we wish to be able to toggle through criteria– perhaps create a mini interactive network where the user selects the criterion.
💭 Incorporate visual citations: Consider pictures released to the public by shooters that contain visual references to other shooters. Most of these are to Columbine. We have the “wrath” & “natural selection”-style “uniforms”, the Dylan Klebold gun-finger wave, and so on.

Coming soon… Perils and adventures encountered putting my first toe in the web scraping waters!