Parsing scripts

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 

<pre>
...
    <b> INT. MOS EISLEY SPACEPORT - DOCKING BAY 94 </b>

Chewbacca leads the group into a giant dirt pit that is 
Docking Bay 94. Resting in the middle of the huge hole is a 
large, round, beat-up, pieced-together hunk of junk that 
could only loosely be called a starship.

    <b> LUKE </b>
What a piece of junk.

The tall figure of Han Solo comes down the boarding ramp.
...
</pre>

Parsing with active patterns

1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 

let (|SceneTitle|Name|Word|) (text:string) =
    let scenePattern = "[ 0-9]*(INT.|EXT.)[ A-Z0-9]"
    let namePattern = "^[/A-Z0-9]+[-]*[/A-Z0-9 ]*[-]*[/A-Z0-9 ]+$"
    if Regex.Match(text, scenePattern).Success then
        SceneTitle text
    elif Regex.Match(text, namePattern).Success then
        Name text
    else Word

Parsing scripts with pattern matching

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 

let rec parseScenes sceneAcc characterAcc (items: string list) =
   match items with
   | item::rest ->
       match item with
       | SceneTitle title -> 
            // add the finished scene to the scene accumulator
            let fullScene = List.rev characterAcc
            parseScenes (fullScene::sceneAcc) [] rest
       | Name name -> 
            // add character's name to the character accumulator
            parseScenes sceneAcc (name::characterAcc) rest
       | Word -> // do nothing
            parseScenes sceneAcc characterAcc rest
   | [] -> List.rev sceneAcc     

(| Active patterns |)

hide complexity behind readable code

Jupyter notebooks on Azure with F#

notebooks.azure.com/evelina

Number of common mentions

⬇

Number of interactions

Theorem:

There's an API for everything.

Proof:

swapi.co

fsharp-swapi

github.com/evelinag/fsharp-swapi

swapi.co

scenes

+

characters

visualization

the phantom menace

a new hope

the force awakens

Network structure

How do the the movies differ?

Size

Density

Density

Density

\[\begin{align} \text{Density} &= \frac{\text{Existing connections}}{\text{Potential connections}} \\ & \\ &= \frac{\text{Existing connections}}{\frac{1}{2}N(N-1)} \end{align}\]

Density

Clustering coefficient

Clustering coefficient

Clustering coefficient

Clustering coefficient

Clustering coefficient

Clustering coefficient

Clustering coefficient

\[K_v = \text{Number of neighbours of $v$} \\ E_v = \text{Number of links between neighbours of $v$} \\ \\ \text{Clustering}(v) = \frac{E_v}{\frac{1}{2} K_v (K_v - 1)}\]

Clustering coefficient

\[K_v = \text{Number of neighbours of $v$} \\ E_v = \text{Number of links between neighbours of $v$} \\ \\ \text{Clustering}(\text{network}) = \frac{1}{N} \sum_v \frac{E_v}{\frac{1}{2} K_v (K_v - 1)}\]

Clustering Coefficient

Degree

Degree

Degree

Degree

\[\text{Degree}(v) = \text{Number of links }v \leftrightarrow v' \\ v \neq v'\]

Betweenness

Betweenness

Betweenness

Betweenness

Betweenness

Betweenness

\[S_v = \text{Number of shortest paths between $a$ and $b$ through $v$} \\ S = \text{Number of shortest paths between $a$ and $b$} \\ \\ \text{Betweenness}(v)_{ab} = \frac{S_v}{S}\]

Betweenness

\[S_v = \text{Number of shortest paths between $a$ and $b$ through $v$} \\ S = \text{Number of shortest paths between $a$ and $b$} \\ \\ \text{Betweenness}(v) = \sum_{ab} \frac{S_v}{S}\]

R

1: 
2: 
3: 

library(igraph)

b <- betweenness(network)

F#

1: 
2: 
3: 

open RProvider.igraph

let b = R.betweenness(network)

Centrality

Star Wars network in Neo4j

Network analysis in the real world

Parsing scripts in a functional way

Polyglot data science

Type providers

Network science

Having fun with interesting data

Star Wars resources

• Internet Movie Script Database

• swapi : Star Wars API

• The Star Wars social network

• Star Wars social networks: The Force Awakens

• Star Wars Neo4j demo

• Azure notebooks with parsing scripts