Scalable mechine learning: Lecture one: Introduction to Spark and HPC

Structured: Relational database, Formatted messages
semi-structured: Documents XML, Tagged Text/media, .csv document, Json
Unstructured: Plain text, Media

Data base: relational data model
Schema: the organizastion of data as a blueprint of how the database is constructed(The programmer must statically specify the schema)
SQL: Structure Query Language

Self describing rather than formal constructured

tags/markers to separate semantic elements

Schema: the column types

impose structure from unstructured data:

Extract, Transform, Load

Fast and general cluster computing system:

interoperable with hadoop

improve efficiency:
a. In memory computing primitives
b. general computation graphs

improve usability:
a. Rich apis in scala, python, Java
b. Interactive shall

Write programs in terms of transformations on distributed datasets

Resilient Distributed Datasets(RDDs):
a. collections of objects that can be store on memory or disk across a cluster
b. parallel functional transformations(map, filter…)
c. automatically rebuilt on failure