Ron Fredericks writes: Dr. Saptarshi Guha created an open-source interface between R and Hadoop called the R and Hadoop Integrated Processing Environment or RHIPE for short. LectureMaker was on the scene filming Saptarshi’s RHIPE presentation to the Bay Area’s useR Group, introduced by Michael E. Driscoll and hosted at Facebook’s Palo Alto office on March 9’th 2010. Special thanks to Jyotsna Paintal for helping me film the event.
Saptarshi received his Ph.D from Purdue University in 2010, having been advised by Dr. William S. Cleveland. Saptarshi works at Revolution Analytics in Palo Alto, as of the last update to this blog post.
Hadoop is an open source implementation of both the MapReduce programming model, and the underlying file system Google developed to support web scale data.
The MapReduce programming model was designed by Google to enable a clean abstraction between large scale data analysis tasks and the underlying systems challenges involved in ensuring reliable large-scale computation. By adhering to the MapReduce model, your data processing job can be easily parallelized and the programmer doesn’t have to think about the system level details of synchronization, concurrency, hardware failure, etc.
Reference: “5 common questions about Hadoop” a cloudera blog post May 2009 – by Christophe Bisciglia
RHIPE allows the R programmer to submit large datasets to Hadoop for a Map, Combine, Shuffle, and Reduce to process analytics at a high speed. See the figure below as an overview of the video’s key points and use cases.
The RHIPE Video
Note: On May 30, 2013, LectureMaker’s new video player version 4.4 was put into use on this page… “This new version works with LectureMaker’s eCommerce, massive streaming storage, and eLearning plugins. In addition, now you can use the full screen feature, and see automated tool tip displays in the timeline” — Ron Fredericks, Co-founder LectureMaker LLC.
Video Topics and Navigation Table
| Navigation Dot | Elapsed Time | Description of Topics (plus hot links into the video) |
|---|---|---|
| 22 | 0.00% | Beginning |
| 21 | 1.00% | Introduction to Dr. Saptarshi Guha |
| 20 | 3.00% | Introduction to RHIPE |
| 19 | 4.40% | Analysis of very large data sets |
| 18 | 8.10% | Overview of Hadoop |
| 17 | 13.6% | High performance computing with existing R packages |
| 16 | 15.6%, | High performance computing with RHIPE: MapReduce interface to R |
| 15 | 24.4% | Case study: VOIP |
| 14 | 30.3% | Case study, step 1: Convert raw data to R dataset |
| 13 | 35.4% | Case study, step 2: Feed data to a reducer |
| 12 | 41.6% | Case study, step 3: Compute summaries |
| 11 | 45.7% | Case study, step 4: Create new objects |
| 10 | 49.3% | Case study, step 5: Statistical routines across subsets |
| 9 | 52.7% | Case study: VOIP summary |
| 8 | 54.6% | Another example: Dept. of Homeland Security |
| 7 | 62.0% | RHIPE on EC2: Indiana bio-terrorism project |
| 6 | 72.8% | Q: What is the discrepancy between sampled data? |
| 5 | 78.7% | RHIPE on EC2: Simulation timing |
| 4 | 80.6% | RHIPE Todo list |
| 3 | 86.3% | RHIPE Lessons learned |
| 2 | 95.0% | Q: What optimization methods were used? |
| 1 | 99.9% | Credits |
Code Examples from the Video
Source code highlighter note: R and RHIPE language constructs are color coded and hot-linked to appropriate online resources. Click on these links to learn more about these programming features. I manage the R/RHIPE source code highlighter project on my engineering site here: R highlighter.
Move Raw Data Into Hadoop File System for Use In R Data Frames
-
## Case Study – VoIP
-
## Copy text data to HDFS
-
## Use RHIPE to convert text data :
-
## 1079089238.075950 IP UDP 200 67.17.54.213 6086 67.17.50.213 15074 0
-
## …
-
## to R data frames
-
## create components (direction, id.ip,id.port) from Sys.getenv("mapred.input.file")
-
,value0[id.port[2]],direction,sep=".")
-
})})
Submit a MapReduce Job Then Retrieve semi-calls
-
## Case Study – VoIP
-
## We can run this from within R:
-
## This takes 40 minutes for 70 gigabytes across 8 computers(72 cores).
-
## Saved as 277K data frames(semi-calls) across 14 gigabytes.
-
## We can retrieve semi-calls:
-
## a list of lists(key,value pairs)
Compute Summaries With MapReduce
-
## Case Study – VoIP
-
## make key from map.keys[[i]]
-
})
-
})
-
r<-expression(
-
pre={
-
},reduce={
-
},post={
-
in.start<-ifnull(mydata[‘in.start‘])
-
…..
-
out.end<-ifnull( mydata[‘out.end‘] )
-
out.start<-ifnull(mydata[‘out.start‘])
-
})
Compute Summaries With MapReduce Across HTTP and SSH Connections
-
## Example Compute total bytes, total packets across all HTTP and SSH connections.
-
})})
-
r < – expression(
-
pre <- { sums <- 0 }
-
reduce <-{
-
},post={
-
})
Load RHIPE on an EC2 Cloud
-
})
References:
“I just watched the Saptarshi Guha video. It looks great!! Thank you! The picture is incredibly crisp, and the timeline tab is a nice touch for reviewing the film. Thank you!” — Matt Bascom
VMware’s open-source partnership with Cloudera offers you a virtual machine with Hadoop, PIG, and HIVE – download
The University of Purdue hosts the documentation and open-source code base for RHIPE – download
This is what the share text above looks like…
Presented by Saptarshi Guha
Pingback: Bay Area R User Group 2009 Kickoff Video | LectureMaker, LLC
Pingback: RHIPE: An Interface Between Hadoop and R for Large and Complex Data Analysis | R User Groups
Pingback: Tweets that mention RHIPE: An Interface Between Hadoop and R for Large and Complex Data Analysis | LectureMaker, LLC -- Topsy.com
Pingback: A video from the Creator of Rhipe « Baoqiang Cao's blog
Pingback: News | LectureMaker, LLC
It’s always a relief when someone with obvious expertise ansrwes. Thanks!
Pingback: RHIPE: An Interface to Hadoop and R for Large a...
This web website is certainly a walk-through for all the info Hadoop
Online Training .
As I’ve seen the only variation for learning at hadoop online training Hyderabad and learning from the informative blogs like this is that, here in these blogs we can connect to more examples along case-studies.