LectureMaker, LLC » Michael E. Driscoll http://www.lecturemaker.com Teach, Market, and Sell From a Broadcast Video Studio, Supported by Influential Speech Direction, and Distributed on Smart Internet Video Publishing Platforms Sat, 04 Feb 2012 04:39:59 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 RHIPE: An Interface to Hadoop and R for Large and Complex Data Analysis http://www.lecturemaker.com/2011/02/rhipe/ http://www.lecturemaker.com/2011/02/rhipe/#comments Sun, 13 Feb 2011 20:17:06 +0000 admin http://www.lecturemaker.com/?p=3005 Dr. Saptarshi Guha's event video demonstrates his open-source interface between R and Hadoop called RHIPE to a packed Bay Area R User Group audience at Facebook. Continue reading →]]>


Ron Fredericks writes: Dr. Saptarshi Guha created an open-source interface between R and Hadoop called the R and Hadoop Integrated Processing Environment or RHIPE for short. LectureMaker was on the scene filming Saptarshi’s RHIPE presentation to the Bay Area’s useR Group, introduced by Michael E. Driscoll and hosted at Facebook’s Palo Alto office on March 9′th 2010. Special thanks to Jyotsna Paintal for helping me film the event.

Saptarshi received his Ph.D from Purdue University in 2010, having been advised by Dr. William S. Cleveland. Saptarshi works at Revolution Analytics in Palo Alto, as of the last update to this blog post.

Hadoop is an open source implementation of both the MapReduce programming model, and the underlying file system Google developed to support web scale data.

The MapReduce programming model was designed by Google to enable a clean abstraction between large scale data analysis tasks and the underlying systems challenges involved in ensuring reliable large-scale computation. By adhering to the MapReduce model, your data processing job can be easily parallelized and the programmer doesn’t have to think about the system level details of synchronization, concurrency, hardware failure, etc.

Reference: “5 common questions about Hadoop” a cloudera blog post May 2009 – by Christophe Bisciglia

RHIPE allows the R programmer to submit large datasets to Hadoop for a Map, Combine, Shuffle, and Reduce to process analytics at a high speed. See the figure below as an overview of the video’s key points and use cases.


The RHIPE Video

Note: On August 5′th 2011, LectureMaker’s new video player version 4.1 was put into use on this page… “Now you can hot link to video content, just like the Video Topics and Navigation Table below suggests.” — Ron Fredericks, Co-founder LectureMaker LLC.

Get Adobe Flash player

 

Video Topics and Navigation Table

Elapsed TimeDescription of Topics (plus hot links into the video)
0.00%Beginning
1.00%Introduction to Dr. Saptarshi Guha
3.00%Introduction to RHIPE
4.40%Analysis of very large data sets
8.10%Overview of Hadoop
13.6%High performance computing with existing R packages
15.6%,High performance computing with RHIPE: MapReduce interface to R
24.4%Case study: VOIP
30.3%Case study, step 1: Convert raw data to R dataset
35.4%Case study, step 2: Feed data to a reducer
41.6%Case study, step 3: Compute summaries
45.7%Case study, step 4: Create new objects
49.3%Case study, step 5: Statistical routines across subsets
52.7%Case study: VOIP summary
54.6%Another example: Dept. of Homeland Security
62.0%RHIPE on EC2: Indiana bio-terrorism project
72.8%Q: What is the discrepancy between sampled data?
78.7%RHIPE on EC2: Simulation timing
80.6%RHIPE Todo list
86.3%RHIPE Lessons learned
95.0%Q: What optimization methods were used?
99.9%Credits

 

Code Examples from the Video

Source code highlighter note: R and RHIPE language constructs are color coded and hot-linked to appropriate online resources. Click on these links to learn more about these programming features. I manage the R/RHIPE source code highlighter project on my engineering site here: R highlighter.

Move Raw Data Into Hadoop File System for Use In R Data Frames

Code (r)
  1. ## Case Study – VoIP
  2. ## Copy text data to HDFS
  3. rhput(‘/home/sguha/pres/voip/text/20040312-105951-0.iprtp.out‘,’/pres/voip/text‘)
  4. ## Use RHIPE to convert text data :
  5. ##   1079089238.075950 IP UDP 200 67.17.54.213 6086 67.17.50.213 15074 0
  6. ##   …
  7. ##   to R data frames
  8. input < – expression({
  9. ## create components (direction, id.ip,id.port) from Sys.getenv("mapred.input.file")
  10. v <- lapply(seq_along(map.values),function(r) {
  11. value0 <- strsplit(map.values[[r]]," +")[[1]]
  12. key <- paste(value0[id.ip[1]],value0[id.port[1]],value0[id.ip[2]]
  13.    ,value0[id.port[2]],direction,sep=".")
  14. rhcollect(key,value0[c(1,9)])
  15.  })})

 

Submit a MapReduce Job Then Retrieve semi-calls

Code (r)
  1. ## Case Study - VoIP
  2. ## We can run this from within R:
  3. mr< -rhmr(map=input,reduce=reduce, inout=c('text','map'), ifolder='/pres/voip/text', ofolder='/pres/voip/df',jobname='create'
  4.    ,mapred=list(mapred.reduce.tasks=5))
  5. mr <- rhex(mr)
  6. ## This takes 40 minutes for 70 gigabytes across 8 computers(72 cores).
  7. ##    Saved as 277K data frames(semi-calls) across 14 gigabytes.
  8.  
  9. ## We can retrieve semi-calls:
  10. rhgetkey(list('67.17.50.213.5002.67.17.50.6.5896.out'),paths='/pres/voip/df/p*')
  11. ## a list of lists(key,value pairs)

 

Compute Summaries With MapReduce

Code (r)
  1. ## Case Study - VoIP
  3. lapply(seq_along(map.values),function(i){
  4. ## make key from map.keys[[i]]
  5. value<-if(tmp[11]=="in")
  6. c(in.start=start,in.end=end,in.dur=dur,in.pkt=n.pkt)
  8. c(out.start=start,out.end=end,out.dur=dur,out.pkt=n.pkt)
  9. rhcollect(key,value)
  10. })
  11. })
  13. pre={
  14. mydata<-list()
  15. ifnull <- function(r,def=NA) if(!is.null(r)) r else NA
  16. },reduce={
  17. mydata<-append(mydata,reduce.values)
  18. },post={
  19. mydata<-unlist(mydata)
  20. in.start<-ifnull(mydata['in.start'])
  21. .....
  22. out.end<-ifnull( mydata['out.end'] )
  23. out.start<-ifnull(mydata['out.start'])
  24. value<-c(in.start,in.end,in.dur,in.pkt,out.start,out.end,out.dur,out.pkt)
  25. rhcollect(reduce.key,value)
  26. })

 

Compute Summaries With MapReduce Across HTTP and SSH Connections

Code (r)
  1. ## Example Compute total bytes, total packets across all HTTP and SSH connections.
  2. m < - expression({
  3. w <- lapply(map.values,function(r)
  4. if(any(r[1,c('sport','dport')] %in% c(22,80))) T else F)
  5. lapply(map.values[w],function(v){
  6. key <- if(22 %in% v[1,c('dport','sport')]) 22 else 80
  7. rhcollect(key, c(sum(v[,'datasize']),nrow(v)))
  8. })})
  9.  
  10. r < - expression(
  11. pre <- { sums <- 0 }
  12. reduce <-{
  13. v <- do.call("rbind",reduce.values)
  14. sums <- sums+apply(v,2,sum)
  15. },post={
  16. rhcollect(reduce.key,c(bytes=sums[1],pkts=sums[2]))
  17. })

 

Load RHIPE on an EC2 Cloud

Code (r)
  1. library(Rhipe)
  2. load("ccsim.Rdata")
  3. rhput("/root/ccsim.Rdata","/tmp/")
  4. setup < - expression({
  5. load("ccsim.Rdata")
  6. suppressMessages(library(survstl))
  7. suppressMessages(library(stl2))
  8. })
  9. chunk <- floor(length(simlist)/ 141)
  10. z <- rhlapply(a,cc_sim, setup=setup,N=chunk,shared="/tmp/ccsim.Rdata"
  11.    ,aggr=function(x) do.call("rbind",x),doLoc=TRUE)
  12. rhex(z)

 

References:

“I just watched the Saptarshi Guha video. It looks great!! Thank you! The picture is incredibly crisp, and the timeline tab is a nice touch for reviewing the film. Thank you!” -- Matt Bascom

VMware's open-source partnership with Cloudera offers you a virtual machine with Hadoop, PIG, and HIVE - download

The University of Purdue hosts the documentation and open-source code base for RHIPE - download

This is what the share text above looks like...

RHIPE: An Interface Between Hadoop and R
Presented by Saptarshi Guha

Video Link

Technorati Tags: , , , , , , , , , , , , , , ,

]]> http://www.lecturemaker.com/2011/02/rhipe/feed/ 6 Bay Area R User Group 2009 Kickoff Video http://www.lecturemaker.com/2009/02/r-kickoff-video/ http://www.lecturemaker.com/2009/02/r-kickoff-video/#comments Wed, 25 Feb 2009 23:39:28 +0000 admin http://www.lecturemaker.com/2009/02/r-and-science-of-predictive-analytics-lecture-production/ Continue reading →]]>
Ron Fredericks writes: In February I attended the Bay Area R User Group meeting recently held at Predictive Analytics World 2009. Michael E. Driscoll, one of the two kickoff meeting co-chairs, was gracious enough to let LectureMaker capture the video for the event as a technical marketing “lighthouse” project.

Editor’s Note: New Bay Area useR Group Video can be found here:
RHIPE: An Interface Between Hadoop and R for Large and Complex Data Analysis

Today I am happy to present this video to my readers.

If you manage the marketing, feature roll-out, or web site design, for a social network or professional ecosystem, then you need the techniques presented in this video.

Watch this video to learn about:

  1. The open-source analytics programming language called R
  2. How Google and Facebook approach analytics to predict their web user community’s behavior
  3. Where to download R and get enterprise level support
  4. How the meeting co-chairs use R

The R and Science of Predictive Analytics: Four Case Studies in R – the Video

Get Adobe Flash player 10If you prefer not to upgrade to flash 10, you may try playing the flash movie from this link: lmpremovie.swf
Email this video Link to this video LectureMaker Video Player product and support page

Panel of four recognized R users from industry:

  • Bo Cowgill, Google
  • Itamar Rosenn, Facebook
  • David Smith, Revolution Computing
  • Jim Porzak, The Generations Network

Moderator and co-chair of Bay Area R User Group:

  • Michael E. Driscoll, Dataspora LLC



A live R demo
The co-chairs, Michael and Jim, presented a great overview of the R language. Here at LectureMaker, source code highlighting is supported with automatic links back to language documentation for 132 languages plus my own R highlighter.

For example, here is a short R program:

Code (r)
  1. x <- rnorm(100)            # 100 random numbers from a normal(0,1) distribution
  2. y <- exp(x) + rnorm(100)   # an exponential function with error
  3. result <- lm(y ~ x)        # regress x on y and store the results
  4. summary(result)            # print the regression results
  5. plot(x,y)                  # pretty obvious what this does
  6. abline(result)             # add the regression line to the plot
  7. lines(lowess(x,y), col=2)  # add a nonparametric regression line (a smoother)
  8. hist(result$residuals)     # histogram of the residuals from the regression
  9. # source code reference: http://bayes.math.montana.edu/Rweb/Rweb.general.html

You can click on the highlighted methods, or copy the code above and paste it into a web version of R called RWeb

How I developed the video
The LectureMaker video capture, editing, publishing, and technical marketing service starts with the on location video capture of a live event. Druing the process I become emersed in the presenter(s) and the audience participation. My empathy acquired during the project combined with my own professional background allows me to turn video into technical marketing.

I watched the video presented here myself several times: Starting with the content through the lens of my Canon XL-H1S 3CCD HDV High Definition Professional Camcorder with 20x HD Video Zoom Lens III
prosumer digital video camera during the event, followed by several editing iterations on my eVGA e-GeForce GTX280 1GB DDR3 PCI-Express 2.0 Graphics Card-Lifetime Warranty with Free Special Edition EVGA Precision Overclocking Utility
graphics powered PC, ending with comprehensive testing from several Linux web servers connected to Mac, Linux, and Windows client PCs.

To create the video I took the following steps to edit and publish the video, primarily with Adobe Creative Suite 4 Master Collection
software:

  • Edit the video with Adobe After Effects and Premier Pro CS4. I incorporated the features of the Neat Video Pro noise reduction plug-in to correct for low light high gain raw camera issues
  • Edit and optimize the sound track separately using Adobe Soundbooth CS4 along with brightening of the speech itself as only Toastmasters International experience can provide
  • Design and develop a state of the art Video Player with custom hot spot navigation dots, intelligent preloader, and client-server progressive content management using Adobe Flash Professional CS4 ActionScript 3.0
  • Customize the LectureMaker Video Player with external preloader graphic image, personalized playback faceplate, selection of a playback control skin, and custom buttons using Adobe Illustrator and Photoshop CS4 Extended
  • Compress the 75 minute 200 GB high definition 1440 x 1080 mpeg video down to a 85 MB 800 x 600 flash video using both Adobe Media Encoder CS4 and Sorenson Squeeze 5 Pro
  • Lastly, I hosted the video inside this WordPress.org blog package currently at version 2.71 along with KFE SWFObject flash movie publisher plug-in

    • Some thoughts for next time: I did not have ideal placement for the video camera so I struggled with noise, light, who is talking when, and tripod stability. I made a few mistakes with sound and white balance during live experiments with the 100 or so buttons on this new camera too. I have since solved these issues with more experience, an after-market Sony XEL-1 11-Inch OLED Digital TV
    and wired remote control. I apologize to you the viewer, the R presenters, and attendees who participated with their questions. Hopefully these errors will seem small thanks to the excellent delivery of great material during the R User Group event itself. This was one of the best panel discussions I have attended in either engineering or science.

Contact LectureMaker to develop a video enhanced technical marketing program
This blog post’s content and its longtail of moderated comments over time demonstrates LectureMaker’s live video capture, video editing, social networking outreach, and technical marketing service. It might be based on a single live event, or a series of events integrated into a year-long community outreach program. What’s more, most of the cost for the LectureMaker service is already built into your current editorial events calender! Yet adding LectureMaker service to your existing events program could make all the difference in meeting your goals around ecosystem or sales development. You can get a brief overview here:
http://www.lecturemaker.com/products_services/

LectureMaker targets this service to exceptional people, brands, and products. Turn your high value live but localized events into reusable global marketing and education programs

How the panelists use R for predictive analytics
Each panelist came prepared to discuss R’s strengths and weaknesses as a tool, along with example case studies. Mike has a great blog post summarizing this meeting. So I invite those who would like to learn more about the techniques presented in this video to jump over to read it now:

How Google and Facebook are using R
by Michael E. Driscoll | February 19, 2009
http://dataspora.com/blog/predictive-analytics-using-r/

At the end of this process I learned a great deal about the material presented: including R packages, where to download it, who to contract as project mentors, who to lead group training, and some additional open-source packages such as Octave and R’s interface with Python. I Developed a passion for the techniques discussed in this video, and every video product I put together. But in this case I also plan to get started with R along with some of the integrated tools that were discussed in the near future.

Want to learn R? Here is a link to the book recommended by the R User Group:
Software for Data Analysis: Programming with R (Statistics and Computing)

Technorati Tags: , , , , , , , , , , , , , , , , , , , , , , ,

]]> http://www.lecturemaker.com/2009/02/r-kickoff-video/feed/ 15