LectureMaker, LLC » Professional Associations

ACM Data Mining Camp 2010 – Expert Panel

admin — Tue, 20 Sep 2011 20:55:22 +0000

Ron Fredericks writes: LectureMaker was on the scene again this year to record the annual event held at eBay Saturday November 13′Th, 2010. Watch the video and review some selected event photos – both are presented below on this page. Use the navigation table below the video to jump into the video or share selected highlights with others. Special thanks to eBay, ACM Data Mining Special Interest Group, and Austin Anderson of Spice Pictures, LLC. Austin supported me with his time, expertise, and professional video camera filming the day’s events.

The next ACM Data Mining Camp for 2011 will be held October 15′th at eBay: More Info

See all of LectureMaker’s video enhanced math publications here: www.lecturemaker.com/category/mathematics/

Expert Panel Discussion

The panel was just as lively as last year’s discussion. Follow the full video with all the Q/A to see what was interesting to the experts and students in attendance.

Moderator
Dr. Patricia Hoffman, Research Scientist

Panelists
and their signature question

Dr. Neel Sundaresan, Sr. Director and Head, eBay Research Labs at eBay
How does data mining apply to social and incentive networks?

Mr. Dean Abbott, Chief Scientist and Co-Founder at SmarterRemarketer, LLC
What tools are available to the data miner today?

Dr. Mike Bowles, Research Scientist and Start-up Executive
How has the field of stock market prediction changed over the past two years?

Dr. Hans Dolfing, Pattern Recognition Manager at Apple
What are the current challenges in Speech and handwriting recognition?

Dr. Susan Holmes, Stanford Professor, Statistics Department
What recommendations do you have for people starting out in data mining today?

Dr. Omid Madani, Senior Computer Scientist at SRI International
How has data mining changed as the scale of data has increased so dramatically in the last few years?

Dr. Lionel Jouffe, President/CEO at BAYESIA
Can you give us a brief example where a Bayesian network did a great job in diagnostics? [poor audio, the answer to this question not included in video]

The ACM Data Mining Camp Expert Panel Video

This video is being hosted with LectureMaker’s Cross-domain Video Platform. It’s designed specifically for sharing lectures. Learn more about the real-time streaming, direct sharing of short video segments, and content metrics here

Video Topics and Navigation Table

Elapsed Time	Description of Topics (plus hot links into the video)
0.0%	Dr. Neel Sundaresan, Sr. Director and Head, eBay Research Labs at eBay
2.0%	How does data mining apply to social and incentive networks?
12.0%	Mr. Dean Abbott, Chief Scientist and Co-Founder at SmarterRemarketer, LLC
13.5%	What tools are available to the data miner today?
15.7%	Dr. Mike Bowles, Research Scientist and Start-up Executive
16.4%	How has the field of stock market prediction changed over the past two years?
21.8%,	Dr. Hans Dolfing, Pattern Recognition Manager at Apple
22.7%	What are the current challenges in Speech and handwriting recognition?
26.8%	Dr. Susan Holmes, Stanford Professor, Statistics Department
27.9%	What recommendations do you have for people starting out in data mining today?
34.8%	Dr. Omid Madani, Senior Computer Scientist at SRI International
35.6%	How has data mining changed as the scale of data has increased so dramatically in the last few years?
43.3%	Dr. Lionel Jouffe, President/CEO at BAYESIA
44.2%	Audience Q/A
44.6%	What is the next major unexplored frontier in data mining?
50.6%	What are the difficulties in managing incentive networks?
55.2%	Can you comment on using economic metrics in incentive networks?
60.2%	How can you measure emotion or sentiment in text or through other means?
68.0%	What tools are best for terabyte data mining?
72.4%	What are the alternatives to data mining with terabytes?
79.3%	Can you comment on how you approach a new data mining problem?
83.5%	Is there a way to use incremental algorithms in data mining?
90.4%	Can you comment on the effects of anonymizing data?
92.9%	Can you comment on using domain knowledge in generating recommendations from large data sets?
99.5%	Credits

This is what the share text above looks like…

ACM Data Mining Camp 2010 – Expert Panel
Moderated by Dr. Patricia Hoffman

Photo Gallery

You can click on any thumbnail to start a slide show, then right click to enlarge selected photos captured during the event.

References

Dr. Patricia Hoffman Blog:
patriciahoffmanphd.com/

DJ Cline write-up, links, and photos:
www.djcline.com/2010/11/17/nov-13-2010-acm-data-mining-camp/

Revolution Analytics write-up:
blog.revolutionanalytics.com/2010/11/acm-data-mining-camp-1.html

Analytic Bridge write-up:
www.analyticbridge.com/group/conferences/forum/topics/acm-data-mining-camp-nov-13

Technorati Tags: Anonymizing Data, Apple, Austin Anderson, BAYESIA, Data, Data Mining, Dean Abbott, Domain Knowledge, eBay, Economic Metrics, Emotion, Executive, Handwriting, Hans Dolfing, Incentive Networks, LectureMaker, Lionel Jouffe, Metrics, Mike Bowles, Neel Sundaresan, Omid Madani, Patricia Hoffman, Pattern Recognition, Research, Ron Fredericks, Scientist, Sentiment, SmarterRemarketer, Social Networking, speech, Spice Pictures, SRI International, Stanford University, Start-up, Statistics, Stock Market Prediction, Susan Holmes

Balloons for Luv Dinner Event

ronf — Sat, 18 Jun 2011 16:21:53 +0000

Balloons For Luv (BFL) held their first annual appreciation dinner for volunteers, staff, and key donors in November, 2010. The event was a celebration of BFL’s success in meeting its goal of delivering free colorful Mylar balloons to hospitalized children during times of painful and often frighting cancer and other serious illnesses.

Enjoy the night out with the team through these videos and photos. See why there is so much passion sharing a balloon with a seriously ill child in a hospital. Then consider donating to help cheer up more children in more hospitals: www.BalloonsForLuv.org

*Videos and Photos from Balloons for Luv’s Benefit Celebration Dinner: November 11, 2010*
	video Barbara Sayed Introduction video Susan Martinez Testimonial video Douglas Harding Testimonial video Entertainment Highlights photos Event Photo Gallery

Video – Barbara Sayed Introduction

Video – Susan Martinez Testimonial

Video – Douglas Harding Testimonial

Video – Entertainment Highlights

Event Photo Gallery

The support for the event was fantastic. Click on any photo to start a slide show.

Technorati Tags: Balloons for Luv, Barbara Sayed, catastrophic diseases, charity, Children with cancer, Children's Hospital, corporate sponsorships, critically ill children, donations, Douglas Harding, dying, heart, Hope Flight Foundation, lifting children’s spirits, love, luv, medical procedures, miracle, Mylar balloons, non-profit, Oakland Children's Hospital, Research Center, Research Hospital, serious illnesses, suffering, Susan Martinez, terminal illnesse, transplant, treatment, Zambra Restaurant

Workshop: How to Conduct Productive Meetings

admin — Thu, 26 May 2011 07:21:31 +0000

LectureMaker and Blue Cube Toastmasters Club are sponsoring a 1 hour workshop on how to conduct productive meetings Tuesday night June 7′th from 5:30pm to 7pm.

Why attend? Become a better speaker, be more proficient with your time, and be recognized as a leader within your organization.

Seating is limited, and registration is required for all attendees except Blue Cube Toastmasters members:

$6.11 per guest, advance registration through EventBrite
$10.00 per guest, at the door
Guests get $5.00 off if they join Blue Cube Toastmasters Club within one week

Video Introduction

Link to video

References used within the video:

Time is wasted in business meetings

Toastmasters success/leadership kits available to improve teams

What You Get

Toastmasters Workshop Materials

You will learn how to prepare for a meeting, create a concise agenda, lead or participate in the three most common meeting types (give information, get information, and problem solving), how to conclude a meeting, how to build consensus during a meeting, as well as your own workbook and certificate of completion.

Are you FRUSTRATED by:

Meetings that don’t end on time

Action items that never get done

Lack of order or purpose

Inability to make decisions

Then this seminar is for you!

You will LEARN HOW to:

Create an agenda in 10 minutes

Maintain order and respect

Generate ideas

Resolve conflicts

End your meetings on time

Follow up effectively

And much more….

Agenda

5:30 pm Registration, Food, Networking
6:00 pm Workshop begins
7:00 pm Meeting ends

Notes

Meeting Planning Worksheet

Purpose:

Time:

Participants:

Location:

Special arrangements:

Agenda:

Assignments:

Meeting Opening Checklist

Establish a friendly atmosphere
Start on time
Bring everyone up to date
Open with an attention-getter
Introduce the situation or problem
State the purpose of the meeting
Welcome the group
Establish a method for discussion

Technorati Tags: Agenda, Consensus, Effecive Meeting, Meeting, Toastmasters, Workshop

Mental Flexibility Workshop

admin — Thu, 28 Apr 2011 16:48:27 +0000

Ron Fredericks writes: as the C5 Area Governor for 2010-11, I am pleased to announce that the Toasters R Us Toastmasters club will be holding a workshop this coming Wednesday, May 4′th, 2011.

Workshop on Mental Flexibility

Key Speaker: Dr. Pieter Kark, ACS, ALG, Coach, Area C3 Governor 2008-9
Location: Synopsys, Inc., 455 North Mary Ave., Building #2, Room: Lovelace — Sunnyvale, CA
When: Wednesday, May 4′th, 2011 – from 5:30pm to 8:30pm
Cost: $5.00
Seating: 20 max
Extras: Light supper and soft drinks will be served

Registration

RSVP by contacting Stephen Buckhout

stephen.buckhout@synopsys.com
(650) 584-4754

What You Will Learn

Free yourself from unrecognized, subconscious, rigid attitudes
Interpret information in flexible, open-mined ways
Let go of snap judgments and labels that impede flexibility
Develop semantic (verbal) flexibility
Increase ability to think strategically, to have “conceptual fluency” and to think on your feet
Build proficiency with practice during the workshop

Flyer

(click image below to enlarge)

Workshop Flyer

Technorati Tags: Mental Flexibility, Pieter Kark, Synopsys, Toastmasters, Workshop

RHIPE: An Interface to Hadoop and R for Large and Complex Data Analysis

admin — Sun, 13 Feb 2011 20:17:06 +0000

Ron Fredericks writes: Dr. Saptarshi Guha created an open-source interface between R and Hadoop called the R and Hadoop Integrated Processing Environment or RHIPE for short. LectureMaker was on the scene filming Saptarshi’s RHIPE presentation to the Bay Area’s useR Group, introduced by Michael E. Driscoll and hosted at Facebook’s Palo Alto office on March 9′th 2010. Special thanks to Jyotsna Paintal for helping me film the event.

Saptarshi received his Ph.D from Purdue University in 2010, having been advised by Dr. William S. Cleveland. Saptarshi works at Revolution Analytics in Palo Alto, as of the last update to this blog post.

Hadoop is an open source implementation of both the MapReduce programming model, and the underlying file system Google developed to support web scale data.

The MapReduce programming model was designed by Google to enable a clean abstraction between large scale data analysis tasks and the underlying systems challenges involved in ensuring reliable large-scale computation. By adhering to the MapReduce model, your data processing job can be easily parallelized and the programmer doesn’t have to think about the system level details of synchronization, concurrency, hardware failure, etc.

Reference: “5 common questions about Hadoop” a cloudera blog post May 2009 – by Christophe Bisciglia

RHIPE allows the R programmer to submit large datasets to Hadoop for a Map, Combine, Shuffle, and Reduce to process analytics at a high speed. See the figure below as an overview of the video’s key points and use cases.

The RHIPE Video

Note: On August 5′th 2011, LectureMaker’s new video player version 4.1 was put into use on this page… “Now you can hot link to video content, just like the Video Topics and Navigation Table below suggests.” — Ron Fredericks, Co-founder LectureMaker LLC.

Video Topics and Navigation Table

Elapsed Time	Description of Topics (plus hot links into the video)
0.00%	Beginning
1.00%	Introduction to Dr. Saptarshi Guha
3.00%	Introduction to RHIPE
4.40%	Analysis of very large data sets
8.10%	Overview of Hadoop
13.6%	High performance computing with existing R packages
15.6%,	High performance computing with RHIPE: MapReduce interface to R
24.4%	Case study: VOIP
30.3%	Case study, step 1: Convert raw data to R dataset
35.4%	Case study, step 2: Feed data to a reducer
41.6%	Case study, step 3: Compute summaries
45.7%	Case study, step 4: Create new objects
49.3%	Case study, step 5: Statistical routines across subsets
52.7%	Case study: VOIP summary
54.6%	Another example: Dept. of Homeland Security
62.0%	RHIPE on EC2: Indiana bio-terrorism project
72.8%	Q: What is the discrepancy between sampled data?
78.7%	RHIPE on EC2: Simulation timing
80.6%	RHIPE Todo list
86.3%	RHIPE Lessons learned
95.0%	Q: What optimization methods were used?
99.9%	Credits

Code Examples from the Video

Source code highlighter note: R and RHIPE language constructs are color coded and hot-linked to appropriate online resources. Click on these links to learn more about these programming features. I manage the R/RHIPE source code highlighter project on my engineering site here: R highlighter.

Move Raw Data Into Hadoop File System for Use In R Data Frames

Code (r)
## Case Study – VoIP
## Copy text data to HDFS
rhput(‘/home/sguha/pres/voip/text/20040312-105951-0.iprtp.out‘,’/pres/voip/text‘)
## Use RHIPE to convert text data :
##   1079089238.075950 IP UDP 200 67.17.54.213 6086 67.17.50.213 15074 0
##   …
##   to R data frames
input < – expression({
## create components (direction, id.ip,id.port) from Sys.getenv("mapred.input.file")
v <- lapply(seq_along(map.values),function(r) {
value0 <- strsplit(map.values[[r]]," +")[[1]]
key <- paste(value0[id.ip[1]],value0[id.port[1]],value0[id.ip[2]]
   ,value0[id.port[2]],direction,sep=".")
rhcollect(key,value0[c(1,9)])
 })})

Submit a MapReduce Job Then Retrieve semi-calls

Code (r)
## Case Study - VoIP
## We can run this from within R:
mr< -rhmr(map=input,reduce=reduce, inout=c('text','map'), ifolder='/pres/voip/text', ofolder='/pres/voip/df',jobname='create'
   ,mapred=list(mapred.reduce.tasks=5))
mr <- rhex(mr)
## This takes 40 minutes for 70 gigabytes across 8 computers(72 cores).
##    Saved as 277K data frames(semi-calls) across 14 gigabytes.
 
## We can retrieve semi-calls:
rhgetkey(list('67.17.50.213.5002.67.17.50.6.5896.out'),paths='/pres/voip/df/p*')
## a list of lists(key,value pairs)

Compute Summaries With MapReduce

Code (r)
## Case Study - VoIP
m< -expression({
lapply(seq_along(map.values),function(i){
## make key from map.keys[[i]]
value<-if(tmp[11]=="in")
c(in.start=start,in.end=end,in.dur=dur,in.pkt=n.pkt)
else
c(out.start=start,out.end=end,out.dur=dur,out.pkt=n.pkt)
rhcollect(key,value)
})
})
r<-expression(
pre={
mydata<-list()
ifnull <- function(r,def=NA) if(!is.null(r)) r else NA
},reduce={
mydata<-append(mydata,reduce.values)
},post={
mydata<-unlist(mydata)
in.start<-ifnull(mydata['in.start'])
.....
out.end<-ifnull( mydata['out.end'] )
out.start<-ifnull(mydata['out.start'])
value<-c(in.start,in.end,in.dur,in.pkt,out.start,out.end,out.dur,out.pkt)
rhcollect(reduce.key,value)
})

Compute Summaries With MapReduce Across HTTP and SSH Connections

Code (r)
## Example Compute total bytes, total packets across all HTTP and SSH connections.
m < - expression({
w <- lapply(map.values,function(r)
if(any(r[1,c('sport','dport')] %in% c(22,80))) T else F)
lapply(map.values[w],function(v){
key <- if(22 %in% v[1,c('dport','sport')]) 22 else 80
rhcollect(key, c(sum(v[,'datasize']),nrow(v)))
})})
 
r < - expression(
pre <- { sums <- 0 }
reduce <-{
v <- do.call("rbind",reduce.values)
sums <- sums+apply(v,2,sum)
},post={
rhcollect(reduce.key,c(bytes=sums[1],pkts=sums[2]))
})

Load RHIPE on an EC2 Cloud

Code (r)
library(Rhipe)
load("ccsim.Rdata")
rhput("/root/ccsim.Rdata","/tmp/")
setup < - expression({
load("ccsim.Rdata")
suppressMessages(library(survstl))
suppressMessages(library(stl2))
})
chunk <- floor(length(simlist)/ 141)
z <- rhlapply(a,cc_sim, setup=setup,N=chunk,shared="/tmp/ccsim.Rdata"
   ,aggr=function(x) do.call("rbind",x),doLoc=TRUE)
rhex(z)

References:

“I just watched the Saptarshi Guha video. It looks great!! Thank you! The picture is incredibly crisp, and the timeline tab is a nice touch for reviewing the film. Thank you!” -- Matt Bascom

VMware's open-source partnership with Cloudera offers you a virtual machine with Hadoop, PIG, and HIVE - download

The University of Purdue hosts the documentation and open-source code base for RHIPE - download

This is what the share text above looks like...

RHIPE: An Interface Between Hadoop and R
Presented by Saptarshi Guha

Technorati Tags: Algorithm, Bay Area useR Group, Facebook, Hadoop, Incremental Algorithms, Jyotsna Paintal, Large Data Sets, MapReduce, Michael E. Driscoll, R, Revolution Analytics, Ron Fredericks, Saptarshi Guha, streaming, Terabytes, William S. Cleveland

LectureMaker, LLC » Professional Associations

ACM Data Mining Camp 2010 – Expert Panel

Expert Panel Discussion

The ACM Data Mining Camp Expert Panel Video

Video Topics and Navigation Table

Photo Gallery

References

Balloons for Luv Dinner Event

Video – Barbara Sayed Introduction

Video – Susan Martinez Testimonial

Video – Douglas Harding Testimonial

Video – Entertainment Highlights

Event Photo Gallery

Workshop: How to Conduct Productive Meetings

Video Introduction

What You Get

Then this seminar is for you!

Agenda

EventBrite Link:

LectureMaker Link:

Blue Cube Toastmasters Link:

Notes

Meeting Planning Worksheet

Meeting Opening Checklist

Mental Flexibility Workshop

Workshop on Mental Flexibility

Registration

What You Will Learn

Flyer

RHIPE: An Interface to Hadoop and R for Large and Complex Data Analysis

The RHIPE Video

Video Topics and Navigation Table

Code Examples from the Video

Move Raw Data Into Hadoop File System for Use In R Data Frames

Submit a MapReduce Job Then Retrieve semi-calls

Compute Summaries With MapReduce

Compute Summaries With MapReduce Across HTTP and SSH Connections

Load RHIPE on an EC2 Cloud

References: