Download File Firmware and Flashtool plus Video Tutorial compressed:
do not forget there are some additional tips from me:
How To Flashing swipe velocity tab
>> ladies and gentlemen,please welcome c.e.o. google sundar pichai. [ applause ] >> sundar: thank you. thank you all for joining today. it's great to be here. it's an exciting moment for us. we've evolved a longway as a company. i still remember lookingat our early data centers,
and i look at the scaleat which we are today, which essentially representsthe way we have scaled our infrastructure up. we've also come a long wayas an enterprise company. you know, i remember theearly days with google search appliance being our onlyproduct and today we serve our enterprise customersmuch more deeply. we are at the beginning of whatis possible with the cloud, which is why it'sspecial moment for us.
it feels like a special time,the confluence of right people, right technologyand the right time. for us it got particularlyspecial in december because we welcomed diane greene, a leaderwith deep it experience and the pedigree of an entrepreneur. she joined google to runall of cloud and apps, and i can't imagine abetter person to do this. she's already made many positivechanges and you'll hear about all of them today.
when she started vmware, dianehelped change the industry by getting businesses startoff virtual machines. virtualization, the practiceof running several operating systems at the same time,on a single computer, was an entirely new way of usingcomputers more efficiently. it helped businessessave a lot of money. today, we are at anotherinflection point. as you know, businesses are juststarting to adopt the concept of a public cloud.
but what exists today islargely a part of decades-old technology, essentially virtualmachines to a scale server end run, but in the future almosteverything will be done in the cloud, because it simply isa better way to do computing. for years we have been investingin scaling up our infrastructure to do this. because the things we careabout as a business are the same things many of you care about asyou're building your businesses. because of our investments,we have scaled many products.
in fact, we haveover seven products, each of which has abillion users each. and to put the scale inperspective, in gmail alone, every single day we handle over1.4 petabytes of information, that's like 1400 librariesof congress every single day. we do this with99.7% availability. we do this across a privatenetwork by having virtual presence in everycorner of the world. so we have scaled up ourinfrastructure a lot.
and we primarily built it foruse by google for our services to serve our users. but we are at a point where weare opening it up so that every business can build what's next. and we want this to workfor all kinds of businesses, whether you're a large businesslike snapchat or spotify, or a small business, or abusiness of any size in between. we don't think of this astechnology just for silicon valley.
we want this to be technologyfor every developer and every business no matterwhat size or location. for example, in india, anup-and-coming cgi animation studio in mumbai, assemblage, itused the google cloud platform to work on the recent animatedmovie, "norm of the north.". it shows that you no longer needto be a google software engineer to get access togoogle infrastructure. and when we saygoogle infrastructure, we mean bestsecurity, reliability,
analytics and computer research,all of which you'll hear about today. as i said at the start,this is a special moment. we have set up with the rightpeople, the right technology, and at the right time. i know you want to get into moredetails and we want to share what's next, andso to talk more, i'm very happy to introducediane greene who needs no introduction when itcomes to enterprise.
thank you. [ music ] >> diane: so welcomeand thank you, sundar. you know, one of the thingsabout i've discovered since coming to google is just theexcellence in experience brought to bear on absolutelyeverything everybody does. i'm going to talk about thatin terms of our cloud today, but i'm also just going tomention this event and how amazingly wellit's come together.
you know, yesterday thosebeautiful windows on the bay were garage doors, and there'sa kiosk out there with a giant touch screen where you canindicate your interest and it will tell you different partnersthat have relevant technologies. check it out. i'm really looking forwardto scaling that up. so welcome. we've got several thousandpeople here today, here in san franciscoon the wharf,
one of my favoriteplaces in the world, and tens of thousandsmore on live stream. and we have -- and then at watchparties we have people on six different continents. i also want to welcome the190-plus press and analysts from all over the worldthat are here. to all of you in the roomand everywhere, just welcome. it's great to have you here. yeah.
okay. so i founded vmwareand, you know, what we had there was just abetter way of doing things, saved people lotsof time and money. you know, i could go into anybusiness and it was so easy to show the values, the benefits. and what we have here at googlecloud is just so much more. and as sundar said, it's alljust a better way to do things. this has kind of been a voyageof discovery for me over the
last couple of years as i'vegotten to know everybody and learn about allthe technologies. you know, google'stechnical infrastructure, just amazing inventions. that's what allows google topower the biggest search engine in the world. and more recently taking ourmachine learning technologies and infusing them into ourproducts and more which you'll hear about today.
google's had no choice butto do this with amazingly sophisticated security. but what's i think most excitingto me is google now's just so all-in on what we cando for the enterprise. all that we can bring and helporganizations of all sizes all over the world. i also want to add that googleis strengthened by a pretty broad portfolioin the enterprise. you know, we've got google maps.
we have chrome. android. google analytics justlaunched last week. and then our beyond productivityoffice suite, google apps, that all of google runs on. it's behind our culture. we built it. the collaboration and docks. the video conferencing,hangouts, drive, gmail.
it's what lets usbe so communicative, so transparent andso collaborative. now, when i joined google backin december, as a noogler, you know, wearing one of thosepropeller heads -- propeller hats, i got -- on my very firstday i got to go visit google's biggest data center in iowa, andall of us that went were just completely awestruckby the lack of people, mostly security guards,the amazing security, the hvac systems,and how green it was.
the incredible passion aroundbeing energy efficient. and that's kind of why i'vechosen pictures of data centers as the backdrop formost of my slides. our data centers and oursystem's infrastructure are invisible to ourcustomers and partners, but they really underpineverything we do. so why -- you know, it'sbeen really exciting. customers -- we're seeingreal acceleration in customers choosing us, and whyare they doing that?
well, i think it could be putinto three main reasons: better value in terms ofprice and performance. reduced risk in terms ofsecurity, open source software. and one of my favorite, theaccess to the innovation. i'll spend a few minutes goingto each one and i hope all of you do a deep dive over the nexttwo days to learn more details about this. our cloud is faster fromthe bis to the processors, to the time it takes to bringup some queued virtual machine.
a simple feature like snapshotsis one to two ordinates of magnitude faster than the securecopy which is the alternative. and we're delivering on thatperformance with the most competitive price. the efficiency with which we canrun our infrastructure lets us offer this price performanceoften 50% better, or better, and our pricing iscustomer friendly too. we give you discountsautomatically the more you use. we do things like provideconfigurable virtual machines so
that you can configurewhatever machine you want, be it 7 gigabytes, 6cores, whatever you want. we don't require itto be a power of two, which as you might imagine is-- causes some pretty wasteful overprovisioning, so bettervalue through a combination of price and performance. oops. can you go back? i -- i got confused.
reduced risk. reduced risk, firstfrom security. you know, something i've learnedsince coming to google is that just an unbelievable tax that wehave on making all -- ensure all our customers' data is secured. you know, we've been hardenedover the years because we're just in the crosshairs of all ofthe best hackers in the world. and so when you cometo google cloud, you benefit from literallythousands of people-years of
engineering thatwe've put into this. it's not just security, but it'salso the privacy and compliance and the high bar we holdourselves to in terms of protecting the privacy andsecurity of our customers. but the other area of riskreduction comes from our just absolute commitmentto open source. we open source -- wetake our innovations, we put them in our products,we open source them, and then we provide an api, andwe're more than happy to compete
based on the qualityof our implementations. we've open sourced kubernetes,our container engine for managing server instances. we've open sourced dataflow,our batch and streaming data pipeline. it's currently in incubation asan apache beam, on apache beam, and we've opensourced tensor flow. that's our second generationdeep learning algorithms. we're using them to powerservices like image search,
or the smart reply in inbox. and then of course,once you open source, you get a community, a vibrantcommunity where things can improve more and more quickly. the last reason,access to innovation. our customers are alreadybenefiting from our approach. we want to havenoops, automated it. it's based on ourserverless architecture. today we have app engine,bigquery, and container engine,
they just work, without you everhaving to think about whether you've got enough servers orif things are working right. and we're well on your way toa full noops where you can just write and run. one of our best-knowncustomers is snapchat. they went from zero to more than100 million users without even formally hiring an ops team. tim sehn, their vp ofengineering said, yeah, it was just sort ofme and one other guy.
it's just work. we've scaled up. you see this in ourbig data services. there's really nothing close. you type in the query, it'sinstantly distributed across thousands of cords andmagically it just happens. what makes all this work is avery dense network bandwidth, we have widths inbetween our clusters. our whole platform is designedas an integrated architecture,
and without that, the kind ofhighly distributed processing we do is impossible. one of the spotify engineersput it best in a tweet after we announced they were using gcp. "finally i can tell the worldthat bigquery is the best thing that's ever happenedto me." now, people don't say that unlessyou're saving them an incredible amount of time and effort. so for us this access toinnovation is harvesting our r&d
investments in new productionsthat everybody regardless of your size, can use. we'll have more to sayabout innovation today. you'll be hearingabout security, our machine learning platform,new approaches to operations for hybrid cloud, and have since toplatform as a service plans and our regional expansion plans. and now i'll just take a fewminutes to talk about what's ahead at a fairly high level.
you know, i've seen google'sdevelopment accelerate over the past 18 months. i think it's nothing compared towhat we're going to be able to do over the next two years. but i'm going to highlight threeof the areas that we consider particularly important:security, machine learning, and the transition to thenext generation of cloud. how we go to market. how we help ourcustomers be successful.
first, security. there's a lot more wecan do going forward. we can take the tools thatgoogle's developed internally and we can make themavailable to our customers. machine learning. machine learning is pretty muchrevolutionizing every industry. it's really gotten to thepoint where if your customer is embracing machine learning,it would be prudent for you to embrace it too.
and you'll be able to expectconstant upgrades in our machine learning and you'll be able toexpect more and more train data in different verticalssupported over time. moving to cloud. what we want to do here is makeit really nondisruptive for everybody to move to our cloud,to transition in a way that saves them time and money andactually uplevels what your businesses can do. it's really key toour market strategy.
and just say thatin many, many ways, cloud is so different fromselling enterprise hardware and software. we're early in understandingexactly how to sell well and grow our customers. some of the basics are the same. we're definitely setting upprograms to enable our partners, providing training,certification, solutions, also for our customers,and programs to ensure our
customers' success. you know, one area that's reallygetting some reinvention is pricing models, youknow -- you know, the cloud vendors do the --should be the ones doing the capacity planning, not thecustomers with committing long-term out. that's the spirit of the cloud. but what we're really seeing isthe technical expertise and the advance planning required of usto help our customers get ready
to move to the cloud, takeadvantage of the cloud, explore more optimalways to do things. you know, we can see that we'regoing to function as an applied r&d team for ourcustomers, in effect. and it's because of this that wethink we'll be able to give our customers a step functionboost in their velocity. and what do i mean by this? what is this lift? well, i've learned almosteverything i know from racing
sailboats so i'm going touse a boat metaphor here, and with the exceptionof planing, you know, boats are kind of constrainedby how fast they can go by the friction on the hull which isproportional to the square root of the water line length. not unlike -- this is not unlikethe friction existing between computer architecturesand systems imposed on the businesses, but i think thereit's an inverse function of the ability to hire skilledsystems and operations people.
so we've all seen a revolutionover the last few years. we saw it on san franciscobay in sailboats. hydrofoiling catamarans that cango 55 miles an hour and making its way to sailboats,kite boards. so we built the it equivalent soyou don't have to spend the time and money to do it yourself, andit's at scale so we can serve billions of users andmillions of businesses; we can give youthis sort of lift, this sort of force multiplier,permanently accelerate your
progress and onan ongoing basis. so gcp is growing. we're seeing potentially hugegrowths in our core storage services, our core computeservices and we're excited to see the new innovations thatwe're bringing out are just kind of taking off. google container engine usagehas been doubling every 90 days, growing faster thanvirtual machines, and we're expecting thesame from some of our newer
innovations that we'llbe announcing today. we're serious aboutthis business. alphabet invested $9.9billion in cap x in 2015, and that investment expands ouralready-considerable ability to provide cloud infrastructureservices to you today. we're excited to makeour innovation engine, our technology work for you,for companies of every size, and we really want tomake you a success. your success.
you know, it's soexciting to us. so it's terrificto have you here. enjoy the conference, andnow it's just my incredible privilege to invite up our nextspeaker, our alphabet chairman, eric schmidt. >> eric: thank you. thank you very much, diane. and thank you guysfor being here. i was thinking about it; imaginewhat an internet operating
system really should be. it should takecare of everything. and it occurred to me that i'vebeen doing this for 45 years, and that this notion ofincreasing abstraction has been in front of me my entirecareer, literally. so when i started, i was an ibm360 hex assembler programmer. sorry. and c was invented. and i just thought itwas the neatest thing,
because programmingand assembly was hard, and i kept arguing are thecomputers fast enough, right? because i need areally fast computer. and eventually they convincedme that c was fast enough. i know this sounds archaic,but the fact of the matter is, the same has been truefor layer after layer, and so this went from c andc++ and then java, of course, which we announced -- i in factannounced in 1994 next door at moscone, then into the weband then with infrastructure,
and it's also beentrue of hardware, and all the wayseverybody here knows here. soldering on chips, i literallyused to do this to many computers to then the pcarchitecture and now systems on a chip. the 1980s, we announced at sunthe network is the computer. and in 1997, i announcedthe network computer, right? so we've been working on thisfor a very, very long time, but each of these layers isa new layer of abstraction,
and the whole idea is toautomate the tedious detail and empower people, right? that's really what it's about. so along the way, googleinvents google iii. now, i'm often asked whydid we invent google iii, and the answer isbecause we had google ii, and when it had an error, itwould say, grow up, horsy, and then it would stop. this was a problem.
and if you wantedto delete files, you had to make sure that themod file in the file security handle changed without yourpermission from 977 to 0, and aside from that,it was a great product. so google iii was inventedroughly 2002, 2003, internally at google, all right,in order to -- in order to sort of build what you would think ofas the network operating system that we needed. and what's interestingthat this platform,
which we used obviouslyincredibly successfully, and at a time we were investingfour times more capital than the sum of all of the other computercompanies in existence at the time, just to sort of build outthe data structures and build out these platforms, and sowhat would happen if you came to google in a programmerin sort of early 2000s, you didn't have to thinkabout network segments, denial service attacks, beat-offip addresses, scaling, ssl, machine resources.
what happened was the machinetook care of it automatically and you couldbuild gmail, right? or you could build docs. or you could build thenext enhancement of search. so in 2008, we announced appengine, and i figured weird up, right? i mean app engineis beautiful, right? and many, many peoplein this audience use it. and what happened was app enginewas like the apis that went into
google iii. all right. so today, if you'reusing app engine, you're using the same interfacesthat have been used internal at google. i figured okay, we're done, it'sgoing to be hugely successful, and along the way, here is whatwe did: we built mapreduce, you guys can use whatis called data proc, which is sort of a moresophisticated version.
we built dremel, right? you guys can use bigquery, amore sophisticated version. we built somethinginternally called colossus, because we ran out of big names,and you can -- you now know that as google cloud storage. we built stubby, not myfavorite product name. you can use grpc, also notmy favorite product name. we built bigtable;you can use bigtable. now that's a productname that i like.
so the point isit's really easy. just use -- just useapp engine, right? we're done. ah, and along the same way, wegot very interested in doing sharing applications andbuilt this docs architecture, and i remember sitting with theexecutive in charge of docs, and i said, "what's different? what's different? you got to show me what'sdifferent." and he scratches
his head for awhile andhe said "sharing", and i, in my typical stupid way said,"who cares about sharing?" and he said, "do youshare calendars?" i said, "yeah." well, sharing isnow what has driven google docs, apps, architectures,all of that, to 60% of the fortune 500are customers for this stuff. we have more than 2 millionpaying enterprise customers. it's a huge business for us,and growing very dramatically, so very, very proud of it.
but there was somethingfundamentally wrong with my conception in 2008 ofwhat we were doing, which was it's not realisticto expect people to go denovo, right, from an originalarchitecture into app engine, so we decided that we had tochange our strategy, 2009, 2010, to basic -- youknow, it didn't work. they didn't give the rightstepping time to the cloud. it was too hard. it didn't make any sense.
we decided to meetyou where you are, as opposed to where wethink you should be. okay? makes sense. first thing you've got to dois you've got to do a virtual machine, right? and we do that. and that becomes the initialpart of gcp as you know it today, and we -- and i figured,okay, well, we're done now.
we've done internet os. well, that's notquite true either. you know, vm is sortof pretty naked. you've got to do anawful lot of work. very difficult to sort ofcontrol especially in a dynamic environment. we have resource competition. how do you share files? you have all theseissues around security.
in fact, oh, my god, you have togo through the same list again that you just wentthrough with google iii, but now you have todo it with the vm. this is like a disaster. it's like going backin computer science. so then we built containers, andthe core idea of a container is that you can have a littleprotected area where your code runs inside the container, usesall the resources and off you go, and it works, right?
today think docker, right? docker on top, linuxwith extensions, those extensions allowed thecontainers to be properly instantiated and turned on. we had this funny productcalled borg, and i kept saying, "what's the borg?" and everyone wouldsay, "that's the joke". so a borg was the managerof all of these services. so today this containermanagement and scheduling
management system that we'vebuilt is the natural successor to what we invented adecade ago and it works, and it works at a scale google'scomputer cluster is the largest i think in the world, inthe universe, in the planet, and everywhere else it combines. it works at scale you'venever seen before. it's mind-bogglingwhen i see it. so then the problem was, nowwe've got these containers, how do we actually distributethem and manage them,
and we invent kubernetes,and we open source it. it's another name i'm a littleconfused about but you'll get it, i'm sure. we finally got there, and thereason i'm here is i want to say to you right now,45 years later, 20 years after -- 50 yearsafter the internet was invented, 20 years after the networkcomputer was invented, we've finally invented theinternet operating system by meeting you whereyou are, right?
the path is now clear into this. and what we've done is wewill now allow you to build an abstraction layer at the cloud,whatever you have to intersect with. and the reason diane is here,and doing a brilliant job, is because now we have toactually make that happen. we've got the architecture. we were joking about this. the hard part is whatwe have just done now.
the relatively easier part isnow getting the technical links between it, but you've got toget the abstraction right and you've got to get thescalability right, and we have donethat, and i'm very, very proud of thisas you can see. now, if you're not completelywith me -- let me give you a simple example. take a 1960s car and try toremember how to use the choke. easy.
1970s cars, right? i used to drive these things. i still do. you lost control on ice. you had sudden braking. no seat belt. you could bump other carswhen parking the car, right? you had to go to the triplea and get a paper map, which was free, by the way.
you had to remember whenyou had to service your car. and you had to havethese big v6-v8 engines, which of course theysold you at a huge price. the 2000s car tracks andcontrol, antilock, air bags, rear bumper sensors,navigation, service computer, and turbo chargingto deal with scale. it's exactly analogous, right? think about driving a car today. you jump in a car;it just works.
you don't have tothink about it. it alerts you, soforth and so on. go back to that 1960s car. i'm not suggesting that theplatform that the car will drive itself -- oh, yes, i am. in fact in 2020, right? the car drives itself, right? so remember the choke, right? just remember the choke.
now, the serious nerdsin the room here, really liked the choke. nobody else did. that's the secret. this cloud platform we'rebuilding will abstract out all of those exampleslike the choke. what it will do is it will giveyou a platform that you can really express. any times -- and the reasonit's so important to me,
is it's timed perfectly forwhat's going to happen next. and you're going to hear alittle bit about this in a minute from a coupleof other people. this platform is not theend, it's the bottom. and there's something above it. and the thing that is aboveit is machine learning. both narrow ai as it'scalled as well as general ai. and this is thenext transformation. i just want to spend as muchtime as i can on it because i
think it's so profound, andit changes -- i'm basically a programmer who sort of gotlucky, if you will, in google. sorry, and the -- theprogramming paradigm changes. instead of programminga computer, you teach a computer to learnsomething and it does what you want. it's a fundamentalchange in programming. and we have just takenthe initial steps. we -- the neatest example thatmost people can understand about
machine learning areinvolving vision. we built a library, calledvision ml which we announced which will be demoed today. vision ml does theclassification of photos. it's great fun. and if you get confused overwhy this is so important, use google photos. spend a little bit of timeplaying with google photos. it's not the fact that ituploads it to the cloud,
it's not the fact, thatit does face recognition, all of which used to be amazing;it's the fact that it then classifies all your photosinto what you're doing them in. and how it does thatis beyond me, right? i literally stilldon't understand. but perhaps the greatest exampleis what happened in korea a week ago with alphago. alphago as most of you probably-- go is the hardest game known. it's incomputable in itstraditional -- and the team
invented neural network,of which there are two, a value network anda policy network, that could be used tolearn how to play the game. so the most interesting thingabout alphago is not that they're brilliant -- brilliantpeople or brilliant programmers, but that they had to design,as part of their software, learning algorithms that weretailored to the way the game actually worked, andthat's the next layer. it's the layer above.
and 20 years from now, peoplewill say remember the choke, and then they'll say, yeah,yeah, and in 2016, yeah, yeah, we have all of those vmsand all of that other stuff, we don't thinkabout that anymore. but somebody took care of itand now all we're doing is doing various forms oflearning, cognition, and modificationimproving of code. so how do itranslate this today? so i figured i would offersort of -- and put this in
perspective and finish up bysaying i think there's ten things that you allcan do right now. obviously, thankyou all for coming. and i'll read them now. what's the list? if i were right now workingon this right now today, what would i do? well, i'd use linux, i'd usethe google cloud and i'd use kubernetes, right, to writeapplications that treat the
cloud the way i described. in development, i woulduse modern, portable, scalable languages. the one i like is obviously thelanguage go, python, node.js, javascript -- java, and thenuse the google app engine. if i had a clean sheet,i would start there. by the way, forsecurity, use two factor. let's try again. for security, use two factor.
right? don't use one factor. and if you are sufficientlyparanoid, use three. have fun, right? [ laughter ] more factors are better. but if you use ourinfrastructure, which essentially deals with thedistributed denial-of-service attack, you will be protectedfrom this enormous explosion in
cyber issues thateverybody is dealing with. start with machine learning. i mentioned vision ml. you're going to hear a wholebunch more in just a minute. we released a productcalled tensorflow, which is the initial -- thinkof it as nature's multiplication platform for it, and it's thefoundation -- clear to me that this is going to be thefoundation for this next layer of programming, not just atgoogle but everywhere else.
google used sql, right? used h base and h base is nowintegrated in such a way that you think you're using h basebut you're really getting the power of our infrastructure. so again, we've takensomething you use, which is pretty good by the way,sql, been around for 50 years, roughly, inventedin roughly 1970. works very wellfor what it does. use it, but use itin the right way.
and then you get all thebenefit of the architecture. we've extracted that. try cause people. see if you can getthe power of that, and watch the spacefor more from us. what about data warehousing? i think analytics should costless than other transactions, try bigquery, as dianementioned, right? there is no noncloudequivalent of bigquery.
storage. start by deleting nothingunless you're forced to, right? it's so cheap. i estimate that storage isnow a cent a gigabyte a month. by the way, how did this happen? the physicists havebeen busy, right, making discs thatare extraordinary. use google cloudstorage and those apis. data processing.
as diane mentioned, we havesomething called dataflow. dataflow is the naturalsuccessor to mapreduce, right? because it allows -- insteadof sort of back and forth, which is the way tothink of mapreduce, dataflow is more ofan updating process; it keeps you constantly,constantly on, and constantly -- especiallyfor cloud services, it's the only way, and thenfinally i mentioned before use grpc.
right. use a proper rpc mechanism anddo it in such a way that the infrastructure supports it. why is it importantnow to talk about this? because the industrybuilds these platforms, and another platformand another platform, and then have you this enormousamount of code math that has to be brought into it. i'm now convinced that we'vefigured out a way to meet you
where you are, right? and we're investing as acorporation thousands of people enormous amounts of money overthe next few years to build the infrastructure and the partnersto bring you wherever you are into that model. but there's another phenomena,so that's getting to the cloud, getting to -- youcan imagine the song, get to the cloudas fast as you can. but there's another thing,which is this layer above.
and i've become convinced thatthere's a new architecture emerging. i'll describe it like this. what happens -- i spent a lot oftime talking about innovation. it's what i care a lot about,and i talked to a lot of people doing new things. those of you that are doingstartups and new models that are not dealing with some of theinfrastructure and significant issues that are the majorityof our focus are doing the
follow-on. if you're building hardware,you're building in a rapid evaluation model. you're iterating every week. didn't used to bepossible to do that. when you're buildingknowledge systems, you're using crowd sourcingto get the data that you need. i use the googlephotos as an example. and you're puttinginto the google cloud,
using google analytics to dosome kind of very interesting data analysis, and thenfinally, in the next year, you're going to use machinelearning to take that data and do something that's betterthan what the humans are doing. literally, you're going to takehuman crowd source data and you're going tocompute something new. you're going todiscover something new. and i'm convinced that littlemodel i described, rapid eval, odd computing, usinggoogle cloud platform,
machine learning and crowdsourcing the data will be the basis and fundamentals of everysuccessful huge ipo win in five years. in the same sense that thetransition to apps five years ago, which all ofus participated in, created the modern corporationsof uber, snapchat, and others. so it's not just that wemeet you where you are, and it's not just thatyou're taking the cloud, the noncloud data and puttingit into the cloud in these
complicated ways which will makethings get a higher level of traction, but there'sone more thing, and that one more thing is theinvention of the use of crowd sourcing and machineintelligence and rapid evaluation to create huge newplatforms, companies, ipos, wealth, and enormous thingsgoing on in the future. it's a great timeto be in the cloud. it's a great time to be here. thank you, diane,for inviting me.
and i look forward toseeing you all soon. thank you very much. >> ladies and gentlemen, pleasewelcome senior vice president technical infrastructureurs holzle. >> urs: thank you, eric,and good morning, everyone, and welcome to nextin san francisco. since i started atgoogle 17 years ago, we built a whole lotof infrastructure, lots of data centers in 14locations around the world,
and our goal when we started,which is the same goal today, was to build infrastructurethat makes it possible to build amazing applicationsand services. so whether we wereindexing the entire web, or building a 3-d model of theentire world down to the street level, or predicting what usersare going to type next in a search box, the productambitions have forced us to rethink pretty much everythingin infrastructure; hardware, software, and even theway we work together.
and so this continues pressurefrom product innovation, and to stay ahead of thatproduct innovation has been driving what's next forcloud, namely the google cloud platform, gcp. so today i'm going to talkabout three things: first gcp, what it is now, how we builtit, and what's coming up. second, what's just over thehorizon, so what's coming next, so that you can coordinate whatyou're going to do next with us. and then finally, some of thecompanies that are building on
gcp, and i realize thatincludes a lot of you, so thank you for your businessand thank you for partnering with us. so before i talk aboutthat, about what's next, i want to share quickly someof the over 300 improvements we made, just in thelast 12 months, so you can see the paceof our development. together we built a highlydifferentiated platform that has core infrastructure,development tools,
and premium data services,all built into it. there's so many highlights,and i can pick just a few, but here we go. custom machines, dianealready mentioned, they let you choose any virtualsize machine and pay for just what you need. so there's no moreroutine overprovisioning, and the users who are using it,our users are saving on average 19% off of their vm bill.
dataproc is a fully-managedhadoop and spark cluster, lets you spin them up in 90seconds, resize them up or down, and you only pay by the minute. and even betterdataflow, cloud dataflow, lets you build pipelines,you write the code once, and you get both the streamingand the batch analytics pipeline, and you don't evenhave to worry about building the cluster. or about how to correctly handlemissing or late datapoints.
the system does that for you. archival storage hastraditionally meant trading money for access time. so you have to wait hoursto get your data back. well, no more. we launched nearline storagethat combines the price of tape with nearly the speed of discs,and it's using the same apis than regular storage, so youdon't have to change your application after you'vearchived your data.
so no more hours of waiting. it's just seconds away. vm startup times are down toabout 43 seconds on average and falling further. and even better, you can getmore than 100 vms in our cloud in about the time that it takesyou to get one vm in other clouds. that's the difference betweendriving and flying -- or actually it's the differencebetween biking and flying.
high performance vm networking. you can get up to two gigabytesper second throughput to other vms or to storage onevery vm at google. so even data intensiveapplications are never going to be starved for data, and you getthat feature on google in every vm at no extra cost. no other cloud offers suchhigh performance networking. so you get networking. as diane mentioned, you canconnect directly to our network
in more locations thanin any other cloud, and on that network, you getnetwork function as fully managed services that areeasy-to-use, but very powerful. so, for example, our loadbalancer lets you balance across any region in the world, and yetit spins up from zero traffic to million requests persecond instantly. so in just the past year, we'veenhanced our platform with over 300 launches and that doesn'teven include what you're going to hear about today.
and of course, we are stillthe price performance leader. some of our customers see over50% savings versus other clouds, because we allow you to pay foronly what you need when you need it. and that makes it a lot easierfor you to work with us. and it's not just somethingthat is nice to have. if you have a job that onlyruns for a few minutes, but you have to pay for an hour,or if you have a job that needs 20 gigabytes, but youhave to pay for 32,
or if you have abunch of unused vms, but you signed a contractfor them a few months ago, so right there there's someserious savings that you can get when you're on gcp. but cost isn't the only thing. when you're thinking aboutpicking a cloud provider for the next decade, innovation andthe quality of the underlying infrastructure isjust as important. in fact, if you're picking thatcloud provider for a decade,
innovation might actuallybe the most important part, so it's fine tolook at what's next. over the next five years, ithink we will see more change in computing and in clouds thanwe've seen in the last decade or two. literally, an explosionof innovation. now, why am i saying that? i'm basing this prediction on awell-known effect in biological systems.
there, once the basiccapabilities come together and start to function well,they enable the evolution to accelerate. and we're entering a similarlyexplosive period in cloud computing innovation right now. and as software gets woven intoour daily lives and into your company, you all face the samechallenge that google has faced for many years, mainly how torapidly create more functional applications usingthese applications,
operate them at scale,cheaply, efficiently, securely. or in short, how to createbetter software faster. and the user expectations onthat software are ratcheting up rapidly, so today applicationsare constantly being improved with new features, and as users,we're expecting the application to tailor its behavior tous the first time we use it. and good apps delight usby predicting what we want. so for that to happenin every application, we have to have very easy-to-useanalytics and machine learning
frameworks that everyone canembed in their applications. so that's the first client. we've got to do bettersoftware faster. second, noops applicationdevelopment will become mainstream, because developersspend way too much time administering their setups. so next it will be easy to runeven ambitious applications at scale without knowingall the details, so that's the faster partof better software faster.
to do that, we have to automatemost of the work provisioning the running of services, andwith app engine, dataflow, kubernetes, we're alreadywell on the way to that call. last but not least, you know,diane mentioned that security has become more challengingas attack surfaces continue to multiply. very few companiestoday are truly secure. very, very few. and a significant portion ofsecurity vulnerabilities are
created by processvulnerabilities and by human intervention. so a lot of the key of gettingahead of the security challenge is to automatesecurity at scale. now, today, security isoften focused on securing the perimeter, but next,you know, tomorrow, we need systems for securityand not band-aids or topical antiseptics at the skin. and so security is very deeplyembedded in gcp because we have
invested very heavily init, not just for clouds, they're also for allother google products. for example, we were thefirst to offer two factor authentication at scale. the first to provide perfectforward security in essence, and elliptic curve cryptography. last december we announced dataloss prevention for google apps, data rest and gcp is alwaysencrypted by default, because we don't think youshould be forced to choose
between security and performanceor security and cost. and we practicesecurity in depth. our team has discovered heartbleed to name just last year or rowhammer exploits, and these investments are not slowing down. and it's not just about security, but it's also about privacy and compliance. every google product needs tomeet rigorous standards and -- for privacy and compliance, andour customers and our regulators expect third party verificationof these standards.
and us, we regularly undergothird party audits that verified that all of our controls arepresent in our data centers, in our systems,in our operations. now, with gcp, we're reallyafter building a different cloud, and tobetter explain this, let me review where theword "cloud" comes from. the first wave of cloud startedabout 20 years ago with the emergence of co-located hosting. co-located hosting location gaveyou the ability to rent space
rather than to build it, andthe status quo of cloud today actually shares a lot with thoseprivate inventors and co-located data centers decades ago. the components of today'svirtual data centers match the physical building blocksof hosted computing. you have servers, you havediscs, you have load balancers, you have cpus, and so on, exceptthe virtual devices today, and so what had been a purchaseorder now today is an api call, and that's a very profoundchange that has benefited many,
many customers. but the basic building blocks ofthose virtual data centers have not changed yet, and it's stillextraordinarily complex to orchestrate these buildingblocks together at scale, because this model forcesdevelopers to still think much of their time thinking aboutother things other than making their application great. so do we have enough servers? are they all patched?
did we prepurchase toomuch capacity, right? in the virtualized data centerworld, this never gets easier. and actually it's a bit crazythat today's cloud is based on a physical model with all thelimitations that was created 20 years ago, right? it's kind of like the virtualdata center of today still has a manual choke, right? kind of doesn't make sense. and but, you know, i'm a littlebit embarrassed to admit that
actually we used to rungoogle exactly that way, the way i just described. until about ten years ago, ourworld internally looked pretty much like that. but then we realized that wecouldn't really keep up with the rate of changethat google needed, and so we switchedto a new model, distracts away all of thoseservers and is based on containers, not machines.
so today we use a thirdwave cloud, if you want, the combination of automatedservices and scalable data, and we call thisapproach "serverless", because you care about yourcode not -- on your application, not your servers. so while the industry is juststarting to talk about that, serverless architecture is verydeeply embedded in all of gcp. so in app engine, for example,app engine lets you run code snippets, large or small,without specifying where they
run, and your applicationscales all the way down to zero, so if nobody is using yourapplication, you don't pay. in gke, you can run containerswithout specifying the machines they run on. in dataproc, you can createhadoop clusters without actually worrying about the cluster. in dataflow embedder, you don'teven need to worry about the you don't even need topackage container images. you just give us yourcode and it runs.
so the code just runs,and as a developer, you never manage servers. so that's the true meaningof third wave cloud. and gcp has the most integratedapproach to that serverless architecture, so you can starttaking advantage of it without throwing away yourexisting code. so now, to give youthe details about that, it's my pleasure to introducebrian stevens who heads all of our product management.
welcome, brian. >> brian: thanks, urs. so as urs was saying, this nextgeneration of cloud is already happening. it's already bringing outservices that are freeing users from thinking about virtualmachines and storage and blocks on discs. so instead they're startingto focus more on their applications, their usersand then just building their
business. and we have many customers thatare actually coming to gcp today for exactly that reason. so instead of fillingwith infrastructure, they're using full zeropservices like app engine and bigquery, so i want to getstarted really quickly and introduce you to oneof them right now. >> video: we wanted to make thismarketing campaign for coca-cola something that every singleperson in this world,
no matter who you are, couldparticipate in the world cup experience. submit your photo, you couldbe on the pitch at the opening ceremony of the world cup. >> we're talking about 3 millionand a half images from over 200 countries, differentformats, different users, different sources. >> fans could actually findtheir photos to know, okay, i was here on the flag inthe opening of the world cup.
>> it was a gigantic scale. the agency that was creating thephotomosaic printed in fabric, we recreated theequivalent digitally. we did not have time to set upan infrastructure and set up the servers and configurethe machines. by using the googlecloud platform, we sent straight intobuilding the top of it. without that my firstdeliverable would be impossible. >> you have to predict whatthe loads are going to be.
there's a lot of uncertainty. when you look at a cloudprovider like google, the opportunity thereis for autoscaling. you don't have to worryabout disaster recovery, you don't have toworry about backup, because it's providedto you as a service. >> we wanted to send every onethat had an image in that flag a notification the very moment theflag was opened physically in the field.
we had a lot of peopledoing that simultaneously. they're not takingturns to use it, so we were monitoring thetransmission on tv and waiting. i remember clearly seeing thespikes and just hoping please, please, scale up,scale up, and it did. >> fundamentally it gets downto one thing and it's one word; it's trust. the same infrastructure thatpowers the google search engine and search experience iswhat's powering the cloud that
coca-cola is running on. >> the biggest digitalmarketing initiative. >> to participate is like aonce-in-a-lifetime opportunity. >> so we're really lucky todaybecause we have the c.t.o. of coca-cola here alan boehme. so alan, i wanted toask you a few questions. so first of all, like how didyou actually decide to get started with googlecloud platform? >> when you'reworking for coca-cola,
which is one of the largestbrands in the world, almost 2 billion servings aday, and you have google that is serving up over 3.5billion searches a day, it's all about scale. and when you start talking aboutputting something on the world stage for fifa that's seenin 207 different countries, every place in theworld is participating, you need to partner withsomebody that really understands scale, understands reliability.
we can't miss,because if we miss, it's a miss forthe entire world. >> so the video isreally compelling, right? you can get inspired by that. can you tell us a little bitmore about some of the other ways that you're using gcp? >> so we've been advancing withgcp and with google in general for awhile now. we're moving into ananalytics platform.
we're taking advantage of manyof the things that we heard earlier. we're moving forwardwith machine learning. we're real excited about all thenew activities that are going on and the points of opportunitythat are made available to us. in addition, we have createda relationship with google. we've worked on security. we're a very big believer inthe software to find perimeter. google has announcedtheir beyondcorp strategy.
we're partnering at a differentlevel that we have with many other companies in the past,and it's to the benefit of us, our consumers and to everyone. >> so and from your team inside,how is just using cloud changed from how they do thingsfrom five years ago? >> we're 130-year-old company. we're 130-year-old company. we have this culture init that's been going on, not going to say 130 years,but certainly for many,
many generations, and its hard. working with google, workingwith google team and the platform is changingthe culture, changing the skillsets that are required, and it's going to help make uscompetitive going forward in the next century. >> awesome. well, thanks for joining us. >> thank you.
>> brian: so our data center,our cloud is this really dynamic things are always changing. we're constantlyimproving hardware. we're improving software. and we even change thingshow we do operationally, and you can't just shut downthe cloud when you're actually changing out servers andupgrading and working on infrastructure. so we had to support the rateof innovation that we need.
we had to design agilityand change into both our architecture as well as ouroperation proceedures from the beginning. and that's really the reasonwhy we're perhaps the only hyperscale cloud that actuallydepends on live migration. and so that really allows us toupgrade our infrastructure and our software stack and theapplications don't even know about it, because not allapplications are built in this microservice architecture wherethey can -- pieces can be taken
down. so now whether if we'reupgrading that server, if we're applying a patchto a underlying host to us, we can even actually service thepower infrastructure underneath the whole cluster because wehave the spare capacity to live migrate the vms, so applicationsdon't notice when they're running on gcp. and at the same time, gcp isactually the greenest cloud in operation today.
so today in theaverage data center, about 40% of all power actuallygoes for things other than servers. and so that's just waste. and by comparison,in our data centers, only about 10% of power goes foranything other than that server. so you think about howlittle overhead that is. and we actually continueto get a lot better, so we now get three times theamount of compute capacity out
of the same amount of poweras we did just five years ago. so as diane was saying, tojoin her and urs on that trip, a very emotional experience. it was definitely the tenureof my -- of my time here. and you really did expect to seethese football field size rooms full of servers, andit was just that. it was really daunting, but itwas everything else that was around the server thatwas really compelling. it was the power infrastructure.
we were climbing on top of,you know, water coolers, and just the lightsout management. retinal eye scan like to getinside the data center is really impressive. but one of the aha moments tome was -- and i think, you know, everybody else at googletakes this for granted, is when you leave this amazingdata center and you walk outside and the next one isalready being built, and you would think thatonce you've, you know,
delivered perfection youwould just keep cloning it, but the next data center lookedremarkably different than the one that you just left. the size of the floors, theheight of the ceilings were different, they weregoing to four stories, and that just shows you the rateof innovation that's happening in this area today. so if you didn't believethat computing would become a utility, one visit to iowa,and it becomes really obvious.
so we can't bringyou all to iowa, but we actually prepared avirtual reality tour for you today, so here is greg wilson,he leads our developer app gcp team. >> greg: good morning. thanks, brian. so, as brian was saying,we went to a data center. a few months ago we -- our teamgot together to figure out where we wanted to go for a teamretreat and we had a few
options. we could go towhistler, lake tahoe, or the middle of oregonto a google data center, and of course youknow the one we chose. that was absolutelya no-brainer. so we -- i mean you've heardeveryone talk about how much they were blown away by it,and we had the same experience, and it was even inways we didn't imagine. and because we're developeradvocates, you know,
our instant instinct is toshare that with everybody. it's like how in the world canwe get people in here short of doing public tours? they wouldn't let us do that. so what we put together issomething a little new -- if you could switch to -- it'salready on the tablet. so i'm going to be showing 4kvideo using conference wi-fi from a tablet. what could go wrong?
so this video is alittle different. i'm just going to show youa couple of minutes of it. i'll show you how toget to the rest of it. this is a 360video, andi'm doing this on a tablet, so as i turn the tablet -- andyou can do this with any of your devices. as i turn the tablet,you see the view turning. and i'm really trying to besteady here so you don't get nauseated.
and it's not just 360circular; it's spherical. so you can look up, look down. and not playing the soundfor obvious reasons, but we have closed captioningin fourteen languages. by the way, that's chrisfrom the security team. he -- he kept an eyeon us the whole time, and you'll see that inall the other scenes. i want to fast forward and justshow you another quick glimpse, and then i'll tell you howyou can go see it on your own.
so this is the coolingroom in the data center. and they tell you allabout what's going on here. and of course, we want tomake sure that we have proper security. there he is. right over there. so, again, i'm notgoing to -- anyway, this is not the most idealway to experience the video. you can switchback to the slides.
so to view this video, you goon your browser -- using chrome browser on your desktop, you canuse your mouse to look around. you can view it on mobiledevice using the youtube player, and you can do what i was justdoing, move it around like this, or swipe to look left,right, up and down. the most immersive way to doit, the coolest way to do it, is with google cardboard. so you put thedevice in cardboard. you put youtube playerin cardboard mode,
and it's like you're there. as close as wecan get you there. so you'll all receive a googlecardboard tonight at the party. we've got one forevery one of you. we also have a theater out inthe playground area that plays this on an 8-foot screen. it's a circular screen andyou get to hear the whole soundtrack. you can meet sandeepwho stars in the video.
he's on our team. so i encourage you to come overto the program look at that and some of our other experiencesand i hope you guys enjoy it. >> brian: thanks, greg. so when you run on google, yourtraffic runs on the world's largest network backbone, right,with direct connections to thousands of isps innearly every country. and as urs was saying earlier,we have 77 piering points that are in all the major metros,and those are really important
because that's actually where weintegrate directly piering with our network partners and mostrecently our cdm partners and our largest customers. so speaking of thatglobal network, we're actually going to continuebuilding out regional support for a cloud platform, andwe actually just announced yesterday that we're adding twonew regions in tokyo and oregon. so greg just gotto go see oregon. and that's really important,but we're not done, right?
once we get the modeldown, as we are with tokyo, then we can go even faster. so we're actually going tobe adding ten more locations through the end of 2017. so on a hybrid called mini,users are actually using more than one cloud, right, and thatmight be because look at some of the services thatwe're talking about. there's different services thataren't available on all clouds, so they're using multipleservices that might be for --
for risk or cost or otherthings, or even archival. but the -- but the challengeof that is is that all the providers, including onpremise, private cloud, we all build our own tools andkind of put you in this little operational silo, right? and so we think that's,you know, a real problem, and it just makes it harderfor users to consume multiple so google stackdriver is aservice for monitoring, logging, visualizing, and alerting,across cloud services,
across your resources,across your applications. and so we wanted to get youout of that sort of operational silo. so one of the things that we didis we just announced beta for google stackdriver today,and now it will actually be available and it will supportgcp as well as aws with private cloud coming. so gives you that single paneof glass that you would expect where you can, you know, issuesare sort of propagated really
quickly. you can find them, visualizethem, and resolve them. another challenge with -- withhybrid cloud is the applications are just really married intothe underlying infrastructure. whether it's the dependenciesthey have on different operating systems at differentrelease levels, maybe they're building inside ofvirtual machines and their app's trapped inside of vmdk or it'sup on a qem ukvm image or it's on amazon with ami, soapplications really aren't that
lightweight and portable, andthat's a really powerful thing that eric was saying earlierabout containers and namely docker. it's really for the firsttime giving this industry this portability at theapplication level. and so when youthink about that, think about what that meansfor hybrid cloud consumption, because now we're actuallyhelping enterprises consume containers and they'regetting more agile on premise,
at the same time they'reactually able to use support hybrid clouds, and so moreoverwe really realize from our experience back in the borgthat it's not just about the containerization, it's reallyabout the management of thousands of containers and howdo you deal with rollout of new applications, and that waswhat kubernetes was all about. let's open source thebest of what we know, not just so thathackers can have it, so that large enterprises canconsume orchestration at scale,
and by -- by default be ableto support public cloud. so let me introduce now greg,and greg is actually going to take us through everythingthat we've been doing with both google stackdriver aswell as with containers. >> greg: thanks a lot, brian. i'm really happy to be hereto show you today the next generation of monitoring andlogging with google stackdriver. now, i cannot talk andtype at the same time, so i'm very grateful to havedan belcher out on stage here to
help me with this demo. now, you heard a lot aboutopenness being a critical piece of everything we do at google,and it's really no different than monitoring and logging. so to see that in action, let'stake a look at this stackdriver screen. i've got a stackdriverenvironment, and the first thing you'llnotice at the top is i'm monitoring multiplegcp projects.
that's a new capability,and it's really important, because whether you're anindividual developer or you're an operational specialist,you want to have a complete visibility across everythingthat's going in your organization. you don't want your monitoringsilo'd into different parts of your organization. down at the bottom, you'llnotice something that you probably might nothave expected to see,
and that is i have aset of aws accounts, stackdriver's design tolet you do monitoring, logging and alerting acrossboth google cloud and aws. one pane of glass,multiple clouds. so dan is going to switch usover to the main stackdriver console. for this applicationwe're running here, this is a multi-cloudapplication. i've got nginx, and i'm sendingtraffic to both aws and gcp.
so if you lookdown at the bottom, you'll see some request graphs,and you'll notice that we're currently sending about 90% ofthe traffic to aws and about 10% to google. now, i may be biased, butthat seems backwards to me. so i'm going to have dan hererun a script in the background that switches that and startsredistricting the traffic onto gcp. now, that takes a littlebit of time to propagate,
but that's okay, because itgives me a few minutes to show you some other newfeatures of stackdriver. the first thing i want to showyou is intelligent defaults. dan is drilled into a pagespecifically on our elb monitoring, and you'll noticei get a dashboard that's automatically showing me theright events and metrics. it's not showing the uselessthings like cpu load or memory load. it's showing me applicationlatency, 200 response codes.
same thing if i switchover to the nginx piece. i'm getting a set of chartsthat are giving me intelligent defaults. now, the key here iswhat you're not seeing. dan didn't have to go and tellstackdriver that elb is a load balancer, and thereforethese are the right metrics. he didn't have totell it what nginx is. stackdriver has intelligentdefaults for a huge range of both commercial andopen source libraries,
so you can just build yourapplication and you get the right dashboarding, the rightmetrics right out of the box. you can also createcustom dashboards. in this case, maybe i'man operational specialist, and i don't want to drill downinto the specifics of a service. i want to aggregate all of mynetwork traffic across all my services into one pane of glass. and of course, we can -- you cancustomize the user interface, the look and feel.
now, the next thing we want togo to is what happens when there is a problem? now, dashboards aren't enough. i need realtime information. i want to see what'shappening right this second. so we're going toswitch to logging, and that's another importantcapability of stackdriver. dan is going to go ahead and hitthe play button and we're going to get playing in realtimelogging information,
so as errors are loggedin either aws or gcp, i'm getting a viewhere of all of those. and in fact, there are a coupleof errors coming because we're doing the fail over from onecloud provider to another. you can also create metricsand alerts based on logs, so you can make sure that youget an alert when certain error messages appear inyour log systems. but we're actually going toleave that sort of for another day.
i'm not going to actuallycreate one of those right now. if i go back to themain monitoring console, we can see that -- herewe go, as the graph loads, we can see in fact the requestson the aws side are dropping, the requests on thegcp side are coming up. i've got no alerts. no errors otherthan that happening. everything is fine. so today stackdriver'savailable in beta.
it's a great monitoring solutionwhether you're using aws, whether you're using gcpor whether you're using a combination. if you think about what it wouldnormally take to do what i just showed you, we'd have beenrunning through multiple different tools forlogging and monitoring. we would have one tool foreach of our cloud providers. all to do that. stackdriver gives memetrics, logs, dashboards,
up-time monitoringall in one system. it also includes tracing,error reporting, debugging. we're going to have moresessions on that throughout the conference. i encourage you to go to those. so that's monitoringand logging. now, let's turn to the businessof deploying and running applications. you've heard a lot about therole of containers in this whole
notion of a wave 3 cloudapplication and how we know from our experience at googlethat they really are a key to managing systems at scale. this idea of moving from a worldof managing individual servers to managing applicationsand services. that's why we startedthe kubernetes project. that's also why we are invitingthe industry to be part of it, because for containers to takeon the real prominence that we think they're capable of, itwill take the whole industry to
come together tomake that happen. it's also why hybrid cloud wasa fundamental design point from kubernetes from the beginning toallow you to deploy containers on one cloud or another, withvirtualization, with bare metal, on prem, so ray isgoing to come up here, and we're going to giveyou a demo of containers, and we're going to start with ademo of container engine which is our fully hostedkubernetes environment. so kubernetes is designedto be declarative.
you don't deploy applicationswith this complex mess of scripting code thatruns at deployment time, and god help you if it fails andyou have to debug a partially deployed application. kubernetes isentirely descriptive, so ray's going to open upthe file that describes this particular application. i'm not going to drillinto the whole thing, but you'll notice we're sayingthat we need ten instances of a
particular back end server,we're defining what ports we need are open, and again what isrelevant here is what you don't see. i don't see heren 1 standard 8s, or machine sizes or memorysizes or disc sizes. kubernetes will just find theright capacity in my cluster and it will schedulethis job wherever. as a programmer, i just operateat the level of applications and services, not at a bunch ofinfrastructure that i really
don't actually carethat much about. so great, he's going to exit outof his editor and we're going to do a deployment, so he's goingto use the kubernetes command line to deploy thisapplication into gke. now usually at anevent like this, i would plead with you to notget your phone out and not do anything on the wi-fi, but we'regoing to do a little running with scissors here. i'd like you to actually sendsome load to this application
for us. there's a url thereon the screen. hcr.kas.io. really. no, i mean that, likeplease help me out, get your phones out, gonavigate to the site. while people do that -- andstart clicking that button. while you do that, we're goingto see the load go up on this application on the chart there.
i give you guys an a for effort. but nobody at google would call150 queries per second internet scale. luckily, that's what loadtesters are invented for, to give us sort ofa helping hand here. so ray's going to kick off aload tester that's going to send about 20,000 requests persecond to this service. and you'll see it startramping up, and sure enough, that's great, no problems.
now, if i did thisdemo 18 months ago, this would have been, you know,considered an amazing thing, like who has the ability tobuild services that run at that scale? and now it's myhello world demo. but what happens when you wantto update your application? now, you all knowthe right answer. you do a staged update. you do the 1%, 5%, 10%, youmonitor it very closely.
you hope you don'thave to roll back, but in reality that's a hardthing to do while you're serving 20,000 requests per second. kubernetes has built in theability to do automated scalable rollouts so it will just -- itwill slowly roll out updated versions of yourapplication over time. now, to do this right, i shouldmake this take, you know, a day or two. we're going to -- we're goingto go ahead and condense this,
so he's going to do arolling update command, and we're going to tell it toreally push this thing as fast as possible. so what you're going to see isdown there on the bottom of the chart, there's theversion numbers, and you're seeing as kubernetesis one by one replacing one instance of our applicationwith the updated version, so the green slowly shifts to-- the blue shifts to green. if we keep clicking the button,eventually you'll see -- you'll
experience the new version ofthe application and the button will change color. so there we are under a loadof 20,000 queries per second. we're doing a live rollingupdate of the application, and you'll notice what theqps really just stays very, very steady at about20,000 requests per second. our users never evennoticed we did an update. so that's -- go ahead, yes. so that's our applicationrunning on gke.
but we talked about hybrid cloudas being a key design goal, and one of the challenges ofhybrid is how do i bridge my on premise infrastructuredeployments and my new cloud infrastructure deployments? they have differenttools, different language, different ways of working. kubernetes can help with that. ray is going to switchover to another tab here, and this tab has the samesdks, same tools installed,
but instead of beingpointed at a gke cluster, it's being pointed at a clusterright over here on the right part of the stage. that server rack over there. our friends at intel havegraciously loaned us a rack of those aren't evenrunning virtualization. those are bare metal serverswith kubernetes running on them. they use the latest generationintel z on processors. kubernetes is totallyoptimized for that.
so now he's going touse that same command, and now the application is beingdeployed over there on to our rack. now if you go and see --look at the load graph, you will see that we've put aload balancer in front of gke and in front of our onpremise infrastructure. and you'll see seamlessly ourapplication is being deployed. the instances, now there's20 versions of my application running.
we're seamlesslysplitting the traffic, both serving about 20,000 qps. so there you go. five minutes. we took an application. we deployed it. we did a rolling update to it. we bridged that application toour on prem infrastructure all without having to deal withthe differences between the two
environments. thanks, ray, thatwas a great demo. >> you heard from a few of ourcustomers and we have more on the way. but while we're gettingthings set up on stage, we want to watch this shortvideo of what our customers are doing on gcp. >> cloud's fascinating, becausereally what it's done is it's completely changed thespeed of innovation.
>> from hosting websites computeengines to data prediction it offers the flexibilitythat companies need now. it's what makes cloud andgoogle cloud platform very, very important toour businesses. >> i grew up just absolutelyloving visual effects. when we started atomic fiction,one of the most challenging parts of doing visualeffects is rendering. it takes an immense amountof computing horsepower using google cloud platform, we havemassive infrastructure off site
when we need it and thennothing when we don't. it helped us put the resourceson the artist and making sure that every penny that our clientspent made it up on the viewing >> khan academy's mission is afree world class education for anyone, anywhere. google cloud platform allowsus to stay focused on learner outcomes because ourinfrastructure has become this invisible layer that we don'thave to worry about anymore. >> keeping track of that muchdata for millions of students
every day is achallenging problem. and storing all that data andbeing able to quickly act on it and look at it is reallycore to our mission. >> the name pocket gems reallyspeaks to what we do which is make beautiful thingsfor your mobile device. >> for war dragons in minutes orhours you might have hundreds of thousands on playersplaying your game. if you don't have theserver infrastructure, your servers are justgoing to explode.
google cloud platform allowsus to be able to scale automatically. you don't have tothink about that. your backend just works,it's almost magical. >> payments are the connectivetissue of economies and that really resonates withnomonin anywhere. we are providing payments inthe massive underserved markets. >> hosting our application ongoogle cloud platform gives us a much higher reliability thanwe could have gotten hosting
ourselves. >> we're really able to move theneedle and make those payments available andaffordable everywhere. >> the mission at spotifyis really about being a great musical companion and thenmaking it more personal. >> moving to google's cloudplatform enabled us to take use of the skill thatgoogle can provide. >> the reality is, it's quitehard to build great data centers and that was really the driverin to google's cloud platform.
the less time that we canspend solving problems that are already solved, the more timeand energy we can spend on turning our data into value. >> we wanted to make thismarketing campaign for coca-cola something that every singleperson could participate. submit your photo, you can be onthe flag and the opening of the world cup. by using google cloud platform,we went straight into building the top of it.
>> fundamentally it getsdown to one word, it's trust. is what's powering the cloudthat coca-cola is running on. >> well, i hope you enjoyedthat little sneak peek. sorry to cut it off, butwe got to get back inside. the cto and svp of walt disneycompany is on stage right now. >> whoa! >> so, we're really luckyto have mike white here. he's the cto ofdisney interactive. great video.
definitely aninnovation company. so how do you actually think ofthe role of technology and how it applies at creatingeven more innovation. >> yeah, it's ironic being herewhere technology is very front forward. as we look at our jobs andwhen disney is at its best, technology is alwaysin service in the story. it's always been a key aspectto the walt disney company's approach to story telling.
and over 100 --almost 100 years, we have had innovatorsthat have invented, reinvented industry afterindustry from travel to entertainment. so if it's our films, if it'sour parks, if it's our games, and if it's our products,technology is always in service of the story. it's always the scaffolding butthe story telling comes first. and that's what'sreally paramount to us.
and as technology becomes moreand more prevalent to everyone, it also becomes more powerful,it's super important that we have those tools and providethose tools to our story tellers such that they can providethe magic to our guests. >> so how does thecloud fit into that? how does google actuallyhelp you with all of that? >> so, yeah, a couple ofthings to provide context. so i look after consumerproducts and interactive media. we are the part that lives withthe guests in their day-to-day
lives. so we have 1.6 million products. we have a fabulous mobile gamesteam that publishes tons of games that people play millionsand millions of hours on. we also look atour disney stores. we have the largest withthe kids -- largest kid book publisher. and when you lookat our landscape, we need to be able to providemore and more emerging
technologies because of thosetouch points that we have with our guests. we look after the digital frontdoors and all of the social media as well. so with that, we need tobe hooked into the cloud. and we've been leveraging thecloud for sometime and cloud technologies for sometime, butnow we've partnered with google on some neat projects and it'sreally helped us get to where we need to go which was notworrying so much about what's
under the hood and having ourfolks focus on the product themselves and reallythe guest experience. >> and so obviously wethank you for picking gcp. so what's that experience beenlike working with the cloud team? >> well, it's been great. i mean, the first thingyou do, as you guys know, you take a veryscientific approach to it. you send in your most cynicalsoftware developer to go in and
meet with the googlefolks at the googleplex. and i'm sure you guys can closeyour eyes and picture that person in your head. we have them too. and so, go in there -- andreally what we were looking for was culture. you know, it'strendy to say, well, let's fail fast andbe able to experiment, but we truly believe in that.
john lassiter at pixar says, "bewrong quick." and so we really want to take that approach to that. we saw that with the google teamthat we could get up and running very quickly. the second thing welooked at was code wins. and so for us, that is a mantrathrough all of our developers when we say code wins and thatmeans can we get up in your environment? can we get going?
and i've spent my whole careerin silicon valley and a lot of the folks that, you know, workat disney you had a little peek behind the curtain of the folksthat provide the technology for that. have spent a lot of years inhigh-tech companies and they didn't come to disneyto build a data base. and so really what we wanted toget out of is the conversations of plumbing and conversationsof how do you spin things up and more into how do we create thesereally great product experiences
for our guests. >> so like in our pastconversations that was stunning, i don't know ifthe number was 700. the number of projectson app engine was huge. but what are some of the onesthat you're thinking about that actually you're most involvedwith and getting excited? >> yeah, so currently, we haveover 200 projects spun up. a lot of them arecoming out soon, we're super excited about that.
we're going to hold that backfor larger releases with that. but we're excited about that. we do have a product that i'msuper excited about that we launched called disney life. it was in the uk. it's an over the top servicethat not only does videos and movies and television, butalso books and music and games. and so that's been huge. and the partnership withgoogle has been very helpful.
in fact they even participatedin our war room to launch that out. so not just to be able to havethe ability to partner with someone but not think about thescale and knowing that they're going to be there with youto do that has really been instrumental to us. >> awesome, well,thanks for coming. >> yeah, thanks for having me. >> so mike and disney aren'tthat unique in that, you know,
they want to go faster. and we've been working harderto make all developers on app engine to be able tobe more productive. and one of the things that heldus back is to be honest was the run times and the languagesupport that we could bring to app engine. and so just the way we talkedabout like doctorate containers moving the industry forward,we've integrated containers now with app engine.
and what's that's meant is thatwe can now bring out new run times in that zero opsenvironment of app engine much quicker, so we can bring out thelatest version of java, node, and python. so here's -- greg's back andhe's going to show us a demo on what's happeningwith app engine. >> greg: thanks, brian. so next up, we're going to giveyou -- for the developers in the audience a lap aroundgoogle cloud platform.
and along the way i'm going toshow you a few of my favorite developer productivity features. so the scenario we'regoing to do this -- oh, and i've got bret here onstage is going to help me out. thank you very much, bret. we're going to show you anapplication we built at a recent hack-a-thon within google. the application is calledphoto scavenger hunt. it is a mobile game.
it gives you a set of four cluesfor things that it wants you to take pictures of. you take a picture. if it sees the object itasked you for in the picture, you get points. the architecture of theapplication is actually very simple. it is a mobile client that talksvia a rest api to a backend hosted on app engine.
that app engine uses thevision api to see what's in the picture, and it stores theclues in a mongo data base. now app engine has always hadsupport for languages like ruby and python and go. but today we're adding supportfor new languages like no js, and as i said, actually, ruby. so in this case, we've alreadygot the application a little bit of code. so why don't, bret,open up the text editor.
so we're going to highlight justa couple of lines of code here. the first is, there we go,the call to the vision api. you'll see that referenced down,one line down to the vision api. and we're using twofeatures of the vision api. we're using label detection totell us what's in the picture, and since i don't trustmy users entirely, we're using safe searchprotection to make sure there's no naughty bits comingup into our application. we're going to run this locally.
like many of you, our developerssometimes want to just work locally on their laptop. maybe they're on aplane, don't have access. so bret's going to fireup a local copy of node. and we've got this littleservice running locally now. now i got thispicture of a raccoon. so he's going to use curlto just test out the local application and see whathe gets back as a response. and sure enough, 95.7%confidence in an animal and all
the way down to itis a procyonidae, which i had to look that up. and that is the family thatraccoon belongs to in a -- so i actually learned something aboutbiology from our demo today. okay, so we got a localversion of the application. now let's deploy itup into the cloud. now this is where normally youget into the world of having to configure machines,memory, load balancers, what operatingsystem should i use?
is my image fully patched? not with app engine. i just let me justdeploy the code. so he's going to doa g-cloud deploy. now what's happening behindthe scenes here is kind of interesting. that source code is beingdeployed up into app engine where it's getting readyto be -- to be run. that's kind ofstandard app engine.
but what's really interestinghere is for new languages like node, ruby, and java 8, it'sbeing packaged up as a container and then it's being storedin a private google container registry for you. so that means i was ableto build a container-based application without even havingdocker installed on my local system. not only that, because it'sa containerized application, it can run anywhere inthe container ecosystem.
whatever provider you wanton prim or in a cloud. and, of course, it can beorchestrated by kubernetes just as well. so if you step back, whatthat means is that app engine developers can now be a fullparticipant in the exciting developments happening aroundcontainers and container management. so let's actually switchover to the console, to the cloud console.
and let's take a look andfind that image that just got uploaded into gcs. so bret's going to navigateit over to gcs storage, and i don't have permissionsto see the storage section. that's not an error. we've used identity and accessmanagement because i'm just a developer. i shouldn't have access to mycustomer's personal images, so we're using acls and identityand access management so that i
don't have access. now it turns out we have themanager logged in on a different tab. and here as the manager, bretcan go in and expand out the privileges for app engine adminand he could add me as a storage admin for all of those -all of those capabilities. now as the manager, maybei'm actually really paranoid. it's not paranoid, it's calledactually just being a good manager of security.
and what i want to do is seeeverything that's ever been changed about this project. the activity stream, newtoday, gives me just that. i can see every configuration,every policy change, every code push, everydeployment that happens to my application. this is really handy fortracking down compliance concerns or issues and it'sreally important for managers and organizations.
now while we're here, i want tohit you -- i want to show you two productivity features thatcan save you a lot of time. first, ssh from the browser. how many of you have ever hadto ssh into a vm and you have to deal with the hassleof picking a password, recording what the passwordis, configuring keys. not with google cloud. you can just -- bret can justone click on the right side on any vm and immediately havea secure ssh session to that
machine without havingto configure anything. one click secure access. the other time saveris the cloud shell. have you ever found yourselfneeding to do a quick bit of work but you don't have yourlaptop handy or you don't have a machine with an sdk installed? cloud shell gives you as adeveloper a private persistent linux instance that'spreinstalled in google cloud with the latestversion of the sdk.
it's like a gcp command linein the cloud you can access anywhere that you are. all you need is amachine with browser. the last thing i wantto show -- yes, i knew the developerswould like that. another hard part about buildingdistributed application is performance tracking. performance debugging when weused to run on one machine was actually pretty simple.
now individual traces arerouted to different machines. and how do you actually get aconsistent view of how long a request took? stack driver trace is adistributed tracing utility that lets you track down exactlywhere your server time is being spent. it gives you visualizationslike this where you can compare latency across versionsof your application. so you can see if your newversion is five times slower
than your old version. not only that, you can go downdown at the bottom and bret will click on an individual traceand you can see for a given user request exactly how long everycall took to service that request. so he picked a trace here and wesee there's a call to the vision api and the storage api. individual traces down to theindividual application level. okay, let's actually seethis application in action.
it's been deployed,and it's all set go. so at this point, bret needsto fire up the application. and let's take a look at --it's asking him for a person, a hotel, a bank note, or sports. bret, this is your moment. take a selfy on stage. all right, let's see whathappens with the vision api. checking? person, 92% likelihood.
thanks, bret, that was great. that was a great demo. so, yes, please, thank bret. so, if you look atthose two demos, what you're seeing is we'retrying to take you from a world where your developers spendtheir time mucking about with infrastructure and we're optinginto paths means opting into a walled garden of limited choicesto a world where your developers can just write code and yourpaths participants can be
connected to the wholeworld of containers. thanks very much. i'll bring brian back up now. >> thanks, greg. so some companies were born inthe cloud and that's the only world that they know. but then other companies,right, don't have that luxury. they're traditionalcompanies right, that are moving to cloud reallyrapidly and in so doing that are
actually becomingdigital businesses. but in either case, the mostsuccessful companies are the ones that are actually skilledin managing data scale. and gcp excels at storage andarchival but it's a lot more than that. it's really about pullinginsights from data. and that's what separates goodcompanies from great companies. because great companies know howto look at data and understand their markets,understand their users,
how people are usingtheir products; and they respond tothat really rapidly. so managing data scale isobviously one of the things that makes google google. we built data,search, analytics, and that's what powered sevenapplications of a billion users each. and we've actually been prettyopen on it publishing research papers on most of thetechnologies that you see.
and in the case of mapreduceand bigtable and dremel, just those research papers havespawned whole industries because they've been implementedin open source. so as a result, when peoplethink about big data, what do they think of first? they typically think ofopen source solutions. and so now with the cloud,our strategy is a little bit different. we're going to actually exposeall of those internal data
systems as first class cloudservices and that's what actually allowed us, right, thatprior work to create an industry first around on-line storage atthe durability and the price of archival. that's what's allowed us tobring out a full manage to-do platform but beyond that thework that we have done inside because the new projects weuse are typically using data streaming pipelines, we broughtout data flow to integrate well with a dupe as afully managed service.
and now with bigquery, evennon-googlers can process, you know, trillionrow tables in seconds. but as we do this, beingopen isn't optional. we really believe thatbusinesses are going to choose open as the nextplatform, right? this is not a pointproducts decision. this is about choosingyour platform. so we look for every way that wecan actually be even more open than we already are.
and a big part of that is whywhen we put h-space or bigtable on cloud, we didn'tuse our internal api. we adopted the opensource h-space api. and the same thingwith data flow. i mean data flow is reallycompelling and we could have used that to ourproprietary advantage. instead we said let's justopen source the whole model for configuring andexecuting these runners, these data stream pipelinesand creating apache incubator
project. so one part of the job that ithink is most fun besides like standing here, is actuallyworking with customers. and really workingwith customers, not just from asupport perspective, but on anengineering-on-engineering relationship. i mean, so 18 months ago, weactually embarked working with a new customer.
and it meant a lot of travelto new york and to sweden, and for them to mountain view. we built a really greatrelationship with spotify. >> the mission of spotify isreally about being a great musical companion andmaking it more personal. it's not going to be thisstreaming music experience or the online music experience,it's going to be your music >> we created the tools forpeople to actually swim in that sea of music data and findthe information that really
corresponds to whythey are there. >> quite a while ago, we madea decision to focus on data. >> depending on thecomplexity of the query, it could take a whole day. sometimes it wouldhave to run overnight. >> and being able to store thatand process that has been a very big challenge.moving to google's cloudplatform enabled us to take use >> the reality is it's quitehard to build great data centers and we saw an opportunity tospend our focus elsewhere.
and that was really the driverinto google's cloud platform. >> with google cloud tools, wecould re-run the same query and it might take a fewseconds, minutes. >> the less time that we canspend solving problems that are already solved, like scaling ourhadoop cluster when google has already figured out how toscale it for us, the more time and energy we canspend on turning our data into value. and that's really agame changer for us.
when you're talking about atechnology like bigquery and when you're talkingabout data flow, you will feel the spotifyproduct evolve making the user experience great. >> once you break down thedata, you realize that there are nuances, it's not astraightforward path. it's a mosaic rich beyondthe most popular artist. it's a lot more than that. >> music can transcendboundaries of language, culture.
if it moves you then it'sthere and it's there forever. >> so please --[ applause ] so please welcome nick harteau. the vp of infrastructureat spotify. >> how's it going? >> hey, nick.good. >> actually, before we get started with the questions, can we do another round ofapplause for brian and greg and the engineers who helped outfor the amazing live demos?
i know that it's not jumping out of airplanes with the vr goggles on. >> it's not in the script. >> no, it's not in the script. it's not jumping out ofairplanes with vr goggles on, but for me as an engineer,that was real bravery, the live demo stuff. >> well, thank you. >> you're welcome. >> so, really, it's beengreat working with spotify.
we've helped you but i have tosay that you've made us better today we actually handed you anew piece of code for connecting projects securely withouthaving public ips, so just another set of firstsbetween the two companies so thank you, for workingwith us closely. so, why did you decide to movesome of your platform to google cloud platform? >> well, i mean, goingback to your presentation, i think the thing that drewus to google over other cloud
providers actuallywas the data platform. at spotify, we are -- whetherit's like a/v testing product changes or building personalizedplay lists for millions of users every week, we are deeplydependent on data to understand our users and to bringthem great user experience. and google, as wesaw on your timeline, you guys have been a thoughtleader in the data space for as far back as i can remember. and i think you've done a greatjob of bringing that leadership
and that expertise to gcp inthe form of like great tools and services. so the data platform more thananything else was the game changer for us. >> so we know some of the engineis already moved on top of the cloud. so just like how'sthe migration going? any challenges you'vehad along the way? >> yeah, i mean we have a prettybig and complicated backend.
i think we have 250 or soservices that power spotify. so there's not a simpleanswer to that question. you know, some ofit is pretty easy, other stuff is pretty hard. but you know, overall, i wouldsay it's going better than we expected. we have our set of challenges. you know we're figuring outsome stuff related to time syncrhonization.
we are -- we're learning a bunchof new api semantics in adopting the platform. but overall, it'sgoing pretty well. besides the usualstuff about, you know, changing the tires on the carwhile we're driving down the highway. >> easy stuff. >> yeah. >> so we loved the tweets.so we -- definitely that person is a --
>> i think he's pretty happy about not having -- >> he's inaugarated into our marketing hall of fame. so we thank him for that. so what else are you sortof hearing from just the experience? what services are kind ofthe most delightful for your engineers? >> well, there's two thingsi'd want to call out on the migration before we go intothe services side of it.
i would say one thing is -- onething that we're like really concerned about andreally, you know, at the top of our minds in termsof moving to the cloud has been performance. we're a super latencysensitive application. we rely very much on our usersfeeling like they have the world's music like rightthere on their phones. so we've been really watchingthe metrics very carefully as we've been movinginto the cloud.
and we've been reallyhappy with that. the numbers that we're seeing interms of performance are as good as what we can deliverfrom our own data centers. >> cool. >> the other thing to patyou on the back a little bit. one of the big unknowns forus going into the google partnership actually wasthe customer service side. like as i said, it's a big andcomplicated migration and i knew that we would need to rely a loton support from google in order
to do it well. we've really beenblown away there. so thank you for the greatpartnership and the great support. >> awesome, great, thank you. >> one ofspotify's challenges was, can you imagine, is that as bigas that service is is just the pure ingestion ratethat they have. and moving from kafka andtaking advantage of pubsub?
so the ingestionrate is amazing. running services atthat scale is amazing. so we actually wanted to giveyou a demo and show you a version of theseservices in action. >> okay, one last time with me. so, i think nick mentioned this. but at its peak, spotify isserving something like a billion streams. i need a slide update here.
i've got eric schmitt on stageto help me with this demo. not the eric schmittfrom earlier, but a slightlydifferent eric schmitt. you can imagine the funcrossings of e-mail that they have. yeah, exactly. the first step we want to --we want to start with here is ingestion. one of the challenges thatspotify has is how to take data
and bring it in at the kindof rates that they deal with. they use cloud pubsubin order to do that. cloud pubsub is a fully managedservice for ingesting data at now what we're going to show youhere is the same sort of tools and give you a glimpse of howspotify analyzes their data. now, spotify actually treatstheir customer data and privacy very, very securely. so we have generated some testdata of the same volume and the same shape that spotifyactually encounters.
so what you see here may notexactly match what spotify sees in production. so let's start with ingestion. this pipeline is currentlyrunning at about 150,000 to 200,000 events per second. now this actuallyis running now. and throughout the demo,while we do this demo, we're going to continuously beingesting more user events at about 100,000, 200,000events per second.
they can get up to 700,000. and what these events are areall of the interactions that people do with their mobileapplication on spotify. you hit play, you pause,you skip, you share, you make a play list. that data is gold for spotifyin terms of building a better service or building newfeatures like discovery weekly. and with pubsub, they caningest that data at scale, at volume without havingto manage infrastructure.
without having to maintaintheir own kafka clusters. so pubsub is globallyavailable, fully managed. it scales from a few eventsper section up to hundreds of thousands. now the next step after thedata has been ingested is you actually need to extract thatdata and transform it into the various storage systems. so eric's going to switch overand we're going to show data flow.
spotify is actively working onmoving to data flow as their centralized etoprocessing system. this is a dataflow visualization, and it's runninglive as we talk now. and it's takingdata, sharding it, windowing it by hour so thatthey get hour-by-hour updates to the data and then ultimatelywriting that data into multiple data sources, bigquery, gcs, htfs. and it does all this witha fraction of the latency.
what used to take an hournow takes 15 minutes. so they're able to get muchcloser to realtime access to how users are interactingwith their application. so that's our pipelinerunning, and, again, this is running livewhile we're on stage. we can ingest the data,we can run a pipeline, we can store it inthe different systems. now we actually want to findthe needles in the haystack. we want to analyze.
and that's wherebigquery comes in. eric's written a query here thatis going to calculate the top artists that peopleare streaming. now, bear in mind, this is testdata so it may not match what billboard says theactual top artists are. so he's going to goahead and run this query. and while it runs, let me tellyou that this query is going to join over a billion events ofuser interaction with over 20 million tracks of musicwith over a million artists.
and in four seconds, itprocesses 22 gigs of data to come up with that -- justinbieber is the top artist. well, the demo is bieberliciousso far, let's plow on. but again, what's valuablehere is what you're not seeing. there's no recreating indexeswhen the data goes up by a factor of ten. there's no maintaining vms,there's no patching operating they just get to write queriesand get data on the other side. great, so now we'velooked at the data.
let's take a look at howspotify might be able to do collaborative analysis with aproduct like cloud data lab. cloud data lab gives youthese nice visual notebooks. it's based on jupiter, the opensource project formerly known as ipython. and it allows you to combine notjust bigquery data but because it's based on python, youactually can use really popular packages like pandasto do analytics. so at the top, you see that-- let's just scroll back --
there's that same bigqueryquery that we just used in the but if you go downto the bottom, we're piping that into a pandasdata frame in order to be able to do visualizations andcharts right within pandas. so we made a dataframe out of it. we asked pandas to plot it. and you see we've got apretty typical fat head, long tail distribution of music. and that got ericand i thinking.
you know, maybe there's anothermore valuable way to look at this than just top plays. maybe we should do furtheranalysis and take into account the number of tracksthat you have, not just the number of plays. data lab allows me to do thissort of interactive what-if analysis in a visual notebookway that could be shared with other users. so we've added more on -- we'veextended this bigquery to
calculate what we call howinteresting an artist is by taking into account howmany tracks there are. so he's going to run the restof our analysis through to completion. and this query -- thisanalysis will be running live. again, data is being streamed,data flow is continuing to pump that data in. we're getting realtime looks. and by this, it looks likenirvana and torii amos and bob
dylan are more interesting thanjustin bieber so all is right with the world. so, one last thing. what about summarizing the dataand building dashboards for your managers to get realtimelive information? data studio 360 is a new servicethat allows you to do just that. to create beautifuldashboard visualizations. and it works not only with google cloud services like bigquery, but it also works withgoogle analytics so you can
combine your business datawith your web traffic data. and it works with google sheetsso if you have spreadsheets with ad hoc data, all of that canbe brought into data studio. so here's our versionof the dashboard here. and you'll notice it's showingyou the number of events, the number of unique users,the number of unique tracks. we've got some very nicevisualizations of charts about where -- whether somebodyis searching or browsing. so, we're going to extend this-- this dashboard and get some
access to geographicinformation. because one of the things weknow about users is when they registered, what citydid they register. so eric is going to drop a mapcontrol onto our dashboard. great. and he's going to connectthat map up to two pieces of information. he's going to tell it --he's going to tell it -- proof thatthese are live demos.
okay, got an issue. what you would have seen is hewould connect that data up to our user information, be ableto identify the city and state. oh, there it goes. see, i just neededto be patient. connect it up to the user. he's going to tell it thatthe information that we're interested in looking at is thecity that the user is registered in and that the data we wantis the number of streams that
they've listened to. and then as he does this, you'llsee the data start to propagate in in real time with a map. so it automatically showedme a map of the u.s. the circles start to bubble in. he can drill down and say yeah,i really care about california. and, again, he didn't haveto build that, all built in. so there you have, ingestion,processing, analysis, a map dashboarding.
>> thanks, eric.that was a great demo. so, again, we're trying to takeyou from a world where your data scientists have to babysitinfrastructure and spend 80% of their time on that instead of oninsights to a world where your data people can justget the insights. i'll turn it back over to brian. >> so, pretty exciting. for the past few years we'vebeen working on sort of what's next.
and for us that was, can youactually start with raw data and get insights out of raw data? so can you look at just pixelsand understand what the object is? can you look inside -- teach acomputer to actually listen to just sounds, raw sounds, andunderstand what the spoken word or can you have a computer studythe game of go and actally be able to beat the world champion? so two or three years ago, theanswer to all three of those
would have been no. but now we can actually say yes. so at google, we've been rapidlybuilding machine learning into all of our products. and that's why android voicerecognition has gotten so good. that's why google translate cantranslate over 100 languages. and that's why, you know, you'reactually seeing what we're doing with google photos where you canrecognize now it's upwards of tens of thousands of objects.
so on gcp, we're going to giveyou that very same machine learning technology sothat you can use it, right? so whether you want to usethe entire solutions around vision-speech translate. or whether you actually wantto train your own models using tensor flow. so one of the verysame googlers, right, that started this whole dupecraze by becoming one of the first authors to map reduce isactually leading our machine
learning efforts. so my pleasure to welcome jeffdean, the lead of googlebrain. >> all right, thanks, brian. so i think machine learningis actually one of the most important things that'shappening in computing. i think it's comparable inimportance to the rise of the personal computer, with theinternet, or cloud computing, or mobile devices. it's really going to transformnot just computer science,
but lots and lotsof other industries. and for us at google, we've beenusing deep learning for the last several years. and it's really kind of thesecret sauce we've been putting behind more and more products. you can see kind of ourtrajectory of incorporating it into more and moredifferent kinds of areas. and the breadth of differentareas that this kind of technology is applicable to ispretty surprising and startling.
and when we started out in 2012using deep learning in some of our products, it was complicatedand kind of difficult. and we built some infrastructuresystems to make it a bit eaiser. we're now on our secondgeneration system and we've made it a lot easier. and that's why you see thatrise in the different teams, that have been able to applythis to their products. so one of the things that canmake it easy is perhaps you want to use pretrained models thatjust work for a variety of
different kinds of domains andyou want to be able to use them through very simple apis. so we've had the cloud translateapi for quite a while now. it allows you to basically givetext in one language and we give you back correspondingtext in another language. we think it's the besttranslate service in the world. it's very simple to use. and people have likedthat for quite a while. more recently, we introducedthe cloud vision api, which is,
again, a very simple api to use. you give an image, we give youback all kinds of good stuff. so, you know, this is the kindof information we can give you from the raw pixels of an image. so let me give you a demo justover the kinds of stuff you can get from cloud vision. so if we go here -- it wouldbe more helpful if my demo was up. let's see.
uh-oh. there we go, sorry. what we've done here iswe've taken about 100,000 images and run them allthrough the cloud vision api. and so each vertex in thiscluster graph is essentially one image. and if i zoom in here, youknow, you see images flying by. let's just kind ofpick one of them. i have a precannedone, because he's cute.
a little doggy. so if you run thisimage through, this is the kind of data thatyou can get from the cloud vision api. so let's just walk through a fewof the different kinds of data you might be able toextract for images. we have another cuteanimal, that's the internet. so we have a cat. we tell you it's a cat.
but more importantly, we tellyou it's a british short hair and maybe a semilong hair. so we can give you pretty finegrain information about what -- about what this kindof image contains. another thing that's prettyuseful for images is, you know, the visual world oftenhas lots of text in it. and traditionally, it's been abig barrier to using text that you see in images because youcan't actually extract it very well.
but the vision api can actuallyjust give you the ascii text we find in the image. and, you know, when you make thecall, you'll get back, you know, a very structured kind of thinghere where we tell you the labels and there's the actualtext for the thing as an ascii string and you can dowhatever you want with that. faces are another kind ofimportant thing for photographs that people take. so it's actually pretty nicethat we can identify where faces
are in the images. we can tell you if a particularface is happy or sad. you know, some of thepeople here are happy. not everyone. and again, that comes back aslike pretty structured j-sign another kind of important thingis people often take photographs of sort of popularlandmarks around the world. and so, you knowm, here'sa picture of a stadium, we'd like to know where that isso we can click on the landmark
detection facility and thattells us it's citi field, stadium, gives us thelongitude, latitude. that turns out to be in new yorkbecause that's where the mets play. and finally, sometimes it'suseful to be able to look at a logo or an image and understandwhat logos appear in it. and so we can tell you that thatimage is the android 2.2 logo. so that's kind of a tour of thekinds of things you can do with the vision api.
if you have lots of visualimagery, you can run it through, you know, vision api,cluster it, manipulate it, extract information from it. and we think that's a reallypowerful kind of concept for a lot of kinds of thingsyou might want to do. we're not going tostop at vision, though. we've had a really good speechrecognition system for quite a while on our android phones andwe're now exposing this as an additional api.
it's again very simple to use. you essentially just give us anaudio wav form and we give you back a transcript ofwhat is being said. and it works in, you know,dozens of different languages, actually more than 50 languages,and it basically allows you to convert audio to text usingmachine learning under the covers. one of the really nice thingsis it can work in both a batch mode, if you just have a lotof text or audio that's already
been recorded, but it couldalso work in a streaming mode. and so someone could bespeaking into a phone, you send that to the cloud andwe give you back a transcript in real time. so we think that'spretty powerful. so those pretrain models are areally good way to get started using the power of machinelearning and building insights into the data youmight already have. let me show you now,to switch gears a bit,
let's talk about how you mightwant to sort of design and train your own machine learn models. so we've been doing this overthe last five or six years. now, tensorflow is a system thatwe open sourced in november of last year. it's already the number onerepository for github for machine learning in five months. so we're pretty happy with that. the external community hasbeen very receptive to this.
you know, i think one of thenice things about tensorflow's release was we put together abunch of tutorials that come with it that show you how to dodifferent kinds of machine learning tasks. and that's been prettywell receive i think. so we originally designed itas a system and platform to be flexible enough for trying lotsof different kinds of machine learning ideas and research. but also then be robust enoughto take it and then scale those ideas to real world data setsizes and then move them to
deployment in real productsin the same system. so we want really easy andflexible ways of expressing machine learning models and thenmove them to production really easily. so, the outside reception to thetensorflow open source project has been really great. there's already 800 repositorieson get help alone that use tensorflow in some way.i'll just show you one example. so there's a really cool paperfrom a couple of universities in
germany where you can give animage and a painting and then it will render that image inthe style of that painter. so this is vangogh's starry night. plus that image ofthe boston skyline, and it's rendered in the styleof van gogh but it's the boston skyline, which presumablyvan gogh didn't get to see. and that's allexpressed in tensorflow. there's an easy repository thatyou can download it and use. so the power of tensorflow isthat people can use it to do all
kinds of different thingsand people already are. so one of the things thatwe found is that it's really important to bring the power ofmachine learning to all kinds of different users bothinternally and now externally. so i'm really happy to announce that we're introducing a cloud machine learning productbased around tensorflow. so really this product willallow you to understand your data, visualize it, prepareit for machine learning, using the other tools thatalready exist in our cloud,
and then you can use the powerof our system to solve your problems and your particularcontacts or problems. so, and then we allow you totake those models that you've developed and deploythem at scale. so enough talking about that. let's actually see how thismight work in practice with a demo. so if we go back over here. what i have here is, again,another data lab notebook.
so the data lab notebookstyle is a really easy way to collaborate on having multiplepeople work on trying to figure out a good machine learningmodel for a particular problem you might have. what we've done here is put inthis notebook we have downloaded some data from our criteokaggle competition. so this is just a data setof 10 gigabites in size. 45 million records. these are a few ofthe records there.
and what you see is it has abunch of integer fields plus some string fields. these are actually hashedfor anonymization for the competition. but these might be things likethe city the user was in or something. and then we're going to tryto predict whether or not this particular set of othervariables resulted in a click or not.
so our goal was to take all ofthe other data in the record and predict is this a click or not? and you can imagine, you know,that kind of scenario is pretty common in a lot of settingswhere you might have customers and you want to predict ifthey're going to buy something this month or you might havepatients and predict if they might want follow-upvisits or something. so, one of the things in datalab is that it allows you to really easily get a sense ofthe kind of data that exists.
you know, you can get histogramsfor each of the fields in your data set. you can understand which onesmight have missing values and so on. but integrated into this, wesat down -- and a few machine learning colleagues and i satdown on a whiteboard and we designed a model that we want totry for this particular problem. so in this particular case,we're going to use enrollment, because those work prettywell for a lot of problems.
and we're going to takethe string data here, put it through an embedding,take the numeric data, the integer of that field,concatenate all of that data together as a set of featuresand then put a single layer neural network followed by asingle logistic unit at the top to predict yes or no for click. so that's our training label. and then we're going to have anoptimizer that's going to try to optimize this path.
so that's kind of what we cameup with at the white board. this, i won't delve in too much,but this is kind of the code you would write in a tensorflowto express such a model. so, essentially,you have a graph. we're going to have some routinein this little bit of code that creates input to read therecords from the training data set. we're going to create thosehidden layers i talked about with the rectified linear unitsand then we're going to create
our final logistic unit andthen create a training procedure that's going to optimize theloss of that logistic unit. so this is like one of thoseroutines, create a hidden layer. so you get the raw sense. you know, it's nottwo lines of code, but it's not so complicated. so one of the great things aboutthis being tensorflow is that you can actually run thislocally on your laptop. so many of youprobably flew here,
and you might not have internetaccess on the plane home. so you can actually just runyour tensorflow model locally. so you can -- that's thecommand you would run. and then after a little while,we will have a trained model. and this is a graphof training over time. so this is the number ofseconds that our model has been training. the blue line is our losson the training data set. and the yellow line is theerror on the validation side.
and we want theerror to go down. and sure enough at the beginningof training, it was not so good, and then it's graduallybeen creeping down. and obviously being able topredict this metric better could be quite important in theparticular application you might have in mind. so that's great. and after about 30,000 seconds,we got down to an error of .304. so, great.
so let me show you how youmake this use our cloud machine learning product. you take that command linebefore and you add dash, dash cloud. and you can alsooptionally tell us hey, how many replicas do youwant to use for this model. so in this case, we want to use20 replicas so we're going to get 20 way parallelism inoptimizing this particular model.
and now, we get pretty muchthe same looking curve, which is good. and it goes down to about, what is that, .306, 304 at the end. i'll show you acomparative graph. but it did it in about .19 seconds. so that's the difference betweenkind of a full day before you kind of get the answer to yourexperiment versus, you know, 20 minutes, 30minutes, you know, you go get a cup of coffee, comeback, and now you can iterate.
the same way the spotify teamwas talking about how they get much faster itteration and areable to sort of explore their data and make insightsmuch more quickly, this is the same kind ofthing for machine learning. and here you can see graphsof the local processing and instances per second. a 10 replica version in yellowand a 20 replica version in green. and just to kind of show you,this is the magnitude that you
get in improvement of errorover time if you use the single replica versus the 10replica and the 20 replica. and if i switch this to statswhich is a kind of update to the model, you see they'realmost on top of each other. it just happens20 times as fast. so we now have a trained model. what do we want to do with it? well, one thing we might want todo is actually deploy that to a production setting and seedo we actually observe real
improvements. so what i have here is anopen source load testing tool, and it's pointed at makingrequests to get predictions to one of the modelsthat we trained. and you see we're getting about-- we're simulating 100 users kind of concurrently using someservice and we're doing about 12 requests per second. so one of the real nice thingsis to go from, you know, the idea you had to a deployedmodel and then if it's
successful, to not have toworry about scaling anything. you don't want to haveany sort of you know, ops required to add capacity. so let's simulate 100times as many users now. and i'm just going to hitreload on the refresh tool. and all of a sudden,we're hatching new users, apparently they're --they need hatching. so what you see is the latencyis hovering pretty steadily. requests per second are rampingup, and all of a sudden,
you know, in maybe 10or 15 more seconds, we're going to go from low-x to100-x without any work on our part required. and that's kind of theexperience that we think you need for getting machinelearning and trying out lots of ideas, iteratingquickly on the ideas, and then taking that idea andmoving it into a production setting withoutany muss or fuss. so that's what we have there.
so we think that's going tobe pretty exciting for lots of people. and we need kind of earlyusers of this system to give us feedback so we can get it tobe really robust and solid, and even easier to use whenwe go to general availability. but it's in alpha as of today. so we have ready to use machinelearn models, translate, vision, and the cloud speech ati. this is an area we'llbe building out more of.
we think there's a lot more thatcan be done in this particular area. and then what i showed you todayis our cloud machine learning product, which works incollaboration with all of our other products where you'realready storing your data in our you want to start doing somedata lab exploration and then you want to trainmachine learning models. and using tensorflow, you're notlocked into our cloud platform. you can take that same model,run it on your desktop or on
your own premises, deployit on your own premises, deploy it in our cloud,it's very flexible, and allows you to really movehowever -- decide how you want to use this thingin your business. so we're reallyexcited about this. so now the kind of tools wegive you are really the kinds of things that allow you to dobasic things like store and analyze the data. these kinds of tools are reallygoing to be allowing you to get
an understanding of your dataand really give you the insights you need to build theproducts you want. so, thank you. >> and i'll welcome back brian. >> thanks, jeff. so first now we can say allof those demos were live. i didn't want to say that earlybecause you know what happens if you tell you areshowing live demos. and that was because, you know,the engineers and the product
managers, they workfor months on these. and they really want you to seethe technology and not just a lot of presentations. so thanks to all of the teams. so, as you can see,google cloud platform isn't this single service. in our opinion, it's acollection of beautiful and well integrated apis that let youdo the traditional things like spinning up the vm and giveme the os that i care about,
or they're reallyadvanced things. give me a zero-ops auto scalerand spread my data across five regions. but it's also the place wheresecurity and operational control come together so that the mostrisk-averse of enterprises can put the first application ontop of cloud or integrate our network directly intotheir own premise. and it's also the first placethat new technology is being born.
things like deep learning andput directly in the hands of developers that never wouldhave been able to access that technology in the past. so we really want you to takeadvantage of the next two days, right. so we have some greatsessions for you. we've got an eveningevent tonight. tomorrow we have -- inthe general session, we have three talks.
you're going to hear from theperson that leads all of our data centers,really fascinating. you're going to hear the personthat's going to tell you where the tl for ware containers andwhere this is all going as well as from niels on what'shappening inside of google but most importantly as well,like we've sent a lot of engineers here that aren'tgiving formal presentations. so really take advantage of thehallway conversations, right? meet the engineerslet the -- you know,
have some conversationsaround that. and there's developers fromother companies that have come here all over the world. so it's really allabout the conversations.
0 comments:
Post a Comment