There are three types of crawling, all of which provide useful data. Internet-wide crawlers are for large-scale link indexing. It's a complicated and often expensive process but, as with social listening, the goal is for SEO experts, business analysts, and entrepreneurs to be able to map how websites link to one another and extrapolate larger SEO trends and growth opportunities. Crawling tools generally do this with automated bots continuously scanning the web. As is the case with most of these SEO tools, many businesses use internal reporting features in tandem with integrated business intelligence (BI) tools to identify even deeper data insights. Ahrefs and Majestic are the two clear leaders in this type of crawling. They have invested more than a decade's worth of time and resources, compiling and indexing millions and billions, respectively, of crawled domains and pages.
I think ranking in organic listings is a lot about trusted links making trusted pages rank, making trusted links making trusted pages rank ad nauseam for various keywords. Some pages can pass trust to another site; some pages cannot. Some links can. Some cannot. Some links are trusted enough to pass ranking signals to another page. Some are not. YOU NEED LINKS FROM TRUSTED PAGES IF YOU WANT TO RANK AND AVOID PENALTIES & FILTERS.
Cross-platform measurement: The number of marketing channels continues to expand, as measurement practices are growing in complexity. A cross-platform view must be used to unify audience measurement and media planning. Market researchers need to understand how the Omni-channel affects consumer's behaviour, although when advertisements are on a consumer's device this does not get measured. Significant aspects to cross-platform measurement involves deduplication and understanding that you have reached an incremental level with another platform, rather than delivering more impressions against people that have previously been reached (Whiteside, 2016).[43] An example is ‘ESPN and comScore partnered on Project Blueprint discovering the sports broadcaster achieved a 21% increase in unduplicated daily reach thanks to digital advertising’ (Whiteside, 2016).[43] Television and radio industries are the electronic media, which competes with digital and other technological advertising. Yet television advertising is not directly competing with online digital advertising due to being able to cross platform with digital technology. Radio also gains power through cross platforms, in online streaming content. Television and radio continue to persuade and affect the audience, across multiple platforms (Fill, Hughes, & De Franceso, 2013).[46]

Over the past few days or so, I have been seeing chatter both at WebmasterWorld and Black Hat World around ranking and traffic changes related to Google organic search. I got to say, I doubt Google pushed anything specific this week or even next week. Rankings do change by itself and new content is always being added to the index but an algorithm update - I doubt it.