WO2014089417A2 - Physical context and cookies - Google Patents

Physical context and cookies Download PDF

Info

Publication number
WO2014089417A2
WO2014089417A2 PCT/US2013/073541 US2013073541W WO2014089417A2 WO 2014089417 A2 WO2014089417 A2 WO 2014089417A2 US 2013073541 W US2013073541 W US 2013073541W WO 2014089417 A2 WO2014089417 A2 WO 2014089417A2
Authority
WO
WIPO (PCT)
Prior art keywords
user
information
data
cookie
shopper
Prior art date
Application number
PCT/US2013/073541
Other languages
French (fr)
Other versions
WO2014089417A3 (en
Inventor
Tony F. Rodriguez
Mark J. PETRIE
Original Assignee
Digimarc Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Digimarc Corporation filed Critical Digimarc Corporation
Publication of WO2014089417A2 publication Critical patent/WO2014089417A2/en
Publication of WO2014089417A3 publication Critical patent/WO2014089417A3/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0241—Advertisements
    • G06Q30/0251—Targeted advertisements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0241—Advertisements
    • G06Q30/0251—Targeted advertisements
    • G06Q30/0255—Targeted advertisements based on user history

Definitions

  • DSPs Demand Side Platforms
  • a company using a Demand Side Platform provides information about the target audience to which its ads should be directed, and a budget (e.g., daily or weekly) for the ad spend.
  • the DSP software spends the budget to buy ad space on online sites where it determines the company's ads will yield the best return.
  • SSPs are software tools used by online publishers who have ad space to fill (e.g., seattletimes ⁇ dot>com).
  • the SSP software discerns information about a user who visits the web site (typically through use of cookie data), and determines which ad should fill an available ad slot in the web page delivered for that user's visit.
  • the SSP software In placing third party ads, the SSP software typically conducts a quick online auction to identify the vendor willing to pay the most.
  • SSP software is also used by retailers in placing ads promoting their own
  • Joe Public requests a page be loaded from seattletimes ⁇ dot>com
  • a cookie on his computer is read and allows access to an associated dossier of information.
  • This dossier - typically a file in a remote database maintained by an SSP vendor - contains history data about Joe's other online activities/experiences. It may also include other demographic data, from public and proprietary databases, etc.
  • This information about Joe prompts a flurry of activity to fill an available slot on the seattletimes ⁇ dot>com web page with an ad from a brand that wants to tempt Joe.
  • the cookie is a gateway to stored context data about Joe, allowing the SSP to identify an advertiser to whom Joe represents a high-value customer. Naturally, the SSP wants to sell the ad slot for the best available price. The more candidate advertisers know about Joe, the more confident they can be in determining whether Joe is a close match to their target customer. The more advertisers know about Joe, the more confident they can be in paying top dollar to present Joe their ads.
  • a cookie assigns an identity to a user, and allows access to information about the user's activities. But these activities are always digital.
  • certain analog activities of the user are also memorialized, and help identify particular ads best suited for presentation to that user.
  • Fig. 1 is a flow chart of an illustrative embodiment, from the viewpoint of a user device.
  • Fig. 2 is a flow chart of an illustrative embodiment, from the viewpoint of a remote service.
  • Fig. 3 is a diagram of another illustrative embodiment.
  • Fig. 4 is a block diagram showing components of an illustrative system.
  • a first embodiment involves a smartphone or tablet "second screen" app.
  • apps are often used in conjunction with television programs (and some radio broadcasts) to present auxiliary content that is complimentary to the primary television (radio) content.
  • auxiliary content that is complimentary to the primary television (radio) content.
  • such an app may present player stats to viewers of a football game, or present trivia quizzes to viewers watching a sitcom.
  • Shazam, Zeebox and IntoNow are exemplary second screen apps.
  • Other second screen apps are more specialized, sometimes being tailored to a particular television show (e.g., Grey's Anatomy or NCIS), or to a specific broadcaster.
  • Second screen apps typically identify the primary content to which the user is being exposed through use of audio watermarking or fingerprinting technology. These techniques process sampled content to derive corresponding identification information. Alternatively, program identification information can be broadcast (e.g., by Apple's Bonjour service) on a local wireless network to the second screen app from the television system, or from another device involved in delivering the primary content to the user.
  • Apple's Bonjour service e.g., by Apple's Bonjour service
  • second screen apps complement such content by identifying one or more corresponding items of secondary content from which the user can choose.
  • These secondary content items are typically identified by querying a database using the program identification information. Sometimes, the secondary content items are identified (or conveyed) with the primary content stream.
  • KXYZ-FM that distributes a tablet app with second screen features (somewhat a misnomer in the case of radio, since there is no primary "screen”).
  • an audio fingerprint or watermark detector e.g., JavaScript
  • This detector listens to ambient audio, and seeks to identify it (e.g., by computing fingerprint data, and forwarding to a database for matching against a collection of fingerprints for reference content). For example, ambient audio in the user's environment may be identified as a Bob Dylan song (e.g., "Po' Boy").
  • a database of secondary content is next consulted and may indicate, e.g., that the app should respond by presenting an on-screen quiz testing the user's knowledge of Bob Dylan trivia.
  • the app may successfully identify the primary content being rendered in the user's environment (e.g., a song by The Eagles, or audio from local television channel 6), but find that the secondary content database does not identify any secondary content that corresponds.)
  • the primary content e.g., a song by The Eagles, or audio from local television channel 6
  • the secondary content database does not identify any secondary content that corresponds.
  • the tablet app When the tablet app transmits detected watermark or fingerprint data (or other identification information) to a remote server for response (if any), it also transmits an identifier of the tablet - or its user. Commonly, but not necessarily, this identifier takes the form of a cookie.
  • KXYZ-FM uses a Supply Side Platform from SuperCo
  • the tablet's data exchange can be arranged so that a 'superco' cookie is transmitted from the tablet to the SuperCo server.
  • An exemplary cookie comprises a small file containing random data, e.g., A86FC2, with a descriptive name, e.g., superco.txt. The random data allows cookies from different users to be distinguished.
  • this information can comprise a song title or other metadata identified by looking-up a decoded watermark identifier in a metadata database (Bob Dylan's "Like a Rolling Stone;” The Eagles' "Hotel California”), or it can be title or other metadata information for a television program, identified by matching derived fingerprint data against reference data in a fingerprint database (e.g., the October 7, 2012, 60 Minutes broadcast).
  • the identification information can comprise a TV program title as broadcast by the television system, etc. (It may also comprise the derived audio fingerprint data, or extracted watermark ID, although this is less common.)
  • This identifying information is sent to SuperCo by the tablet (e.g., the tablet app), or by KXYZ-FM, or by a third party involved in associating identification information with metadata.
  • the identified content not only is the identified content associated with the cookie; so is the delivery channel by which the content was delivered to the user.
  • fingerprinting may be employed to identify the title of the content (e.g., Seinfeld, episode 125), and watermark decoding can reveal its distribution channel (e.g., broadcast by KABC-TV, on October 5, 2012, at 10:00 pm).
  • a broadcast from a television system may indicate that the television is tuned to Cox Cable channel 47.
  • information that identifies the content recognition software or second screen app on the user's tablet can also be sent and stored in association with the cookie. All of this information is context data - useful, e.g., in identifying advertising that is relevant to the user.
  • a small payment may be due to KXYZ-FM (or a larger payment, if the user actually completes a purchase of the Dylan boxed set). Additionally or alternatively, if a watermark or other information identifies the distribution channel through which the sensed Dylan audio was provided to the user's environment, that entity may also deserve a tribute payment.
  • the system can swap-out the Dylan ad and replace it with an advertisement for Mad Men-related merchandise available on Amazon.
  • Such updating of ads in real time is facilitated by web 3.0 standards, which enable ongoing communication between a browser and a web page server.
  • Knowledge about the user's instantaneous environment allows still better tailoring of promotional content - better suiting the user, the advertiser, and the supply side provider.
  • the context information that can be fed-back to the web page server (or other destinations) is not limited to information about media content in the user's environment.
  • Information about mouse movement or other user interaction can also be relayed from the user's device, and serves to signal that the user is active on the web page - and is not off in the kitchen, etc. Again, more accurate characterization of the user and his context leads advertisers to bid higher for available ad slots.
  • cookies historically have been used to make assertions about users' digital activities, e.g., User A visited web sites X, Y and Z; User A clicked on a news article about Lance Armstrong and on an online catalog item offering a Shimano derailleur.
  • the present technology extends this capability to aspects of the user's physical world, e.g.: User B listened to Bob Dylan music (even if on an analog radio), and watched a documentary film about industrialized farms (even if in an analog cinema).
  • content is typically identified "on-demand” (e.g., by pressing a ListenNow button on a second screen app).
  • tablets and other devices may routinely perform content identification as a background operation, e.g., as an operating system service rather than as part of a second screen application.
  • these content identifiers can all be stored in association with a user identifier (e.g., cookie), providing a rich source of context by which the user' s needs may be better met.
  • cookies are presently used primarily for ad serving, their basic concepts and usage models can be employed more extensively.
  • a user who consumes Spanish language media primarily. This fact can be discerned by examination of the content IDs that are stored in association with her cookie identifier (e.g., identifying newscasts from the Univision network, telenovelas, etc.).
  • the cookie data can be sent and serve as a type of profile information.
  • the Acme web site can determine, from a listing of media content consumed by the user, that Spanish is a favored language. Accordingly, a Spanish language version of the Acme web page can be delivered to the user instead of a default English language version.
  • Age can be similarly inferred from content consumption habits. (Audiences for movies directed by Tim Burton tend to be under 30 years old; audiences for movies directed by Federico Felini tend to be over 40 years old; etc.) Educational background may also be correlated to media consumption (relatively few high school dropouts favor films by Akira Kurosawa).
  • web or other content delivered to the user may be tailored in accordance with demographic classifications, for which the user's history of media consumption can serve as a proxy.
  • iCloud enables sharing about history of browsing states between a user's devices.
  • a user who leaves a web page on a desktop computer to commute home can open the same web page on a tablet (using the iCloud infrastructure) and continue browsing on the ride home, from the point she left off.
  • a history of the user's analog content consumption can be stored in the user' s cloud account and employed to tailor other content delivered to the user on the same or other devices.
  • no cookie is used. Instead, the user is identified by the cloud account into which she is logged- in, and in which content consumption history is logged.
  • Cookies can form part of metadata statements made by the user's device.
  • a user's tablet which is listening continuously to the user's environment and identifying the content it hears.
  • the device spews a cookie, associated with a content identifier (e.g., an IS AN identifier: International Standard Audiovisual Number).
  • the device memory may have a stored cookie with the file name SSP123.txt, which stores the value A86FC2, issued by Supply Side Provider Superco.
  • the device Each time content is identified, the device writes a metadata statement that includes the file name or value of the cookie, combined with the content's IS AN Identifier - forming a historical record of the user's environmental state.
  • This information may be stored locally on the device, and/or transmitted for storage a cloud account associated with the user (e.g., iCloud), and/or transmitted for storage in a database maintained by Superco, etc.
  • the metadata statement may take the form of a linked data RDF (Resource Description Framework) triple.
  • audio context can serve as a proxy for location information.
  • a user shopping in Wal-Mart with a Wal-Mart app running on his smartphone.
  • the user has logged- in with his Sam's Club member ID and password.
  • the app While shopping, the user pauses in front of a display of Rubbermaid products - inspecting storage bins.
  • the app includes an audio WM detector, and the store plays differently- watermarked background music in different zones of the store.
  • the app's watermark detector decodes a watermark indicating that the user is in the Housewares-Bin Storage section of the store.
  • the app - or a remote server with which it communicates - notes this fact, and sends it for storage in a SSP database, associated with a supply side cookie from the user's smartphone. The fact that the user remained in that part of the store for ten minutes is also recorded in association with the cookie.
  • Wal- Mart Based on the user's apparent interest in such a product, and his failure to purchase, Wal- Mart expresses its interest to present a Rubbermaid advertisement to that user, the next time an available ad slot comes up on a web site the user browses. Sure enough, at home that evening, the user fires up a tablet computer and navigates to a sports news site. That web site receives the user's cookie as part of the initial data exchange, and advertises availability of an ad slot on that user's tablet to the highest bidder. With its knowledge of the user's physical presence at the Rubbermaid display in its store earlier that day, Wal-Mart makes an offer to buy the ad slot at a price that no other advertiser could justify to pay.
  • a system can publish the user's sensed context information, with cookie information.
  • the context information can be sampled audio, identifying information (e.g., fingerprint or watermark data) derived from the sampled data, and/or metadata (such as song title or movie title) identified by reference to the identifying information.
  • the context information can additionally include other context variables, such as location information, motion data, etc.
  • the cookie information can serve as an anonymous identifier of the user, by which other profile information about the user can be obtained.
  • Camera data can be used to identify physical objects near the user, or with which the user interacts. (This will become increasingly prevalent as head- worn computing apparatus proliferates.) Similarly with NFC data, sensed from NFC (RFID) chips in the user's
  • Olfactory sensors can provide further information about the user's environment. Cookies are suitable to represent all such information.
  • the web site may launch a Java app (or call a Java Native Interface) that talks to one or more of the tablet's sensors (with user approval) and collects sensor data.
  • a Java app or call a Java Native Interface
  • This information is stored in association with a cookie (which is written, if not pre-existing).
  • this sensor information - polled at the request of a remote computer associated with the web site may be hashed to yield a Globally Unique Identifier (GUID).
  • GUID Globally Unique Identifier
  • the GUID may, for example, be a function of events/information that one or more of the above-noted sensors has observed in the prior interval of time, e.g., 5 or less, 10, 30, 60, or 300 or more, seconds or minutes.
  • This GUID can be written to a cookie (i.e., stored in a database in association with an identifier). This can be useful when it is desirable to place the user and/or device in a particular physical context, without revealing details,
  • GUIDs For devices of two users, both in the same physical environment (i.e., having shared sensor context) at the same time, create such GUIDs, they should match, or be within an error tolerance apart (e.g., within a specified Euclidean distance or percentage of each other). If the fact of such co-presence is thereby established, a computer device might allow the two users to interact in a manner that might normally be restricted. For example, music or other entertainment content available to one user might be made available to the other user. If the two users visit the same web site/service, it may invite the users to share information, chat, listen to the same audio stream, etc., based on the common physical/sensor history (which may have been at a previous time).
  • a user's context information (or a hash of such information), stored at different times at a cloud repository, in association with a unique, anonymous identifier of the user (or the user's system, e.g., a cookie), enables a variety of powerful capabilities.
  • third parties e.g., supply side providers/platforms, in currently-popular systems
  • employ cookies to leverage historical context about the user.
  • the cookie contains such historical information, or provides a key into a contextual database in which a record of the previous contextual history of the user/device is stored.
  • Steve's smartphone or wearable computer system which periodically reports context information to a cloud database.
  • Steve's device senses information about his physical context - what the microphone is hearing, what the camera is seeing, the barometric pressure, Steve's GPS coordinates, etc.
  • Related information is written to a cloud repository, with a cookie identifier.
  • the device sends hash/fingerprint data FS.
  • the information stored in the cloud may not identify Steve by name, but it indicates that a person/device associated with a particular cookie identifier experienced sound that yields a fingerprint FS, at a particular date and time (and, if authorized, location).
  • Sensed sound information yields derivative data SOUND- DF34A967EA; a recognized Bob Dylan song yields derivative data SONG- 8FF7A9D66C; a decoded digital watermark from The Daily Show playing in the background yields derivative data
  • DWMAUDIO- 0BEF838E26 a voice recognized to be Steve's friend Bob yields derivative data SPEAKER- 2D2A54A4DF; recognized speech by Bob yields derivative data SPEECH- 1A7BC15AA6; Steve's location at 45°18'18"N 122°58'2"W yields derivative data GPS- 9389DEB5C3; the current barometric pressure yields derivative data BARO-EF0B0E93F5; a can of Coke glimpsed by the smartphone camera yields derivative data PRODUCT- CFB9800146; a garlic aroma from food simmering in the kitchen and sensed by the phone's olfactory sensor yields derivative data SMELL- AD33A3E8CC; Steve's heart-rate, sensed by a biometric sensor, yields derivative data HEART- 3F88B65334; etc., etc.
  • a cloud database may contain entries including:
  • Each such entry is typically time- and date-stamped (and less frequently location- stamped), but such notation is omitted above, for clarity.
  • the database naturally includes similar records for other individuals and/or devices (i.e., other cookie identifiers). Thus, similar contextual assertions are stored for Tom (cookie
  • Steve is surfing the web, as before.
  • the cookie/context information is sampled and uploaded during a visit to a particular web site or service
  • it is routinely and periodically logged (e.g., every five minutes, or other interval as noted above) by the operating system of Steve's device, e.g., Android, during the period that Steve is using the device.
  • Steve may be using a tablet that is provided to him free of monthly charges, by Google's subsidiary DoubleClick.
  • Steve's operating system uploads to DoubleClick the stored, historical DoubleClick cookie information gathered by the operating system since the last such upload.
  • code in the operating system or a resident application program
  • the ad tag takes the form of an http string that - in addition to including the host address for the DoubleClick ad server (http: //ad ⁇ dot>doubleclick ⁇ dot>net/), also includes other information, including a code identifying the New York Times web site, a topic or zone code indicating that Steve is within the sports section of the paper, and a sub-topic/sub-zone code indicating the requested article relates to baseball.
  • This hierarchy allows more precise targeting of advertising, and optimization of ad revenue.
  • the New York Mets baseball team may pay a penny to present an ad (offering upcoming game tickets) to a reader of the New York times, but may pay 2 pennies with knowledge that the reader is in the sports section, and may pay a dime with knowledge that the reader is interested in baseball.)
  • the URL may be created - or modified - dynamically, as a function of context.
  • a software component on Steve's device e.g., resident in the operating system, or part of a browser plug-in, or Java code running with a web page
  • Steve's device can collect sensor information about Steve's physical context, or can read context data gathered earlier and stored locally. It can then dynamically form an ad tag URL, e.g., by appending such context information to the ad tag provided from the web site. That is, whereas prior art ad tags were static, a tag can instead start with a static part (e.g., from the web site), and build from it a dynamic ad tag URL employing physical context information.
  • POS point of sale
  • Each POS terminal has one or more sensors, such as a barcode scanner, a camera, a microphone, a scale, etc.
  • Shopper Steve carries a smartphone or other such device with its own set of sensors, and code (e.g., in the operating system or an application) that publishes contextual information in an anonymous fashion.
  • Objectl is a six-pack of Sprite soft drink.
  • Object2 is ajar of Old El Paso salsa.
  • Object3 is a can of Science Diet cat food. Etc.
  • the barcode scanner is linked to the POS system, and reports a GTIN (Global Trade Item Identifier) number as each object is scanned (e.g., 549410582762, 923364619460,
  • the POS system relays each of these GTIN identifiers, as it is received from the scanner, to a cloud-based database, together with information identifying the store and checkout terminal (lane) from which the data originated.
  • the database time- and date-stamps this information, and stores it.
  • the barcode scanner also emits feedback signals that are sensed by Steve's smartphone.
  • each time the barcode scanner reads a GTIN code it emits an audio signal that encodes the GTIN identifier.
  • a chord of 13 different tones can be sounded for 200 milliseconds - with each tone drawn from a different library of ten tones, identifying a 0-9 digit at a different position in the GTIN string.
  • an audio signal can be frequency-shift-keyed to convey 13 ASCII character codes coiTesponding to the 13 GTIN digits, at a data rate of 300 bits per second, including error correction overhead.
  • other forms of feedback signals can be employed, such as Bluetooth and other wireless data, and ultrasonic audio.
  • Steve's smartphone senses this feedback signal (with a microphone, in the foregoing arrangement), and decodes the GTIN identifiers.
  • These decoded identifiers comprise a sequence that is temporally aligned with the time-stamped sequence of GTIN identifiers received by the cloud database from the store's POS system.
  • the smartphone stores these decoded identifiers, and also sends the sequence of decoded identifiers to the cloud database (with an anonymous identifier assigned to Steve, or hashed from information specific to Steve; such an identifier may be the letters DFGHJ).
  • the database matches the smartphone-sent GTIN sequence with a GTIN sequence from the POS terminal in Lane 4 of that grocery store. Anonymous Steve is thus associated with the purchase of the soda, salsa, and cat food, in Lane 4, through the smartphone' s publication of sensed audio context information to the cloud repository.
  • the set of identifiers thus serves like a fingerprint, by which Steve's checkout transaction can be identified, and distinguished from other shoppers' checkout transactions.
  • Step 2 The next time Steve enters the store, this historical record of purchases can be recalled based on the anonymous identifier DFGHJ, and used to provide coupons, targeting advertising, etc.
  • Step's smartphone may have app software, distributed by the grocery store, in which the DFGHJ identifier is stored, and which serves to present coupons and other information.
  • This provides loyalty card-like functionality, without any use of a loyalty card.
  • Steve's credit card or debit card number (a technique on which some other shopper- identification systems are based).
  • the DFGHJ identifier can be used like a cookie - to permit access to this record of physical shopping history elsewhere. For example, Steve may later view a Monday night football game, while surfing the web on the smartphone. The smartphone microphone senses the game audio, which enables identification of the football broadcast by audio fingerprinting or digital watermark decoding. A corresponding contextual assertion about Steve's activity is written to cloud storage. When Steve surfs to the front page of the New York Times, the phone authors an ad tag URL that conveys information about his activity (watching football), and also conveys Steve's DFGHJ identifier. The GTIN codes that are cloud- associated with this identifier reveal Steve's brand preferences.
  • Part of each GTIN code is a plural-digit "Global Company Prefix" field, identifying the company that provided the product.)
  • the ad server can take into account Steve's football-watching activity, and Steve's historical preference for Coca Cola products over Pepsi products (as evidenced by all the Coke-prefixed GTIN codes associated with the DFGHJ identifier). It can then, e.g., select a football-themed Coke ad for presentation on the New York Times front page that Steve's device is presently loading.
  • the POS terminal provides feedback data memorializing the GTIN codes for the objects Steve is buying.
  • the same functionality can be achieved without such feedback GTIN code data.
  • each time the barcode scanner in Lane 4 decodes a barcode it can emit a two-tone, 200 millisecond beep, e.g., 1200 & 1300 Hz. (Other lanes can do likewise, with different tone pairings.)
  • each time the POS terminal reads an object's GTIN code it sends the code to the database, which makes a time- stamped record.
  • Steve's smartphone senses these beeps, and reports each such detection to the database, with his DFGHJ identifier. Again, the database time-stamps and stores such information.
  • a temporal fingerprint is defined by the intervals between GTIN reports from the POS terminal (e.g., 1.3 seconds, .9 second, .9 seconds, 1.1 seconds). This temporal fingerprint is matched with a corresponding temporal fingerprint defined by the smartphone's report of beep detections. Again, Steve's shopping history is discerned, and stored in association with his anonymous identifier.
  • Steve's smartphone may detect and report the scanner beeps while still in Steve's pocket.
  • the cloud database can look up any electronic coupons stored in a cloud wallet associated with the DFGHJ identifier, and can inform the POS terminal of their particulars (e.g., 50 cents off a six-pack of Sprite).
  • the POS terminal can make the adjustment in the checkout tally - without Steve having identified himself in the store, and with the phone still in his pocket.
  • the database can query the smartphone for its coupons.
  • the smartphone can send its collection of coupon data, and the cloud can relay any that apply to the POS terminal. Once they have been redeemed, such information can be reported by the POS terminal to the cloud, which in turn confirms such redemption to the smartphone wallet.
  • the matching of temporal sequences can be performed by the smartphone, rather than the cloud database.
  • the store can broadcast - on its WiFi network - GTIN codes decoded by each POS terminal barcode scanner in the store, with each GTIN code being time-stamped and paired with an identifier of the POS terminal.
  • the smartphone can derive temporal fingerprints for each POS terminal from such information, and match one such fingerprint to the temporal sequence of beeps it detects during checkout.
  • the smartphone transmits its coupon data to the store POS system (over the WiFi network, or otherwise), noting the POS terminal tally to which the coupon credit should be applied.
  • the list of GTIN codes reported by the POS terminal in Lane 4 can be associated with Steve, simply by Steve's smartphone reporting its GPS location to the database, with his DFGHJ identifier.
  • Steve's phone can gather information indicating his location in Lane 4 by a short range wireless beacon that marks that lane, or by the distinctive frequency of confirmation beeps issued by the barcode scanner in that lane, or by an RFID chip positioned in that lane, as sensed/reported by Steve's phone.
  • the smartphone serves as a physical token - an ownership factor, and the beeps or other context that the phone senses serves as a knowledge factor.
  • Steve may return nearly every day to the same grocery store - each time paying with cash.
  • the store notes his frequent visits, and works to lock-in his continued patronage by offering a branded credit card that provides a 2% cash rebate on purchases made at that grocery chain.
  • the offer is made the next time Steve checks out with a clerk- attended POS terminal, with the clerk verbally extending the offer, and pointing out that details - including a calculation of how much his cash rebate would be based on an interval of past purchases - are printed on his register receipt.
  • roving store clerks e.g., at a Home Depot hardware store. Steve wants help concerning a particular item he is thinking about buying (if I buy this Moen shower head, will I require a metric wrench set to install it?), and taps a "Help" button on the Home Depot app on his smartphone. The app directs him to take a picture of the item, from which it then may extract identification information. The app also directs a sensor in the phone to capture information that serves to identify Steve's location.
  • Information about the customer's situation is sent to the clerk's smartphone (e.g., the shopper's location, and a picture of the product, or the name of the product as discerned from a barcode on the package or by object recognition).
  • the smartphone is also sent any other data gleaned from this customer today (such as other Help requests from that anonymous ID, information about other images captured using the store app for price-lookup, information about web sites recently visited by a device having the same IP address on the store's WiFi network as the IP address from which the Help request was received, etc.).
  • the clerk may thereby learn that the customer recently reviewed an Amazon web page about a Kohler shower head, and used a barcode scanning feature in the Home Depot app to learn the price of a waterproof indoor can light fixture.
  • the clerk walks to Steve's aisle location, and looks for a person puzzling over shower heads.
  • the background information about the customer's interest in shower hardware suggests to the clerk that the customer is involved in bathroom remodeling - helping the clerk better serve the customer.
  • DoubleClick selects a Moen shower head advertisement to display on the home improvement web site.
  • POS system that acquires visual information from a retail product (e.g., by a barcode scanner), and relays data derived from such information (e.g., a GTIN identifier) in audio form to a smartphone.
  • data derived from such information e.g., a GTIN identifier
  • This may be regarded as a form of synesthesia - a phenomenon detailed in a Wikipedia article by that name.
  • the car senses its location, processing information from a GPS radio receiver,
  • the car emits audio or ultrasonic tones representing this location data, and the smartphone senses it.
  • the Samsung Smart Gear watch is powered by a single-core 800 MHz processor, with accelerometer, multiple microphones and a camera.
  • a single-core 800 MHz processor with accelerometer, multiple microphones and a camera.
  • new possibilities for always-on sensing are enabled.
  • the device is always exposed to both the user's environment (as opposed to being in a pocket or purse) and in physical contact with the user,
  • the placement on the body enables additional opportunities for activity recognition.
  • more advanced gait analysis can be performed providing additional insight in the user's subsequent search queries.
  • Sensors that are not battery powered and can store sensor information, similar to a more advanced RFID chip, can also be used. Such sensors can take the form of jewelry, eye- ware, etc. Such sensors when activated in the presence of an electromagnetic field can report sensor results, such as number of activations (or power-ups) since the last time a download was occurred and the ID of those devices that were powering up the sensor.
  • the above can be thought of as a distributed sensing ecosystem, which can be created by using sensors in-place and shared by multiple users, in combination with battery-less, RFID enabled recorders for each user.
  • Prototype, 3D printed rings have been proposed that carry all of a user's stored value or transit credentials, allowing the wearer to seamless move through turnstiles in many cities. If such a ring was also able to report on activations when queried, a snapshot of the day's commute would emerge. If sensors in the wearer's office building were made available, the ring could store average indoor air-quality during the work day. When queried at home, a picture of the wearer' s environment could be created.
  • the resulting information can be used in the same form described earlier and made available to the marketplace.
  • On-body sensors are increasingly being paired with consumer smartphones for fitness and health. Beyond heart-rate monitoring for exercise, EKG, electro muscular signals, temperature and blood chemistry are being sensed in non-invasive fashions. Such sensors create new opportunities to collect and share context. Blood chemistry, respiratory health
  • heart-rate all provide insight into what services or products the user may benefit from.
  • One method includes sensing information about a user's physical - as opposed to computational - environment.
  • the sensed information - or data derived from such information - is transmitted to a remote service for storage, in association with identifier data associated with the user (e.g., a cookie identifier).
  • identifier data associated with the user e.g., a cookie identifier.
  • identifier data associated with the user allows access to the data about the user's earlier- sensed physical environment. This enables information presented to the user (e.g., a web page, digital signage, etc.), to be customized based on the information about the user's earlier physical environment.
  • Another method involves, at a first time, receiving information corresponding to audio or visual content sensed from a user's physical environment. Then, at a second, subsequent time, identifying advertising for presentation to the user, based at least in part on the identified information.
  • the second time may follow the first time by a few seconds, but more typically follows it by several hours, or days or more.
  • a further aspect of the technology is a method in which a visual sensor of a first system acquires visual information representing first data.
  • An audio sensor in a second system is then used to receive audio information representing the first data, where the received audio was emitted by the first system based on its acquisition of the visual information.
  • the first data is extracted from the received audio, using a hardware processor configured to perform such act.
  • Cookie data is then stored, based on the extracted first data.
  • Another such method involves a first system that includes a first sensor responsive to a first type of stimulus, which acquires information - representing first plural-bit data - conveyed by the first type of stimulus.
  • a second system using a second sensor responsive to a second type of stimulus different than the first type of stimulus, then receives information - again
  • the first plural-bit data represented by the information received by the second sensor is extracted, using a hardware processor configured to perform such act.
  • Cookie data is then stored, based on the extracted first plural-bit data.
  • Another method includes, at a first time, receiving information about entertainment ambient content sensed by a microphone in a user's portable device, where the received information is accompanied by an identifier of the user, or the user's device. From this received information, a language apparently understood by the user is determined, and data related to this language is stored. Then, at a later time, a language-specific version of content to be provided to the user (or to the user's portable device) is selected, based on the stored data.
  • a related method involves, at a first time, receiving information about entertainment ambient content sensed by a microphone in a user's portable device, where the received information is accompanied by an identifier of the user, or the user's device. Based on the received information about ambient content, an age or education of the user is estimated, and related information is stored. At a later time, content to be provided to the user is selected, based on the stored data. (By such arrangement, the user's history of media consumption serves as a proxy for information about age or education.) Yet another method includes sending a request for a web page from a user's device to a first web server. Responsive to this request, the device receives first information, including an ad tag URL.
  • Data - including the ad tag URL (or a modified version of the ad tag URL) - is then sent to a second web server, responsive to which the user's device receives second information.
  • a display is then presented to the user, based on the received first and second information.
  • This method is characterized by sensing physical context data about the user or the user's environment, using a sensor in the user's device, and including information about the sensed physical context data with the data sent to the second web server.
  • Still another method involves discerning a set of item identifiers from items presented for purchase during a checkout operation by a shopper, using apparatus in a bricks and mortar store operated by a retailer.
  • the set of item identifiers serves as a fingerprint by which that checkout operation can be distinguished from other checkout operations.
  • Information related to the fingerprint is transmitted to a portable device conveyed by the shopper.
  • the device relays this data - together with first information that serves as an identification of the shopper - to a computer system (which may be the store POS system, or another system).
  • This fingerprint information is received, and matched with fingerprint information discerned by the apparatus.
  • the first information which serves as an identification of the shopper - is associated with the set of item identifiers discerned by the apparatus.
  • purchased items are associated with a particular shopper, without the shopper directly providing shopper- identifying information to the retailer during the checkout operation.
  • the set of item identifiers can comprise an ordered set of such identifiers, to better avoid confusion with similar items purchased at other checkouts, albeit in different orders.
  • a related method involves, from a point of sale terminal in a bricks and mortar store operated by a retailer, emitting a signal for detection by a shopper's portable device, each time an item presented for purchase in a checkout operation is sensed.
  • These emitted signals define a temporal sequence that serves as a fingerprint by which that checkout operation can be identified, and distinguished from other checkout operations.
  • the shopper's device receives these signals, and sends data including first sequence information based on its detection of the emitted signals, and also including first information that serves as an identification of the shopper. This data sent by the shopper's device is received, and matched with corresponding second sequence information generated as part of the checkout operation.
  • the first information - that serves as an indication of the shopper - is then associated with the second sequence information, to thereby associate the first information with a particular checkout operation.
  • the first information - which serves as an identification of the shopper - is associated with a particular checkout operation, without the shopper directly providing shopper- identifying information to the retailer during the checkout operation.
  • a further aspect of the technology comprises compiling, in a data structure, two or more types of information from the group consisting of: (a) a user's online activities, including web sites visited; (b) entertainment content sensed by a microphone-equipped device conveyed by the user; and (c) a record of items purchased by the user in a bricks and mortar store. Advertising is then selected for presentation to the user, based on these two or more types of information.
  • Yet a further method includes, at a first time, receiving sensor information from an apparatus worn on a wrist of a user. Data related to this received sensor information is stored in a data structure remote from the user, in association with cookie data that serves as an identifier of the user. Then, at a second time, the stored data is accessed by reference to the cookie data, and used in identifying information to present to the user.
  • FIG. 1-4 illustrate aspects of the foregoing arrangements.
  • video can be sampled, e.g., using the camera of a cell phone.
  • Watermarks and fingerprints can be derived from the captured image/video data, and used as detailed above.
  • processing other than watermark- and fingerprint-based content identification can be used.
  • One such alternative is speech recognition.
  • Another is speaker recognition.
  • Still another is audio classification.
  • a system may thereby discern, e.g., that the user is in a crowded public place - such as a busy shopping venue - based on the sampled audio (e.g., a jumble of speech-like phonemes that can't be recognized), and systems interacting with the user/user device can tailor their behavior accordingly. (Again, combined use of media content information with location information allows still more accurate context classification.)
  • augmented reality glasses Such devices can overlay logos and other computer-generated indicia over a real-world scene presented to the user. Different augmentations can be presented to different users, based on their respective historical- and currently-sensed context information.
  • the sensing of physical context information can occur at one time, and its use can occur at a second time, where the second time is the same as the first time, or follows the first time by 5, 10, 30, 60, or 300 or more, seconds or minutes.
  • Ad serving companies typically have maintained the backend databases that aggregate context information about a user' s digital browsing history. However, different companies may emerge to aggregate the other types of context information detailed herein.
  • cookie data needn't be sent to identify the user.
  • a remote system can identify the user (or the user device) otherwise, such as by an IP address included in the packet stream that conveys the data about the user's physical environment, or that conveys a request for a web page.
  • the JP address can be associated with the user (and/or the user's cookie data) using a table, database, or other data structure.
  • cookie data isn't used at all. The user's identity is conveyed or discerned otherwise in each transaction between the user device and the remote system.
  • remote system While the remote system is sometimes referenced as a unitary entity (as in the preceding sentence), it is more often a distributed system - involving multiple computer servers at multiple locations, operated by multiple different parties. (The appendix begins to illustrate the many different parties that may be involved.)
  • a watermark detector forms part of a tablet app distributed by radio station KXYZ-FM.
  • software code for a watermark detector may form part of a Java Native Interface (JNI) library downloaded to a user's device with a web page. Thereafter, when that web page, or another, wants information about the user's context, it can invoke the earlier- stored code with JNI.
  • JNI Java Native Interface
  • the Java instructs the device to activate its microphone and associated software modules, decode any watermark, and write cookies (or call out to another service that writes cookies) accordingly.
  • Context is sometimes defined as any information useful in characterizing the situation of an entity.
  • An entity is a person, place or object that is considered relevant to an interaction with a user.
  • Such context information can be of many sorts, including computing context (network connectivity, memory availability, processor type, CPU contention, etc.), user context (user profile, location, actions, preferences, nearby friends, social network(s) and situation, etc.), physical context (e.g., lighting, noise level, traffic, etc.), temporal context (time of day, day, month, season, etc.), history of the above, etc.
  • computing context network connectivity, memory availability, processor type, CPU contention, etc.
  • user context user profile, location, actions, preferences, nearby friends, social network(s) and situation, etc.
  • physical context e.g., lighting, noise level, traffic, etc.
  • temporal context time of day, day, month, season, etc.
  • While the present technology has been described mainly in the context of third-party, cookie-based arrangements, it is applicable in other systems as well.
  • Microsoft, Facebook, Google, and Apple are each promoting their respective technologies for identifying consumers on the web - without use of third-party cookies.
  • one Microsoft system employs device-specific identifiers, which are associated together in a cloud database as used by one particular individual.
  • Facebook' s technology relies on its unique user logins.
  • Google's system (AdID) and Apple's system (Identifier for Advertising, or ADFA) similarly aim to supplant third-party cookies with identification technologies that they themselves govern - allowing more granular usage and privacy controls.
  • Applicant's invention encompasses the technology described herein, as applied to such alternatives to third-party cookies.
  • HTTP cookies In addition to the above-noted alternatives to classic (HTTP) cookies, other means of identifying an online user (and device) include IP address (noted above), URL (query string), hidden form fields, HTTP authentication data (based on user name and password, etc.), and the DOM (Document Object Model) property "window.name,” which are familiar to artisans in the field (and are detailed, e.g., in the Wikipedia article for HTTP Cookie dated December 4, 2013). Unless used with the adjective "HTTP,” the term “cookie” herein should be construed to encompass such alternative forms of user or device identification.
  • Sample smartphones include the Apple iPhone 5; smartphones following Google's Android specification (e.g., the Galaxy S4 phone, manufactured by Samsung, and the Google Moto X phone, made by Motorola), and Windows 8 mobile phones (e.g., the Nokia Lumia 1020, which features a 41 megapixel camera).
  • Google's Android specification e.g., the Galaxy S4 phone, manufactured by Samsung, and the Google Moto X phone, made by Motorola
  • Windows 8 mobile phones e.g., the Nokia Lumia 1020, which features a 41 megapixel camera.
  • Apple iPhone including its touch interface, are provided in Apple's published patent application 20080174570.
  • each includes one or more processors, one or more memories (e.g. RAM), storage (e.g., a disk or flash memory), a user interface (which may include, e.g., a keypad, a TFT LCD or OLED display screen, touch or other gesture sensors, a camera or other optical sensor, a microphone, etc., together with software instructions for providing a graphical user interface), and an interface for communicating with other devices (which may be wireless, as noted above, and/or wired, such as through an Ethernet local area network, a T-l internet connection, etc.).
  • memories e.g. RAM
  • storage e.g., a disk or flash memory
  • a user interface which may include, e.g., a keypad, a TFT LCD or OLED display screen, touch or other gesture sensors, a camera or other optical sensor, a microphone, etc., together with software instructions for providing a graphical user interface
  • an interface for communicating with other devices which may be wireless, as noted above, and/or wired
  • the processes and system components detailed in this specification may be implemented as instructions for computing devices, including general purpose processor instructions for a variety of programmable processors, including microprocessors (e.g., the Intel Atom, the ARM A5, the Qualcomm Snapdragon, and the NVidia Tegra 4; the latter includes a CPU, a GPU, and NVidia' s Chimera computational photography architecture), graphics processing units (GPUs, such as the NVidia Tegra APX 2600, and the Adreno 330 - part of the Qualcomm Snapdragon processor), and digital signal processors (e.g., the Texas Instruments TMS320 and OMAP series devices), etc.
  • microprocessors e.g., the Intel Atom, the ARM A5, the Qualcomm Snapdragon, and the NVidia Tegra 4; the latter includes a CPU, a GPU, and NVidia' s Chimera computational photography architecture
  • GPUs such as the NVidia Tegra APX 2600, and the Adreno 330 - part of the Qualcomm Snapdragon processor
  • digital signal processors
  • processor circuitry including programmable logic devices, field programmable gate arrays (e.g., the Xilinx Virtex series devices), field programmable object arrays, and application specific circuits - including digital, analog and mixed analog/digital circuitry.
  • Execution of the instructions can be distributed among processors and/or made parallel across processors within a device or across a network of devices. Processing of data may also be distributed among different processor and memory devices. As noted, cloud computing resources can be used as well.
  • references to "processors,” “modules” or “components” should be understood to refer to functionality, rather than requiring a particular form of implementation.
  • Smartphones and other devices can include software modules for performing the different functions and acts.
  • Known browser software, communications software, imaging software, and media processing software can be adapted for use in implementing the present technology.
  • Software and hardware configuration data/instructions are commonly stored as instructions in one or more data structures conveyed by non-transitory tangible media, such as magnetic or optical discs, memory cards, ROM, etc., which may be accessed across a network.
  • Some embodiments may be implemented as embedded systems -special purpose computer systems in which operating system software and application software are indistinguishable to the user (e.g., as is commonly the case in basic cell phones).
  • the functionality detailed in this specification can be implemented in operating system software, application software and/or as embedded system software.
  • data can be stored anywhere: local device, remote device, in the cloud, distributed, etc.
  • the present technology can be used in connection with wearable computing systems, including headworn devices.
  • Such devices typically include display technology by which computer information can be viewed by the user - either overlaid on the scene in front of the user (sometimes termed augmented reality), or blocking that scene (sometimes termed virtual reality), or simply in the user's peripheral vision.
  • augmented reality an overlaid on the scene in front of the user
  • virtual reality blocking that scene
  • Exemplary technology is detailed in patent documents 7,397,607, 20100045869, 20090322671, 20090244097 and 20050195128.
  • watermark technology can be used in various embodiments.
  • Technology for encoding/decoding watermarks is detailed, e.g., in Digimarc's patents 6,614,914, 6,590,996 and 6,122,403; in Nielsen's patents 6,968,564 and 7,006,555; and in Arbitron's patents 5,450,490, 5,764,763, 6,862,355, and 6,845,360.
  • Content fingerprinting can also be used in various embodiments.
  • audio fingerprinting are detailed in patent publications 20070250716, 20070174059 and 20080300011 (Digimarc), 20080276265, 20070274537 and 20050232411 (Nielsen), 20070124756 (Google), 7,516,074 (Auditude), and 6,990,453 and 7,359,889 (both Shazam).
  • image/video fingerprinting are detailed in patent publications 7,020,304 (Digimarc), 7,486,827 (Seiko- Epson), 20070253594 (Vobile), 20080317278 (Thomson), and 20020044659 (NEC).
  • SIFT, SURF, ORB and CONGAS are some of the most popular algorithms.
  • SIFT, SURF and ORB are each implemented in the popular OpenCV software library, e.g., version 2.3.1.
  • CONGAS is used by Google Goggles for that product's image recognition service, and is detailed, e.g., in Neven et al, "Image Recognition with an Adiabatic Quantum Computer I, Mapping to Quadratic
  • Bag of Features or Bag of Words, methods.
  • Such methods extract local features from patches of an image (e.g., SIFT points), and automatically cluster the features into N groups (e.g., 168 groups) - each corresponding to a prototypical local feature.
  • a vector of occurrence counts of each of the groups i.e., a histogram
  • a vector occurrence count is then determined, and serves as a reference signature for the image.
  • To determine if a query image matches the reference image local features are again extracted from patches of the image, and assigned to one of the earlier-defined N-groups (e.g., based on a distance measure from the corresponding prototypical local features).
  • a vector occurrence count is again made, and checked for correlation with the reference signature.
  • Digimarc has various other patent filings relevant to the present subject matter. See, e.g., patent publications 8,498,627, 8,412,577, 6,947,571, 20130150117, 20120284012,
  • Interactive ads are everywhere these days, but when it comes to the technical process of getting an ad on the page and how publishers and marketers verify it delivered, not many people can explain what actually happens in detail. Read this article though and you'll be one of them! Below I've detailed step-by-step how a browser gets from the initial call to a publisher's website to the final ad creative, and when and how each party counts an impression. You can view a diagram of the ad serving process at the bottom of this post - the numbers in the text refer to the steps labeled in the diagram.
  • the publisher's web server sends back a bunch of HTML code (2) that tells the browser where to get the content (3) and how to format it.
  • Part of the HTML code returned to the browser (4) will include a coded link known as an ad tag.
  • the ad tag points the browser to the Publisher's Ad Server (5), a system designed exclusively for delivering and tracking advertising.
  • the Publisher's Ad Server is actually a network of cloud servers owned and maintained by a separate company.
  • the content server tells the browser to fetch the ad from Doubleclick, a company owned by Google, that then makes the very complex decision on which ad to serve using a program called an Ad Selector.
  • the ad server is deciding among thousands upon thousands of potential options in mere milliseconds.
  • the computational power behind the Ad Selector is mind blowing - Atlas, the major rival to Doubleclick calls the supercomputer running its Ad Selector "WARP" and it is among the most powerful in the world, making billions of decisions a day and trillions in its lifetime.
  • the Ad Server makes a decision, and in most cases sends back another ad tag (6), or redirects the browser by pointing it to the Marketer's Ad Server. These redirects are technically speaking 302 redirects, which tells the browser the page has been "temporarily moved”. This allows Ad Servers to count the 302 call as an impression and host the actual ad content on a different server.
  • the publisher's adserver sends the browser a redirect to the marketer, it counts a delivered impression in its own database (star). The only exception here is if the publisher decides to deliver a house ad or the marketer has asked the publisher to "site-serve" the ads, both of which requires the publisher load the actual creative files into their ad server, meaning the publisher is the final destination, and the browser can skip the loop through the marketer side (steps 7,8,11, 12).
  • the browser now calls the Marketer's Ad Server (7) and is redirected yet again to a Content Delivery Network, or CDN, (8) a global network of cloud servers that actually house the raw creative graphics to fetch the actual ad. Why, you ask? Well, as powerful as ad servers are, they just aren't equipped to handle the volume and bandwidth required to deliver content as heavy as image files. Redirects are often nothing more than a lxl pixel requiring just a few bytes of memory. Image files on the other hand are kilobytes or even megabytes in size, could be called millions of times a day, and require a much faster and robust infrastructure.
  • CDN Content Delivery Network
  • Ad Servers might maintain three to six data centers across the world, but a CDN can process the heavy bandwidth and deliver the content faster because they operate hundreds of data centers and can route requests to the one nearest to the user, no matter where they are on earth. You can think of the ad server as the brain and the CDN as the brawn. Ad Servers aren't the only companies that use CDNs, in fact many websites host their bandwidth intensive files in these cloud networks. A CDN is almost always another independent company, such as Akamai, that hosts the heavy creative assets so the Ad Server doesn't have to. There used to be a handful of these companies out there, but Akamai has acquired almost all of them and is the largest player by far in the space.
  • the Marketer's Ad Server In addition to sending back the redirect to the CDN, the Marketer's Ad Server also appends a second redirect (10) back to itself with a query string to fetch a lxl pixel (11) after the ad content has been called.
  • the browser fires this last redirect calling a lxl pixel from the Marketer's Ad Server (11)
  • the Ad Server knows the ad was successfully downloaded and it finally counts an impression in its own database (star).
  • your browser has to make at least four calls for site served ads and six in the case of third-party served ads for this whole process to work, if not even more, but shouldn't take more than a second regardless of the number of parties involved.
  • the diagram below - 302 redirects are highlighted in blue, and the ad creative is highlighted in red.
  • Tweet ; i5 Tagged as: 302 redirects, ad tags, adserving, Profile
  • Clicks are tracked in much the same way as impressions, with redirects.
  • the publisher's ad server populates multiple click tags into the href of the ad, concatenated with a query string so that when a user clicks the ad, the browser calls all three links more or less simultaneously. But yes, the specific order with any 3rd party system is always publisher ad server, marketer ad server, final destination.
  • 4th party tracking tends to happen when the marketer will have a handful of rich media ads that are served through a different system than their primary ad server.
  • the rich media vendor would be the 3rd party from a publisher perspective, and the primary ad server, perhaps DART, would be the 4th party.
  • publishers do not allow advertisers to bill on a 4th party tracker because they incur a dual discrepancy, but the practice of using them is quite common to centralize reporting.
  • the impression is counted for the ad.server. But in the case of the marketer, the impression isn't counted till the actual ad is served (having bounced the query pixel off the browser and back) which is after the initial redirect of browser to content network.
  • the ad server tells the browser to fetch the creative file from the CDN through a 302 redirect. If you were to view a page through a web development tool like Firebug, you would be able to look at both the HTTP request and response headers for each piece of the ad call, and you would see that the initial call starts with a 200 request, or a standard GET request to the publisher's ad server, that call responds with a '302 'Object moved' header, and the location of that file, redirecting the browser to the marketer's ad server, which responds in kind with a 302 'Object moved' header, and the location of the file on the CDN, which it can finally fetch.
  • CostPerAction I know we can divide it in different subcategories: CostPerLead, CostPerSale,
  • CostPerEngagement For example, I don't want to pay just because someone arrives on my landing page by clicking on a banner, but I want to pay if someone arrive on my landing page and complete a registration to my web site (CPL) or, even better, if he complete a purchase (CPS). How can the ad server "understand" that the customer has completed a purchase? How can it know that he performed a registration?
  • CostPerCall especially in the mobile area, where an advertiser pays when someone calls, for example, a call center.
  • apps that, after a click on a banner, automatically redirect you to your phone app, with the number already composed, you just have to press the "call” button.
  • my question is: how can the ad server be sure that I complete the call process? How can it know that I have not pushed the "cancel" button?
  • Google's bid per call in the desktop environment requires the advertiser to use Google Voice generated tracking numbers (usually positioned to the user as an extension), which allows it to track the conversion.
  • Google Voice generated tracking numbers usually positioned to the user as an extension
  • banner swf file is dynamically calling on three separate jpg files using absolute paths (i.e. http://www.mydomam.com/frame 1 -jpg). These banner frame jpg's are hosted on our own internal server so that we can update the banner creative on demand "without" having to edit, change, or resubmit new swf files to 3rd party vendors.
  • Ad servers provide publishers with a very simple technical mechanism called an ad tag which is easy to implement on any website, even if the ad server has never served on that site before. In fact, in the majority of cases, the ad server won't have any direct relationship with the publisher running its tags, and in many cases may actually have competing interests.
  • Atlas- serviced advertiser tags are running on Doubleclick serviced publisher site right now, for example, and you can repeat the scenario with virtually any combination of ad servers out there. The only exception I can really think of would be from an ethical point of view - most ad servers wouldn't knowingly let you run ads on adult content, pirated content, hate speech content, or things like that, but I assume that's irrelevant in your case.
  • the ad server can make quick decisions for any user from only one or two geolocations because it only processes a few bytes of data before passing the user on to a CDN based service to fetch the actual content, which might be thousands or even millions of times larger than what the ad server needs to process. It makes more sense to have a service that keeps the heavy data physically closer to a user, and on a system that is setup to load balance for much heavier volume.
  • the CDN doesn't have to make any variable-based decision, it just has to provide access to the file quickly.
  • publisher's ad server decides what ad to serve, and redirects the user to a marketer's ad server
  • the agency is the marketer in this case, or is the marketer's agent. If you, a media planner, work at an agency, the marketer is your client, and the company actually paying for all the advertising the agency is buying. There's always a marketer, they may just be an indirect participant if they've hired an agency to handle their advertising campaigns. I think it's just a case of semantics - when I say marketer, I really just mean the buy-side of the equation - that could be an agency working on behalf of a marketer, or it could be a marketer directly, as some do their own media buying and don't contract with an agency.
  • the publisher decides what client needs to serve in order to fulfill delivery quotas based on the goals and campaigns in their system, and for what the incoming ad request qualifies.
  • 3rd party tags When 3rd party tags are used, correct - and for most paid ads on the internet, 3rd party tags are used, so your understanding is spot on.
  • the publisher can host and serve the creative themselves - this is the difference between 3rd party ad serving (when a publisher redirects to a marketer's ad server) and 1st party ad serving (where the publisher hosts everything themselves).
  • 3rd party ad serving when a publisher redirects to a marketer's ad server
  • 1st party ad serving where the publisher hosts everything themselves.
  • the pricing is the same regardless of the setup - it doesn't really save the marketer or publisher any money to have it setup one way or another unless the marketer doesn't want to serve their ads at all, in which case they don't need an ad server and I guess would save that money. It's too painful operationally for a large organization to do that though, and would actually be more expensive to try and keep track of reporting, etc. For small marketers it might make sense, but it won't save them all that much money - ad serving can be had for extremely cheap these days.
  • the IO is a legal contract which should append the IAB Terms and Conditions I mentioned above, which states that all payments should happen on Net 90 terms, meaning the Advertiser has to send payment within 90 days of being billed.
  • Many agencies are notoriously late with payments, usually because they are also waiting for their clients to pay them for the work.
  • Publishers will suspend all campaign activity if a large balance builds up and the agency stops paying, but this is rare.
  • Publisher know the agency is playing the waiting game themselves, and extends a certain amount of goodwill for important clients, who they know will eventually pay up. If you can't get paid though and you have a signed IO, you can pursue legal action, or refer the case to a collections agency. Hope that helps - good
  • Ad Ops Jobs o Ad Ops Jobs Ad Ops Resources o AdMonsters
  • Ad Exchanges have been around for a few years, but have exploded in importance in the last year. Along with Demand Side Platforms (DSPs) and Supply Side Platforms (SSPs), Ad Exchanges are dramatically changing the way digital media is bought and sold. If you are a digital marketer or publisher, it is a very exciting time to be working in the industry.
  • DSPs Demand Side Platforms
  • SSPs Supply Side Platforms
  • the publisher's web server When a browser navigates to a publisher website (1), the publisher's web server sends back a bunch of HTML code (2) that tells the browser where to get the content (3) and how to format it. Part of the HTML code returned to the browser (4) will include a coded link known as an ad tag. That part is the same as in regular third-party ad serving. But instead of returning a DFA or Atlas tag, the Publisher Ad server will return a tag that points to a RTB -enabled SSP, typically through a dynamic Javascript tag that passes information like the publisher's ID, the site ID, and ad slot dimensions.
  • the user calls the SSP server (5) where the SSP reads that user's SSP cookie ID, likely already oil their machine. Assuming the user already has that SSP's cookie on their machine (and most users will, given that the largest SSPs boast 80 - 90% reach rates to the US internet population), the SSP starts the auction (6), by requesting bids from a host of demand sources, the DSPs (7). If the user does not have an SSP cookie on their machine, their ad inventory can technically still be auctioned, but since nothing is know about that user, the price will be very low and more related to the site context than the user's attributes. For the DSPs to truly value the impression though, they need to know something about the user that is going to see it. This is where the SSP cookie ID comes in - packaged with the bid request is the SSP's cookie ID, along with the url the impression will deliver on, and what the current user's frequency is on the site.
  • DSPs are able to match the SSP's cookie ID to their own cookie on that user, which is tied to a huge cache of marketer data and 3rd party data. What kind of 3rd party data? Using the cookie ID, the DSP will be able to know if that user recently priced out a car, is flying to Paris in the next 90 days, was recently shopping for shoes, and even more general demographic information about the user such as their age, gender, income range, credit score, and much, much more. I'll cover how that all works in a later post. But suffice to say, rich data is far and away the driver of higher bids, and the cookie ID is the mechanism through which data is associated to a user.
  • the URL is also important. Many brands don't want their ads to appear on just any site, even if they want that user. If the user is on a site with PG-13 content, for example, the advertiser might bid a lower amount or not at all. Similarly, the frequency of that user to the site they are on is also important to valuation. Advertisers are willing to pay a premium to reach users on their first or second pageview on a site vs. their 50th pageview for the simple fact that users are less engaged with site content and more likely to respond to an ad during their first few page views.
  • the DSPs all value that impression and submit a bid back to the SSP (8) as well an ad redirect to send the user should their bid win the auction.
  • the SSP picks the winning bid (9) and passes the DSP's redirect back to the user (10). From here the process is basically the same as third party ad serving -
  • the user calls the DSP (11), the DSP sends the user the marketer's ad server redirect (12), and user calls the marketer's ad server (13) and the marketer serves the user the final ad (14). Whew!
  • the matching process of the SSP cookie ID to the DSP cookie ID happens through a parallel process to serving ads called cookie-syncing.
  • Cookie-syncing is necessary because as a standard security process, web servers of any kind can only request cookies that are set to their own domain. Since the SSP sits between the end-user and all the DSP bidders in a real-time auction however, the DSP needs a way to identify the users it is looking for.
  • SSP123 After SSP123 selects a winning bidder though, it runs one last piece of javascript that forces userl23 to call out to a handful of regular bidders, including DSP456. In that redirect, using a query string, the SSP passes its cookie ID on userl23 (SSPcookieXYZ). Now the user IS calling DSP456's web server and the DSP can request its own cookie from userl23 in a process the industry refers to as "piggybacking". Eureka! SSPcookieXYZ also has DSPcookie789 - it's user 123 !
  • the DSP knows the SSP's cookie ID because of the query string in the piggyback call, and it can read its own cookie ID because that user called its web server as the end destination with the piggyback call.
  • Pubmatic http : / / ads . pubmatic. com/AdServer / j s sync uppixels . html It looks aS though VCode IS the item passing the cookie ID.
  • Admeld appears to take a slightly different solution, passing a browser partner by partner with callbacks to Admeld on each run rather than calling out to a handful at once like the SSPs above. I'm not able to identify the logic here as easily, but they must accomplish the same end result.
  • the SSP's tag is a chunk of JavaScript that in turn references other scripts. That's pretty common with JS ad tags in general, actually - they usually have to do a few things to get the ad to work as designed. Sometimes they need to reference jQuery libraries, access browser plugins, compile information from other servers, reference cookies values, and a bunch of other things.
  • the first sub-script usually redirects the user to the SSP's server which starts the auction process.
  • the last sub-script is a call which tells the browser to go call a bunch of other companies for the sake of cookie syncing users.
  • this last script in practice may run more or less simultaneous to the auction / ad load, because by then, the browser has been redirected downstream to other parties and has moved on to the next part of the tag.
  • Most modern browsers can keep 6 - 8 HTTP connections active at any given time, so once it executes the first sub-script, it starts following that path with some of the available connections and starts executing the next sub-script with others, depending on what else it has to load on the page.
  • DSPs and SSPs must stay as separate systems - many expect clients to abandon Admeld because of the conflict of interest for Google to represent both a buying and selling technology, but I can't say I know that it's actually happened in practice. As far as I know Google keeps those two companies very separate, and there's no reason to think that Invite and Admeld don't effectively operate as two separate, independent entities.
  • an advertiser may have offline data and work through a match partner and data management platform (DMP) to bring that data online to a cookie, in which case the DMP would need to sync with the DSP before those users would be active in the exchange. That's a much less common use case however, and in that instance, the advertiser doesn't really have a cookie in the classic sense, dropped from their own domain, but really reusing someone else's cookie space for their own purposes.
  • DMP match partner and data management platform
  • DMPs integrate with DSPs through cookie syncing.
  • a system to system sync only needs to happen once however, not once per system per client.
  • the bridge benefits all the clients on both systems, who can move securely move proprietary data from one place to the other through a server to server integration.
  • the DSP and the SSP have a mutual benefit in syncing cookies - browsers only allow a cookie to be read by the domain from which it was set, meaning the SSP can only read its own cookie on a user, and can't know if a DSP also has a cookie on that user.
  • the SSP can only show bidders its own cookie ID, it can't provide the DSP's cookie ID. So it's critical then that the DSP have a way to know if it has a cookie on that user, and what it knows about that cookie, from the SSP's cookie ID only. The way it does this is through the cookie syncing process.
  • the DSP doesn't really have anything to gain in that scenario. It's not as if the SSP knows anything meaningful about that user through the match, they just know if the user exists in the DSPs system, and vice versa.
  • the cookie match simply facilitates the transaction. And the fact of the matter is that DSPs don't bid more for un-synced cookies, they bid less, because they don't know if they know anything about that user. Even if they were lying some of the time, in general, the DSP will likely bid less for unsynced users overall, so the SSP won't favor them at all in that case.
  • lxl pixel is basically, "you have a mechanism to track a conversion event" - that might manifest as a simple image tag for a transparent lxl, or a piece of javascript may configure that conversion beacon call in some kind of dynamic fashion, or you may simply drop a cookie on that user through an image call. But whatever the specific technical execution, the point is the same, to track a conversion, and associate it to a specific user and that user's other actions and touchpoints.
  • Ad Tags are the HTML code a browser uses to fetch an advertisement from an Ad Server - it is a redirect to content rather than content itself.
  • click tags, action tags, view tags, and other more specific variants to the general ad tag category For this particular example, we'll look at publisher side tag, because our purpose is to show how ad tags help publishers organize their content into targetable products.
  • this is the site code that Doubleclick uses to distinguish one publisher property from another. For example, the New York Times owns NYTimes.com, About.com, and Boston.com among other properties. If they are a client of Doubleclick, the corporation likely pays the bill, but each site would have its own site code so ads could be targeted to a specific paper and not the entire network.
  • the zone is akin to a channel level, so the Homepage vs. the Arts page, vs. the Sports page.
  • These content verticals are likely to attract different advertisers, so it's important for publishers to be able to target to this kind of granularity.
  • zone level the last level in which you can pull historical reporting. So you might have sports/baseball or even sports/baseball nymets so you can pull traffic statistics going back months or years.
  • zones are vertical structures, so if you had multiple verticals on your site that all had a games section, you would have to select each games zone every time you wanted to target all games when trafficking the ads, rather than just targeting a single "games" key value. This sounds easy on paper, but adds up to lots of extra time for your trafficking staff if you have lots of subcategories in each zone. It wouldn't be difficult to imagine needing 50 zones or more per content vertical to tag to the lowest level of granularity.
  • topic level next in the hierarchy is the topic level.
  • k xyz - the keyword segment isn't really another level in the hierarchy but a way to describe the page for contextual targeting.
  • the benefit here is multiple keywords are allowed. These are typically used in guides and directories like a recipe, where you would want to be able to target chicken recipes vs. vegetarian recipes vs. winter recipes, and etc, allowing some overlapping targeting.
  • tile l - the tile variable sets a unique value for each ad call on a specific page. If there were two or more of the same size ads on a page, separate tile values would prevent the browser from trying to serve the same ad to multiple ad slots at the same time.
  • siot 728x90.
  • i - typically defines the location of the ad tag, but is really just another type of key-value. While this may seem duplicative with the tile value, it isn't.
  • websites use different templates all the time so the homepage may not have as many ad calls as a category page which may have a different number of calls than an article page, so the tile value isn't designed to be a consistent variable for use in targeting. The slot however, is.
  • the slot value allows them to target specifically to one or the other. That said, the publisher could just as easily set the value of this to anything they want, and it's common to see sites re-purpose this keyvalue for another purpose, use a text value such as "leaderboard", or not use it at all. See Jared's post in the comment thread below for more detail.
  • slot 728x90.1— this is just another keyvalue. you could easily make the difference between the top and bottom space by making it pos-top or pos-bottom on each ad call, keyvalues are totally open so based on our preference in work flow you could make it whatever you want.
  • utm campaif3 ⁇ 4n kzfoxlxphltlltov2xrn

Abstract

Users' browsing histories and other online activities are commonly tracked using cookies, and employed to customize users' web experiences. In accordance with certain aspects of the present technology, microphones, cameras, and other sensors of portable computing apparatuses are employed to gather information about users' offline experiences. This information can be used - alone, or in conjunction with traditional cookie data - to enable systems to adapt their behaviors based on a fuller view of user's circumstances. A great variety of other features and arrangements are also detailed.

Description

PHYSICAL CONTEXT AND COOKIES
Related Application Data
This application claims priority to U.S. provisional applications 61/734,763, filed December 7, 2012, 61/738,632, filed December 18, 2012, and 61/903,559, filed November 13, 2013, the disclosures of which are incorporated herein by reference.
Background and Introduction
Much of the online economy is driven by advertising; some reports estimate the amount spent on internet advertising in 2011 exceeded $80 billion.
In addition to end users, there are two classes of players in online advertising: companies that want their ads seen (e.g., Dell and Delta), and publishers that have online ad space available for sale (e.g., seattletimes<dot>com). The former companies are often termed the "demand side," and the latter publishers are often termed the "supply side."
Software tools are commonly used to automate both the supply and demand sides of online advertising.
"Demand Side Platforms" (DSPs) are software tools used by companies buying ad space (e.g., Dell and Delta). A company using a Demand Side Platform provides information about the target audience to which its ads should be directed, and a budget (e.g., daily or weekly) for the ad spend. The DSP software spends the budget to buy ad space on online sites where it determines the company's ads will yield the best return.
"Supply Side Platforms" (SSPs) are software tools used by online publishers who have ad space to fill (e.g., seattletimes<dot>com). The SSP software discerns information about a user who visits the web site (typically through use of cookie data), and determines which ad should fill an available ad slot in the web page delivered for that user's visit. (In placing third party ads, the SSP software typically conducts a quick online auction to identify the vendor willing to pay the most. SSP software is also used by retailers in placing ads promoting their own
merchandise.)
When Joe Public requests a page be loaded from seattletimes<dot>com, a cookie on his computer is read and allows access to an associated dossier of information. This dossier - typically a file in a remote database maintained by an SSP vendor - contains history data about Joe's other online activities/experiences. It may also include other demographic data, from public and proprietary databases, etc. This information about Joe prompts a flurry of activity to fill an available slot on the seattletimes<dot>com web page with an ad from a brand that wants to tempt Joe.
The cookie is a gateway to stored context data about Joe, allowing the SSP to identify an advertiser to whom Joe represents a high-value customer. Naturally, the SSP wants to sell the ad slot for the best available price. The more candidate advertisers know about Joe, the more confident they can be in determining whether Joe is a close match to their target customer. The more advertisers know about Joe, the more confident they can be in paying top dollar to present Joe their ads.
A cookie assigns an identity to a user, and allows access to information about the user's activities. But these activities are always digital.
In accordance with certain aspects of the present technology, certain analog activities of the user are also memorialized, and help identify particular ads best suited for presentation to that user.
The foregoing and other features and advantages of the present technology will be more readily apparent from the following detailed description, which proceeds with reference to the accompanying drawings.
Brief Description of the Figures
Fig. 1 is a flow chart of an illustrative embodiment, from the viewpoint of a user device. Fig. 2 is a flow chart of an illustrative embodiment, from the viewpoint of a remote service.
Fig. 3 is a diagram of another illustrative embodiment.
Fig. 4 is a block diagram showing components of an illustrative system.
Detailed Description
The present technology has broad applicability, but necessarily is described by reference to a limited number of embodiments. The reader should understand that the technology can be employed in various other forms - many quite different than the arrangements detailed in the following discussion. A first embodiment involves a smartphone or tablet "second screen" app. Such apps are often used in conjunction with television programs (and some radio broadcasts) to present auxiliary content that is complimentary to the primary television (radio) content. For example, such an app may present player stats to viewers of a football game, or present trivia quizzes to viewers watching a sitcom.
Shazam, Zeebox and IntoNow (Yahoo!) are exemplary second screen apps. Other second screen apps are more specialized, sometimes being tailored to a particular television show (e.g., Grey's Anatomy or NCIS), or to a specific broadcaster.
Second screen apps typically identify the primary content to which the user is being exposed through use of audio watermarking or fingerprinting technology. These techniques process sampled content to derive corresponding identification information. Alternatively, program identification information can be broadcast (e.g., by Apple's Bonjour service) on a local wireless network to the second screen app from the television system, or from another device involved in delivering the primary content to the user.
Regardless of how the primary content is identified, second screen apps complement such content by identifying one or more corresponding items of secondary content from which the user can choose. These secondary content items are typically identified by querying a database using the program identification information. Sometimes, the secondary content items are identified (or conveyed) with the primary content stream.
Consider a radio station, KXYZ-FM, that distributes a tablet app with second screen features (somewhat a misnomer in the case of radio, since there is no primary "screen"). When the app is launched, it downloads the latest version of an audio fingerprint or watermark detector (e.g., JavaScript) from the KXYZ-FM web site. This detector listens to ambient audio, and seeks to identify it (e.g., by computing fingerprint data, and forwarding to a database for matching against a collection of fingerprints for reference content). For example, ambient audio in the user's environment may be identified as a Bob Dylan song (e.g., "Po' Boy"). A database of secondary content is next consulted and may indicate, e.g., that the app should respond by presenting an on-screen quiz testing the user's knowledge of Bob Dylan trivia.
(Sometimes the app may successfully identify the primary content being rendered in the user's environment (e.g., a song by The Eagles, or audio from local television channel 6), but find that the secondary content database does not identify any secondary content that corresponds.)
When the tablet app transmits detected watermark or fingerprint data (or other identification information) to a remote server for response (if any), it also transmits an identifier of the tablet - or its user. Commonly, but not necessarily, this identifier takes the form of a cookie. If KXYZ-FM uses a Supply Side Platform from SuperCo, the tablet's data exchange can be arranged so that a 'superco' cookie is transmitted from the tablet to the SuperCo server. (An exemplary cookie comprises a small file containing random data, e.g., A86FC2, with a descriptive name, e.g., superco.txt. The random data allows cookies from different users to be distinguished.)
Also transmitted to the SuperCo server, for association with the user's cookie, is information identifying the ambient audio. For example, this information can comprise a song title or other metadata identified by looking-up a decoded watermark identifier in a metadata database (Bob Dylan's "Like a Rolling Stone;" The Eagles' "Hotel California"), or it can be title or other metadata information for a television program, identified by matching derived fingerprint data against reference data in a fingerprint database (e.g., the October 7, 2012, 60 Minutes broadcast). Alternatively, the identification information can comprise a TV program title as broadcast by the television system, etc. (It may also comprise the derived audio fingerprint data, or extracted watermark ID, although this is less common.)
This identifying information is sent to SuperCo by the tablet (e.g., the tablet app), or by KXYZ-FM, or by a third party involved in associating identification information with metadata.
In some instances, not only is the identified content associated with the cookie; so is the delivery channel by which the content was delivered to the user. For example, fingerprinting may be employed to identify the title of the content (e.g., Seinfeld, episode 125), and watermark decoding can reveal its distribution channel (e.g., broadcast by KABC-TV, on October 5, 2012, at 10:00 pm). Similarly, a broadcast from a television system may indicate that the television is tuned to Cox Cable channel 47.
Still further, information that identifies the content recognition software or second screen app on the user's tablet (e.g., the KXYZ-FM second screen app), can also be sent and stored in association with the cookie. All of this information is context data - useful, e.g., in identifying advertising that is relevant to the user.
For example, consider what happens if the user of the KXYZ tablet app later visits the music page of the amazon<dot>com web site. Amazon wants to place a relevant ad on the web page delivered to this user. If Amazon uses the SuperCo SSP software, the user's tablet will send the superco cookie to the SuperCo server as part of loading the Amazon web page. By reference to stored data associated with the cookie, SuperCo will see the user's audio environment has included music by Bob Dylan and the Eagles. Knowing this context about the user, SuperCo can select - from the Amazon inventory of available ads - an advertisement promoting a newly-released boxed set of Bob Dylan music CDs.
If the user clicks the Dylan ad, credit may be due to KXYZ-FM as one of the
infrastmcture players whose involvement led to the user's click. A small payment may be due to KXYZ-FM (or a larger payment, if the user actually completes a purchase of the Dylan boxed set). Additionally or alternatively, if a watermark or other information identifies the distribution channel through which the sensed Dylan audio was provided to the user's environment, that entity may also deserve a tribute payment.
In the example just-given, the selection of an ad for a Dylan boxed set was based on the user's content consumption history. But the same principles can adapt the presented
advertisement (or other content) to the content detected in real-time in the user' s environment. For example, if the ad for the Dylan boxed set is initially detected and then, while the user is still on that Amazon web page, his device detects audio from the television show Mad Men, the system can swap-out the Dylan ad and replace it with an advertisement for Mad Men-related merchandise available on Amazon.
Such updating of ads in real time is facilitated by web 3.0 standards, which enable ongoing communication between a browser and a web page server. Knowledge about the user's instantaneous environment allows still better tailoring of promotional content - better suiting the user, the advertiser, and the supply side provider.
Moreover, the context information that can be fed-back to the web page server (or other destinations) is not limited to information about media content in the user's environment.
Information about mouse movement or other user interaction, for example, can also be relayed from the user's device, and serves to signal that the user is active on the web page - and is not off in the kitchen, etc. Again, more accurate characterization of the user and his context leads advertisers to bid higher for available ad slots.
It will be recognized that cookies historically have been used to make assertions about users' digital activities, e.g., User A visited web sites X, Y and Z; User A clicked on a news article about Lance Armstrong and on an online catalog item offering a Shimano derailleur. The present technology extends this capability to aspects of the user's physical world, e.g.: User B listened to Bob Dylan music (even if on an analog radio), and watched a documentary film about industrialized farms (even if in an analog cinema).
Presently, content is typically identified "on-demand" (e.g., by pressing a ListenNow button on a second screen app). In the future, however, tablets and other devices may routinely perform content identification as a background operation, e.g., as an operating system service rather than as part of a second screen application. With the user's permission, these content identifiers can all be stored in association with a user identifier (e.g., cookie), providing a rich source of context by which the user' s needs may be better met.
While cookies are presently used primarily for ad serving, their basic concepts and usage models can be employed more extensively. Consider a user who consumes Spanish language media, primarily. This fact can be discerned by examination of the content IDs that are stored in association with her cookie identifier (e.g., identifying newscasts from the Univision network, telenovelas, etc.). When this user visits the web site of Acme Appliances for the first time, the cookie data can be sent and serve as a type of profile information. For example, the Acme web site can determine, from a listing of media content consumed by the user, that Spanish is a favored language. Accordingly, a Spanish language version of the Acme web page can be delivered to the user instead of a default English language version.
(Language can also be determined by an audio classifier, which takes audio-related information as input, and outputs a signal indicating the language of the input audio.)
Age can be similarly inferred from content consumption habits. (Audiences for movies directed by Tim Burton tend to be under 30 years old; audiences for movies directed by Federico Felini tend to be over 40 years old; etc.) Educational background may also be correlated to media consumption (relatively few high school dropouts favor films by Akira Kurosawa).
Again, web or other content delivered to the user may be tailored in accordance with demographic classifications, for which the user's history of media consumption can serve as a proxy.
It should be recognized that traditional cookies are not needed to practice the present technology. In some instances, use of traditional cookies is actually an impediment, since browsers and other applications typically place strict limitations on their use (e.g., a Superco cookie can be sent only to Superco - not to Acme Appliances, etc.) But other identifiers can serve in a similar capacity.
An implementation that does not use traditional cookies involves the Apple iCloud service. iCloud enables sharing about history of browsing states between a user's devices. A user who leaves a web page on a desktop computer to commute home, can open the same web page on a tablet (using the iCloud infrastructure) and continue browsing on the ride home, from the point she left off. Similarly, a history of the user's analog content consumption, as determined by one device, can be stored in the user' s cloud account and employed to tailor other content delivered to the user on the same or other devices. Here, no cookie is used. Instead, the user is identified by the cloud account into which she is logged- in, and in which content consumption history is logged.
Cookies, or other identifiers, can form part of metadata statements made by the user's device. Consider a user's tablet, which is listening continuously to the user's environment and identifying the content it hears. Each time a new item of content is identified, and/or each short interval of time (e.g., every 2 minutes, 5 minutes, or 10 minutes), the device spews a cookie, associated with a content identifier (e.g., an IS AN identifier: International Standard Audiovisual Number). The device memory may have a stored cookie with the file name SSP123.txt, which stores the value A86FC2, issued by Supply Side Provider Superco. Each time content is identified, the device writes a metadata statement that includes the file name or value of the cookie, combined with the content's IS AN Identifier - forming a historical record of the user's environmental state. This information may be stored locally on the device, and/or transmitted for storage a cloud account associated with the user (e.g., iCloud), and/or transmitted for storage in a database maintained by Superco, etc. The metadata statement may take the form of a linked data RDF (Resource Description Framework) triple. (See, e.g., patent publication 20120154633, and references cited therein, for more about linked data.) Just as content identified from the user's environment can serve as context, so, too, can other sensor data from a device, e.g., accelerometer, gyroscope, temperature, barometric pressure, GPS/location, etc. All such information is useful in serving up relevant ads, and otherwise tailoring experiences delivered to users.
In some instances, audio context can serve as a proxy for location information. Consider a user shopping in Wal-Mart, with a Wal-Mart app running on his smartphone. To use the app, the user has logged- in with his Sam's Club member ID and password.
While shopping, the user pauses in front of a display of Rubbermaid products - inspecting storage bins. The app includes an audio WM detector, and the store plays differently- watermarked background music in different zones of the store. The app's watermark detector decodes a watermark indicating that the user is in the Housewares-Bin Storage section of the store. The app - or a remote server with which it communicates - notes this fact, and sends it for storage in a SSP database, associated with a supply side cookie from the user's smartphone. The fact that the user remained in that part of the store for ten minutes is also recorded in association with the cookie.
When the user later goes to check-out, he presents his Sam's Club membership card. The user's items being purchased are tallied for checkout, and corresponding item identifiers are also stored in a database association with the user's Sam's Club ID. Curiously, the user did not buy any Rubbermaid storage bin - despite pausing in that section for ten minutes.
Based on the user's apparent interest in such a product, and his failure to purchase, Wal- Mart expresses its interest to present a Rubbermaid advertisement to that user, the next time an available ad slot comes up on a web site the user browses. Sure enough, at home that evening, the user fires up a tablet computer and navigates to a sports news site. That web site receives the user's cookie as part of the initial data exchange, and advertises availability of an ad slot on that user's tablet to the highest bidder. With its knowledge of the user's physical presence at the Rubbermaid display in its store earlier that day, Wal-Mart makes an offer to buy the ad slot at a price that no other advertiser could justify to pay. It wins the auction and presents to the user a display ad for Rubbermaid storage units - touting free shipping on orders over $25. The user was earlier chided by his wife for not having brought one home from the store, so clicks the ad and completes a purchase. Everyone is happy. Wal-Mart is glad to conclude that sale, but is concerned that the user - and two dozen other shoppers this week - lingered at the Rubbermaid display, but did not purchase such a product while they were in the store. That's an opportunity for improvement. Alerted by this information from the system, the Wal-Mart store manager goes out on the floor to assess the display, and decides it might do better re-positioned to be closer to office supplies, rather than amidst picture frames and pillows. The next week the manager's move is vindicated - sales of Rubbermaid storage units are up, and fewer shoppers lingered without purchasing.
Applicant's previous work, detailed in patent publications 20110212717 and
20110161076, described how information about a user's auditory or visual environment can be published to an auction marketplace, where bidders compete for the opportunity to provide related services. The present technology is well suited for use in such applications. In particular, a system can publish the user's sensed context information, with cookie information. The context information can be sampled audio, identifying information (e.g., fingerprint or watermark data) derived from the sampled data, and/or metadata (such as song title or movie title) identified by reference to the identifying information. The context information can additionally include other context variables, such as location information, motion data, etc. The cookie information can serve as an anonymous identifier of the user, by which other profile information about the user can be obtained.
Additional Disclosure
While the present disclosure has focused on cookie-like use of physical context information sensed by a microphone, it will be understood that there are many different types of physical sensors, and all may be used with the present technology. For example, the technology can be practiced with camera data, magnetometer data, RFID/Near Field Chip sensors, etc.
Camera data can be used to identify physical objects near the user, or with which the user interacts. (This will become increasingly prevalent as head- worn computing apparatus proliferates.) Similarly with NFC data, sensed from NFC (RFID) chips in the user's
environment. Olfactory sensors can provide further information about the user's environment. Cookies are suitable to represent all such information.
In accordance with another aspect of the technology, when a user visits a web site (e.g., using a tablet computer), the web site may launch a Java app (or call a Java Native Interface) that talks to one or more of the tablet's sensors (with user approval) and collects sensor data. This information is stored in association with a cookie (which is written, if not pre-existing).
Additionally, or alternatively, this sensor information - polled at the request of a remote computer associated with the web site, may be hashed to yield a Globally Unique Identifier (GUID). The GUID may, for example, be a function of events/information that one or more of the above-noted sensors has observed in the prior interval of time, e.g., 5 or less, 10, 30, 60, or 300 or more, seconds or minutes. This GUID can be written to a cookie (i.e., stored in a database in association with an identifier). This can be useful when it is desirable to place the user and/or device in a particular physical context, without revealing details,
If devices of two users, both in the same physical environment (i.e., having shared sensor context) at the same time, create such GUIDs, they should match, or be within an error tolerance apart (e.g., within a specified Euclidean distance or percentage of each other). If the fact of such co-presence is thereby established, a computer device might allow the two users to interact in a manner that might normally be restricted. For example, music or other entertainment content available to one user might be made available to the other user. If the two users visit the same web site/service, it may invite the users to share information, chat, listen to the same audio stream, etc., based on the common physical/sensor history (which may have been at a previous time).
In such an arrangement, a user's context information (or a hash of such information), stored at different times at a cloud repository, in association with a unique, anonymous identifier of the user (or the user's system, e.g., a cookie), enables a variety of powerful capabilities.
For example, third parties (e.g., supply side providers/platforms, in currently-popular systems) employ cookies to leverage historical context about the user. The cookie contains such historical information, or provides a key into a contextual database in which a record of the previous contextual history of the user/device is stored.
Consider Steve's smartphone or wearable computer system, which periodically reports context information to a cloud database. (Headworn computer devices are discussed, e.g., in applicant's patent application 13/651,182, filed October 12, 2012.) When visiting a web site, Steve's device senses information about his physical context - what the microphone is hearing, what the camera is seeing, the barometric pressure, Steve's GPS coordinates, etc. Related information is written to a cloud repository, with a cookie identifier. For example, on hearing a particular sound S, the device sends hash/fingerprint data FS. The information stored in the cloud may not identify Steve by name, but it indicates that a person/device associated with a particular cookie identifier experienced sound that yields a fingerprint FS, at a particular date and time (and, if authorized, location).
As Steve browses the web with his smartphone, additional information is sensed (perhaps triggered by queries from the web server or another remote computer, which Steve's device is authorized to answer). This information, or derivative information, is sent to the cloud for storage - as part of a cookie cache, or associated with a cookie identifier. The cookie may identify Steve to a particular web site, or to a particular service (e.g., Google, or another ad network). Sensed sound information yields derivative data SOUND- DF34A967EA; a recognized Bob Dylan song yields derivative data SONG- 8FF7A9D66C; a decoded digital watermark from The Daily Show playing in the background yields derivative data
DWMAUDIO- 0BEF838E26; a voice recognized to be Steve's friend Bob yields derivative data SPEAKER- 2D2A54A4DF; recognized speech by Bob yields derivative data SPEECH- 1A7BC15AA6; Steve's location at 45°18'18"N 122°58'2"W yields derivative data GPS- 9389DEB5C3; the current barometric pressure yields derivative data BARO-EF0B0E93F5; a can of Coke glimpsed by the smartphone camera yields derivative data PRODUCT- CFB9800146; a garlic aroma from food simmering in the kitchen and sensed by the phone's olfactory sensor yields derivative data SMELL- AD33A3E8CC; Steve's heart-rate, sensed by a biometric sensor, yields derivative data HEART- 3F88B65334; etc., etc.
Thousands of contextual assertions are thus made in connection with Steve's cookie identifier. If Steve's cookie identifier is A9C1B87, then a cloud database may contain entries including:
A9C1B87:S0UND-DF34A967EA
A9C1B87:S0NG-8FF7A9D66C
A9C1B87:DWMAUDIO-OBEF838E26
A9C1B87:SPEAKER-2D2A54A4DF
A9C1B87:SPEECH-1A7BC15AA6
A9C1B87:GPS-9389DEB5C3
A9ClB87:BARO-EF0B0E93F5
A9C1B87:PRODUCT-CFB9800146 A9C1B87:SMELL-AD33A3E8CC
A9C1B87:HEART-3F88B65334
Each such entry is typically time- and date-stamped (and less frequently location- stamped), but such notation is omitted above, for clarity.
The database naturally includes similar records for other individuals and/or devices (i.e., other cookie identifiers). Thus, similar contextual assertions are stored for Tom (cookie
B56789C), Dick (cookie C581505) and Harry (ED7FE8B). The aggregate collection of such entries can be inverted, and sorted by the contextual statement (rather than by cookie). Such operation may reveal that Tom and Dick - like Steve - listen to Bob Dylan music. Such operation may further reveal that Bob's voice has also been recognized in audio sensed by Harry; and that Steve and Dick sometimes spend noon hours on weekdays together. (Again, nearest- neighbor constructs can be employed to deal with derivative data that is slightly different, but corresponds to the same or similar information - like GPS information.) Social network linkages can be gleaned from such information (e.g., the apparent social relationship between Steve and Dick, and Steve's and Harry's evident exposure to Bob).
In another arrangement, Steve is surfing the web, as before. However, instead of the cookie/context information being sampled and uploaded during a visit to a particular web site or service, it is routinely and periodically logged (e.g., every five minutes, or other interval as noted above) by the operating system of Steve's device, e.g., Android, during the period that Steve is using the device. Steve may be using a tablet that is provided to him free of monthly charges, by Google's subsidiary DoubleClick. In exchange, whenever Steve visits a web site for which DoubleClick is an advertising supplier, Steve's operating system uploads to DoubleClick the stored, historical DoubleClick cookie information gathered by the operating system since the last such upload. Alternatively, code in the operating system (or a resident application program) can upload logged cookie information to DoubleClick in response to other another trigger, such as the first browsing session of each day.
All such arrangements serve to create a historical record of context, so that subsequent supply side advertising events can be better matched to Steve's circumstances.
Reviewing the prior art, consider what happens when Steve's browser navigates to an article in the sports section of the online New York Times, concerning new safety standards for wooden baseball bats. The New York Times web server replies with HTML code that tells Steve's browser where to get content for that page, and how to format it. As is familiar, part of this returned code includes an ad tag URL that, e.g., directs Steve's browser to a DoubleClick ad server. The ad tag takes the form of an http string that - in addition to including the host address for the DoubleClick ad server (http: //ad<dot>doubleclick<dot>net/), also includes other information, including a code identifying the New York Times web site, a topic or zone code indicating that Steve is within the sports section of the paper, and a sub-topic/sub-zone code indicating the requested article relates to baseball. This hierarchy allows more precise targeting of advertising, and optimization of ad revenue. (E.g., the New York Mets baseball team may pay a penny to present an ad (offering upcoming game tickets) to a reader of the New York times, but may pay 2 pennies with knowledge that the reader is in the sports section, and may pay a dime with knowledge that the reader is interested in baseball.)
In accordance with the certain embodiments of the present technology, the URL may be created - or modified - dynamically, as a function of context. For example, a software component on Steve's device (e.g., resident in the operating system, or part of a browser plug-in, or Java code running with a web page) can collect sensor information about Steve's physical context, or can read context data gathered earlier and stored locally. It can then dynamically form an ad tag URL, e.g., by appending such context information to the ad tag provided from the web site. That is, whereas prior art ad tags were static, a tag can instead start with a static part (e.g., from the web site), and build from it a dynamic ad tag URL employing physical context information.
(Such an arrangement can alternatively, or additionally, recall cookie information for the web site earlier stored on Steve's device, or recall an ad tag cached on the device from an earlier visit to that web site, and author an ad tag URL with such information as a starting point.)
Consider, next, a point of sale (POS) system in a bricks-and-mortar grocery store, which identifies purchases with users -without use of a store "loyalty" card. Each POS terminal has one or more sensors, such as a barcode scanner, a camera, a microphone, a scale, etc. Shopper Steve carries a smartphone or other such device with its own set of sensors, and code (e.g., in the operating system or an application) that publishes contextual information in an anonymous fashion.
As Steve is checking-out at one of the store's several POS terminals (e.g., in checkout Lane 4), purchased items are scanned by the POS barcode scanner. Objectl is a six-pack of Sprite soft drink. Object2 is ajar of Old El Paso salsa. Object3 is a can of Science Diet cat food. Etc. The barcode scanner is linked to the POS system, and reports a GTIN (Global Trade Item Identifier) number as each object is scanned (e.g., 549410582762, 923364619460,
837280103520, etc.)
The POS system relays each of these GTIN identifiers, as it is received from the scanner, to a cloud-based database, together with information identifying the store and checkout terminal (lane) from which the data originated. The database time- and date-stamps this information, and stores it.
The barcode scanner, or the POS terminal, also emits feedback signals that are sensed by Steve's smartphone. In one particular arrangement, each time the barcode scanner reads a GTIN code, it emits an audio signal that encodes the GTIN identifier. For example, a chord of 13 different tones can be sounded for 200 milliseconds - with each tone drawn from a different library of ten tones, identifying a 0-9 digit at a different position in the GTIN string. Or an audio signal can be frequency-shift-keyed to convey 13 ASCII character codes coiTesponding to the 13 GTIN digits, at a data rate of 300 bits per second, including error correction overhead. (Of course, other forms of feedback signals can be employed, such as Bluetooth and other wireless data, and ultrasonic audio.)
Steve's smartphone senses this feedback signal (with a microphone, in the foregoing arrangement), and decodes the GTIN identifiers. These decoded identifiers comprise a sequence that is temporally aligned with the time-stamped sequence of GTIN identifiers received by the cloud database from the store's POS system. The smartphone stores these decoded identifiers, and also sends the sequence of decoded identifiers to the cloud database (with an anonymous identifier assigned to Steve, or hashed from information specific to Steve; such an identifier may be the letters DFGHJ). The database matches the smartphone-sent GTIN sequence with a GTIN sequence from the POS terminal in Lane 4 of that grocery store. Anonymous Steve is thus associated with the purchase of the soda, salsa, and cat food, in Lane 4, through the smartphone' s publication of sensed audio context information to the cloud repository.
The set of identifiers thus serves like a fingerprint, by which Steve's checkout transaction can be identified, and distinguished from other shoppers' checkout transactions.
The next time Steve enters the store, this historical record of purchases can be recalled based on the anonymous identifier DFGHJ, and used to provide coupons, targeting advertising, etc. (Steve's smartphone may have app software, distributed by the grocery store, in which the DFGHJ identifier is stored, and which serves to present coupons and other information.) This provides loyalty card-like functionality, without any use of a loyalty card. Nor does it make any use of Steve's credit card or debit card number (a technique on which some other shopper- identification systems are based).
(Loyalty rewards may similarly be provided to Steve outside the grocery store, e.g., a discount on fuel purchased at a gas station, based on a prior month's tally of purchases made at the grocery store.)
Moreover, this information has utility outside the grocery store. The DFGHJ identifier can be used like a cookie - to permit access to this record of physical shopping history elsewhere. For example, Steve may later view a Monday night football game, while surfing the web on the smartphone. The smartphone microphone senses the game audio, which enables identification of the football broadcast by audio fingerprinting or digital watermark decoding. A corresponding contextual assertion about Steve's activity is written to cloud storage. When Steve surfs to the front page of the New York Times, the phone authors an ad tag URL that conveys information about his activity (watching football), and also conveys Steve's DFGHJ identifier. The GTIN codes that are cloud- associated with this identifier reveal Steve's brand preferences. (Part of each GTIN code is a plural-digit "Global Company Prefix" field, identifying the company that provided the product.) When the ad server is queried for an ad, it can take into account Steve's football-watching activity, and Steve's historical preference for Coca Cola products over Pepsi products (as evidenced by all the Coke-prefixed GTIN codes associated with the DFGHJ identifier). It can then, e.g., select a football-themed Coke ad for presentation on the New York Times front page that Steve's device is presently loading.
In the barcode scanning example given above, the POS terminal provides feedback data memorializing the GTIN codes for the objects Steve is buying. The same functionality can be achieved without such feedback GTIN code data. For example, each time the barcode scanner in Lane 4 decodes a barcode, it can emit a two-tone, 200 millisecond beep, e.g., 1200 & 1300 Hz. (Other lanes can do likewise, with different tone pairings.) As before, each time the POS terminal reads an object's GTIN code, it sends the code to the database, which makes a time- stamped record. Meanwhile, Steve's smartphone senses these beeps, and reports each such detection to the database, with his DFGHJ identifier. Again, the database time-stamps and stores such information.
In this case, a temporal fingerprint is defined by the intervals between GTIN reports from the POS terminal (e.g., 1.3 seconds, .9 second, .9 seconds, 1.1 seconds...). This temporal fingerprint is matched with a corresponding temporal fingerprint defined by the smartphone's report of beep detections. Again, Steve's shopping history is discerned, and stored in association with his anonymous identifier.
Steve's smartphone may detect and report the scanner beeps while still in Steve's pocket. After a match is established with the temporal sequence of GTIN codes reported by the POS terminal (e.g., after six or eight items have been scanned), the cloud database can look up any electronic coupons stored in a cloud wallet associated with the DFGHJ identifier, and can inform the POS terminal of their particulars (e.g., 50 cents off a six-pack of Sprite). The POS terminal can make the adjustment in the checkout tally - without Steve having identified himself in the store, and with the phone still in his pocket. (Or, if the coupon data are stored in the smartphone instead of in the cloud, once the temporal sequences are matched, the database can query the smartphone for its coupons. The smartphone can send its collection of coupon data, and the cloud can relay any that apply to the POS terminal. Once they have been redeemed, such information can be reported by the POS terminal to the cloud, which in turn confirms such redemption to the smartphone wallet.)
In other arrangements, the matching of temporal sequences can be performed by the smartphone, rather than the cloud database. For example, the store can broadcast - on its WiFi network - GTIN codes decoded by each POS terminal barcode scanner in the store, with each GTIN code being time-stamped and paired with an identifier of the POS terminal. The smartphone can derive temporal fingerprints for each POS terminal from such information, and match one such fingerprint to the temporal sequence of beeps it detects during checkout. When a matching sequence is identified, the smartphone transmits its coupon data to the store POS system (over the WiFi network, or otherwise), noting the POS terminal tally to which the coupon credit should be applied.
In alternative arrangements, the list of GTIN codes reported by the POS terminal in Lane 4 can be associated with Steve, simply by Steve's smartphone reporting its GPS location to the database, with his DFGHJ identifier. Alternatively, Steve's phone can gather information indicating his location in Lane 4 by a short range wireless beacon that marks that lane, or by the distinctive frequency of confirmation beeps issued by the barcode scanner in that lane, or by an RFID chip positioned in that lane, as sensed/reported by Steve's phone.
(It should be recognized that the arrangements described above provides multi-factor authentication of the user - reducing fraud potential. That is, the smartphone serves as a physical token - an ownership factor, and the beeps or other context that the phone senses serves as a knowledge factor.)
Steve may return nearly every day to the same grocery store - each time paying with cash. The store notes his frequent visits, and works to lock-in his continued patronage by offering a branded credit card that provides a 2% cash rebate on purchases made at that grocery chain. The offer is made the next time Steve checks out with a clerk- attended POS terminal, with the clerk verbally extending the offer, and pointing out that details - including a calculation of how much his cash rebate would be based on an interval of past purchases - are printed on his register receipt.
Related technology can be used with roving store clerks, e.g., at a Home Depot hardware store. Steve wants help concerning a particular item he is thinking about buying (if I buy this Moen shower head, will I require a metric wrench set to install it?), and taps a "Help" button on the Home Depot app on his smartphone. The app directs him to take a picture of the item, from which it then may extract identification information. The app also directs a sensor in the phone to capture information that serves to identify Steve's location. (This can be done, e.g., by decoding a digital watermark in music playing in that part of the aisle, or by an ultrasonic or wireless radio beacon in that part of the aisle, or by LED lighting modulated to convey location information, or other indoor location technology.) All such information is written to a remote database, together with an anonymous ID assigned to Steve by the app. Locations of the store's clerks are similarly determined, and a nearby clerk is alerted.
Information about the customer's situation is sent to the clerk's smartphone (e.g., the shopper's location, and a picture of the product, or the name of the product as discerned from a barcode on the package or by object recognition). The smartphone is also sent any other data gleaned from this customer today (such as other Help requests from that anonymous ID, information about other images captured using the store app for price-lookup, information about web sites recently visited by a device having the same IP address on the store's WiFi network as the IP address from which the Help request was received, etc.). The clerk may thereby learn that the customer recently reviewed an Amazon web page about a Kohler shower head, and used a barcode scanning feature in the Home Depot app to learn the price of a waterproof indoor can light fixture.
The clerk walks to Steve's aisle location, and looks for a person puzzling over shower heads. The background information about the customer's interest in shower hardware suggests to the clerk that the customer is involved in bathroom remodeling - helping the clerk better serve the customer.
All such context information about Steve and his interests is stored with a cookie identifier by the Home Depot app, e.g., Doubleclick's cookie XYZ243. Information from the store's POS system is also written to such a cloud database, and memorializes that Steve left the store purchasing only a drill bit.
A week later, Steve is at the airport - still thinking about his remodeling project, and surfs to a home improvement web site while waiting for his flight. Through the DoubleClick cookie, it is found that this user was recently in Home Depot and asked a store clerk about a Moen shower head, but did not purchase it. As a consequence, DoubleClick selects a Moen shower head advertisement to display on the home improvement web site.
(The reader is presumed to be familiar with personalized retargeting of online advertising, using cookies. The foregoing arrangement extends such methods from the online world to the physical realm.)
Reference was earlier made to a POS system that acquires visual information from a retail product (e.g., by a barcode scanner), and relays data derived from such information (e.g., a GTIN identifier) in audio form to a smartphone. This may be regarded as a form of synesthesia - a phenomenon detailed in a Wikipedia article by that name.
There are many instances where such arrangements are useful. One is in a car, driving. The car senses its location, processing information from a GPS radio receiver, The car emits audio or ultrasonic tones representing this location data, and the smartphone senses it.
Reciprocally, information gathered by one type of smartphone sensor can be relayed to a car, and received using a sensor of a different type in the car. Always-On/W earable
Multiple wearable devices are now available in the market, with Sony, Samsung, Pebble and others first to market at scale using a watch form factor. These devices contain multiple sensors and are capable of creating actionable context similar to a smartphone or other mobile device.
By example, the Samsung Smart Gear watch is powered by a single-core 800 MHz processor, with accelerometer, multiple microphones and a camera. In this form factor new possibilities for always-on sensing are enabled. The device is always exposed to both the user's environment (as opposed to being in a pocket or purse) and in physical contact with the user,
Beyond having microphones and other sensors exposed, the placement on the body enables additional opportunities for activity recognition. In addition to acting as a simple pedometer, more advanced gait analysis can be performed providing additional insight in the user's subsequent search queries.
Unlike standalone training tools, such as a Garmin GPS watch or a Nike Fuel band, newer classes of wearable devices are always connected (via Bluetooth, WiFi, etc.). This means that the ability for a user's wearable device to provide information to a supply-side provider regarding training habits, gait analysis, even cardiovascular information can inform the user's subsequent search for a running shoe.
Other, less smart-phone like, architectures can also participate in the described architecture. Sensors that are not battery powered and can store sensor information, similar to a more advanced RFID chip, can also be used. Such sensors can take the form of jewelry, eye- ware, etc. Such sensors when activated in the presence of an electromagnetic field can report sensor results, such as number of activations (or power-ups) since the last time a download was occurred and the ID of those devices that were powering up the sensor.
The above can be thought of as a distributed sensing ecosystem, which can be created by using sensors in-place and shared by multiple users, in combination with battery-less, RFID enabled recorders for each user. Prototype, 3D printed rings have been proposed that carry all of a user's stored value or transit credentials, allowing the wearer to seamless move through turnstiles in many cities. If such a ring was also able to report on activations when queried, a snapshot of the day's commute would emerge. If sensors in the wearer's office building were made available, the ring could store average indoor air-quality during the work day. When queried at home, a picture of the wearer' s environment could be created.
Independent of the embodiment of the sensors and how sensor data is collected, the resulting information can be used in the same form described earlier and made available to the marketplace.
Body Sensors
On-body sensors are increasingly being paired with consumer smartphones for fitness and health. Beyond heart-rate monitoring for exercise, EKG, electro muscular signals, temperature and blood chemistry are being sensed in non-invasive fashions. Such sensors create new opportunities to collect and share context. Blood chemistry, respiratory health
(microphones), heart-rate, all provide insight into what services or products the user may benefit from.
A long day of travel as sensed using a user's tie-clip or other jewelry, which has observed significant changes in GPS location, altitude, temperature, humidity, respiratory behavior, etc., can be a valuable source of context for specific brands. Emergen-C travel vitamins might be very interested in approaching travelers within 48 hours of completing an airplane trip, as may companies that sell products to the business traveler.
Review
A small sampling of some of the inventive arrangements detailed herein are reviewed in the following discussion.
One method includes sensing information about a user's physical - as opposed to computational - environment. The sensed information - or data derived from such information - is transmitted to a remote service for storage, in association with identifier data associated with the user (e.g., a cookie identifier). Then, in connection with a subsequent transaction, identifier data associated with the user allows access to the data about the user's earlier- sensed physical environment. This enables information presented to the user (e.g., a web page, digital signage, etc.), to be customized based on the information about the user's earlier physical environment.
Another method involves, at a first time, receiving information corresponding to audio or visual content sensed from a user's physical environment. Then, at a second, subsequent time, identifying advertising for presentation to the user, based at least in part on the identified information. The second time may follow the first time by a few seconds, but more typically follows it by several hours, or days or more.
A further aspect of the technology is a method in which a visual sensor of a first system acquires visual information representing first data. An audio sensor in a second system is then used to receive audio information representing the first data, where the received audio was emitted by the first system based on its acquisition of the visual information. The first data is extracted from the received audio, using a hardware processor configured to perform such act. Cookie data is then stored, based on the extracted first data.
Another such method involves a first system that includes a first sensor responsive to a first type of stimulus, which acquires information - representing first plural-bit data - conveyed by the first type of stimulus. A second system, using a second sensor responsive to a second type of stimulus different than the first type of stimulus, then receives information - again
representing the first plural-bit data - conveyed by the second type of stimulus from the first system. The first plural-bit data represented by the information received by the second sensor is extracted, using a hardware processor configured to perform such act. Cookie data is then stored, based on the extracted first plural-bit data.
Another method includes, at a first time, receiving information about entertainment ambient content sensed by a microphone in a user's portable device, where the received information is accompanied by an identifier of the user, or the user's device. From this received information, a language apparently understood by the user is determined, and data related to this language is stored. Then, at a later time, a language-specific version of content to be provided to the user (or to the user's portable device) is selected, based on the stored data.
A related method involves, at a first time, receiving information about entertainment ambient content sensed by a microphone in a user's portable device, where the received information is accompanied by an identifier of the user, or the user's device. Based on the received information about ambient content, an age or education of the user is estimated, and related information is stored. At a later time, content to be provided to the user is selected, based on the stored data. (By such arrangement, the user's history of media consumption serves as a proxy for information about age or education.) Yet another method includes sending a request for a web page from a user's device to a first web server. Responsive to this request, the device receives first information, including an ad tag URL. Data - including the ad tag URL (or a modified version of the ad tag URL) - is then sent to a second web server, responsive to which the user's device receives second information. A display is then presented to the user, based on the received first and second information. This method is characterized by sensing physical context data about the user or the user's environment, using a sensor in the user's device, and including information about the sensed physical context data with the data sent to the second web server.
Still another method involves discerning a set of item identifiers from items presented for purchase during a checkout operation by a shopper, using apparatus in a bricks and mortar store operated by a retailer. The set of item identifiers serves as a fingerprint by which that checkout operation can be distinguished from other checkout operations. Information related to the fingerprint is transmitted to a portable device conveyed by the shopper. The device relays this data - together with first information that serves as an identification of the shopper - to a computer system (which may be the store POS system, or another system). This fingerprint information is received, and matched with fingerprint information discerned by the apparatus. The first information, which serves as an identification of the shopper - is associated with the set of item identifiers discerned by the apparatus. This associates the purchased items with a particular shopper - information which is stored in a database. By such arrangement, purchased items are associated with a particular shopper, without the shopper directly providing shopper- identifying information to the retailer during the checkout operation. (The set of item identifiers can comprise an ordered set of such identifiers, to better avoid confusion with similar items purchased at other checkouts, albeit in different orders.)
A related method involves, from a point of sale terminal in a bricks and mortar store operated by a retailer, emitting a signal for detection by a shopper's portable device, each time an item presented for purchase in a checkout operation is sensed. These emitted signals define a temporal sequence that serves as a fingerprint by which that checkout operation can be identified, and distinguished from other checkout operations. The shopper's device receives these signals, and sends data including first sequence information based on its detection of the emitted signals, and also including first information that serves as an identification of the shopper. This data sent by the shopper's device is received, and matched with corresponding second sequence information generated as part of the checkout operation. The first information - that serves as an indication of the shopper - is then associated with the second sequence information, to thereby associate the first information with a particular checkout operation. By such arrangement, the first information - which serves as an identification of the shopper - is associated with a particular checkout operation, without the shopper directly providing shopper- identifying information to the retailer during the checkout operation.
A further aspect of the technology comprises compiling, in a data structure, two or more types of information from the group consisting of: (a) a user's online activities, including web sites visited; (b) entertainment content sensed by a microphone-equipped device conveyed by the user; and (c) a record of items purchased by the user in a bricks and mortar store. Advertising is then selected for presentation to the user, based on these two or more types of information.
Yet a further method includes, at a first time, receiving sensor information from an apparatus worn on a wrist of a user. Data related to this received sensor information is stored in a data structure remote from the user, in association with cookie data that serves as an identifier of the user. Then, at a second time, the stored data is accessed by reference to the cookie data, and used in identifying information to present to the user.
Figs. 1-4 illustrate aspects of the foregoing arrangements.
Concluding Remarks
Having described and illustrated the principles of our technology by reference to certain embodiments, it will be apparent that the technology is not so limited.
For example, while reference was made to sampling audio output from a radio or television, in other embodiments video can be sampled, e.g., using the camera of a cell phone. Watermarks and fingerprints can be derived from the captured image/video data, and used as detailed above.
Similarly, processing other than watermark- and fingerprint-based content identification can be used. One such alternative is speech recognition. Another is speaker recognition. Still another is audio classification. A system may thereby discern, e.g., that the user is in a crowded public place - such as a busy shopping venue - based on the sampled audio (e.g., a jumble of speech-like phonemes that can't be recognized), and systems interacting with the user/user device can tailor their behavior accordingly. (Again, combined use of media content information with location information allows still more accurate context classification.)
Moreover, while certain of the implementations contemplate outputting a web page to the user on a tablet (or other) display screen, other types of information (including non- visual) can be presented, using other devices.
One particular example is augmented reality glasses. Such devices can overlay logos and other computer-generated indicia over a real-world scene presented to the user. Different augmentations can be presented to different users, based on their respective historical- and currently-sensed context information.
Consider a baseball stadium, with advertising display screens arrayed in a border ringing the field. These screens present corporate logos and other familiar forms of advertising to the general public. Users with augmented reality glasses, however, find that their glasses overlay, in those locations, content that is better tailored to their interests - again by reference to personal historical and real-time context data indicated by a user- identifying cookie. Again, an auction model can be employed, whereby different people see different presentations in this viewing real estate, based on what their cookies respectively reveal.
While the disclosure has focused on presentation of visual information tailored to the user, the same principles can likewise be used to tailor auditory information presented to the user.
As noted earlier, the sensing of physical context information can occur at one time, and its use can occur at a second time, where the second time is the same as the first time, or follows the first time by 5, 10, 30, 60, or 300 or more, seconds or minutes.
Ad serving companies typically have maintained the backend databases that aggregate context information about a user' s digital browsing history. However, different companies may emerge to aggregate the other types of context information detailed herein.
While one of the detailed arrangements contemplates the user device sending cookie data twice - once in connection with transmitting data about the physical environment (e.g., audio), and once in connection with a subsequent transaction (e.g., requesting a web page), this is not necessary. In alternative embodiments cookie data needn't be sent to identify the user. For example, a remote system can identify the user (or the user device) otherwise, such as by an IP address included in the packet stream that conveys the data about the user's physical environment, or that conveys a request for a web page. The JP address can be associated with the user (and/or the user's cookie data) using a table, database, or other data structure. In some implementations, cookie data isn't used at all. The user's identity is conveyed or discerned otherwise in each transaction between the user device and the remote system.
While the remote system is sometimes referenced as a unitary entity (as in the preceding sentence), it is more often a distributed system - involving multiple computer servers at multiple locations, operated by multiple different parties. (The appendix begins to illustrate the many different parties that may be involved.)
In one of the earlier examples, a watermark detector forms part of a tablet app distributed by radio station KXYZ-FM. In other embodiments, software code for a watermark detector (or a fingerprint engine) may form part of a Java Native Interface (JNI) library downloaded to a user's device with a web page. Thereafter, when that web page, or another, wants information about the user's context, it can invoke the earlier- stored code with JNI. The Java instructs the device to activate its microphone and associated software modules, decode any watermark, and write cookies (or call out to another service that writes cookies) accordingly.
While the emphasis of the disclosure has been on environmental context, sensed by device sensors, it will be recognized that the present technology is useful with all other forms of context.
Context is sometimes defined as any information useful in characterizing the situation of an entity. An entity is a person, place or object that is considered relevant to an interaction with a user.
Such context information can be of many sorts, including computing context (network connectivity, memory availability, processor type, CPU contention, etc.), user context (user profile, location, actions, preferences, nearby friends, social network(s) and situation, etc.), physical context (e.g., lighting, noise level, traffic, etc.), temporal context (time of day, day, month, season, etc.), history of the above, etc.
Although disclosed as complete systems, subcombinations of the detailed arrangements are also separately contemplated.
While consumers have been trained to think of automated content recognition (such as by the Shazam app), as being performed occasionally - when identification of a particular content object is requested by a user, the inventors expect that such recognition will eventually become ubiquitous and continuous. Physical sensors will be free-running, and sensed data (and its derivatives - such as recognized content information) will be always available (hopefully with some automated destruction after a suitable period of time.) The present technology works in both scenarios - with physical context being sensed in response to a user action, or being sensed continuously. The latter provides a richer set of context data by which system responses to the user can more accurately be customized.
While certain of the detailed embodiments focused on audio sampled from a television or radio, it will be recognized that these are illustrative only and not limiting. For example, such audio may be sampled in a movie theatre, in a nightclub, etc.
While the present technology has been described mainly in the context of third-party, cookie-based arrangements, it is applicable in other systems as well. For example, Microsoft, Facebook, Google, and Apple, are each promoting their respective technologies for identifying consumers on the web - without use of third-party cookies. For example, one Microsoft system employs device-specific identifiers, which are associated together in a cloud database as used by one particular individual. Facebook' s technology relies on its unique user logins. Google's system (AdID) and Apple's system (Identifier for Advertising, or ADFA) similarly aim to supplant third-party cookies with identification technologies that they themselves govern - allowing more granular usage and privacy controls.
Related technologies by these companies are detailed in patent documents 20090119167, 20110167079, 20110307323, 20110321167, 20120116875, 20120316956, 8,060,402, 8,082,179, and 8,484,073. Applicant's invention encompasses the technology described herein, as applied to such alternatives to third-party cookies.
In addition to the above-noted alternatives to classic (HTTP) cookies, other means of identifying an online user (and device) include IP address (noted above), URL (query string), hidden form fields, HTTP authentication data (based on user name and password, etc.), and the DOM (Document Object Model) property "window.name," which are familiar to artisans in the field (and are detailed, e.g., in the Wikipedia article for HTTP Cookie dated December 4, 2013). Unless used with the adjective "HTTP," the term "cookie" herein should be construed to encompass such alternative forms of user or device identification.
While reference has been made to smartphones, it will be recognized that this technology finds utility with all manner of devices - both portable and fixed. Tablets, laptop computers, digital cameras, wrist- and head-mounted systems and other wearable devices, servers, etc., can all make use of the principles detailed herein. (The term "smartphone" should be construed herein to encompass all such devices, even those that are not telephones.)
Sample smartphones include the Apple iPhone 5; smartphones following Google's Android specification (e.g., the Galaxy S4 phone, manufactured by Samsung, and the Google Moto X phone, made by Motorola), and Windows 8 mobile phones (e.g., the Nokia Lumia 1020, which features a 41 megapixel camera).
Details of the Apple iPhone, including its touch interface, are provided in Apple's published patent application 20080174570.
The design of smartphones, tablets, and other devices/computers referenced in this disclosure is familiar to the artisan. In general terms, each includes one or more processors, one or more memories (e.g. RAM), storage (e.g., a disk or flash memory), a user interface (which may include, e.g., a keypad, a TFT LCD or OLED display screen, touch or other gesture sensors, a camera or other optical sensor, a microphone, etc., together with software instructions for providing a graphical user interface), and an interface for communicating with other devices (which may be wireless, as noted above, and/or wired, such as through an Ethernet local area network, a T-l internet connection, etc.).
The processes and system components detailed in this specification may be implemented as instructions for computing devices, including general purpose processor instructions for a variety of programmable processors, including microprocessors (e.g., the Intel Atom, the ARM A5, the Qualcomm Snapdragon, and the NVidia Tegra 4; the latter includes a CPU, a GPU, and NVidia' s Chimera computational photography architecture), graphics processing units (GPUs, such as the NVidia Tegra APX 2600, and the Adreno 330 - part of the Qualcomm Snapdragon processor), and digital signal processors (e.g., the Texas Instruments TMS320 and OMAP series devices), etc. These instructions may be implemented as software, firmware, etc. These instructions can also be implemented in various forms of processor circuitry, including programmable logic devices, field programmable gate arrays (e.g., the Xilinx Virtex series devices), field programmable object arrays, and application specific circuits - including digital, analog and mixed analog/digital circuitry. Execution of the instructions can be distributed among processors and/or made parallel across processors within a device or across a network of devices. Processing of data may also be distributed among different processor and memory devices. As noted, cloud computing resources can be used as well. References to "processors," "modules" or "components" should be understood to refer to functionality, rather than requiring a particular form of implementation.
Software instructions for implementing the detailed functionality can be authored by artisans without undue experimentation from the descriptions provided herein, e.g., written in C, C++, Visual Basic, Java, Python, Tel, Perl, Scheme, Ruby, etc., in conjunction with associated data. Smartphones and other devices according to certain implementations of the present technology can include software modules for performing the different functions and acts.
Known browser software, communications software, imaging software, and media processing software can be adapted for use in implementing the present technology.
Software and hardware configuration data/instructions are commonly stored as instructions in one or more data structures conveyed by non-transitory tangible media, such as magnetic or optical discs, memory cards, ROM, etc., which may be accessed across a network. Some embodiments may be implemented as embedded systems -special purpose computer systems in which operating system software and application software are indistinguishable to the user (e.g., as is commonly the case in basic cell phones). The functionality detailed in this specification can be implemented in operating system software, application software and/or as embedded system software.
Different of the functionality can be implemented on different devices. For example, in a system in which a smartphone communicates with a computer at a remote location, different tasks can be performed exclusively by one device or the other, or execution can be distributed between the devices. Extraction of fingerprint and watermark data from content is one example of a process that can be distributed in such fashion. Thus, it should be understood that description of an operation as being performed by a particular device (e.g., a smartphone) is not limiting but exemplary; performance of the operation by another device (e.g., a remote server), or shared between devices, is also expressly contemplated.
In like fashion, description of data being stored on a particular device is also exemplary; data can be stored anywhere: local device, remote device, in the cloud, distributed, etc.
As indicated, the present technology can be used in connection with wearable computing systems, including headworn devices. Such devices typically include display technology by which computer information can be viewed by the user - either overlaid on the scene in front of the user (sometimes termed augmented reality), or blocking that scene (sometimes termed virtual reality), or simply in the user's peripheral vision. Exemplary technology is detailed in patent documents 7,397,607, 20100045869, 20090322671, 20090244097 and 20050195128.
Commercial offerings, in addition to the Google Glass product, include the Vuzix Smart Glasses M100, Wrap 1200AR, and Star 1200XL systems. An upcoming alternative is augmented reality contact lenses. Such technology is detailed, e.g., in patent document 20090189830 and in Parviz, Augmented Reality in a Contact Lens, IEEE Spectrum, September, 2009. Some or all such devices may communicate, e.g., wirelessly, with other computing devices (carried by the user or otherwise), or they can include self-contained processing capability. Likewise, they may incorporate other features known from existing smart phones and patent documents, including electronic compass, accelerometers, gyroscopes, camera(s), projector(s), GPS, etc.
As noted, watermark technology can be used in various embodiments. Technology for encoding/decoding watermarks is detailed, e.g., in Digimarc's patents 6,614,914, 6,590,996 and 6,122,403; in Nielsen's patents 6,968,564 and 7,006,555; and in Arbitron's patents 5,450,490, 5,764,763, 6,862,355, and 6,845,360.
Content fingerprinting can also be used in various embodiments. Examples of audio fingerprinting are detailed in patent publications 20070250716, 20070174059 and 20080300011 (Digimarc), 20080276265, 20070274537 and 20050232411 (Nielsen), 20070124756 (Google), 7,516,074 (Auditude), and 6,990,453 and 7,359,889 (both Shazam). Examples of image/video fingerprinting are detailed in patent publications 7,020,304 (Digimarc), 7,486,827 (Seiko- Epson), 20070253594 (Vobile), 20080317278 (Thomson), and 20020044659 (NEC).
Other fingerprint-based content identification techniques are well known. SIFT, SURF, ORB and CONGAS are some of the most popular algorithms. (SIFT, SURF and ORB are each implemented in the popular OpenCV software library, e.g., version 2.3.1. CONGAS is used by Google Goggles for that product's image recognition service, and is detailed, e.g., in Neven et al, "Image Recognition with an Adiabatic Quantum Computer I, Mapping to Quadratic
Unconstrained Binary Optimization," Arxiv preprint arXiv:0804.4457, 2008.)
Still other fingerprinting techniques are detailed in patent publications 20090282025, 20060104598, WO2012004626 and WO2012156774 (all by LTU Technologies of France).
Yet other fingerprinting techniques are variously known as Bag of Features, or Bag of Words, methods. Such methods extract local features from patches of an image (e.g., SIFT points), and automatically cluster the features into N groups (e.g., 168 groups) - each corresponding to a prototypical local feature. A vector of occurrence counts of each of the groups (i.e., a histogram) is then determined, and serves as a reference signature for the image. To determine if a query image matches the reference image, local features are again extracted from patches of the image, and assigned to one of the earlier-defined N-groups (e.g., based on a distance measure from the corresponding prototypical local features). A vector occurrence count is again made, and checked for correlation with the reference signature. Further information is detailed, e.g., in Nowak, et al, Sampling strategies for bag-of-features image classification, Computer Vision-ECCV 2006, Springer Berlin Heidelberg, pp, 490-503; and Fei-Fei et al, A Bayesian Hierarchical Model for Learning Natural Scene Categories, IEEE Conference on Computer Vision and Pattern Recognition, 2005; and references cited in such papers.
Digimarc has various other patent filings relevant to the present subject matter. See, e.g., patent publications 8,498,627, 8,412,577, 6,947,571, 20130150117, 20120284012,
20100046842, 20070156726, 20080049971, and 20070266252, and pending applications 12/125,840, filed May 22, 2008; 13/946,968, filed July 19, 2013; 14/074,072, filed November 7, 2013; 61/838,165, filed June 21, 2013; and 61/818,839, filed May 2, 2013.
Additional information about ad serving is provided in a series of articles published by adopsinsider<dot>com, attached as an appendix.
This specification has discussed several different embodiments. It should be understood that the methods, elements and concepts detailed in connection with one embodiment can be combined with the methods, elements and concepts detailed in connection with other embodiments. While some such arrangements have been particularly described, many have not - due to the large number of permutations and combinations. Applicant similarly recognizes and intends that the methods, elements and concepts of this specification can be combined, substituted and interchanged - not just among and between themselves, but also with those known from the cited art. Moreover, it will be recognized that the detailed technology can be included with other technologies - current and upcoming - to advantageous effect.
Implementation of such combinations is straightforward to the artisan from the teachings provided in this disclosure.
While this disclosure has detailed particular ordering of acts and particular combinations of elements, it will be recognized that other contemplated methods may re-order acts (possibly omitting some and adding others), and other contemplated combinations may omit some elements and add others, etc.
Although disclosed as complete systems, sub-combinations of the detailed arrangements are also separately contemplated (e.g., omitting various of the features of a complete system).
While certain aspects of the technology have been described by reference to illustrative methods, it will be recognized that apparatuses configured to perform the acts of such methods are also contemplated as part of applicant's inventive work. Likewise, other aspects have been described by reference to illustrative apparatus, and the methodology performed by such apparatus is likewise within the scope of the present technology. Still further, tangible computer readable media containing instructions for configuring a processor or other programmable system to perform such methods is also expressly contemplated.
The present specification should be read in the context of the cited references. (The reader is presumed to be familiar with such prior work.) Those references disclose technologies and teachings that the inventors intend be incorporated into embodiments of the present technology, and into which the technologies and teachings detailed herein be incorporated.
To provide a comprehensive disclosure, while complying with the statutory requirement of conciseness, applicant incorporates-by-reference each of the documents referenced herein. (Such materials are incorporated in their entireties, even if cited above in connection with specific of their teachings.) These references disclose technologies and teachings that can be incorporated into the arrangements detailed herein, and into which the technologies and teachings detailed herein can be incorporated. The reader is presumed to be familiar with such prior work.
In view of the wide variety of embodiments to which the principles and features discussed above can be applied, it should be apparent that the detailed embodiments are illustrative only, and should not be taken as limiting the scope of the invention. Rather, we claim as our invention all such modifications as may come within the scope and spirit of the following claims and equivalents thereof. • Home
• About
• Ad Ops Jobs
Ad Ops Insider
How Does Ad Serving Work? by Ben on September 3, 2010
Interactive ads are everywhere these days, but when it comes to the technical process of getting an ad on the page and how publishers and marketers verify it delivered, not many people can explain what actually happens in detail. Read this article though and you'll be one of them! Below I've detailed step-by-step how a browser gets from the initial call to a publisher's website to the final ad creative, and when and how each party counts an impression. You can view a diagram of the ad serving process at the bottom of this post - the numbers in the text refer to the steps labeled in the diagram.
So, without further ado -
When a browser navigates to a publisher website ( 1 ), the publisher's web server sends back a bunch of HTML code (2) that tells the browser where to get the content (3) and how to format it. Part of the HTML code returned to the browser (4) will include a coded link known as an ad tag.
Here's an example of what an ad tag from Doubleclick, one of the major ad serving companies, looks like: http : / / ad . doubleclick . net/ABC/publisher/ zone topic=abc; sbtpc=def ca =ghi kw=xyz ; tile=l ; slot=728x90.1;sz=728x90;ord=7268140825331981?
Click here to read more about ad tags and how they are constructed.
The ad tag points the browser to the Publisher's Ad Server (5), a system designed exclusively for delivering and tracking advertising. In most cases, the Publisher's Ad Server is actually a network of cloud servers owned and maintained by a separate company. In this case, the content server tells the browser to fetch the ad from Doubleclick, a company owned by Google, that then makes the very complex decision on which ad to serve using a program called an Ad Selector.
In many cases the ad server is deciding among thousands upon thousands of potential options in mere milliseconds. The computational power behind the Ad Selector is mind blowing - Atlas, the major rival to Doubleclick calls the supercomputer running its Ad Selector "WARP" and it is among the most powerful in the world, making billions of decisions a day and trillions in its lifetime. The Ad Server makes a decision, and in most cases sends back another ad tag (6), or redirects the browser by pointing it to the Marketer's Ad Server. These redirects are technically speaking 302 redirects, which tells the browser the page has been "temporarily moved". This allows Ad Servers to count the 302 call as an impression and host the actual ad content on a different server. Once the publisher's adserver sends the browser a redirect to the marketer, it counts a delivered impression in its own database (star). The only exception here is if the publisher decides to deliver a house ad or the marketer has asked the publisher to "site-serve" the ads, both of which requires the publisher load the actual creative files into their ad server, meaning the publisher is the final destination, and the browser can skip the loop through the marketer side (steps 7,8,11, 12).
Click here to read more about why Publishers and Marketers have their own Ad Servers.
The browser now calls the Marketer's Ad Server (7) and is redirected yet again to a Content Delivery Network, or CDN, (8) a global network of cloud servers that actually house the raw creative graphics to fetch the actual ad. Why, you ask? Well, as powerful as ad servers are, they just aren't equipped to handle the volume and bandwidth required to deliver content as heavy as image files. Redirects are often nothing more than a lxl pixel requiring just a few bytes of memory. Image files on the other hand are kilobytes or even megabytes in size, could be called millions of times a day, and require a much faster and robust infrastructure. Ad Servers might maintain three to six data centers across the world, but a CDN can process the heavy bandwidth and deliver the content faster because they operate hundreds of data centers and can route requests to the one nearest to the user, no matter where they are on earth. You can think of the ad server as the brain and the CDN as the brawn. Ad Servers aren't the only companies that use CDNs, in fact many websites host their bandwidth intensive files in these cloud networks. A CDN is almost always another independent company, such as Akamai, that hosts the heavy creative assets so the Ad Server doesn't have to. There used to be a handful of these companies out there, but Akamai has acquired almost all of them and is the largest player by far in the space.
Here's what a CDN redirect to an Akamai server hosting a flash file looks like:
http://spe.atdmt.com/ds/ABCDEF12334/filenamel23_300x250. swf
In addition to sending back the redirect to the CDN, the Marketer's Ad Server also appends a second redirect (10) back to itself with a query string to fetch a lxl pixel (11) after the ad content has been called. When the browser fires this last redirect calling a lxl pixel from the Marketer's Ad Server (11), the Ad Server knows the ad was successfully downloaded and it finally counts an impression in its own database (star).
In many cases, your browser has to make at least four calls for site served ads and six in the case of third-party served ads for this whole process to work, if not even more, but shouldn't take more than a second regardless of the number of parties involved. To visualize the process explained above, please see the diagram below - 302 redirects are highlighted in blue, and the ad creative is highlighted in red.
How 3rd Party Ad Serving Works
Figure imgf000035_0001
In the next post I'll write about how this process changes when an impression is sold in real-time (RTB) on a Supply Side Platform or Ad Exchange.
Tweet ; i5 Tagged as: 302 redirects, ad tags, adserving, Profile
Sign in with Twitter Sign in with Faccbook
or
Name
Email Not published
Website
Comment
Post It
• 39 Replies
• 38 Comments
• .0-3 weets
• 0 Facebook
• 1 Ping back
Last reply was 1 month . ί David
View October 5, 2010
Brillant description. Concise and effective. Thank you ! · Get Pixel Tracking Transparency with Ghoster
View October 8, 2010
[...] not exactly. In fact, thanks to the off-site redirects inherent to 3rd party adserving, publishers often have no idea when an advertiser or marketer attempts to redirect the user within [...] . I %Matthieu
View March 24. 2011
Hello,
Good explanations indeed
Can you explain clickcount in detail as well?
Is it ; publisher adserver -> marketer adserver -> site ?
Or ; marketer adserver -> publisher adserver -> site ?
How clicktags in flash are handled ?
Thanks ! . ί "tBen
View March 25, 2011
Hi Matthieu,
Clicks are tracked in much the same way as impressions, with redirects. Typically the publisher's ad server populates multiple click tags into the href of the ad, concatenated with a query string so that when a user clicks the ad, the browser calls all three links more or less simultaneously. But yes, the specific order with any 3rd party system is always publisher ad server, marketer ad server, final destination. Highly simplified for sake of readability, the href to track clicks on a standard .gif ad might look something like this: href='¾ttp://ad.doubleclick.net activity;id% 123456789 redir%http://marketeradserver.net click; id 123456789%redir%http://www.destinationurl.com/". Flash and javascript ads approach how the code functions slightly differently, but the method of using redirects to count the same action in multiple databases is the same.
Hope that helps -
Ben . W 'tPooja Singh
View June 21, 2011
Brilliant website indeed! It's especially useful for people like me who are very new to this industry and working as ad-ops engineer. I'm still trying to learn the ropes of this industry and this website has been a great help to me, a life-savior almost ¾?:;
Thank you. And continue with the great job! f Partho Ghosh
View August 6. 20 ! i
Awesome, fantastic i really enjoyed while reading this and clear all my doubt and comcepts how an ad is delivered while an ad call is made ....thanks Ben ϊ tErika I. ens
View November 1 , 201 1
I've been wondering how ad servers worked, and this is by far the clearest explanation I can find. Very well written, thank you, Ben! f f Brendan David
View November 1 , 2011
Job well done, Ben. Thanks for posting. This is one the clearest descriptions of the overall process I've seen. f tGbolahan
View November 23, 2011
Great article! Please I have 2 questions:
1. If a marketer has asked a publisher to "site-serve" the marketer's ad, can the marketer still use a marketer ad server to verify the impressions and clicks delivered by the publisher?
2. If a marketer is using a marketer ad server to serve their ad, can they still use a second marketer ad server to verify the impressions and clicks delivered by the first marketer ad server?
Figure imgf000037_0001
View November 26, 2011
Hi Gbolahan,
Thanks -
1. 1 suppose so - they could ask the publisher to host the creative, but still embed some type of tracking pixel in the ad. I'm not sure why anyone would do this though - usually creative is site served because the advertiser is too small to have a need for their own ad server. Perhaps if the publisher was responsible for developing the creative on the advertiser's behalf, they would host the creative, but still embed some kind of 3rd party tracking mechanism for the advertiser. Both the publisher and advertiser ad servers will end up hosting the raw creative asset on a CDN like Akamai, so there isn't a benefit from a latency standpoint. 2. They can, this is called 4th party tracking and tends to happen when the marketer will have a handful of rich media ads that are served through a different system than their primary ad server. In this scenario, the rich media vendor would be the 3rd party from a publisher perspective, and the primary ad server, perhaps DART, would be the 4th party. Generally, publishers do not allow advertisers to bill on a 4th party tracker because they incur a dual discrepancy, but the practice of using them is quite common to centralize reporting.
Hope that helps -
Ben
Gbolahan
View December 1
Thanks a lot for your reply Ben. I'm clear now. ί 'Milan
View December 2, 2011
Loving your write ups, quick question.
With regards to how ad.serving works between the marketer's ad.server and content delivery network, does the marketer send a 302 redirect to the content network via the browser or is it a different type of redirect?
As in you say once the 302 redirect has occurred the impression is counted for the ad.server. But in the case of the marketer, the impression isn't counted till the actual ad is served (having bounced the query pixel off the browser and back) which is after the initial redirect of browser to content network.
Cheers,
Allan
Figure imgf000038_0001
View December 3. 2011
Hi Allan,
Yes, the ad server tells the browser to fetch the creative file from the CDN through a 302 redirect. If you were to view a page through a web development tool like Firebug, you would be able to look at both the HTTP request and response headers for each piece of the ad call, and you would see that the initial call starts with a 200 request, or a standard GET request to the publisher's ad server, that call responds with a '302 'Object moved' header, and the location of that file, redirecting the browser to the marketer's ad server, which responds in kind with a 302 'Object moved' header, and the location of the file on the CDN, which it can finally fetch.
To be clear though, it's usually only that simple with simple ,gif or .jpeg ads. It can get a little confusing in practical application because the process may not look so straightforward if, for example, the tag loads a flash creative. In that case, the tag may redirect to a script, and the confirmation pixel via a 302 response, and the script may have the subsequent calls to the CDN already embedded in the file. In that case, it isn't technically a 302 redirect, but likely just looks like another 200 GET request. You can start to wander down the technical rabbit hole pretty quickly here, so my goal was to explain the process as simply and in a way that was as easy as possible to understand. To really dig in, I would recommend you speak to a sales engineer at your ad server, who will know the specifics for your setup with their technology.
Hope that helps -
Ben Bob
View 11 months ago
Ben,
I built a .net site for a client, and created a custom ad rotator that logs click throughs to a database and then redirects the browser to the target url. It works great for all the jpg's in the rotator, EXCEPT for a Doubleclick.net ad. I can log the image click, but can't response.redirect to the target url. I just get a "page can't be displayed" error. I've read that Doubleclick ads need to open in a new window, and tried several variations of window.location... and window.open(), but they go nowhere. Any help would be greatly appreciated. The ad url is below, and the TimeStamp var correctly inserts the proper format of the date, as required in the url.
http://ad.ctoubleclick.net iump N2886.291385..COM B 5246111 ;sz= 190x 190;ord=?"
Thanks very much!
Bob ί' 1-Alex
View 11 months ago
Hi Ben,
I find your explanation extremely interesting, I was really looking for something like that.
I would like to ask you other informations, in particular I'm focusing my attention on CPA:
CostPerAction. I know we can divide it in different subcategories: CostPerLead, CostPerSale,
CostPerEngagement. For example, I don't want to pay just because someone arrives on my landing page by clicking on a banner, but I want to pay if someone arrive on my landing page and complete a registration to my web site (CPL) or, even better, if he complete a purchase (CPS). How can the ad server "understand" that the customer has completed a purchase? How can it know that he performed a registration?
Another new type is CostPerCall, especially in the mobile area, where an advertiser pays when someone calls, for example, a call center. I have seen different apps that, after a click on a banner, automatically redirect you to your phone app, with the number already composed, you just have to press the "call" button. But my question is: how can the ad server be sure that I complete the call process? How can it know that I have not pushed the "cancel" button?
Thank you very much
Cheers Alex
Figure imgf000040_0001
View 11 months ago
Hi Boh,
Thanks for your comment - I'm not sure I'm technical enough to understand what you're trying to do, but it sounds like you're trying to append your own click tracker on the fly to an ad tag by standing in the middle of the redirect path? That particular doubleclick tag won't open for me, so there's not much advice I can give you at this point without seeing more details, and even then I'm not sure I have enough development skills to really help. You might try looking through the doubleclick documentation online, or connecting with a broader forum of Ops people at AdMonsters, which has a free forum where you can ask questions.
Sorry I couldn't be of more help - good luck!
Ben
Figure imgf000040_0002
View 11 months ago
Hi Alex,
Thanks - ad servers track conversions, however you want to define it, through pixels. Effectively, you would get a lx 1 image file from the ad server that you would place on the confirmation page for your conversion - this might be a thank you page or an order summary page, whatever destination would confirm the user actually took the desired action. Now, when a user converts, they land on that page and calls that pixel, which tells the ad server they converted, and allows the ad server to associate the right media to that user by looking at the cookie it has on that user's machine. It's effectively the same process used to track an impression, just without the ad itself involved in the process. You would have to do a little extra work in trafficking the ad to link everything up correctly (I would refer you to your ad server's documentation for the specific details that apply to you and your ad server configuration), but it's fairly straightforward to do, and a common strategy, so it should be well documented.
In terms of cost per call, I believe that is exclusively a search concept vs. the display ad concepts I tend to focus on here, but as I understand it, there are a few ways to do that, depending on how you plan to execute. For example, Google's bid per call in the desktop environment requires the advertiser to use Google Voice generated tracking numbers (usually positioned to the user as an extension), which allows it to track the conversion. For mobile devices, I'm less familiar with the approach, but I imagine it works either through a click tracker that fires when a user clicks on a phone number in a search result, or also depends on a Google generated call tracking number. Google is the only company I'm aware of that actually offers this type of service.
Hope that helps -
Ben ί chris
View 11 months ago
This is great, thank you. Do you know if there are best practices as a publisher re: working with multiple ad exchanges? Should i be calling each exchange concurrently or sequentially based on their response?
Thanks.
Figure imgf000041_0001
View 10 months ago
Hi Ben,
First off, excellent job on this article. I've searched high and low for simple explanations of ad serving and this is by far the easiest and most intuitive read yet. Second, I had a quick question about using fourth-party calls in Google swf banners being served up via the Google Ad Network.
In short, my banner swf file is dynamically calling on three separate jpg files using absolute paths (i.e. http://www.mydomam.com/frame 1 -jpg). These banner frame jpg's are hosted on our own internal server so that we can update the banner creative on demand "without" having to edit, change, or resubmit new swf files to 3rd party vendors.
The problem I'm running into is that when I upload my banner swf into the Google Ad Network I get denied as they don't permit 4th-party server calls. Is there any sort of workaround for this by chance? We still want to use the Google Ad Network for banner tracking ect (clickTag) but on the other hand we'd also like to host our own banner creative. Any advice is much appreciated and thanks again for such an awesome article.
Best regards,
NM
Figure imgf000041_0002
View J jnonths ago
Hi Chris,
Ideally, you would call each exchange concurrently, and take the highest paying bid from either. Without an SSP to sit as a layer between the exchanges and help you manage this process, I'm not sure exactly how you would do this. If you don't have the scale to make it working with an SSP make sense, my advice would be to try and test a few exchanges at the same time and see who can pay the most. The best test setup would be to run rotate a cookie value on your users for however many exchanges you want to test, and silo the exchange calls per cookie value so that each exchange gets the entire user session. If you mix impressions between exchanges from the same user session, you'll confuse the frequency tracking of buyers, and potentially throw a wrench in your final results.
That said, my guess is that if you execute the test successfully, you'll see similar results from every exchange, since most bidders are on all the exchanges, so it shouldn't really matter all that much which one you pick.
Good luck!
Ben
Figure imgf000042_0001
View 10 months ago
Thanks NM,
Probably not - Google Engineers are pretty smart - they're not likely to allow a loophole, at least not for very long. That said, Google does allow some advertisers to 3rd party serve, and that would pretty much solve your problem http:/ysupport.gooele.coni/adwoi,dspolicy/biiT/answei,.py?hl=en&answer=:94230. With a 3rd party tag, you could swap out the display creative whenever you want, without having to change the actual ad tag in the Network. Google says this functionality is available only by invitation, but you could ask support if you really wanted, and probably find a way to get there as long as you can work with an approved 3rd party. Of course, this probably doesn't make sense unless you are a big advertiser. If you're a small shop, I doubt there's a workaround. This is really a question for Google Support - I'm just not this intimately familiar with the terms and policies enough to give you a solid answer here.
Best of luck!
Ben f 'tTim Goudie
View 9 months ago
Ben,
I have read your 3rd party ad server explanation with great interest and as a marketer, found it very helpful. Thank you.
Question: If I chose to work through only Doubleclick as my ad server am I limited to a finite number of sites that Doubleclick has contractual relationships with or would I be able to have my ad served up to any website(s) I decide on? So as a logical extension of that same thinking does Atlas or iBlaster similarly have their own finite (or favourite) set of websites they can deliver ads to?
Let me know.
Thanks,
Tim Goudie
■ Sitrojii Mukerji
View 8 months ago
Ben,
This piece of content is fabulous. But something which still keeps ringing is why Akamai (or equivalents). What would the possible issues be if I as a publisher wanted to host ads for my customers, taking images and videos as inputs. I have complete control over the content. I can host these content on my superfast cloud based content server with no need having to carry out 9, 10. 1 can pass the storage charges to my customers.
Could you please provide in realistic terms, what savings we derive out of using Akamai et.al.
Many thanks and look forward to hearing more in the days t come.
Figure imgf000043_0001
View 8 months ago
Hi Tim,
Glad to be of help - to answer your question, no, any ad server should be able to serve on any webpage. Ad servers provide publishers with a very simple technical mechanism called an ad tag which is easy to implement on any website, even if the ad server has never served on that site before. In fact, in the majority of cases, the ad server won't have any direct relationship with the publisher running its tags, and in many cases may actually have competing interests. Plenty of Atlas- serviced advertiser tags are running on Doubleclick serviced publisher site right now, for example, and you can repeat the scenario with virtually any combination of ad servers out there. The only exception I can really think of would be from an ethical point of view - most ad servers wouldn't knowingly let you run ads on adult content, pirated content, hate speech content, or things like that, but I assume that's irrelevant in your case.
Hope that helps - Ben
Figure imgf000043_0002
View 8 months ago
Hi Surojit,
Thanks - Akamai, or any CDN for that matter isn't necessary per se, but does help reduce latency. Akamai has done the very difficult and cost intensive work of setting up and maintaining physical data centers around the globe so that content loads extremely fast to a user. See, the ad server can make quick decisions for any user from only one or two geolocations because it only processes a few bytes of data before passing the user on to a CDN based service to fetch the actual content, which might be thousands or even millions of times larger than what the ad server needs to process. It makes more sense to have a service that keeps the heavy data physically closer to a user, and on a system that is setup to load balance for much heavier volume. The CDN doesn't have to make any variable-based decision, it just has to provide access to the file quickly.
Practically speaking, unless you were to setup your own ad server, I don't know that you can get around using a CDN - the ad server just automatically moves it onto those servers for their customers, it's not as if you have a choice from a customer point of view. But even if you were to build your own ad server from the ground up and decide to not use a CDN, it's hard to say what your cost savings would be. At the end of the day, you still have to carry out the service that steps 9 & 10 provide, which is deliver the creative to the user, and verify that it fully rendered on the page. So I don't know that you really save any steps, you just change where those steps happen. The question to you would be will you not geolocate your data service at all - if so, what is your cost on the user experience and your system performance to customers? You'll almost certainly have an inferior product to your competition. If you will geolocate your data service, why bear the entire cost of that yourself, when you could just outsource it to a company that provides for that need as their primary service? It would seem to not make economic sense to me to go about it that way - I would expect it would actually cost you more not to use a CDN service like an Akamai.
So that would be my take - hope that helps, and best of luck!
Ben Anish
View 7 months ago
Hi Ben,
Could you clarify where a digital media agency would fall in the 'Ad-serving' process you had elaborated. I remember seeing, planners dealing with the Publishers directly, and not with marketers(or advertisers); nor had I ever once heard them talking about marketer ad-servers. Does the process work w/o marketer ad-server's in picture?
Please do clarify.
Thanks,
Anish
Figure imgf000044_0001
View 7 months ago
Hi Anish,
Sure - basically the chain of events is as follows:
1. User calls a publisher - publisher's web server redirects the user to the publisher's ad server
2. User calls publisher's ad server - publisher's ad server decides what ad to serve, and redirects the user to a marketer's ad server
3. User calls marketer's ad server - marketer ad server knows what ad to serve based on the incoming call and delivers the ad to the user (or more likely redirects the user once more to a Content Distribution Network like Akamai, which hosts the actual creative.)
The agency is the marketer in this case, or is the marketer's agent. If you, a media planner, work at an agency, the marketer is your client, and the company actually paying for all the advertising the agency is buying. There's always a marketer, they may just be an indirect participant if they've hired an agency to handle their advertising campaigns. I think it's just a case of semantics - when I say marketer, I really just mean the buy-side of the equation - that could be an agency working on behalf of a marketer, or it could be a marketer directly, as some do their own media buying and don't contract with an agency.
Hopefully that makes sense - Ben f Chintan
View 4 months ago
Ben,
Excellent! Really glad you took this initiative.
Could you help clarify my understanding. This is what i think happens
1) Publisher AdServer will only determine that the add needs to be served for a specific advertiser. But we dont know what ad at this point.
2) One the request is re-directed to marketer's (advertisers) ad server, at this point the creative is identified that gets served. And the location to fetch the creative is sent in the redirect. The location is typically a CDN (Akamai) path.
3) User browser does the final lookup to locate the most nearby CDN server and load the add at this point. Is the above correct?
If for some reason, ads were loading fast, and suddenly they start to load very slowly, what could be the potential bottleneck?
When a publisher approaches a network like doubleclick for ad serving, does it always do a redirect to advertiser ad server, or can it serve the ad itself? And if it can serve the ad itself, how is the pricing for advertiser determined.
Thanks a ain.
Figure imgf000045_0001
View 4 months ago
Thanks Chintan - let me see if I can answer your questions:
1. Right, the publisher decides what client needs to serve in order to fulfill delivery quotas based on the goals and campaigns in their system, and for what the incoming ad request qualifies.
2. If a 3rd party tag is used, that's correct - the marketer will determine which creative actually renders on the page, which they house at a CDN. If a publisher's house ad or locally hosted ad is served, the publisher will simply redirect to the CDN themselves, where the raw creative is hosted.
3. When 3rd party tags are used, correct - and for most paid ads on the internet, 3rd party tags are used, so your understanding is spot on.
4. If the ads were loading slowly, there could be a few reasons - first, there could be more parties involved in the ad serving process. So, if the publisher redirects to a network or an exchange, there will be additional latency. Alternatively, it could be a matter of lots of pixels involved in the serving process - for example if a verification script like AdSafe is used to scan the page before the ad is allowed to load. Finally, it could also simply be a matter of bandwidth on the user side.
5. The publisher can host and serve the creative themselves - this is the difference between 3rd party ad serving (when a publisher redirects to a marketer's ad server) and 1st party ad serving (where the publisher hosts everything themselves). Usually the pricing is the same regardless of the setup - it doesn't really save the marketer or publisher any money to have it setup one way or another unless the marketer doesn't want to serve their ads at all, in which case they don't need an ad server and I guess would save that money. It's too painful operationally for a large organization to do that though, and would actually be more expensive to try and keep track of reporting, etc. For small marketers it might make sense, but it won't save them all that much money - ad serving can be had for extremely cheap these days.
Hope that helps -
Ben f tChin an
View 4 months ago
Once again, brilliant. Thanks alot! iMzfce
View 4 months ago
Hi,
Thanks for the explanation!
I am working on a website that dynamically generates content, from user's queries. There are literally billions of different pages that can be generated, each with a set of associated keywords (constructed form the queries and the relevant content). For example, a page might have the keywords "kitten cat feline mammal", or "judge jury sentence law" etc.
I want the appropriate adverts to be served, such as for cat food products, or lawyers, or whatever, depending on the keywords. The site cannot be spidered, as all the pages are virtual, and dynamically generated. (Kind of like a search engine, where Google results pages have related adverts.)
I am having great trouble finding anywhere that can do this, and have met several dead-ends, where I explain and explain, and finally they say ..."Ah! ... No, we cannot do that."
If you have any advice or pointers, or industry terms that I should know, that would be very helpful!
Thanks,
Mike
Figure imgf000046_0001
View 4 months ago
Hi Mike,
I'm still not sure why this would be a deal breaker - plenty of sites dynamically generate content and their URLs. Perhaps I'm not fully grasping what you're trying to do, but there's nothing new about dynamic pages. Your problem could be if you use the same URL for multiple content iterations, in which case, I'd suggest you change your approach and just dynamically generate a unique page URL along with whatever content you have. This should be fairly straightforward for someone with experience coding PHP.
But all that said, you still need someone with the demand liquidity and breadth to serve relevant ads on the fly from everything from cat food to lawyers to take your examples - only the exchange, or Google Adsense can provide that kind of granular relevancy, and really only Adsense is that focused on the context. The exchange will care more about the user data, regardless of what content they are reading. So I would say there's no reason you can't get Adsense to work the way you want - you may just need to change your coding approach somewhat. Read this: http ://support. google.com/adsense
/bia/answer.py?hi-en&answei-2Q763 As long as you fill your impressions and generate revenue, is it really that critical how the ads do that? My advice would just be to read up on how Google works with dynamic content, and ensure you follow their best practices.
Good luck!
Ben
UMarcia
View 3 months ago
We have been running an ad tag and dont see anything in the reporting. What could the issue be? Thanks.
Figure imgf000047_0001
View 2 months ago
Hi Marcia,
It's very tough to tell - there could be all kinds of reasons. I would try asking your ad server for help, or you may find this article on identifying and fixing report discrepancies helpful:
http://w w.adopsiiisider.com/ori^
Ben I Ylo' m
View 2 months ago
Ben, this is by far the best explanation of how this entire process works that I've been able to find. Thank you so much for all your insights and advice.
I am trying to further understand the process regarding the how the payments and timing of when a "publisher" (website operator) gets paid for the ad revenue they've generated off their site. I know there are many dimensions to it, but here are my basic questions that perhaps you can help with:
- 1 understand that both the marketer and publisher ad servers keep track and ensure a successful "impression", but how to the parties reconcile the numbers if each is reporting something different?
- How is the tracking of the "billing" done and ultimately, how does the marketer know which publisher to pay— and how do they "pay'Vtransfer the money to the publisher? I think if this is done via Google's network then the publisher gets paid via their AdSense account, but how does Google in this case "bill" and collect from the marketer? How does it work if not via Google?
- How long does it take for a publisher to get paid and what contractual commitment is in place to ensure the publisher can pursue the marketer if they refuse to pay?
Thanks so much again for all your insights to how this works!
Figure imgf000048_0001
View 2 months ago
Hi Tom,
Thanks - to answer your questions:
1. Reconciling discrepancies is something covered in the IAB's standard Terms & Conditions, which is in version 3.0, Sec XII.D. You can read it here: http://www.iab.net giiidelines 508676/tscs3 It basically says that the Publisher and Advertiser will make a good-faith effort to resolve any technical problems should they arise, but that the Publisher should be able to bill in full if the reporting systems are within 10%. In practice, that's not what happens - instead, to keep their relationship with the Advertiser on good terms, Publishers usually add what is called an "ovemin" and basically bonus every campaign 10% on top of what was contracted to ensure the Advertiser's server reports full delivery. Since this is the common practice, expect advertisers to balk at the suggestion they pay in full if their server hasn't reported full delivery. It is on the publisher to check the advertiser's reporting and ensure there isn't a major technical problem. You can read more about resolving 3rd party discrepancies here: http ://www. adopsinsider.com /online-ad-m.easurement-tracldag/resolving-3rd-party-discrepancies/
2. So for Ad Sense, Marketers have to pay upfront, they basically put a credit in their Ad Words account that runs down as their campaigns run. So billing is pretty easy compared to most agency deals - publishers for their part each have a unique ID in their tags, so that when the tag makes a request for an ad to Google's servers, it passes an ID that Google can log in order to attribute the revenue to them specifically. Google then wires the money to the publishers bank account, which they've had to link and verify in their AdSense account on a monthly basis, assuming they can accrue more than $100 in revenue that month. If they can't, the money just rolls over and waits until the Publisher meets that threshold, at which point a payment gets sent. In the traditional Publisher to Agency or Advertiser direct sale, a contract called an Insertion Order, or IO is cut. Check out the IAB document, Section III. Usually bills get sent once a month by the Publishers for the prior month's activity. So on Oct 1st, lots of publisher will log in to the Advertiser's reporting portal, pull the 3rd party reporting for the campaign activity for September, and will use those numbers to generate invoices. Sometimes agencies request specific payment terms, perhaps they want to be billed evenly, regardless of actual delivery in a given month. That is, if you had a 6-month campaign for $60,000, you'd just bill $10,000 each month, even if you delivered let's say $10, 108.33 that particular month and $9,891.67 the next. The IO should specify a billing contact for the Advertiser.
3. That's all in the IO. The IO is a legal contract which should append the IAB Terms and Conditions I mentioned above, which states that all payments should happen on Net 90 terms, meaning the Advertiser has to send payment within 90 days of being billed. Many agencies are notoriously late with payments, usually because they are also waiting for their clients to pay them for the work. In certain cases, Publishers will suspend all campaign activity if a large balance builds up and the agency stops paying, but this is rare. In many cases, Publisher know the agency is playing the waiting game themselves, and extends a certain amount of goodwill for important clients, who they know will eventually pay up. If you can't get paid though and you have a signed IO, you can pursue legal action, or refer the case to a collections agency. Hope that helps - good
Ben
I tlbm
View 2 months ago
Thanks Ben, very helpful ... I appreciate the links you included for finding out more.
Thanks !
Tom . I" fed novas
View 1 month ago
Say if I was not accessing a website but using a piece of software installed on my desktop which I was logged in to and had ads served to...
Would anyone in the process have an identifier that the ad was served to me/my computer/user account? Who in the process outlined above would have this?
Figure imgf000049_0001
View 1 month
Hi Ed,
It's hard to say without seeing the exact program, but I wouldn't expect it would be all that difficult to assign a unique ID to you. After all, if the software is serving ads, it must be connecting to the internet to contact an ad server, and probably has some kind of browser based functionality which would allow it to assign a cookie to you. It can deliver the ad back, so there's no reason it shouldn't also be able to deliver a cookie back. In this scenario any ad server could assign you a cookie - either the software, or the software plus a 3rd party ad server (if one is in use). Again though, it's hard to know how it might work for your specific program.
Ben
17
Follow Ad Ops Insider
Figure imgf000049_0002
• Search To search, type and hi
AD OPERATIONS JOBS
Online IVtedia Account Representative
Stony Brook, NY - Private
Agile Project Manager - Video & Advertising - B
New York, NY - Project One
Account Executive - Online Advertising
New York, NY - Onward Search
Advertising Sales Representative
New York, NY - Recruitarrow
SelLQnjine Advertising^
Publishers
New York, NY
See Ail Jobs
Post a job for $50
Categories o Ad Exchanges
o Ad Ops 101
o Ad Ops Strategy
0 Ad Ops Tools
o Ad Serving
o Data Leakage
o Data Management
o Inventory & Yield
o Measurement & Tracking
Latest Tweets
o Why beat around the bush ©delta, just put ' zone last" on my boarding pass already - posted on 12/07/2012
o Really solid series on programmatic buying on @ admonsters courtesy of @AdMonsterGavin right now htt : // t.co/zRue5Xrr - posted on 12/06/2012
o We're all about to bear witness to the overnight implosion of a brand http://t.co/GbuTeVEZ - posted on 11/15/2012
o "It's not an ad server - it's an offer server." - posted on 11/02/2012
o Test Driving YouTube's Secret New Site Design by @FrazsE http ;//t.c o/460 gmdl g via @RWW - posted on 11/01/2012
Ad Ops Jobs o Ad Ops Jobs Ad Ops Resources o AdMonsters
o Adtools
0 Fiddler
o HTTP Watch
o Run of Network
Digital Marketing News
0 AdAge
0 AdExchanger
0 ClickZ
o DigiPay
o e arketer
o MediaPost.
0 Mike On Ads
Online Advertising Associations o Interactive Advertising Bureau
0 Online Publishers Association
Recent Posts o What is Holistic Ad Serving?
o The Future of Geotargeting is Hyperlocal
o Limitations of IP-Based Geolocation
0 Geotargeting Expl ined: How Ad Servers Understand Physical Locations
o How Geotargeting Ads Works
Get smart with the Thesis WordPress Theme from DIYthemes.
Copyright © AdOpsInsider.com 2009-2012
• Home
• About
• Ad Ops Jobs
Ad O s Insider
Figure imgf000053_0001
How Real Time Bidding, DSPs, SSPs, and Ad
Exchanges Work by Ben on December 8, 2010
Let's say you're online one day and decide to do a little shoe shopping. You navigate to your favorite store, check out a pair of boots, add something to your cart and just as you're about to checkout, the phone rings and you get distracted. By the time you're done talking to your friend, it's late and you decide to buy the shoes later.
Then, the next day, you check to see who won the big game last night and you notice an ad from the shoe store you were on last night. Not only is it an ad for the store, but the exact pair of boots you were looking at are in the ad! Weird. You decide to check your email and see that your mom sent you a link to a news article. You go to read the article and staring you in the face on the page is another ad for the same pair of boots, this time tempting you with a 10% discount!
How did the ads know you were shopping for shoes last night, and how did they wind up on all those different sites? The answer is, probably through an ad exchange.
Ad Exchanges have been around for a few years, but have exploded in importance in the last year. Along with Demand Side Platforms (DSPs) and Supply Side Platforms (SSPs), Ad Exchanges are dramatically changing the way digital media is bought and sold. If you are a digital marketer or publisher, it is a very exciting time to be working in the industry.
What makes these companies so innovative is how they allow buyers and sellers to value inventory on an impression by impression basis and in real-time. That's right, real time. That means that when you clicked on your mom's link to the news article and your browser requested an ad from the news site, the publisher put that ad up for auction on an exchange, marketers bid on that impression, and it was served to your browser in about 50 milliseconds, so fast it was indistinguishable from the time it took any other image on the page to render. Welcome to the world of real-time bidding, or RTB, where marketers value each impression as it is created and the Ad Exchange is where it all happens.
From a technical perspective though, how does the ad exchange process differ from regular ol' third-party adserving? I'll answer that question and diagram the process in my next post, similar to what I showed you in my diagram for how third-part)' ad serving works.
Tweet 11 i Tagged as: ad exchange, dsp, ssp Profile
Sign in with Twitter Sign in with Facebook
or
Name
Email Not published
Website
Comment
Post It
• 1 Reply
• 0 Comments
• 0 Tweets
• 0 Facebook
• 1 Ping back
Last reply was March 17, 2011
1 · Biagr mmin tht SSP^^
View March 17, 2011
[...] wrote about the basics of how DSPs, SSPs, and Ad Exchanges work from a high level and the value they add to the marketplace in my last post - for this piece [...]
• Follow Ad Ops Insider
Figure imgf000054_0001
• Search
To search, type and hi
Figure imgf000055_0001
Categories
0 Ad Exchanges
0 Ad Ops 101
o Ad Ops Strategy
° Ad Ops Tools
0 Ad Serving
0 Data Leakage
0 Data Management
o Inventory & Yield
o Measurement & Tracking
Latest Tweets o Why beat around the bush @ delta, just put ' zone last" on my boarding pass already - posted on 12/07/2012
o Really solid series on programmatic buying on @admonsters courtesy of ©AdMonsterGavin right now htt : //t.co/zR ue5Xrr - posted on 12/06/2012
o We're all about to bear witness to the overnight implosion of a brand http://t.co/GbuTeVEZ - posted on 11/15/2012
o "It's not an ad server - it's an offer server." - posted on 11/02/2012
o Test Driving YouTube's Secret New Site Design by @FruzsE http://t.co/460gmdlg via @RWW - posted on 11/01/2012
Ad Ops Jobs
0 Ad Ops Jobs
Ad Ops Resources
0 AdMonsters
o Adtoojs
o Fiddler
o HTTP Watch
o Run of Network • Digital Marketing News
0 AdAge
o AdExchanger
0 CUckZ
o DigiDay
o eMarketer
0 MediaPost
o Mike On Ads
• Online Advertising Associations o Interactive Advertising Bureau
0 Online Publishers Association
• Recent Posts
° What is Holistic Ad Serving?
o The Future of Geotargeting is Hyperlocal
0 l io iiai i r.-- il IP-Based Geolocation
0 Geotargetine Explained: How Ad Servers Understand Physical Locations o How Geoiarseime Ads Works
Get smart with the Thesis WordPress Theme from PIYthemes.
Copyright © AdOpsInsider.com 2009-2012 • Home
• About
• Ad Ops Jobs
Ad Ops Insider
Diagramming the SSP, DSP, and RTB Redirect Path by Ben on December 15, 2010
Ho DSPs, SSPs,. and Ad Exchanges Work
Figure imgf000057_0001
I wrote about the basics of how DSPs. SSPs, and Ad Exchanges work from a high level and the value they add to the marketplace in my last post - for this piece we're going to dig into the nitty gritty of how they work from a technical perspective.
In my explanation of third-party ad serving, I outlined a 12-step process to get from publisher ad call to marketer ad creative. In an Exchange environment the process is basically the same, but with three new parties involved: the SSP, the DSP, and the Ad Exchange itself, which all act as the ad selector, instead of the publisher's ad server. You can view a diagram of the ad serving process at the top of this post - the numbers in the text refer to the steps labeled in the diagram.
When a browser navigates to a publisher website (1), the publisher's web server sends back a bunch of HTML code (2) that tells the browser where to get the content (3) and how to format it. Part of the HTML code returned to the browser (4) will include a coded link known as an ad tag. That part is the same as in regular third-party ad serving. But instead of returning a DFA or Atlas tag, the Publisher Ad server will return a tag that points to a RTB -enabled SSP, typically through a dynamic Javascript tag that passes information like the publisher's ID, the site ID, and ad slot dimensions.
From there, the user calls the SSP server (5) where the SSP reads that user's SSP cookie ID, likely already oil their machine. Assuming the user already has that SSP's cookie on their machine (and most users will, given that the largest SSPs boast 80 - 90% reach rates to the US internet population), the SSP starts the auction (6), by requesting bids from a host of demand sources, the DSPs (7). If the user does not have an SSP cookie on their machine, their ad inventory can technically still be auctioned, but since nothing is know about that user, the price will be very low and more related to the site context than the user's attributes. For the DSPs to truly value the impression though, they need to know something about the user that is going to see it. This is where the SSP cookie ID comes in - packaged with the bid request is the SSP's cookie ID, along with the url the impression will deliver on, and what the current user's frequency is on the site.
All these factors help the DSP value the impression. First, through a rather complex cookie-synching process, DSPs are able to match the SSP's cookie ID to their own cookie on that user, which is tied to a huge cache of marketer data and 3rd party data. What kind of 3rd party data? Using the cookie ID, the DSP will be able to know if that user recently priced out a car, is flying to Paris in the next 90 days, was recently shopping for shoes, and even more general demographic information about the user such as their age, gender, income range, credit score, and much, much more. I'll cover how that all works in a later post. But suffice to say, rich data is far and away the driver of higher bids, and the cookie ID is the mechanism through which data is associated to a user.
In addition to the cookie though, where the ad will appear, the URL, is also important. Many brands don't want their ads to appear on just any site, even if they want that user. If the user is on a site with PG-13 content, for example, the advertiser might bid a lower amount or not at all. Similarly, the frequency of that user to the site they are on is also important to valuation. Advertisers are willing to pay a premium to reach users on their first or second pageview on a site vs. their 50th pageview for the simple fact that users are less engaged with site content and more likely to respond to an ad during their first few page views.
Using those pieces of data, the DSPs all value that impression and submit a bid back to the SSP (8) as well an ad redirect to send the user should their bid win the auction. The SSP picks the winning bid (9) and passes the DSP's redirect back to the user (10). From here the process is basically the same as third party ad serving - The user calls the DSP (11), the DSP sends the user the marketer's ad server redirect (12), and user calls the marketer's ad server (13) and the marketer serves the user the final ad (14). Whew!
Tweet 34
Profile
Sign in with Twitter Sign in with Facebook
or
Name
Email Not published Website
Comment
• 54 Replies
• 54 Comments
• 0 Tweets
• 0 Facebook
• O Pingbacks
Last re ly was 4 months ago
Figure imgf000059_0001
View 4 months ago
Sure - chances are an advertiser doesn't work with an exchange directly, regardless of the channel, but relies on a network or DSP to buy on their behalf. They likely spread budget among multiple supply sources depending on their needs and how much they want to spend, and then the network or DSP figures out which impressions to buy via the exchanges or in mobile, more likely with publishers directly at this point. Don't get caught up in the exchange, it's just a gateway player that facilitates liquidity and transactions. It's kind of like asking what the relationship between an investor and the NYSE is; changes are, not much because they buy through a broker like E*Trade or something, and same thing with the seller. The exchange is almost invisible to the user; it's the technology platforms like the DSPs, SSPs, and Networks (the E*Trades in this case) that have to worry about the exchange.
In terms of getting more money, it's all about performance and relationships on the buy side service provider front. For the exchange, it's usually about how much scale they bring to the table. I don't believe the prices are meaningfully different for any of the exchanges.
Hope that helps -
Ben
2. %Nilin
View 4 months ago
Hi Ben,
Thanks for the detailed explanation and the quote on the one failed relationship. Fully understood your views.
Regards
Nitin
3.
Figure imgf000059_0002
View 4 months ago
Hey Ben - This was an awesome read. I have been reading bits and pieces at various places but never got such a ood grasp of the process before reading this. Thanks a ton.
Figure imgf000060_0001
View 4 months ago
Thanks Rohit, Tm glad you found it helpful!
Comment navigation
<-- Older Comments
• Follow Ad Ops Insider
Figure imgf000060_0002
• Search
To search, type and hi'
Figure imgf000060_0003
Sell Online Advertising-Sell Video Content io
Publishers
New Ybr NY
Sea All Jobs
Categories
0 Ad Exchanges
° Ad Ops 101
o Ad Ops Strategy
o Ad Ops Tools
o Ad Serving
0 Data Leakage
0 Data Management
o Inventory & Yield
0 Measurement & Trading
Latest Tweets o Why beat around the bush @ del la, just put ' zone last" on my boarding pass already - posted on 12/07/2012
o Really solid series on programmatic buying on @admonsters courtesy of @AdMonsterGavin right now http://t.cQ zRue5Xrr - posted on 12/06/2012
o We're all about to bear witness to the overnight implosion of a brand http://t.co/GbuT VEZ - posted on 11/15/2012
o "It's not an ad server - it's an offer server." - posted on 11/02/2012
o Test Driving YouTube's Secret New Site Design by @FruzsE http ://t.eo/460 gmdl g via @KWW - posted on 11/01/2012
Ad Ops Jobs
0 Ad Ops Jobs Ad Ops Resources o AdMonsters
o Adtools
° Fiddler
o HTTP Watch
o Run of Network
Digital Marketing News
0 AdAge
o AdExchanger
o ClickZ
o DigiDay
o eMarketer 0 MediaPost
o Mike On Ads
• Online Advertising Associations
0 Interactive Advertising Bureau
• Recent Posts o What is Holistic Ad Serving?
o The Future of Geotargeting is Hyperlocal
o Limitations of IP-Based Geolocation
0 Geotargeting Explained: How Ad Servers Understand Physical Locations 0 How Geotar eting Ads Works
Figure imgf000062_0001
Get smart with the Thesis WordPress Theme from DIYthemes.
Copyright © AdOpsInsider.com 2009-2012
• Home
• About
• Ad Ops Jobs
Ad O s Insider
Figure imgf000063_0001
SSP to DSP Cookie-Synching Explained by Ben on May 1, 2011
The matching process of the SSP cookie ID to the DSP cookie ID happens through a parallel process to serving ads called cookie-syncing. Cookie-syncing is necessary because as a standard security process, web servers of any kind can only request cookies that are set to their own domain. Since the SSP sits between the end-user and all the DSP bidders in a real-time auction however, the DSP needs a way to identify the users it is looking for.
So let's take a simple example on a store trying to retarget their users. Let's say that you run storeABC, and userl23 drops in and adds a pair of $ 150 shoes to their shopping cart, but never makes it to the checkout. You want to retarget that user and serve them an ad directing them back to your store to try and close the sale. Since you work with DSP456, you have a lxl pixel sitting on your shopping cart page, which forces the user to call out to DSP456's web server as they load the shopping cart page, giving DSP456 a chance to drop a cookie on userl23. That cookie ID is DSPcookie789. Now, userl23 is surfing around the web, and lands on
awesomesite.com, which is using SSP 123 to monetize their ad inventory, awesomesite.com serves a 3rd party redirect to SSPL23, which drops a cookie on userl23. That cookie ID is SSPcookieXYZ. SSP123 now requests a bid from DSP456 among other bidders for the impression that SSPcookieXYZ is about to view. But wait, how does DSP456 know that SSPcookieXYZ = DSPcookie789? On this first ad, it doesn't, so your DSP doesn't bid on the impression. Bummer. Remember, the SSP can only read and pass its own cookie ID to bidders.
After SSP123 selects a winning bidder though, it runs one last piece of javascript that forces userl23 to call out to a handful of regular bidders, including DSP456. In that redirect, using a query string, the SSP passes its cookie ID on userl23 (SSPcookieXYZ). Now the user IS calling DSP456's web server and the DSP can request its own cookie from userl23 in a process the industry refers to as "piggybacking". Eureka! SSPcookieXYZ also has DSPcookie789 - it's user 123 ! The DSP knows the SSP's cookie ID because of the query string in the piggyback call, and it can read its own cookie ID because that user called its web server as the end destination with the piggyback call. DSP456 now writes into its database that DSPcookie789 = SSPcookieXYZ for bid requests from SSP123. The next time userl23 hits a page that SSP123 is helping to monetize, DSP456 will know it is the same user that abandoned their shopping cart and can bid appropriately. The process repeats for sites using other SSPs.
It sounds complex, but all it means is that for every ad served through RTB, about lOx as many technology companies are involved in the cookie-synch process. The reason the synching process occurs after the auction happens and not before it is because of latency and user experience. If user 123 had to call 10 DSPs, wait for those DSPs to cookie their machine, write the matching SSP id into their database, and then bid, it would dramatically slow down the entire auction process for everyone. If the cookie synch fails on the other hand, well there will be many more opportunities for that.
If you are interested, here are the URL's that run the piggyback script for each major SSP - these pages look blank, so you'll have to 'view source' to see the code.
Pubmatic: http : / / ads . pubmatic. com/AdServer / js sync uppixels . html It looks aS though VCode IS the item passing the cookie ID.
Rubicon Project: ttp: / t.apZ-cdn. r biconpro-i ct .com /partner/s n p-s rubi con -;emily . html It looks as though nid is the item passing the cookie ID.
Admeld appears to take a slightly different solution, passing a browser partner by partner with callbacks to Admeld on each run rather than calling out to a handful at once like the SSPs above. I'm not able to identify the logic here as easily, but they must accomplish the same end result.
T eet < 37 ; Tagged as: cookie-synching, dsp, ssp Profile
Sign in with Twitter Sign in with Facebook
or
Name
Email Not published
Website
Comment
Post It
• 12 Replies
• 12 Comments
• 0 Tweets
• QjFa_cebook
• 0 Plngbacks
Last reply was 2 months ago
1. S' Rolin ' Hl i.'i
View November 7, 201 i
This is great. Thanks for a detailed step-by-step explanation.
2.
Figure imgf000064_0001
Cooper
View 9 months ago
Could you expand on this statement: "After SSP123 selects a winning bidder though, it runs one last piece of javascript that forces userl23 to call out to a handful of regular bidders, including DSP456"
So how exactly does that happen?
Thx
Figure imgf000065_0001
View 9 months ago
Hi Jason,
Basically speaking, the SSP's tag is a chunk of JavaScript that in turn references other scripts. That's pretty common with JS ad tags in general, actually - they usually have to do a few things to get the ad to work as designed. Sometimes they need to reference jQuery libraries, access browser plugins, compile information from other servers, reference cookies values, and a bunch of other things. Specific to the SSPs, the first sub-script usually redirects the user to the SSP's server which starts the auction process. The last sub-script is a call which tells the browser to go call a bunch of other companies for the sake of cookie syncing users. For most browsers, this last script in practice may run more or less simultaneous to the auction / ad load, because by then, the browser has been redirected downstream to other parties and has moved on to the next part of the tag. Most modern browsers can keep 6 - 8 HTTP connections active at any given time, so once it executes the first sub-script, it starts following that path with some of the available connections and starts executing the next sub-script with others, depending on what else it has to load on the page.
Hope that makes sense -
Ben Ψ IWes Kim
View 9 months ago
Hi Ben,
Can one company be both SSP and DSP, then no syncing is necessary? Is it because of privacy concern?
Figure imgf000065_0002
View 9 months ago
Hi Wes,
No, no company plays both sides of the fence and is at once a DSP and an SSP. Google does own both types of companies - Invite Media is a DSP and Admeld is an SSP, but those companies still maintain separate cookie spaces on separate domains, so still need to cookie sync with one another. Their is no real privacy concern, the reason that there isn't a DSP / SSP hybrid is because having both advertisers and publishers as a client on the same platform is a large conflict of interest - how would the platform represent both sides fairly - that if such a platform existed, I doubt they would have any customers. You wouldn't hire a real estate agent to sell your house if they also represented the buyer, would you? Why would the agent have any incentive to get a fair market value for your house? For the same reason, DSPs and SSPs must stay as separate systems - many expect clients to abandon Admeld because of the conflict of interest for Google to represent both a buying and selling technology, but I can't say I know that it's actually happened in practice. As far as I know Google keeps those two companies very separate, and there's no reason to think that Invite and Admeld don't effectively operate as two separate, independent entities.
Ben f " khil
View 9 months ago
Thanks for the great explanation, just curious to know
1. Will DSP always have their code on advertiser website or there will be cases when the DSP cookie needs to be synched with advertiser's cookie ?
2. As explained in your another post, DMP will have cookies synched with various 3rd party systems, now how will DMP integrate with DSP, will it require cookie synching ?
Wen
View 9 months ago
Hi Akhil,
To answer your questions -
1. There's no reason I can see why an advertiser wouldn't fire their DSP's cookie at the same time they drop their own cookie for any users they want to target in the exchange. In some cases however, an advertiser may have offline data and work through a match partner and data management platform (DMP) to bring that data online to a cookie, in which case the DMP would need to sync with the DSP before those users would be active in the exchange. That's a much less common use case however, and in that instance, the advertiser doesn't really have a cookie in the classic sense, dropped from their own domain, but really reusing someone else's cookie space for their own purposes.
2. Yes, DMPs integrate with DSPs through cookie syncing. A system to system sync only needs to happen once however, not once per system per client. Once the users are synced, the bridge benefits all the clients on both systems, who can move securely move proprietary data from one place to the other through a server to server integration.
Hope that helps -
Ben
Figure imgf000066_0001
View 8 months ago
Hi,
I am not sure I understand this: "The DSP knows the SSP's cookie ID because of the query string in the piggyback call, and it can read its own cookie ID because that user called its web server as the end destination with the piggyback call. DSP456 now writes into its database that DSPcookie789 = SSPcookieXYZ for bid requests from SSP123. The next time userl23 hits a page that SSP123 is helping to monetize, DSP456 will know it is the same user"
- wouldn't it be in the DSP's favor to 'fake' the cookie match? Say for e.g. the DSP456 says that the DSPcookie999 = SSPcookieXYZ, then SSP has no way of knowing whether DSP has genuine match or not and it will have to favor DSP456 for the subsequent bids.
[here I am assuming that winning the bid matters the most to the DSP rather than winning the right bid, for the urpose of getting visibility or some other xyz reason...]
Figure imgf000067_0001
View 8 months ago Hi Rohan,
It's important to realize that the DSP and the SSP have a mutual benefit in syncing cookies - browsers only allow a cookie to be read by the domain from which it was set, meaning the SSP can only read its own cookie on a user, and can't know if a DSP also has a cookie on that user. In the bid request then, the SSP can only show bidders its own cookie ID, it can't provide the DSP's cookie ID. So it's critical then that the DSP have a way to know if it has a cookie on that user, and what it knows about that cookie, from the SSP's cookie ID only. The way it does this is through the cookie syncing process.
To your question about faking the cookie match, though, the DSP doesn't really have anything to gain in that scenario. It's not as if the SSP knows anything meaningful about that user through the match, they just know if the user exists in the DSPs system, and vice versa. The cookie match simply facilitates the transaction. And the fact of the matter is that DSPs don't bid more for un-synced cookies, they bid less, because they don't know if they know anything about that user. Even if they were lying some of the time, in general, the DSP will likely bid less for unsynced users overall, so the SSP won't favor them at all in that case. But again, there's not much point - it's far more important for the DSP to win the right bids than to simply win any bid, that's the entire point of using a DSP. Otherwise, you could just use an ad network to buy as many impressions for the cheapest price possible, regardless of any knowledge about the user set.
Hope that clears things up -
Ben i' Wltve Page
View 2 months ago
Hi Ben,
What other information can be passed via an SSP ID to the DSP to add value in the bidding process. From your description, the cookie syncing only supports a binary decision from the DSP as to whether or not there is a DSP cookie on the users machine - yes or no. It is commonly referenced within the industry that publishers have a wealth of data points on their users, and the onus should be on the publishers to expose this data with a view to better monetizing their sites. With the above definition, all of the information used as the basis for the bidding decision is on the advertiser side; through the retargeting pixel. Is there any other information that can be passed via the SSP cookie which would give the bidders greater insight into the user and their behaviour, so for example; the segment IDs (and segment definitions) to which the user belongs as classified by the publisher or the SSP. . Wqf
View 2 months ago
"you have a lxl pixel sitting on your shopping cart page"
Hello,
you mean embedded javascript right? not a real pixel ?
Ty
Franck
Figure imgf000068_0001
View 2 months ago
Hi Ty,
What I mean by that lxl pixel is basically, "you have a mechanism to track a conversion event" - that might manifest as a simple image tag for a transparent lxl, or a piece of javascript may configure that conversion beacon call in some kind of dynamic fashion, or you may simply drop a cookie on that user through an image call. But whatever the specific technical execution, the point is the same, to track a conversion, and associate it to a specific user and that user's other actions and touchpoints.
Ben
• Follow Ad Ops Insider
Figure imgf000068_0002
• Search
To search, type and hi AD OPERATIONS JOBS
Agile Project Manager - Video & Advertising - BM
New York, NY - Project One
Online IVSsdia Account Representative
Figure imgf000069_0001
New York, NY - Onward Search
Advertising Sales Representative
New York, NY - Recruitarrow
Sail Online Advertising-Sell Video Content to
Publishers
New York, NY
Sas All Jobs
Post a job for $50
Categories o Ad Exchanges
o Ad Ops 101
o Ad Ops Strategy
0 Ad_Orjs_ ools
0 Ad Serving
o Data Leakage
0 Data Management
0 Iirvejitor jfc_Yi d
o Measurement & Tracking
Latest Tweets o Why beat around the bush @ delta, just put ' zone last" on my boarding pass already - posted on 12/07/2012
o Really solid series on programmatic buying on @admonste-rs courtesy of @AdMonsterGavm right now http://t.co/zRue5Xrr - posted on 12/06/2012
o We're all about to bear witness to the overnight implosion of a brand http://t.co/GbuTeVEZ - posted on 11/15/2012
o "It's not an ad server - it's an offer server." - posted on 11/02/2012 o Test Driving YouTube's Secret New Site Design by @FruzsE http://t.co/460gm<ll¾ via ( KW'W - posted on 11/01/2012
Ad Ops Jobs
0 Ad Ops jobs
Ad Ops Resources o AdMonsters
o Adtools
o Fiddler
0 HTTP Watch
0 Run of Network
Digital Marketing News
° AdAge
o AdExchanger
0 ClickZ
o DigiPay
o eMarketer
o MediaPost
o Mike On Ads
Online Advertising Associations o Interactive Advertising Bureau
o Online Publishers Association
Recent Posts
0 What is Holistic Ad Serving ?
0 The Future of Geotargeting is Hyperlocal
o Limitations of IP-Based Geolocation
o Geotargeting Explained: How Ad Servers Understand Physical Locations
o How Geotargeting Ads Works
Figure imgf000071_0001
• Home
• About
• Ad Ops Jobs
Ad Ops Insider
Figure imgf000072_0001
How to Read Doubleclick Ad Tags and Ad Tag
Variables by Ben on February 25, 2010
The term ad tag is thrown around quite a bit, and can usually refer to any link involved in the ad serving process, on the publisher, or marketer side. Strictly speaking, Ad Tags are the HTML code a browser uses to fetch an advertisement from an Ad Server - it is a redirect to content rather than content itself. There are also click tags, action tags, view tags, and other more specific variants to the general ad tag category. For this particular example, we'll look at publisher side tag, because our purpose is to show how ad tags help publishers organize their content into targetable products.
Ad Tag Components
So, without further ado, feast your eyes on this example a Doubleclick ad tag:
http : / / ad . doubleclick . net/ADJ/publisher/ zone ; topic=abc; sbtpc=def ; cat=ghi ; kw=xyz ; tile=l ;
slot=728x90. I;sz=728x90;ord=7268140825331981?
An ad tag can tell you quite a bit about how which ad ends up on a page - if you want, navigate to any major publisher and look at the source code; you can probably find a real-life example of a working ad tag. So how can you tell what the ad tag says about the publisher hierarchy and ad targeting? Let's break it down piece by piece: http://ad.doubleclick.net / - this is the host address for the Ad Server - you can see that it is not a publisher's website, but an independent technology company that has nothing to do with publishing content. In this example, we're talking about Doubleclick, the Ad Serving powerhouse that was acquired by Google for $3.1 billion dollars in 2007.
/ADJ - this code defines a specific type of ad call, and what the response can be, i.e., images vs. XML vs.
scripts. For this example, the code 'ADJ' is the most common, and only returns images, which will serve via JavaScript. Other responses can include ADF (only image creatives in a frame), ADX (only image creatives served through streaming technologies), as well as others. (Thanks to Jared & Paul for correcting!)
/publisher - this is the site code that Doubleclick uses to distinguish one publisher property from another. For example, the New York Times owns NYTimes.com, About.com, and Boston.com among other properties. If they are a client of Doubleclick, the corporation likely pays the bill, but each site would have its own site code so ads could be targeted to a specific paper and not the entire network.
/zone - the zone is akin to a channel level, so the Homepage vs. the Arts page, vs. the Sports page. These content verticals are likely to attract different advertisers, so it's important for publishers to be able to target to this kind of granularity.
Zone-Based Hierarchy vs. Topic Based Hierarchy
Here is where tagging logic starts to diverge in Doubleclick. Some publishers prefer to deeply categorize at the zone level, while others keep moving down the hierarchy to the topic level. The benefit of using zones over the topic, subtopic, category, or keyword levels that we'll talk about in just a minute is that the zone is the last level in which you can pull historical reporting. So you might have sports/baseball or even sports/baseball nymets so you can pull traffic statistics going back months or years.
The downside with this method is that zones are vertical structures, so if you had multiple verticals on your site that all had a games section, you would have to select each games zone every time you wanted to target all games when trafficking the ads, rather than just targeting a single "games" key value. This sounds easy on paper, but adds up to lots of extra time for your trafficking staff if you have lots of subcategories in each zone. It wouldn't be difficult to imagine needing 50 zones or more per content vertical to tag to the lowest level of granularity.
Which is why most Publishers tag at a higher level, and leave the granularity to the topic variable and below. A great benefit of granular topic tagging opposed to granular zone tagging aside from being able to use the same topic tag across multiple zones is the ability for topic tags to handle wild cards when trafficking. This means if you had topic=newyorkmarathon and topic=bostonmarathon, you could simply target topic=*marathon* and ads would automatically fall into both areas. This makes trafficking much easier, but has the downside of no historical reporting, which can be a challenge for your Yield or Inventory teams.
topic=abc - next in the hierarchy is the topic level. As mentioned above you can use the topic level to tag similar content across zones. For example, games in multiple content verticals or within them.
sbtpc=def - next in the hierarchy is the subtopic level. You might use this to target sportsgames vs.
adventuregames for example. Again, you can use this to target across content verticals or within them. k =xyz - the keyword segment isn't really another level in the hierarchy but a way to describe the page for contextual targeting. The benefit here is multiple keywords are allowed. These are typically used in guides and directories like a recipe, where you would want to be able to target chicken recipes vs. vegetarian recipes vs. winter recipes, and etc, allowing some overlapping targeting.
tile=l - the tile variable sets a unique value for each ad call on a specific page. If there were two or more of the same size ads on a page, separate tile values would prevent the browser from trying to serve the same ad to multiple ad slots at the same time.
siot=728x90. i - typically defines the location of the ad tag, but is really just another type of key-value. While this may seem duplicative with the tile value, it isn't. For example, tile values are often set dynamically, in the order they appear on the page. So the first call is tile=l, the second is tile=2, and so on. But websites use different templates all the time so the homepage may not have as many ad calls as a category page which may have a different number of calls than an article page, so the tile value isn't designed to be a consistent variable for use in targeting. The slot however, is. For example, if a publisher had two of the same ad units on a given page, say a 728x90 unit at the top of the page and a 728x90 at the bottom of the page, the slot value allows them to target specifically to one or the other. That said, the publisher could just as easily set the value of this to anything they want, and it's common to see sites re-purpose this keyvalue for another purpose, use a text value such as "leaderboard", or not use it at all. See Jared's post in the comment thread below for more detail.
s z=728x90 - defines the ad size of the unit for the ad server logic. To be clear, this doesn't restrict the size of the ad in the unit, it just provides a targeting attribute for the ad server. If a trafficker were to mistakenly target a 300x250 ad to a market segment with a sz=728x90 attribute however, the 300x250 creative would still serve to the 728x90 call, it would just be cut off. It isn't uncommon to catch one of these mistakes from time to time as you surf around the web. Additionally, you can actually include multiple values into this attribute, separated by commas. (Thanks to Jared for correcting !) ord=7268140825331981 - this number is a random value better known as a cache-buster. As users move back and forth between pages of content, they often return to pages they've seen before, especially navigational pages like the homepage. Browsers today try to save as much content as possible to speed up load times. To prevent browsers from reloading the same ad multiple times (so publishers can maximize revenue and advertisers can get accurate reporting), a random number is tacked on to the end of each ad call so it looks unique to a browser and forces a new series of calls through the ad server. Click here to read more about what a cache-buster is and how it works.
Tweet < 7 Tagged as: ad tags, doubleclick
Profile
Sign in with Twitter Sign in with Facebook
or
Name
Email Not published
Website
Comment
Post It
• 20 Replies
• 20 Comments
• 0 Tweets
• 0 Facebook
• 0 Pingbacks
Last reply was 2 months ago
1. f tkas
View June 14, 2010
Great article. I have been on the publishing side. I used to track ads diligently for my clients, and provide their agencies with stats, and then work out the differences. Now that I'm on the marketing side, I have an agency that uses only the stats from the publishers. The agency thought I was crazy when I asked for verification - or at least numbers from their own system. And then I thought I was crazy for asking. Not so. Looks like the landscape is changing a bit, but in order for us to get our $ worth, I'll be asking them for some additional level of verification. Thanks for bringing back some sanity © . f UDerek
View June 8, 2011
Hello,
I would like to know how "key value targeting" is integrated into a SMART adserver adtag? Can anybody help please? An example would be most welcome!
Many thanks,
Derek
Figure imgf000075_0001
View June 13, 2011
Hi Derek,
I'm sorry but I don't have any experience w/ the SMART ad server - but hopefully someone else knowledgeable will be able to answer your question. If you are still having trouble figuring this out I would suggest posting a topic on the AdMonsters publisher forum. I would also request documentation from the ad server itself, I would think this would be part of a FAQ...good luck!
Ben 'if fAllison
View Mv 14, 2011
Great article! I have a blog and we are looking to work with an adnetwork to garner higher CPM's vs. the CPC Google Adsense ads we have been serving. They noted "We're currently offering our $X CPM for all the inventory we can fill and whatever inventory we cannot fill, we'll send it back to you through the default tags you provide us.
Where can I get default tags? We are using OIO publisher and I am a bit confused as to what she wants me to send her.
Figure imgf000075_0002
View July 15. 2011
Hi Allison,
Right, since networks cannot always fill every impression you generate on your site, they need a way to refuse the ad call, letting you choose to serve something else instead. I hadn't heard of OIO before, but looked them up (very cool!) and I would say you have two options.
1. If you are only going to work with this one network and Adsense, my recommendation would be to get the Adsense ad code from Google, and give that to the network as your default tag. Essentially you are saying to the network, "if you can't serve anything, serve my Google Adsense code instead". You can get this code in the Adsense UI under My Ads > Content > Ad Units, and then clicking "get code" under each ad. Be sure that you give your network the right ad code for the right ad size so you don't serve a 728x90 unit in a 300x250 space.
2. Your other option is if you plan to work with more than one network and is the way most large publishers operate (or outsource through an SSP). What you would do there is give them your OIO code by looking in your WordPress account at OIO Publisher > Settings, and then providing the Javascript Output code you see at the top of the page. What you are saying to the network there is "if you can't serve anything, call my regular ad code again". I have to stress however that you must setup a new zone with the first network relationship excluded from that targeting, otherwise your ad code will just continue to call the first network, which will default back to you, which will call the first network again, and you'll be stuck in an ad serving loop until the call fails or the user abandons the page.
It sounds like you are just working with this one network and Adsense, so my advice would be to go with the former, and simply give this network your Adsense code.
Hope that helps,
Ben f % jarred
View My 28, 201 1
i hate to be negative but the article is not 100% accurate about what the ad call means htip://ad.doubleclick.net/AB
sz=728x9Q;or< =7268140825331981 ?
/ABC -this is not a publisher unique code. It will always be (ad, adi, adx, pfadx, adj, etc...) the differences represent types of ad calls DFP makes. Some will only return an image, XML, scripts etc..)
sz=728x90 - defines the ad size of the unit.— to some extent this is true, you could easily serve a 1000x1000 into that ad call, the size doesn't matter but for an ad to serve to that ad call the line must match the value set in the ad call
slot=728x90.1— this is just another keyvalue. you could easily make the difference between the top and bottom space by making it pos-top or pos-bottom on each ad call, keyvalues are totally open so based on our preference in work flow you could make it whatever you want.
Figure imgf000076_0001
View My 28, 201 1
Thanks for the feedback, Jared - after further research, you're absolutely correct. I will correct this article accordingly, and credit you for the information. Thanks again!
Ben
Figure imgf000077_0001
View Aueust 17. 2011
Hi Ben!
Awesome helpful articles, thanks a lot!
Could you tell me, for which products does this info stand, doubleclick for advertiser, doubleclick for publisher (dfp small business/premium) or the old DART thing?
Thanks m" Beti
View Aueust 17, 2011
Hi Bob,
Thanks for the compliments! I wrote this article originally about DFP, the enterprise, cloud-based ad server designed for major publishers. An ad tag from the DFA, or advertiser perspective is much simpler, because it doesn't need to collect any dynamic information about where the ad is running, as the publisher tag does.
Ben
Figure imgf000077_0002
View March 22. 2012
Hi Ben
Is there any way to differentiate between DART and DoubleClick for Publisher ad tags?
Thanks !
Figure imgf000077_0003
View March 30, 2012
Hi Ajit,
Hrm, an interesting question, and the answer is no, not that I can think of, at least not reliably or universally. Part of the issue is there are multiple versions of both systems, there are lots of ways to customize the implementation, and there are lots of different tag types, so it is very difficult for me to think of an attribute that is reliable across everything. Sounds like a question for the AdMonsters Forum, or perhaps the DoubleClick Boards. Sorry I couldn't be of any help!
Ben W IPaw/
View April 2, 2012
Hey
Great fan of the site. Slight correction on the "/adj" section as this can accept Flash, rich media or image creatives and will serve them by JavaScript. Image only would be "/ad"
atlp://support.google.com dfp. preniium/biiVanswe.f.py?fal~en^
utm campaif¾n=kzfoxlxphltlltov2xrn
Cheers
Paul
Figure imgf000078_0001
View April 10, 2012
Thanks Ben. I will post the question on Ad Monsters and DoubleClick forums
Regards
A it
Figure imgf000078_0002
View April 22. 2012
Hi Paul,
Thanks very much for the link and correction - I've edited the post to reflect your comments Thanks again!
Ben
Shij
View May 17, 2012
Hi Ben,
Great work with this blog. Do you have a blog post that explains the ad tag format for 3rd party marketer ad server and how the ad serving and statistics logging happens ?
Thanks ί tNilish
View July 11.
Thanks for the great article Ben
I am new into advertising and the website really helps 1
Figure imgf000079_0001
View My 12, 2012
Thanks Nitish, glad you find it useful!
Figure imgf000079_0002
View My 12, 2012
Hi Shiju,
The nearest thing I have to explaining tag format is the post above - in terms of counting methodology and process though, you might find these articles helpful:
hiip://www.adopsmsk1er.conx½i-sem
ht^:/ www.adopsinsider.com/ad-serving/diagramrrii
Hope you find them useful -
Ben f f rbaea.org
View 2 months ago
Hey! Quick question that's totally off topic. Do you know how to make your site mobile friendly? My web site looks weird when viewing from my iphone. I'm trying to find a theme or plugin that might be able to fix this problem.
If ou have any suggestions, please share. Appreciate it!
Figure imgf000079_0003
View 2 months ago
Hi there - I'm using WPTouch, developed by the fine folks at BraveNewCode to make my site mobile friendly. It's just a handy plugin for WordPress. http://wordpress.ore/plugins/wptouch/
Hope that helps!
Ben
8* 4 Follow Ad Ops Insider
Figure imgf000080_0001
Search
To search type and hi
Figure imgf000080_0002
AD OPERATIONS JOBS
J u nior Campaign Trafficker
BrookiynTNY-- VICE Media
Associate Advertising Solutions Manager
Kingdom - TripAdvisor LLC
:al iechnoiogy
New York - TMP Worldwide
Digital Marketing Manager
New York, NY - Ready Set Rocket
French Account Executive - Get Online Team Kingdom - TripAdvisor LLC
See All Jobs
Post a job tor $50
Categories o Ad Exchanges
0 Ad Ops 101
0 Ad Ops Strategy
0 Ad Ops Tools
o Ad Serving
o Data Leakage
o Data Management
0 Inventory & Yield
0 Measurement & Tracking Latest Tweets o Biggest pet peeve to see people who digital advertising using @adblockplus - don't these people know who pays their salary? - posted on 06/03/2013
o C'mon people. Scope, Resources, Time - you can pick two. - posted on 05/09/2013
o we're all adults here @pandora radio - go ahead and play the explicit version. - posted on
04/28/2013
o Love cheeky headlines? You can't write them, but at least you'll see them 1st as the new #adops manager @nydailyncws htlp://t,co/wL3QOKWsLT - posted on 04/26/2013
o But seriously, the RTB $ isn't going to optimize itself - why don't you do it @ match as the new Dir. of Yield? #adops http://t.co/vur7DjYnCL - posted on 04/26/2013
Ad Ops Jobs o Ad Ops Jobs
Ad Ops Resources o AdMonsters
0 Adtools
0 Fiddler
o GeoEdge
o HTTP Watch
o Run of Network
Digital Marketing News o AdAge
o AdExchanger
0 lickZ
o DigiDay
o eMarketer
o MediaPost
o Mike On Ads
Online Advertising Associations
0 Interactive Advertising Bureau
o Online Publishers Association
Recent Posts o Lookalike Modeling Your Ad Ops Team Can Build With a DMP
o How Ad Serving Works - Mobile vs. Web Environments
0 What is Holistic Ad Serving?
0 Th Future of Geotargeling is Hyperlocal
o Limitations of IP-Based Geolocation
1 2 3 4 5 6 7 8 9 Next Page » 0 PTI Ml ΙΙΙΤΑΜΕΪ | ATT R I BUTE
Figure imgf000082_0001
Get smart with the Thesis WordPress Theme from DIYthemes.
Copyright © AdOpsInsider.com 2009-2013

Claims

WE CLAIM:
1. A method practiced by a user's computing apparatus, comprising the acts: sensing information about a user's physical - as opposed to computational - environment;
transmitting the sensed information, or data derived therefrom, to a remote service, and also transmitting identifier data associated with the user;
in connection with a subsequent transaction, transmitting said identifier data associated with the user; and
as part of said subsequent transaction, receiving information for rendering by the user's computing apparatus;
wherein the information received for rendering is customized based on the information sensed about the user's earlier physical environment.
2. The method of claim 1 wherein the subsequent transaction occurs more than an hour after said sensing.
3. The method of claim 1 wherein the sensing employs a sensor component in the computing apparatus.
4. The method of claim 3 in which the sensor component comprises a headworn camera.
5. The method of claim 1 that includes hashing the sensed information, and transmitting the hashed information to said remote service.
6. The method of claim 1 in which the transmitting is performed in responsive to a request by a web site with which the user's computing apparatus is in communication.
7. The method of claim 1 that includes discerning a social relationship between the user and another individual based, in part, on said transmitted sensed information, wherein the information received for rendering is customized based on said discerned social relationship.
8. The method of claim 1 wherein:
the physical environment includes at least audio, and the sensing employs a sensor in the user's computing apparatus to sense said audio;
the identifier data associated with the user comprises cookie data associated with the user, and corresponds to a cookie file stored in storage of the user's computing apparatus;
and the method includes, in connection with the subsequent transaction, transmitting the stored cookie file from the user's computing apparatus.
9. The method of claim 1 wherein the information received for rendering has been customized by a remote system based on the information sensed about the user's earlier physical environment.
10. A method practiced by a system remote from a user's computing device, comprising the acts:
receiving data about the user's physical - as opposed to computational - environment; storing the received data about the user's physical environment in association with identifier information for the user;
in connection with a subsequent transaction, receiving identifier information for the user; and
tailoring information to be sent to the user device based on the stored data about the user's earlier physical environment.
11. The method of claim 10 wherein the subsequent transaction occurs more than an hour after said act of receiving data.
12. The method of claim 10 wherein: the physical environment includes at least audio, the received data being based on information sensed by a sensor in the user's computing device; and
the identifier information comprises cookie information.
13. A method comprising the acts:
in connection with a first transaction, determining context information about a user' s environment, and transmitting the context information, or data based thereon, to a remote service, for storage in association with user identifier information identifying the user or a user's computing device;
in connection with a subsequent transaction, transmitting user identifier data from a user apparatus; and
as part of said subsequent transaction, receiving information at the user apparatus that depends, in part, on the earlier- sensed context information.
14. The method of claim 13 in which the sensed context information is indicative of a first item of entertainment content presented to the user.
15. The method of claim 13 wherein the identifier information comprises cookie information.
16. The method of claim 13 in which the determining comprises determining attribute information about a user's environment, said attribute information originating from a consumer electronic device distinct from said computing device.
17. The method of claim 13 in which:
the determining includes sensing audio originating from a consumer electronic device using a microphone in the user's computing device, and deriving identification information from the sensed audio, the derived identification information being indicative of a first item of entertainment content presented to the user; the transmitting of the context information, or data based thereon, further includes transmitting identifier data associated with the user to the remote service, the transmitted cookie data corresponding to a cookie file stored in storage of the user's computing device;
the subsequent transaction comprises the user's computing device requesting delivery of data for rendering by the user's computing device;
the transmitting of the user cookie data from the user apparatus comprises transmitting the stored cookie file from the user's computing device; and
the receiving information comprises receiving the requested delivery of data together with advertising, the advertising depending, in part, on the first item of entertainment content earlier presented to the user.
18. A method comprising:
in connection with a first transaction, sensing content using a microphone or camera in a user's computing device;
deriving identification information from the sensed content, the derived identification information being indicative of the content being presented to the user;
providing the derived identification information to a remote computer, and also providing identification data associated with the user to said processor;
in connection with a subsequent transaction in which the user's computing device requests delivery of data for rendering by the user's computing device, transmitting the identification information from the user's computing device; and
as part of said subsequent transaction, receiving the requested data together with advertising;
wherein the advertising depends, in part, on the content earlier presented to the user.
19. The method of claim 18 in which the deriving comprises calculating fingerprint data from the sensed content.
20. The method of claim 18 in which the deriving comprises decoding digital watermark data steganographically encoded in the sensed content.
21. A method comprising:
at a first time, receiving information corresponding to audio or visual content sensed from a user's physical environment; and
at a subsequent time, identifying advertising for presentation to the user, based at least in part on said identified information.
22. The method of claim 21 in which the receiving further comprises receiving cookie data from a user device; and in which the method further includes, by reference to the received cookie data, identifying information about audio or visual content sensed from the user' s environment.
23. The method of claim 21 in which an hour or more separates the first time from the subsequent time.
24. The method of claim 21 in which a day or more separates the first time from the subsequent time.
25. The method of claim 21 in which a week or more separates the first time from the subsequent time.
26. The method of claim 21 in which a month or more separates the first time from the subsequent time.
27. A method in which a visual sensor of a first system acquires visual information representing first data, the method including the acts:
using an audio sensor in a second system to receive audio information representing said first data, the received audio having been emitted by the first system based on its acquisition of said visual information;
extracting the first data represented by the received audio, using a hardware processor configured to perform such act; and
storing cookie data based on said extracted first data.
28. A method in which a first system includes a first sensor responsive to a first type of stimulus, and acquires information conveyed by said first type of stimulus, the acquired information representing first plural-bit data, the method including the acts:
in a second system, using a second sensor responsive to a second type of stimulus different than the first type of stimulus, to receive information conveyed by said second type of stimulus from the first system, said received information representing said first plural-bit data; extracting the first plural-bit data represented by the information received by the second sensor, using a hardware processor configured to perform such act; and
storing cookie data based on said extracted first plural-bit data.
29. The method of claim 28 in which:
the first type of stimulus is light;
the second type of stimulus is sound; and
the first plural-bit data comprises a multi-digit identifier of a retail product.
30. A method comprising:
at a first time, receiving information about entertainment ambient content sensed by a microphone in a user' s portable device, said received information being accompanied by an identifier of the user or the user's device;
determining, from the received information, a language apparently understood by the user, and storing data related thereto; and
at a later time, selecting a language- specific version of content to be provided to the user, or the user's portable device, based on said stored data.
31. The method of claim 30 wherein an hour or more separates the first time from the later time.
32. The method of claim 30 wherein a day or more separates the first time from the later time.
33. The method of claim 30 wherein a week or more separates the first time from the later time.
34. The method of claim 30 wherein a month or more separates the first time from the later time.
35. A method comprising:
at a first time, receiving information about entertainment ambient content sensed by a microphone in a user' s portable device, said received information being accompanied by an identifier of the user or the user's device;
estimating, from the received information, an age or education of the user, and storing data related thereto; and
at a later time, selecting a specific version of content to be provided to the user, or the user's portable device, based on said stored data;
wherein the user's history of media consumption thereby serves as a proxy for information about age or education.
36. The method of claim 35 wherein said later time is at least an hour after the first time.
37. The method of claim 35 that includes estimating an age of the user, and at said later time, selecting said specific version of content based on said estimated age.
38. The method of claim 35 that includes estimating an education of the user, and at said later time, selecting said specific version of content based on said estimated education.
39. A method comprising the acts:
sending a request for a web page from a user' s device to a first web server;
responsive to said request, receiving first information including an ad tag URL;
sending data, including the ad tag URL, or a modified version of the ad tag URL, to a second web server; responsive to said sent data, receiving second information; and
presenting a display to the user based on the received first and second information; wherein the method further includes sensing physical context data about the user or the user's environment using a sensor in the user's device, and including information about the sensed physical context data with the data sent to the second web server.
40. The method of claim 39 that includes modifying the received ad tag URL based on the sensed physical context data, and sending said modified ad tag URL to the second web server.
41. A method including the acts:
using an apparatus in a bricks and mortar store operated by a retailer, discerning a set of item identifiers from items presented for purchase during a checkout operation by a shopper, the set of item identifiers serving as a fingerprint by which that checkout operation can be distinguished from other checkout operations, the shopper not directly providing shopper- identifying information to the retailer during said checkout operation;
transmitting information related to said fingerprint to a portable device conveyed by the shopper;
receiving data from the shopper's device, the received data relaying said information related to said fingerprint, and also including first information that serves as an identification of the shopper;
matching the information related to said fingerprint received from the shopper's device, with fingerprint information discerned by said apparatus; and
associating the first information - that serves as an identification of the shopper, with the set of item identifiers discerned by the apparatus, to thereby associate said purchased items with a particular shopper, and storing said association between said items and said first information in a database;
wherein purchased items are associated with a particular shopper, without the shopper directly providing shopper-identifying information to the retailer during the checkout operation.
42. The method of claim 41 in which said discerning comprises discerning an ordered set of item identifiers from items presented for purchase during a checkout operation by a shopper, the ordered set of item identifiers serving as said fingerprint by which that checkout operation can be distinguished from other checkout operations.
43. The method of claim 41 that includes providing a loyalty award to the shopper, based on a history of shopping with the retailer, as indicated by data in said database.
44. The method of claim 41 in which said transmitting comprises transmitting tones for sensing by a microphone in the shopper's portable device.
45. The method of claim 41 that further comprises employing information in said database about items or item brands selected by the shopper for purchase in the bricks and mortar store, to tailor advertising presented to the shopper during a shopper visit to a web site.
46. The method of claim 45 in which said web site is not otherwise affiliated with said retailer.
47. A method including the acts:
from a point of sale terminal in a bricks and mortar store operated by a retailer, emitting a signal for detection by a shopper' s portable device, each time an item presented for purchase in a checkout operation is sensed, the emitted signals defining a temporal sequence that serves as a fingerprint by which that checkout operation can be identified;
receiving data sent by the shopper' s device, the received data including first sequence information based on detection of said emitted signals by said device, and also including first information that serves as an identification of the shopper;
matching the first sequence information received from the shopper' s device with corresponding second sequence information generated as part of said checkout operation; and associating the first information - that serves as an indication of the shopper, with the second sequence information, to thereby associate the first information with a particular checkout operation; wherein the first information that serves as an identification of the shopper, is associated with a particular checkout operation, without the shopper directly providing shopper-identifying information to the retailer during the checkout operation.
48. A method comprising the acts:
compiling, in a data structure, two or more types of information from the group consisting of:
(a) a user's online activities, including web sites visited;
(b) entertainment content sensed by a microphone-equipped device conveyed by the user; and
(c) a record of items purchased by the user in a bricks and mortar store; and
selecting advertising for presentation to the user based on said two or more types of information.
49. The method of claim 48 that includes compiling all three of said types of information in said data structure, and selecting advertising for presentation to the user based on said three types of information.
50. A method comprising:
at a first time, from an apparatus worn on a wrist of a user, receiving sensor information; storing data related to said sensor information in a data structure remote from the user, in association with cookie data that serves as an identifier of the user; and
at a second time, accessing the stored data by reference to the cookie data, and using the accessed stored data in identifying information to present to the user.
51. The method of claim 50 that includes performing an activity recognition operation using said received sensor information, and storing activity recognition data in said data structure, in association with said cookie data.
52. The method of claim 50 in which the second time is more than an hour after the first time.
53. The method of claim 50 in which the information comprises advertising to be inserted in a web page for presentation to the user.
54. The method of claim 50 that further includes storing data about a web browsing history of said user in said data structure, and using the accessed stored data - including both said data related to sensor information, and said data about the user's web browsing history, in identifying information to present to the user.
PCT/US2013/073541 2012-12-07 2013-12-06 Physical context and cookies WO2014089417A2 (en)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US201261734763P 2012-12-07 2012-12-07
US61/734,763 2012-12-07
US201261738632P 2012-12-18 2012-12-18
US61/738,632 2012-12-18
US201361903559P 2013-11-13 2013-11-13
US61/903,559 2013-11-13

Publications (2)

Publication Number Publication Date
WO2014089417A2 true WO2014089417A2 (en) 2014-06-12
WO2014089417A3 WO2014089417A3 (en) 2014-07-31

Family

ID=50881978

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2013/073541 WO2014089417A2 (en) 2012-12-07 2013-12-06 Physical context and cookies

Country Status (2)

Country Link
US (2) US20140164111A1 (en)
WO (1) WO2014089417A2 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190130047A1 (en) * 2017-10-31 2019-05-02 Microsoft Technology Licensing, Llc Centralized client-side component for retrieving remote content items

Families Citing this family (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11599915B1 (en) * 2011-10-25 2023-03-07 Auddia Inc. Apparatus, system, and method for audio based browser cookies
US20150019334A1 (en) * 2013-07-09 2015-01-15 Infolinks, Inc. Systems and methods for providing targeted messaging when targeting terms are unavailable
US10776834B2 (en) 2013-07-15 2020-09-15 Criteo Sa Domain selection for advertisement data
US9554267B2 (en) * 2014-11-21 2017-01-24 Facebook, Inc. Techniques to associate user data with a mobile device
US9892478B2 (en) 2015-03-06 2018-02-13 Digimarc Corporation Digital watermarking applications
JP5931242B1 (en) * 2015-03-20 2016-06-08 ヤフー株式会社 Terminal device, information transmission method, and information transmission program
JP5931243B1 (en) * 2015-03-20 2016-06-08 ヤフー株式会社 Information processing device, terminal device, information processing method, and information processing program
EP3096282A1 (en) * 2015-05-21 2016-11-23 Tata Consultancy Services Limited Multi-dimensional sensor data based human behaviour determination system and method
US10045169B2 (en) * 2015-07-24 2018-08-07 Google Llc Systems and methods for personalizing public devices
US9836535B2 (en) * 2015-08-25 2017-12-05 TCL Research America Inc. Method and system for content retrieval based on rate-coverage optimization
US10564794B2 (en) * 2015-09-15 2020-02-18 Xerox Corporation Method and system for document management considering location, time and social context
US10360569B2 (en) * 2015-10-16 2019-07-23 International Business Machines Corporation Utilizing physical cookies to connect physical world experiences to digital world outcomes
US10193988B2 (en) * 2015-11-06 2019-01-29 Criteo Sa Setting a first-party user ID cookie on a web servers domain
US10592913B2 (en) 2015-12-14 2020-03-17 Google Llc Store visit data creation and management
US10872353B2 (en) * 2015-12-14 2020-12-22 Google Llc Providing content to store visitors without requiring proactive information sharing
US10193882B2 (en) 2016-06-12 2019-01-29 Criteo Sa Provision of cross-device identification
FR3054055B1 (en) * 2016-07-12 2019-09-06 Ingenico Group METHOD FOR PROCESSING AT LEAST ONE PAYMENT MEASUREMENT DATA, PAYMENT TERMINAL AND CORRESPONDING COMPUTER PROGRAM
US10769670B2 (en) 2016-08-17 2020-09-08 Criteo Sa Runtime matching of computing entities
US11184766B1 (en) * 2016-09-07 2021-11-23 Locurity Inc. Systems and methods for continuous authentication, identity assurance and access control
WO2018084851A1 (en) * 2016-11-04 2018-05-11 Google Llc Realtime busyness for places
US11080756B2 (en) 2016-11-13 2021-08-03 The Nielsen Company (Us), Llc Methods and apparatus to deliver targeted advertising
CN106792985B (en) * 2016-12-13 2017-12-12 北京邮电大学 A kind of mobile terminal access point Forecasting Methodology and device
US11328308B1 (en) * 2017-05-31 2022-05-10 Juniper Networks, Inc Systems and methods for matching and delivering account-level data from multiple customer-facing platforms based on contact-level touchpoints
US11170408B2 (en) * 2017-07-28 2021-11-09 Ncr Corporation Geofenced selection with targeted interaction
CN107730250A (en) * 2017-09-19 2018-02-23 维沃移动通信有限公司 A kind of method and mobile terminal for showing payment information
US10554616B1 (en) 2017-12-08 2020-02-04 Criteo S.A. Generating mobile device-specific identifiers across native mobile applications and mobile browsers
US20190206102A1 (en) * 2017-12-29 2019-07-04 Facebook, Inc. Systems and methods for enhancing content
US10902461B2 (en) 2018-08-03 2021-01-26 International Business Machines Corporation Environmental modification using tone model analysis
RU2766134C2 (en) 2020-02-26 2022-02-08 Акционерное общество "Лаборатория Касперского" Method of anonymously sending data from a user device

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090070797A1 (en) * 2006-03-31 2009-03-12 Arun Ramaswamy Methods, systems, and apparatus for multi-purpose metering
US20110161076A1 (en) * 2009-12-31 2011-06-30 Davis Bruce L Intuitive Computing Methods and Systems
US20110212717A1 (en) * 2008-08-19 2011-09-01 Rhoads Geoffrey B Methods and Systems for Content Processing
US20110283306A1 (en) * 2009-02-12 2011-11-17 Davis Bruce L Media Processing Methods and Arrangements
US20120158491A1 (en) * 2000-05-18 2012-06-21 Goulden David L Techniques for displaying impressions in documents delivered over a computer network
US20120198328A1 (en) * 2007-02-06 2012-08-02 5O9, Inc. Contextual data communication platform
US20120212593A1 (en) * 2011-02-17 2012-08-23 Orcam Technologies Ltd. User wearable visual assistance system

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8516533B2 (en) * 2008-11-07 2013-08-20 Digimarc Corporation Second screen methods and arrangements
US10180572B2 (en) * 2010-02-28 2019-01-15 Microsoft Technology Licensing, Llc AR glasses with event and user action control of external applications

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120158491A1 (en) * 2000-05-18 2012-06-21 Goulden David L Techniques for displaying impressions in documents delivered over a computer network
US20090070797A1 (en) * 2006-03-31 2009-03-12 Arun Ramaswamy Methods, systems, and apparatus for multi-purpose metering
US20120198328A1 (en) * 2007-02-06 2012-08-02 5O9, Inc. Contextual data communication platform
US20110212717A1 (en) * 2008-08-19 2011-09-01 Rhoads Geoffrey B Methods and Systems for Content Processing
US20110283306A1 (en) * 2009-02-12 2011-11-17 Davis Bruce L Media Processing Methods and Arrangements
US20110161076A1 (en) * 2009-12-31 2011-06-30 Davis Bruce L Intuitive Computing Methods and Systems
US20120212593A1 (en) * 2011-02-17 2012-08-23 Orcam Technologies Ltd. User wearable visual assistance system

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190130047A1 (en) * 2017-10-31 2019-05-02 Microsoft Technology Licensing, Llc Centralized client-side component for retrieving remote content items

Also Published As

Publication number Publication date
WO2014089417A3 (en) 2014-07-31
US20170243246A1 (en) 2017-08-24
US20140164111A1 (en) 2014-06-12

Similar Documents

Publication Publication Date Title
WO2014089417A2 (en) Physical context and cookies
KR101525417B1 (en) Identifying a same user of multiple communication devices based on web page visits, application usage, location, or route
US9535577B2 (en) Apparatus, method, and computer program product for synchronizing interactive content with multimedia
US20180197198A1 (en) Social network system and method
US9402099B2 (en) Arrangements employing content identification and/or distribution identification data
US9223893B2 (en) Updating social graph data using physical objects identified from images captured by smartphone
US20170185596A1 (en) Trigger-based content presentation
US20180033045A1 (en) Method and system for personalized advertising
US20170286999A1 (en) Personal device-enabled lifestyle, commerce and exchange tracking system
US20130191394A1 (en) System and method for dynamically forming user groups
US20220277356A1 (en) Systems and methods for associating advertisers and content creators
US20130262233A1 (en) Apparatuses, methods and systems for generating 3d content based on detected mobile device capability
US10366419B2 (en) Enhanced digital media platform with user control of application data thereon
JP2008530632A (en) System and method for interactive marketing
Karpischek et al. my2cents—Digitizing consumer opinions and comments about retail products
US20130246182A1 (en) Apparatuses, methods and systems for acquiring information from objects in a virtual space
US20160189226A1 (en) Method and system for enabling skipping advertisements for a content
Stokes eMarketing-The Essential Guide to Marketing in a Digital World (Stokes)
US20180204246A1 (en) Social network system and method
TWM583181U (en) Short message service marketing system
Karpischek et al. Digitizing consumer opinions and comments about retail products

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 13859802

Country of ref document: EP

Kind code of ref document: A2

122 Ep: pct application non-entry in european phase

Ref document number: 13859802

Country of ref document: EP

Kind code of ref document: A2