943 stories

Introducing Datashades.info, a CKAN Community Service

1 Share

Do you use CKAN to power an open data portal? In this guest post Link Digital explains how you can take advantage of their latest open data initiative Datashades.info.

Datashades.info is a tool designed to deliver insights for researchers, portal managers, and the wider tech community to inform and support open data efforts relating to data hosted on CKAN platforms.

Link Digital created the online service through a number of alpha releases and considers datashades.info, now in beta, as a long term initiative they expect to improve with more features in future releases.

Specifically, Datashades.info provides a publicly-accessible index of metadata and statistics on CKAN data portals across the globe. For each portal, a number of statistics are aggregated and presented surrounding number of datasets, users, organisations and dataset tags. These statistics give portal managers the ability to quickly compare the size and scope of CKAN data portals to help inform their development roadmaps. Moreover, for each portal, installed plugin information is collected along with the relative penetration of those plugins across all portals in the index. This will enable CKAN developers to quickly see what extensions are the most popular and on what portals they are being used. Finally, all historical data is persisted and kept publically accessible, allowing researchers to analyse historical data trends in any indexed CKAN portal.

Datashades.info was built to support a crowd-sourced indexing scheme. If a visitor searches for a CKAN portal and it is not found within the index, the system will immediately query that portal and attempt to generate a new index entry on-the-fly. Aggregation of a new portal’s statistics into Datashades.info also happens automatically.

Maximise the tool and gain interesting information with the following features:

Globally Accessible open data

With Datashades.info, you can easily access an index of metadata and statistics on CKAN data portals across the globe. To do this, simply type in the portal’s URL on the homepage then click “Search“.

Integrated Values of All Metrics

After entering a portal’s URL, Datashades.info will load its information. After a few seconds, you will be able to see a range of data on portal users, datasets, resources, organisations, tags and plugins. Portal managers can access these via the individual portal page found on the site.

Easily-tracked Historical Data

Want to revisit data you previously explored? The tool also keeps old data in a historical index which users can explore any time on any portal page or by clicking “View All Data Portals” on the homepage.


Datashades.info uses crowdsourcing to build its index. This means users can easily add any CKAN data portal not found on the site. To do this, simply search for a portal you know and it’ll be automatically added to the site and global statistics.

As the project remains at a beta level of maturity, it is still wanting of improvements in many areas. But with the continuous feedback coming from the CKAN community, expect that more data and features will be added in future releases. For now, have a look around and stay tuned!


Read the whole story
22 days ago
Share this story

What Do you Mean You Write Code EVERY DAY?

1 Share

Every so often, I ask folk in the department when they last wrote any code; often, I get blank stares back. Write code? Why would they want to do that? Code is for the teaching of, and big software engineering projects, and, and, not using it every day, surely?

I disagree.

I see code as a tool for making tools, often disposable ones.

Here’s an example…

I’m writing a blog post, and I want to list the file types recognised by Jupytext. I can’t find a list of the filetypes it recognises as a simple string that I can copy and paste into the post, but I do find this:

Copying out those suffixes is a pain, so I just copy that text string, which in this case happens to play nicely with Python (because it is Python), sprinkle a bit of code:

and here’s the list of filetypes supported by Jupytext: .py, .R, .r, .jl, .cpp, .ss, .clj, .scm, .sh, .q, .m, .pro, .js, .ts, .scala.

Note that is doesn’t have to be nice code, and there may be multiple ways of solving the problem (in the example, I use a hybrid “me + the computer” approach where I get the code to do one thing, I copy the output, paste that into the next cell and then hack code around that, as well as “just the computer” approach. The first one is perhaps more available to a novice, the second to someone who knows about .join()).

So what?

I tend use code without thinking anything special of it; it’s just a tool that’s to hand to fashion other tools from, and I think that colours my attitude towards the way in which we teach it.

First and foremost, if you come out of a coding course not thinking that you now have a skill you can use quite casually to help get stuff done, you’ve been mis-sold…

This blog post took much longer to write than it took me to copy the _SCRIPT_EXTENSIONS text and write the code to extract the list of suffixes… And it didn’t take long to write the post at all…

See also: Fragment – Programming Privilege.

Read the whole story
58 days ago
Share this story

Exploring Jupytext – Creating Simple Python Modules Via a Notebook UI

1 Share

Although I spend a lot of my coding time in Jupyter notebooks, there are several practical problems associated with working in that environment.

One problem is that under version control, it can be hard to tell what’s changed. On the one hand, the notebook .ipynb format, which saves as a serialised JSON object, is hard to read cleanly:

The .ipynb format also records changes to cell execution state, including cell execution count numbers and changes to cell outputs (which may take the form of large encoded strings when a cell output is an image, or chart, for example:

Another issue arises when trying to write modules in a notebook that can be loaded into other notebooks.

One workaround for this is to use the notebook loading hack described in the official docs: Importing notebooks. This requires loading in a notebook loading module that then allows you to import other modules. Once the notebook loader module is installed, you can run things like:

  • import mycode as mc to load mycode.ipynb
  • `moc = __import__(“My Other Code”)` to load code in from `My Other Code.ipynb`

If you want to include code that can run in the notebook, but that is not executed when the notebook is loaded as a module, you can guard items in the notebook:

In this case, the if __name__=='__main__': guard will run the code in the code cell when run in the notebook UI, but will not run it when the notebook is loaded as a module.

Guarding code can get very messy very quickly, so is there an easier way?

And is there an easier way of using notebooks more generally as an environment for creating code+documentation files that better meet the needs of a variety of users? For example, I note this quote from Daniele Procida recently shared by Simon Willison:

Documentation needs to include and be structured around its four different functions: tutorials, how-to guides, explanation and technical reference. Each of them requires a distinct mode of writing. People working with software need these four different kinds of documentation at different times, in different circumstances—so software usually needs them all.

This suggests a range of different documentation styles for different purposes, although I wonder if that is strictly necessary?

When I am hacking code together, I find that I start out by writing things a line at a time, checking the output for each line, then grouping lines in a single cell and checking the output, then wrapping things in a function (for example of this in practice, see Programming in Jupyter Notebooks, via the Heavy Metal Umlaut). I also try to write markdown notes that set up what I intend to do (and why) in the following code cells. This means my development notebooks tell a story (of a sort) of the development of the functions that hopefully do what I actually want them to by the end of the notebook.

If truth be told, the notebooks often end up as an unholy mess, particularly if they are full of guard statements that try to separate out development and testing code from useful code blocks that I might want to import elsewhere.

Although I’ve been watching it for months, I’ve only started exploring how to use Jupytext in practice quite recently, and already it’s starting to change how I use notebooks.

If you install jupytext, you will find that if you click on a link to a markdown (.md)) or Python (.py), or a whole range of other text document types (.py, .R, .r, .Rmd, .jl, .cpp, .ss, .clj, .scm, .sh, .q, .m, .pro, .js, .ts, .scala), you will open the file in a notebook environment.

You can also open the file as a .py file, from the notebook listing menu by selecting the notebook:

and then using the Edit button to open it:

at which point you are presented with the “normal” text file editor:

One thing to note about the notebook editor view over the notebook is that you can also include markdown cells, as you might in any other notebook, and run code cells to preview their output inline within the notebook view.

However, whilst the markdown code will be saved into the Python file (as commented out code), the code outputs will not be saved into the Python file.

If you do want to be able to save notebook views with any associated code output, you can configure Jupytext to “pair” .py and .ipynb files (and other combinations, such as .py, .ipynb and .md files) such that when you save an open .py or .ipynb file from the notebook editing environment, a “paired” .ipynb or .py version of the file is also saved at the same time.

This means I could click to open my .py file in the notebook UI, run it, then when I save it, a “simple” .py file containing just code and commented out markdown is saved along with a notebook .ipynb file that also contains the code cell outputs.

You can configure Jupytext so that the pairing only works in particular directories. I’ve started trying to explore various settings in the branches of this repo: ouseful-template-repos/jupytext-md. You can also convert files on the command line; for example, <span class="s1">jupytext --to py Required\ Pace.ipynb will convert a notebook file to a python file.

The ability to edit Python / .py files, or code containing markdown / .md files in a notebook UI, is really handy, but there’s more…

Remember the guards?

If I tag a code cell using the notebook UI (from the notebook View menu, select Cell Toolbar and then Tags, you can tag a cell with a tag of the form active-ipynb:

See the Jupytext docs: importing Jupyter notebooks as modules for more…

The tags are saved as metadata in all document types. For example, in an .md version of the notebook, the metadata is passed in an attribute-value pair when defining the language type of a code block:

In a .py version of the notebook, however, the tagged code cell is not rendered as a code cell, it is commented out:

What this means is that I can tag cells in the notebook editor to include them — or not — as executable code in particular document types.

For example, if I pair .ipynb and .py files, whenever I edit either an .ipynb or .py file in the notebook UI, it also gets saved as the paired document type. Within the notebook UI, I can execute all the code cells, but through using tagged cells, I can define some cells as executable in one saved document type (.ipynb for example) but not in another (a .py file, perhaps).

What that in turn means is that when I am hacking around with the document in the notebook UI I can create documents that include all manner of scraggy developmental test code, but only save certain cells as executable code into the associated .py module file.

The module workflow is now:

  • install Jupytext;
  • edit Python files in a notebook environment;
  • run all cells when running in the notebook UI;
  • mark development code as active-ipynb, which is to say, it is *not active* in a .py file;
  • load the .py file in as a module into other modules or notebooks but leaving out the commented out the development code; if I use `%load_ext autoreload` and `%autoreload 2` magic in the document that’s loading the modules, it will [automatically reload them](https://stackoverflow.com/a/5399339/454773) when I call functions imported from them if I’ve made changes to the associated module file;
  • optionally pair the .py file with an .ipynb file, in which case the .ipynb file will be saved: a) with *all* cells run; b) include cell outputs.

Referring back to Daniele Procida’s insights about documentation, this ability to have code in a single document (for example, a .py file) that is executable in one environment (the notebook editing / development environment, for example) but not another (when loaded as a .py module) means we can start to write richer source code files.

I also wonder if this provides us with a way of bundling test code as part of the code development narrative? (I don’t use tests so don’t really know how the workflow goes…)

More general is the insight that we can use Jupytext to automatically generate distinct versions of a document from a single source document. The generated documents:

  • can include code outputs;
  • can *exclude* code outputs;
  • can have tagged code commented out in some document formats and not others.

I’m not sure if we can also use it in combination with other notebook extensions to hide particular cells, for example, when viewing documents in the notebook editor or generating export document formats from an executed notebook form of it. A good example to try out might be the hide_code extension, which provides a range of toolbar options that can be used to customise the display of a document in a the notebook editor or HTML / PDF documents generated from it.

It could also be useful to have a very simple extension that lets you click a toolbar button to set an active- state tag and style or highlight that cell in the notebook UI to mark it out as having limited execution status. A simple fork of, or extension to, the freeze extension would probably do that. (I note that Jupytext responds to the “frozen” freeze setting but that presumably locks out executing the cell in the notebook UI too?)

PS a few weeks ago, Jupytext creator Marc Wouts posted this handy recipe for *rewriting* notebook commits made to a git branch against markdown formatted documents rather than the original ipynb change commits: git filter-branch --tree-filter 'jupytext --to md */*.ipynb && rm -f */*.ipynb' HEAD This means that if you have a legacy project with commits made to notebook files, you can rewrite it as a series of changes made to markdown or Python document versions of the notebooks…

Read the whole story
58 days ago
Share this story

Manhattan DA Made Google Give Up Information on Everyone in Area as They Hunted for Antifa

2 Comments and 10 Shares

When Gavin McInnes—founder of the violent, far-right group The Proud Boys—spoke to a Manhattan Republican club last October, the neighborhood response was less than welcoming. Protesters took to the normally sedate Upper East Side block with chants and spray paint. The Proud Boys responded with fists and kicks. Nearly a year later, as the assault and riot charges against four Proud Boys go to trial, prosecutors revealed that they had turned to an alarming new surveillance tool in this case: a reverse search warrant.

The Manhattan District Attorney's Office admitted it demanded Google hand over account information for all devices used in parts of the Upper East Side. They didn’t do this to find the Proud Boys; they did it to find Antifa members.

Reverse search warrants have been used in other parts of the country, but this is the first time one was disclosed in New York. Unlike a traditional warrant, where law enforcement officials request information on a specific phone or individual, reverse warrants allow law enforcement to target an entire neighborhood. Police and prosecutors create a “geofence”—a map area—and demand information on anyone standing in the zone. This flips the logic of search warrants on its head. Rather than telling service providers the name or phone number of a suspect, reverse search warrants start with the location and work backwards.

It’s a big change. Depending on the size and location of the geofence, a reverse search warrant can easily target hundreds or even thousands of bystanders. That scale is what makes reverse search warrants so enticing to law enforcement and so concerning to civil liberties groups. One concern is that the more broadly law enforcement uses surveillance, the higher the risk for “false discovery.” That’s a clinical way to say that the more people you spy on, the more innocent people will wrongly go to jail.

The phenomenon is well-documented in the sciences, where researchers have long known that “high false discovery rates occur when many outcomes of a single intervention are tested.” Essentially, when you look for too many patterns at the same time, you increase the danger that the data will fool you. When police officers request the data for hundreds or even thousands of devices, there’s a higher chance that they’ll wrongly think that one of those bystanders is a suspect.

This isn’t just theoretical. That’s what Jorge Molina discovered in 2018, when Arizona detectives wrongly arrested him for a brutal murder, jailing him for nearly a week before he was exonerated. Officers demanded that Google hand over information on every single laptop, phone, tablet, and smart device in a two-block area. We don’t know how many accounts that includes, but it’s no surprise that while sifting through that many devices that they quickly found a “match.” Only he was innocent.

In response to the Manhattan DA’s reverse search warrant, Google provided information that investigators used—along with images given to a private facial recognition company—to target two people who turned out to be innocent bystanders. Thankfully, unlike in Molina’s case, the two “matches” in Manhattan were never arrested—and the Antifa members have not been identified, even as several Proud Boys have stood trial.

But with the seal broken now in Manhattan, there are likely to be more geofence warrants and more false discoveries. While a judge needs to sign off on a reverse warrant, that formality provides little protection to the public. A traditional warrant application asks for information about the individual being targeted and the reasons they are suspected. With reverse warrants, judges don’t even know how many people’s data will be compromised. They simply don’t have enough information to do their job.

It’s also unclear how judges will evaluate reverse warrants around sensitive sites: political protests, houses of worship or medical facilities, among others. The practice is even more alarming when you consider the ways that ICE and other federal agencies could use a reverse warrant to pursue their deportation campaigns and target American immigrants.

None of this is to say that reverse search warrants are unique, they are just the latest example of how the surveillance capitalism that powers tech firms can become a tool for the government. Maybe some users who happily hand their data to the tech giants will second guess that choice when they realize how quickly their digital sidekicks can morph into a big brother.

Albert Fox Cahn is the executive director of The Surveillance Technology Oversight Project at the Urban Justice Center, a New York-based civil rights and privacy organization. On Twitter @cahnlawny. 

Read the whole story
58 days ago
62 days ago
Share this story
2 public comments
62 days ago
this is the scary stuff
Brooklyn, New York
38 days ago
https://keramatzade.com/Earn-wealth-with-amazing-business-ideals https://keramatzade.com/Law-of-Attraction-of-Wealth https://keramatzade.com/Ways-to-make-money https://modirebimeh.ir/online-calculation-of-iranian-life-insurance/ https://modirebimeh.ir/engineers-professional-liability-insurance/ https://modirebimeh.ir/third-party-insurance-calculation/ https://modirebimeh.ir/iran-liability-insurance-have-you-not-yet-insured-your-business-with-iran-liability-insurance/ https://modirebimeh.ir/iran-life-insurance-ganji-for-the-future-of-children-and-families/ https://modirebimeh.ir/iran-car-body-insurance-the-best-and-most-prestigious-in-the-iranian-insurance-industry/ https://modirebimeh.ir/the-most-reliable-and-unrivaled-third-party-car-insurance-in-iran/ https://keramatzade.com/14-ways-to-increase-revenue https://keramatzade.com/8-ways-to-increase-revenue https://keramatzade.com/25-jobs-with-which-you-can-earn-up-to-a-million-dollars https://keramatzade.com/success-secret-1 https://keramatzade.com/Make-Money-Online-Effective-step-by-step-money-making-techniques https://keramatzade.com/Make-money-at-home https://keramatzade.com/Ways-to-make-money-without-capital https://keramatzade.com/Creative-Money-Making-Ideas https://keramatzade.com/The-law-of-attracting-money https://keramatzade.com/Ways-to-Make-Money-at-Home https://keramatzade.com/Immediate-absorption-of-wealth-in-10-minutes-and-attractive-ways-to-get-rich https://keramatzade.com/The-secret-of-attracting-money-in-Iran-to-achieve-creative-money-maker-ideas https://keramatzade.com/Ways-to-get-rich-in-Iran-with-the-most-wonderful-business-ideas https://keramatzade.com/Astonishing-economic-intelligence-test-to-increase-financial-intelligence
63 days ago

SAMBA versus SMB: Adversarial Interoperability is Judo for Network Effects

1 Share

Before there was Big Tech, there was "adversarial interoperability": when someone decides to compete with a dominant company by creating a product or service that "interoperates" (works with) its offerings.

In tech, "network effects" can be a powerful force to maintain market dominance: if everyone is using Facebook, then your Facebook replacement doesn't just have to be better than Facebook, it has to be so much better than Facebook that it's worth using, even though all the people you want to talk to are still on Facebook. That's a tall order.

Adversarial interoperability is judo for network effects, using incumbents' dominance against them. To see how that works, let's look at a historical example of adversarial interoperability role in helping to unseat a monopolist's dominance.

The first skirmishes of the PC wars were fought with incompatible file formats and even data-storage formats: Apple users couldn't open files made by Microsoft users, and vice-versa. Even when file formats were (more or less) harmonized, there was still the problems of storage media: the SCSI drive you plugged into your Mac needed a special add-on and flaky driver software to work on your Windows machine; the ZIP cartridge you formatted for your PC wouldn't play nice with Macs.

But as office networking spread, the battle moved to a new front: networking compatibility. AppleTalk, Apple's proprietary protocol for connecting up Macs and networked devices like printers, pretty much Just Worked, providing you were using a Mac. If you were using a Windows PC, you had to install special, buggy, unreliable software.

And for Apple users hoping to fit in at Windows shops, the problems were even worse: Windows machines used the SMB protocol for file-sharing and printers, and Microsoft's support for MacOS was patchy at best, nonexistent at worst, and costly besides. Businesses sorted themselves into Mac-only and PC-only silos, and if a Mac shop needed a PC (for the accounting software, say), it was often cheaper and easier just to get the accountant their own printer and backup tape-drive, rather than try to get that PC to talk to the network. Likewise, all PC-shops with a single graphic designer on a Mac—that person would often live offline, disconnected from the office network, tethered to their own printer, with their own stack of Mac-formatted ZIP cartridges or CD-ROMs.

All that started to change in 1993: that was the year that an Australian PhD candidate named Andrew Tridgell licensed his SAMBA package as free/open source software and exposed it to the wide community of developers looking to connect their non-Microsoft computers—Unix and GNU/Linux servers, MacOS workstations—to the dominant Microsoft LANs.

SAMBA was created by using a "packet sniffer" to ingest raw SMB packets as they traversed a local network; these intercepted packets gave Tridgell the insight he needed to reverse-engineer Microsoft's proprietary networking protocol. Tridgell prioritized compatibility with LAN Manager, a proprietary Network Operating System that enterprise networks made heavy use of. If SAMBA could be made to work in LAN Manager networks, then you could connect a Mac to a PC network—or vice-versa—and add some Unix servers and use a mix of SAMBA and SMB to get them all to play nice with one another.

The timing of Tridgell's invention was crucial: in 1993, Microsoft had just weathered the Federal Trade Commission’s antitrust investigation of its monopoly tactics, squeaking through thanks to a 2-2 deadlock among the commissioners, and was facing down a monopoly investigation by the Department of Justice.

The growth of local-area networks greatly accelerated Microsoft's dominance. It's one thing to dominate the desktop, another entirely to leverage that dominance so that no one else can make an operating system that connects to networks that include computers running that dominant system. Network administrators of the day were ready to throw in the towel and go all-Microsoft for everything from design workstations to servers.

SAMBA changed all that. What's more, as Microsoft updated SMB, SAMBA matched them, relying on a growing cadre of software authors who relied on SAMBA to keep their own networks running.

The emergence of SAMBA in the period when Microsoft's dominance was at its peak, the same year that the US government tried and failed to address that dominance, was one of the most salutary bits of timing in computing history, carving out a new niche for Microsoft's operating system rivals that gave them space to breathe and grow. It's certainly possible that without SAMBA, Microsoft could have leveraged its operating system, LAN and application dominance to crush all rivals.

So What Happened?

We don't see a lot of SAMBA-style stories anymore, despite increased concentration of various sectors of the tech market and a world crying out for adversarial interoperability judo throws.

Indeed, investors seem to have lost their appetite for funding companies that might disrupt the spectacularly profitable Internet monopolists of 2019, ceding them those margins and deeming their territory to be a "kill zone."

VCs have not lost their appetite for making money, and toolsmiths have not lost the urge to puncture the supposedly airtight bubbles around the Big Tech incumbents, so why is it so hard to find a modern David with the stomach to face off against 2019's Goliaths?

To find the answer, look to the law. As monopolists have conquered more and more of the digital realm, they have invested some of those supernormal profits in law and policy that lets them fend off adversarial interoperators.

One legal weapon is "Terms of Service": both Facebook and Blizzard have secured judgments giving their fine print the force of law, and now tech giants use clickthrough agreements that amount to, "By clicking here, you promise that you won't try to adversarially interoperate with us."

A modern SAMBA project would have to contend with this liability, and Microsoft would argue that anyone who took the step of installing SMB had already agreed that they wouldn't try to reverse-engineer it to make a compatible product.

Then there's "anti-circumvention," a feature of 1998's Digital Millennium Copyright Act (DMCA). Under Section 1201 of the DMCA, bypassing a "copyright access control" can put you in both criminal and civil jeopardy, regardless of whether there's any copyright infringement. DMCA 1201 was originally used to stop companies from making region-free DVD players or modding game consoles to play unofficial games (neither of which is a copyright violation!).

But today, DMCA 1201 is used to control competitors, critics, and customers. Any device with software in it contains a "copyrighted work," so manufacturers need only set up an "access control" and they can exert legal control over all kinds of uses of the product.

Their customers can only use the product in ways that don't involve bypassing the "access control," and that can be used to force you to buy only one brand of ink or use apps from only one app store.

Their critics—security researchers auditing their cybersecurity—can't publish proof-of-concept to back up their claims about vulnerabilities in the systems.

And competitors can't bypass access controls to make compatible products: third party app stores, compatible inks, or a feature-for-feature duplicate of a dominant company's networking protocol.

Someone attempting to replicate the SAMBA creation feat in 2019 would likely come up against an access control that needed to be bypassed in order to peer inside the protocol's encrypted outer layer in order to create a feature-compatible tool to use in competing products.

Another thing that's changed (for the worse) since 1993 is the proliferation of software patents. Software patenting went into high gear around 1994 and consistently gained speed until 2014, when Alice v. CLS Bank put the brakes on (today, Alice is under threat). After decades of low-quality patents issuing from the US Patent and Trademark Office, there are so many trivial, obvious and overlapping software patents in play that anyone trying to make a SAMBA-like product would run a real risk of being threatened with expensive litigation for patent infringement.

This thicket of legal anti-adversarial-interoperability dangers has been a driver of market concentration, and the beneficiaries of market concentration have also spent lavishly to expand and strengthen the thicket. It's gotten so bad that even some "open standards organizations" have standardized easy-to-use ways of legally prohibiting adversarial interoperability, locking in the dominance of the largest browser vendors.

The idea that wildly profitable businesses would be viewed as unassailable threats by investors and entrepreneurs (rather than as irresistible targets) tells you everything you need to know about the state of competition today. As we look to cut the Big Tech giants down to size, let's not forget that tech once thronged with Davids eager to do battle with Goliaths, and that this throng would be ours to command again, if only we would re-arm it.

Read the whole story
79 days ago
Share this story

Adblocking: How About Nah?

1 Share

For more than a decade, consumer rights groups (including EFF) worked with technologists and companies to try to standardize Do Not Track, a flag that browsers could send to online companies signaling that their users did not want their browsing activity tracked. Despite long hours and backing from the FTC, foot-dragging from the browser vendors and outright hostility from the big online media companies mean that setting Do Not Track in your browser does virtually nothing to protect your privacy.

Do Not Track grew out of widespread public concern over invasive "behavioral advertising" that relied on tracking to target ads; despite a generation of promises from the ad industry that consumers would welcome more relevant advertising, the consistent result has been that users are freaked out by "relevant" ads because they understand that relevancy is synonymous with privacy invasion. Nothing is so creepy as ads for a product you looked into earlier following you from site to site, then from app to app, as you are tracked and retargeted by a desperate vendor's algorithm.

Internet users didn't take this situation lying down. They wanted to use the Web, but not be tracked, and so they started to install ad-blockers. A lot of ad-blockers, and more every year.

Ad-blockers don't just stop users from seeing ads and being tracked (and indeed, some ad-blockers actually track users!). They can also stop the publishers and marketers who rely on tracking and ad-clicks from earning money. Predictably, industry responded with ad-blocker-blockers, which prevented users from seeing their sites unless they turned off their ad-blocker.

You'll never guess what happened next.

Actually, it's obvious what happened next: users started to install ad-blocker-blocker-blockers.

The Biggest Boycott in History

The rise and rise of ad-blockers (and ad-blocker-blocker-blockers) is without parallel: 26% of Internet users are now blocking ads, and the figure is rising. It’s been called the biggest boycott in human history.

It's also something we've seen before, in the earliest days of the Web, when pop-up ads ruled the world (wide web), and users went to war against them.

In 1994, Hotwired (the defunct online adjunct to Wired magazine) displayed the first banner ad in Internet history. Forty-four percent of the people who saw that ad clicked on it. At the time, it felt like advertising had taken a great leap, attaining a conversion rate that bested print, TV, direct mail, or display advertising by orders of magnitude.

But it turned out that the click-rate on that Hotwired ad had more to do with novelty than any enduring persuasive properties of banner ads. Even as Web companies were raising millions based on the fabulous performance of early ads, the efficacy of those ads was falling off a cliff, with clickthrough rates plummeting to low single digits.

This created a desperate situation, where publishers needed to do something -- anything -- to goose their clickthrough rates.

Enter the Pop-Up Ad

That's when Ethan Zuckermanthen an employee at Tripodinvented the pop-up ad (he has since apologized). These ads spawned in new windows and were much harder to ignorefor a while. Human beings' response to stimulus tends to regress to the mean (the refrigerator hum gets quieter over time because you adapt to it, not because the decibel level decreases) and so pop-up ads evolved into ever-more virulent formspop-under ads, pop-ups with fake "close" boxes, pop-up ads that respawned, pop-up ads that ran away from your mouse when you tried to close them...

At the height of the pop-up wars, it seemed like there was no end in sight: the future of the Web would be one where humans adapted to pop-ups, then pop-ups found new, obnoxious ways to command humans' attention, which would wane, until pop-ups got even more obnoxious.

But that's not how it happened. Instead, browser vendors (beginning with Opera) started to ship on-by-default pop-up blockers. What's more, userswho hated pop-up adsstarted to choose browsers that blocked pop-ups, marginalizing holdouts like Microsoft's Internet Explorer, until they, too, added pop-up blockers.

Chances are, those blockers are in your browser today. But here's a funny thing: if you turn them off, you won't see a million pop-up ads that have been lurking unseen for all these years.

Because once pop-up ads became invisible by default to an ever-larger swathe of Internet users, advertisers stopped demanding that publishers serve pop-up ads. The point of pop-ups was to get people's attention, but something that is never seen in the first place can't possibly do that.

How About Nah?

The Internet is full of take-it-or-leave-it offers: click-through and click-wrap agreements that you can either click "I agree" on or walk away from.

As the online world has grown more concentrated, with more and more power in fewer and fewer hands, it's become increasingly difficult for Web publishers to resist advertisers' insistence on obnoxious tracking ads.

But Internet users have never been willing to accept take-it-or-leave-it as the last word in technological self-determination. Adblockers are the new pop-up blockers, a way for users to do what publishers can't or won't do: demand a better deal from advertisers. When you visit a site, the deal on offer is, "Let us and everyone we do business with track you in every way possible or get lost" and users who install adblockers push back. An adblocker is a way of replying to advertisers and publishers with a loud-and-clear "How about nah?"

Adversarial Interoperability

Adversarial interoperability occurs when someone figures out how to plug a new product or service into an existing product or service, against the wishes of the company behind that existing product or service.

Adblocking is one of the most successful examples of adversarial interoperability in modern history, along with third-party printer ink. When you visit a website, the server sends your browser a bunch of material, including the code to fetch and render ads. Adblockers throw away the ad parts and show you the rest, while ad-blocker-blocker-blockers do the same, and then engage in an elaborate technological game of cat-and-mouse in a bid to fool the server into thinking that you are seeing the ads, while still suppressing them.

Browsers have always been playgrounds for adversarial interoperability, from the pop-up wars to the browser wars. Thanks to open standards and a mutual disarmament rule for software patents among browser vendors, it's very hard to use the law to punish toolsmiths who make adblocking technologies (not that that's stopped people from attempting it).

Adversarial interoperability is often a way for scrappy new upstarts to challenge the established playerslike the company that got sued by IBM's printer division for making its own toner cartridges and grew so big it now owns that printer division (!).

But adversarial interoperability is also a way for the public to assert its rights and push back against unfair practices. Take-it-or-leave it deals are one thing when the market is competitive and you can shop around for someone with better terms of service, but in highly concentrated markets where everyone has the same rotten deal on offer, adversarial interoperability lets users make a counteroffer: "How about nah?"

But for How Long?

Concentration in the tech industry—including the “vertical integration” of browsers, advertising networks, and video content under one corporate umbrella—has compromised the Internet's openness. In 2017, the World Wide Web Consortium published its first-ever "standard" that could not be fully implemented without permission from the giant tech and media companies (who have since refused that permission to anyone who rocks the boat). In publishing that standard, the W3C explicitly rejected a proposal to protect adversarial interoperability by extracting legally binding nonaggression promises from the companies that make up the consortium.

The standard the W3C publishedEncrypted Media Extensions (EME), for restricting playback of videocomes with many dangers for would-be adversarial interoperators, notably the risk of being sued under Section 1201 of the Digital Millennium Copyright Act, which bans tampering with “access controls” on copyrighted works and holds out both criminal and civil liability for toolsmiths who traffic in programs that let you change the rules embodied by EME.

One driving force behind the adoption of EME was the ever-tighter integration between major browser vendors like Google, video distributors, and advertising networks. This created a lopsided power-dynamic that ultimately ended up in the standardization of a means of undoing the configurable Webwhere the user is king. EME is the first crack in the wall that protected browsers from those who would thwart adversarial operability and take "how about nah?" off the table, leaving us with the kind of take-it-or-leave-it Web that the marketing industry has been striving for since the first pop-up ad.

Read the whole story
79 days ago
Share this story
Next Page of Stories