Small ISPs use "malicious" DNS servers to watch Web searches, earn cash

Nearly 2 percent of all US Internet users suffer from "malicious" domain name system (DNS) servers that don't properly turn website names like google.com into the IP addresses computers need to communicate on the 'Net. And, to make matters worse, the problem isn't caused by hackers or malware, but by the local ISPs people pay for access to the Internet.

Though the 2 percent number might sound low, it's astonishingly high for a core Internet function, as is clear from the fact that no other country—apart from Haiti—sees more than 0.17 percent malicious DNS servers. What has gone wrong in America?

According to researchers from Microsoft and from the Polytechnic Institute of NYU, the malicious DNS servers exist to make a little extra cash for Internet providers. A detailed experiment (PDF) carried out between September 1 and October 31 last year found that most of these DNS servers stealthily intercepted and redirected search queries and URL mistakes, but only when these were entered from a Web browser's address bar. Go to Bing.com and everything works as it should; search Bing through a browser address bar and you might be surprised at the results.

What commonly happens is that specific search queries (usually for brand names) made from an address bar no longer return the expected Web search results page from Bing or Google or Yahoo. If your ISP has such DNS servers configured, and your computer points to them (most ISP subscribers will by default), typing "Apple" into a browser search bar will take you directly to Apple's webpage, bypassing the expected search results page.
Why would anyone do this? Well—there's money in it. The researchers found that multiple site redirections took place behind the scenes in these scenarios, with the DNS server in question not passing the query directly to the search engine but through a host of other URLs that "are all related to several online advertising companies," said the research paper. "The companies only get paid when their advertisement links are clicked by users. The extra rounds of inserted redirection are used to generate clicks, as if they are from a large number of real users."

Further investigation by the EFF and UC-Berkeley's International Computer Science Institute (ICSI) claimed that the behavior was courtesy of a company called Paxfire, which says it can help ISPs make money from mistyped URLs. Such schemes have been around for years, but Paxfire allegedly goes further.
"Paxfire's product also includes an optional, unadvertised, and more alarming feature that drastically expands Paxfire's window into users' traffic," wrote the EFF and ICSI researchers. "Instead of activating only upon error, this product redirects the customers' entire Web search traffic destined for Yahoo!, Bing, and sometimes Google, to a small number of separate web traffic proxies." (Though Paxfire's own description of itself does say that it "is the proven industry leader in monetizing Address Bar Search and DNS Error traffic for Network Operators.")

170 brand names trigger the automated redirection that ends with users being taken to those brand pages—and with affiliates pocketing some cash for sending them there. The money is presumably split between Paxfire and the ISPs in question. Paxfire did not respond to our request for comment.

Is this legal?

The Microsoft/Polytechnic research named names, compiling a list of nine ISPs who last year seemed to purposely run the malicious DNS servers: Hughes, Frontier, Cavalier, FiberNet, Spacenet, Onvoy, WOW [Wide Open West], Cincy B., and SDN. The paper noted that end users can switch from their ISP-provided DNS server to a public server (Google runs such servers, for instance, at 8.8.8.8 and 8.8.4.4) to avoid the problems.

But there are other avenues for action; the paper also noted that "complaints can be made to regulatory agencies or legal actions can be taken."
New Scientist yesterday noted a new class action lawsuit against Paxfire over the practice on the grounds that it violates the US Wiretap Act. Similar attempts to monitor user searches at the ISP level have been made in the UK by companies like Phorm, though it was widely believed such services made little traction here in the US.

And regulatory interest will probably be coming, too. This morning, in fact, FCC Chairman Julius Genachowski held a ceremony to announce the winners of the Open Internet Challenge, in which 24 researchers submitted tools to "help consumers foster, measure, and protect Internet openness." One goal of the contest was consumer-level tools that "could, for example, detect whether a broadband provider is interfering with DNS responses."
The action most affected the major search engines, who don't take such behavior kindly. Google, for instance, has systems in place to detect "hijacking" and it throws up a CAPTCHA if it suspects that such tampering has taken place. The system caught real users from Frontier, Huges, WOW, and other ISPs back in March, who then took to Google support pages to complain that they didn't like filling out CAPTCHAs to access Google tools.

But Google blamed the ISPs. "I want to assure you that at Google we are following this very closely, and trying to get Frontier to fix the issue," said a Google rep in response to one complaint. "The root of the problem is that Frontier is intercepting some traffic, so when you try to use Google your search actually goes through a Frontier server first. Google's systems detect this and identify the unusual traffic patterns as abuse, which triggers the captchas. The captchas will go away as soon as Frontier stops intercepting traffic intended for Google.... Unfortunately the interception is a business decision, not a systems error, so they are unlikely to be able to change things until Monday."

One Frontier user complained to the company back in April and "heard back immediately from Maggie Wilderotter, the CEO. She said that this had been done by one of their vendors, in violation of Frontier's business rules, and it's been shut down." Other providers also tweaked their systems so as not to trigger Google's "are you human?" checks.

But according to the EFF, "As of August 2011, all major ISPs involved have stopped proxying Google, but they still proxy Yahoo and Bing."
As for Paxfire, the company's website only appears to make reference to a much more limited tool that responds to URL typos with search result pages. But the implication is clear: most users won't mind when you monkey around with arcane backend systems.
"What feedback you do receive typically will come from a small group of highly technical users," says Paxfire. "Even that feedback tends to fall away after just a few weeks—as they get used to the new behavior."

As for the money, "Some of our customers literally generate millions of dollars a year using the Paxfire Look-up Service... It all depends. That said, no matter how you slice and dice it, the Paxfire Look-up Service will generate good money for you."