Monday, May 1, 2017

For Users of Redis, Running Locally Can Be a Major Security Risk

If you are running Redis locally and, like most people as of this writing, you're using a version older than 3.2.7 (released January 31, 2017), I can most likely copy your entire database, drop an ssh key in your authorized_keys file, overwrite arbitrary files on your computer, and lay a trap that will run arbitrary code next time you open your terminal.
All this can be executed from visiting a malicious web site; or a non-malicious site that's had malicious JavaScript loaded onto it.


I read "How to steal any developer's local database" a while back on Hacker News, and thought it might be a fun project to try out with BugReplay. In the process I’ve learned how powerful DNS rebinding can be as a method of attack and the kind of damage that can be done. This article is an attempt to build on previous works and demonstrate the techniques involved.
When you look at typical authorization mechanisms, the most basic is restricting the ability to access the resource. For example, you probably spend more time making sure people can't steal or mess around with your laptop than crafting the ultimate password for it. In the case of networked software, access restrictions at the network or server layer (ie above the client software layer) are the most important part to secure and are often sufficient. Having a password for your SQL database is important, but security-wise it is way more important that it isn't open to connections from the internet.
Redis has unusually few security features for a database that has seen such wide adoption, even for software designed to run on secured servers. There are no access levels; any connecting user can execute system commands like FLUSHALL or DEBUG SEGFAULT. From antirez, the original developer of Redis:
The Redis security model is: “it’s totally insecure to let untrusted clients access the system, please protect it from the outside world yourself”. The reason is that, basically, 99.99% of the Redis use cases are inside a sandboxed environment.
That may have been true at the time of writing, but nowadays a whole lot of people are running Redis locally, listening on the default port. That is very far from sandboxed.. it's more like a sewer. Unlike a typical remote server, your personal computer typically is used for browsing the web with a JavaScript enabled browser.
Server Side Request Forgery (SSRF) is a method of tricking a vulnerable server into sending out malicious requests that you've crafted to services that you can not access directly. Say you were probing Facebook's open graph page debugger. If you were testing it for SSRF possibilities, the input box (or the api endpoint it is posting to) would be your attack vector. Maybe you would start by plugging in or and seeing what data is exposed.
When you're browsing the web, your browser is virtually as open as that URL input box to any website you visit. Any resource you load, via img tag or script tag or xhr request is a network request originating from your IP address to the wider Internet. That means Redis, sitting there with its default configuration of binding to localhost port 6379 with no password1, is wide open to anyone on the internet.

The Exploit

A combination of factors makes Redis particularly vulnerable to SSRF2. It uses a text based protocol, it won't terminate the connection when fed invalid commands, and the previously mentioned lack of access levels. The combination leads to some interesting possibilities in terms of exploitation. Basically, as has been well known for over 15 years, you can post data to almost any TCP port running on localhost through the browser. Then using a technique known as DNS rebinding, you can sidestep the Same Origin Policy and get data out.
Combined with the lax security model of Redis, it is an interesting exploit. As a developer working on BugReplay, a bug recording tool which tries to capture all the streams of data necessary to understand and reproduce an issue, I'm always looking for new scenarios to test out the software and see how it performs. With all the different moving parts, I was not sure what would come out of recording it in action. I was particularly interested to see how it would record responses from Redis, as Redis isn't a web server and is just incidentally able to communicate back to the browser.
Here's a BugReplay walkthrough of sending arbitrary commands to a local Redis server. In this example we are setting the key “Hi! We’re from the” to the value “Internet.” in Redis using the Redis protocol:
*3 $3 SET $18 Hi! We're from the $9 Internet. *1 $4 QUIT
Redis also supports plain-text notation in its protocol, but using the protocol format makes it easier to transmit over the wire and is more robust (ex. if sending values with newlines).
Notice how the demo shows the response from Redis. Your browser can see that data (and BugReplay is attached to the browser via the Chrome debugging protocol so it also receives the response), but the web page does not thanks to the 'Same Origin Policy.'. I'm actually serving this page locally but because the demo is on a different port than Redis the 'Origin' is different ( http://localhost:5001 and http://localhost:6379 respectively). I can send commands into Redis, but not get data back out.
To get the data back out (exfiltration), we need to use DNS Rebinding. This BugReplay recording demonstrates an attack where I'm injecting a Lua script that retrieves the entire contents of the database.
The script injects a base64 encoding function (with comments crediting the author!), along with a function that will encode the entire database as a JSON hash with the keys corresponding to the keys from the database, and the values corresponding to base64(DUMP(key)). The DUMP command outputs a binary format, so base64 helps for transmitting the data over the wire. You can see the request here.
DNS rebinding is a way of getting around the Same Origin Policy for services where you know the IP address of the service you're trying to access. In the Redis case we are trying to get to (localhost), but other attacks may target your wifi router or something else only accessible on your local network.
To execute a DNS rebinding attack, you set a domain that toggles between a remote address where you are serving up malicious content and the IP address you are trying to access.
With this hack I'm trying to POST data to Redis, and send the results back out to an endpoint I control, but because of the Same Origin Policy I can't just send a request from your computer to and then send back the results. The workaround is serving up some content on port 6379 of a server I control ( that is supposedly posting data to itself. The catch is that I am switching the DNS A Record record behind the scenes to point to, so while the browser is treating it as the same Origin, is switching every 20 seconds from pointing at my server to your local computer.

So I Just Copied Your Database and Added a Key to It, Who Cares?

The demo here is purposely pretty harmless. Following along with Antirez's security writeup, you can see how you can drop files around the OS anywhere the user running Redis has permission. Including your ssh keys file, but that's not the worst of it. This demo goes through overwriting ~/.inputrc, which can trigger a payload from a terminal key binding; ex: you go to your shell and hit the letter "a" and that fires off a background process which installs a backdoor. I was very easily able to bind a key which downloaded an arbitrary url and executed it via the shell.

What Can I Do?

If you are running Redis locally and haven't upgraded to 3.2.7 (stable as of Jan 31 2017), there's a number of things you can do to keep yourself safe.


This one's pretty obvious. Upgrade to latest stable, which aliases Host to QUIT, keeping web browsers from talking to Redis. Upgrading production software is always tricky because of compatibility concerns, plus the hassle of taking the database out of service, but you don't have that excuse if you are running a local database.

Set a Password

The Redis project has always recommended setting a strong password - Redis is fast enough that brute forcing an easy password is not hard.

Rename Sensitive Commands

Even on older versions of Redis, you have the ability to rename sensitive commands. This means you can effectively do what later versions do by default, alias HOST to QUIT so browsers get the boot. You may also just want to do away with being able to delete the entire database while you're in there. And do you really need EVAL?

Bind to a Unix socket Instead for Local Development

Nothing is more secure than being unreachable, so switching to a Unix socket instead of a port closes that vector of attack.


This was a fun blog entry to write. My goal of making BugReplay the best piece of software for recording and diagnosing web application based issues gave me a great excuse to try and hack the crap out of my computer while learning some classic yet still current web based exploits.
As you could probably tell I am in no way a security expert, so any corrections please send my way via twitter.
If you want to check out BugReplay and record some examples of exploits (or bugs) yourself we'd love to have you try it out! Here's a bonus demo of an Ebay info leakage issue that people seemed to enjoy on Hacker News :)

1: Technically the default configuration, what happens when you run redis-server with no arguments, is bound on every interface ( with no password but the config that ships with Redis binds only to as a more sensible default.
2: There does not seem to be an overwhelming consensus as to the classification of this exploit as an example of SSRF. There's a case to be made for classifying it as a CSRF (Cross Site Request Forgery) attack as it is here, because like a typical CSRF attack your browser is the attack vector. However Redis is not a 'Site' in the typical sense; it just happens to be vulnerable to payloads delivered over HTTP.

Thursday, February 23, 2017

Introducing Feedback by BugReplay- A Customer Support Tool

Instead of relying on your users to explain what went wrong, wouldn’t it be easier if you could SEE for yourself?

We’re excited to announce that Feedback by BugReplay (“Feedback”) is now in private beta. With Feedback, you can offer your customers can record the problem they encountered on your website and you’ll receive a detailed report that will help you fix the problem faster. Our browser extension creates a synchronized recording of user actions, network traffic & JavaScript logs.

Your users don’t have to create a BugReplay account and they don’t have access to your account either. All they have to do is press record, retrace their steps, and send the report directly to you.

Feedback fits in seamlessly with all support solutions- drop a Feedback button anywhere you want on your website. It also integrates with Zendesk so users can attach videos to their support tickets.

Check out our website to learn more about Feedback by BugReplay and sign up for our private beta.

Wednesday, November 30, 2016

F**kadblock! How publishers are defeating ad blockers & how ad blockers are fighting back.

A couple of weeks ago, we looked into how PornHub was getting around ad blockers using WebSockets. We thought it would be cool to take a closer look at how publishers like BusinessInsider and Forbes are detecting ad blockers and preventing users from viewing their content.


When visiting a BusinessInsider article while using an ad blocker like AdBlock Plus, after idling on the page for about 10 seconds, a modal blocks you from reading the article:


BusinessInsider uses’s VX product (“The evolution of the paywall”) for detecting ad blockers., helpfully has some documentation on how to detect ad blockersusing their or your own solution. Piano’s solution relies on the open-source library FuckAdblock. They use a safe for work alternate version called BlockAdblock, — “FuckAdBlock same project but with a more convenient name.”
FuckAdblock functions by injecting a dummy div into the webpage that contains css classes and styles that are common to advertisements. Then it performs a variety of checks to see if the div has been hidden from view. If it has, the user has an adblocker enabled.
BusinessInsider’s onpage JavaScript additionally checks to see if the FuckAdBlock js file failed to load, in which case it also detects the user as having an adblocker.


uBlock Origin works around this detection in a clever way. Chrome extensions have the ability to intercept and modify HTTP requests via the webRequest API. uBlock intercepts the request for FuckAdBlock’s JavaScript and responds with an HTTP 307, the code for a temporary redirect. The URL it redirects to is actually a blob of base64-encoded alternate JavaScript. This JavaScript is then loaded instead of FuckAdblock.
Here is a demo in BugReplay that demonstrates how uBlock’s workaround works. I’ve selected the HTTP redirect that is described above, containing the base64 encoded JavaScript. Clicking the download icon next to the ‘Location’ header will display a plaintext version of the code that is loaded instead of FuckAdblock.
uBlock’s alternate JavaScript creates a dummy version of the FuckAdBlock class that doesn’t actually do anything. This way, if any JavaScript external to this file were to check on the status of FuckAdBlock, it would appear as if it loaded successfully. By watching the video above you can see that the ad blocker detection never pops up a modal nagging the user when using uBlock.
FuckAdBlock has a pretty clever response to uBlock’s hack. They recommend utilizing subresource ntegrity for the script tag that loads FuckAdBlock, which causes the browser to fail to load the 307’d version due to it’s hash not matching the integrity check, triggering the onerror for the script which triggers the adblock detector. For now Business Insider doesn’t have that implemented.


When visiting Forbes with AdBlock or Adblock Plus enabled, users are presented with this:
Clicking continue when Adblock or Adblock Plus is enabled just brings you back to the same page. Linking a social account does let you past this adblock wall without disabling your ad blocker, but that’s in exchange for your personal info.


Like BusinessInsider, Forbes relies on a third party company to provide ad blocker detection. LiftDNA provides this functionality for Forbes. Unlike Piano, LiftDNA looks to have built their own solution for detecting ad blockers.
After spending some time puzzling over Forbes’ minified / obfuscated JavaScript , I came across a Github Gist that really helped a lot. The basic functionality of how LiftDNA’s ad blocker check works is very similar to FuckAdblock.
Inside Forbes’ page html is a base64 encoded string that, in its decoded form, contains CSS class names of easily identifiable ads. When ad blockers see dom elements with these class names they immediately hide them.
Forbes’ check is a bit trickier than BusinessInsider, since it creates a div element with between 5 and 10 randomly picked class names from a large list, and then checks to see if it is set to display: none. This way it’s harder to create workarounds since you don’t know which class and styles it will choose in a given run. If the code detects that the div is hidden, the user is using an ad blocker. If not, it sets a cookie and allow the user through when they click continue. The following is the function that accomplishes this:
checkBlockedClasses: function() {
for (var a, b = Math.random(), c = $(“.ads-container>div”), d = c.length, e = Math.floor(b * d), f = “${classes!}” !== fbs_settings.classes ? JSON.parse(base64.decode(fbs_settings.classes)) : [“dynamic-ads”], g = [], h = this, i = 0; i < 5 || Math.random() < .7 && i < 10; i++) g.push(f[Math.floor(Math.random() * f.length)]);
this.rand_classes = g, a = $(’<div class=“’ + g.join(” “) + ’”>’).append(’<div style=“height: 1px;”>’), $(c[e]).before(a), window.navigator.userAgent.indexOf(“Firefox”) > -1 ? setTimeout(function() {
a.height() || (h.triggered_by_classes = !0, h.triggerAdBlockState(!0)), a.remove()
}, 100) : (“none” === a.css(“display”) && (this.triggered_by_classes = !0, this.triggerAdBlockState(!0)), a.remove())


Forbes Gist method
The Forbes gist that I linked earlier which explains how Forbes’ ad blocker detection works also provides a workaround. It involves allowing through all of the ad-spam classes that Forbes is using, so long as the domain is
The only issue here is having to keep up with Forbes’ CSS name list, since the pool they can draw names from is huge.
At the time of this blog post, the workaround code is on version 17. Also, when I checked the list of class names currently being used by Forbes, it differs from the list in the gist.
uBlock Origin workaround
Forbes was able to block Adblock/Adblock Plus, but uBlock origin was able to successfully fool Forbes into thinking I had disabled my ad blocker. How are they getting around the check?
After going through the uBlock source, I tracked down their workaround.
Basically, if the url looks like the Forbes ad welcome page, it sets the cookie ‘welcomeAd’ to true, and if uBlock can determine the article url from a cookie, it automatically redirects to it.
The Anti-Adblock Killer extension functions similarly, except it sets an additional cookie to better mimic what Forbes’ JavaScript is doing.
The request shown is the HTTP request for the Forbes article. As you can see, it immediately redirects with a 302 to Forbes’ adblock page. Clicking continue just redirects you back there. If you look at the HTTP request’s cookies you won’t see welcomeAd or dailyWelcomeCookie, both of which get set by extensions that try to subvert Forbes’ adblock.
In this next example, before visiting the page I used a cookie editing extension to add the following cookie values for
These are the cookies that Forbes would set upon successfully watching their ad.
As you can see in the linked BugReplay report, instead of 302-ing to the adblock page, it loads the article. If you look at the request cookies, you can see both of the above cookie names and values listed. This fools Forbes into taking you to the requested article.
The Future
As ad blockers continue to grow in popularity and publishers continue to rely on advertising dollars, the war of ads, ad blockers, and ad block blockers will continue.
Looking at uBlock Origin’s resources.txt file, you can see just how many custom JavaScript snippets are there neutralizing various web sites’ strategies to bypass ad blockers. It’s an intensive, manual process collecting, implementing and testing all those strategies, and it’s an unpaid job.
Ad blockers have traditionally only had to worry about removing ads while keeping a website functional- and that was difficult enough. Software that actively tries to detect them and disable the hosting website constitutes a new front in the advertising and privacy war, one that Ad Blockers currently seem to be losing.

Tuesday, November 1, 2016

Pornhub Bypasses Ad Blockers With WebSockets

*** Links to discussions on Reddit and Hacker News. Also check out BugReplay on Product Hunt :)
(NSFW level: medium)
We tried to find the most PG page on MindGeek’s network to use as an example- it wasn’t easy.

When I was building the prototype for BugReplay, I was evaluating different methods of capturing and analyzing network traffic from Chrome. One of the first things I saw that looked promising was the chrome.webRequest API.
From the docs: “Use the chrome.webRequest API to observe and analyze traffic and to intercept, block, or modify requests in-flight.”
That seemed to be exactly what I needed.
After experimenting with the Chrome webRequest API, I quickly realized there was a big problem. It didn’t allow me to analyze any WebSocket traffic, something I really wanted to support.
As I was searching the web trying to see if I was misreading the documentation or was looking in the wrong spot, I found a relevant bug report from 2012: “chrome.webRequest.onBeforeRequest doesn’t intercept WebSocket requests.” In the bug report, users were complaining that without the ability to block WebSockets, websites could get around ad blockers fairly easily. If WebSocket data was not visible to Chrome extensions via the webRequest API, they could not be blocked without some heavy duty hacks.
Initially, the risks to ad blockers seemed theoretical; the examples of sites that were employing this technique were very obscure. Then in August 2016, an employee of the company that owns (MindGeek) started arguing against adding the WebSocket blocking capabilities to the Chrome API. Pornhub is the 63rd most visited site on the Internet according to Alexa. I checked out a few of MindGeek’s sites and sure enough, I could see ads coming through even though I had Adblock Plus on. The ads on Pornhub are marked ‘By Traffic Junky,’ which is an ad network owned by MindGeek.
In the screenshot below, you can see a banner at the top of the page announcing that the site is aware that the user is using an Ad Blocker, with an invitation to subscribe to a premium ads free version of the site. On the right side of the page you can see an advertisement.

When you visit, it tries to detect if you have an ad blocker. If it detects one, it opens a WebSocket connection that acts as a backup mechanism for delivering ads.
Watching the BugReplay browser recording, you can see a number of network requests triggered that are blocked by AdBlock: They are marked Failed in the network traffic, and if you click one to inspect the detail pane you can see the failed reason is net::ERR_BLOCKED_BY_CLIENT. That is the error reported by Chrome when an asset is blocked from loading.
You can find the WebSocket frames individually in the network panel or just look at the WebSocket create request which has links to all the individual frames. The name of the domain where the WebSocket connects is “ws://” A decent joke aimed at ad blockers :)
When the WebSocket loads, the browser sends a frame with a JSON encoded payload for each of the spots it has available for ads. Checking out one of the WebSocket frames, you can see in the frame data the advertisement data is sent back with:
  1. The zone_id 13, for where the JavaScript should place the ad.
  2. The media_type image, so the page knows what kind of element to create (most of the ads are videos, I picked an image for this post because it was relatively tame).
  3. The Image itself, transmitted base64 encoded so it can be reconstructed using the data uri scheme
  4. An “img_type” (“image/jpeg”) to pass to the data uri.
Ad Blockers primarily work using the webRequest API, so constructing the ad by transmitting the data over the WebSocket as base64 is a pretty clever way of dodging the blocker.


On October 25th, 2016, there was some new activity on the Chromium ticket. A contributor wrote a patch adding the ability to block WebSockets using the webRequest api. If it’s accepted, it will eventually wind up in Chrome stable.
When or if that rolls out, the ad blocker extension writers can choose to remove the hacks for users of the latest Chrome, leaving content providers like Pornhub to figure out their next move in the ad blocking war.


Since I started looking into this, AdBlock Plus and uBlock Origin have shipped workarounds to block this technique. AdBlock and others still do not.
For AdBlock Plus, “The wrapper performs a dummy web request before WebSocket messages are sent/received. The extension recognizes these dummy web requests as representing a WebSocket message. It intercepts and blocks them if the corresponding WebSocket message should be blocked. The WebSocket wrapper then allows / blocks the WebSocket message based on whether the dummy web request was blocked or not.” via
For uBlock Origin, they shipped a workaround that has the “ability to foil WebSocket using a CSP directive.”