Visualizing the Twitter Social Network
Back in July, BreakingPoint added support for the Twitter API to our Application Simulator. The method I used in developing this AppSim protocol was to reference the API documentation, writeup a small client, and capture network packets. Looking at how API clients and the Twitter servers behave makes developing a realistic simulation of Twitter pretty simple to do. One question we commonly hear, especially concerning our Security and Application Simulator components, is, "How do you verify correctness?" I want to give one specific example of this by talking about a side project I worked on recently.
Once a month, I attend a local meeting of people interested in computer security. The format for the meetings allows anyone to do a talk typically between five and fifteen minutes in length. I always wait until a few days before the meetings to start thinking about prospective topics. Since I'd been working on some Trac tickets related to Twitter, I thought I might do a short topic related to mapping out the Twitter network. My original goal was to find all the cool security celebrities as a way to find all the cool links everyone is talking about.
Twitter is like most social networks, in that it tracks 'friend' relationships between users. I thought it would be interesting to visualize some of the friend/follower relationships on the site. I started to write yet-another-twitter-client. After about 10 minutes, I saw that I was reimplementing the same code I'd already done for AppSim. I decided I might as well leverage the code I'd written for AppSim. This is one way to address the question of application correctness. If I take the code in the BreakingPoint product that implements the Twitter API, and can really communicate with Twitter servers, then I would call our implementation correct. It's also a good way to make sure I don't have any bugs in Twitter, too.
I started implementing a directed graph in ruby. About 15 seconds later I started looking for a library for doing directed graphs. I found RGL, the Ruby Graph Library. There's a ton of usefulness in that library, and I know I'm not scratching the surface of how I could use it. I hadn't even started thinking about visualization yet, but I found that RGL supports Dot files, the input supported by GraphViz. I was further convinced that RGL was going to be great to work with. I had about a day before the meeting, and didn't want to write any layout code. I just wanted something done quickly (and with the least amount of effort on my part).
I wrapped the AppSim code that implements the Friends/Followers Twitter API calls, and started putting the list of Twitter users into a queue. I then dequeue the first user from the front queue, and get her friends and followers, and repeat.
This graph shows the social network from the perspective of the @BreakingPoint Twitter account. This graph adds new nodes in whatever order they are returned in the API calls.
Ok, so there are some users that show prominently, but it's all a jumble and hard to really extract anything useful.
My next method was to order insertion into the queue by the number of follwers each friend had. This technique had an interesting effect; the user I start graphing at (again, @BreakingPoint) remains in the center of the graph. You'll notice that most of the edges are directed out of the @BreakingPoint user. This is due to the fact that I process all the friends first, then all the followers. One attribute of this view is that people we follow that have a large following are prominent. It's a nice side-effect of how GraphViz lays out the graph.
The biggest problem doing this is that you end up with many of the same users populating graphs, even when starting from different initial users. When you order by the number of followers, eventually someone in your network is following @CNN or @the_Onion, and graphs from different runs start looking very similar in terms of what users are prominent. Also, once you hit a user with a follower count in the tens-of-thousands, progress in mapping the network slows as you retrieve followers in batches of 100 at a time, which is a requirement of the API. If anyone in the network you're graphing follows, say, @BarackObama, you should go to your local Alamo Drafthouse and catch a movie.
It became obvious that filtering is key if you want to get any interesting results from the data.
Here is a graph generated by limiting the users included in the graph to those with between one hundred and one thousand followers. I have also modified the graph to show our biggest followers.
An image like this gives you an idea of the most potentially influential followers you have. If you were a marketer like @KyleFlaherty, you could use this information to start trying to influence the biggest influencers that follow you. In the marketing world, word-of-mouth is a goldmine. This seems especially true in social networks where your friends are hand-selected.
This is just a first step into the topic of data visualization for me. I have a feeling I'm going to have to come up with something better than GraphViz for visualization. I've also had a request for making a web application out of this and make it available to a wider audience. I'd like to use the comments to gauge demand for this tool. If it looks like people would find this useful, I might just try and get it cleaned up and make a simple version available.
The number one thing people have said they'd want is interactivity. I think I'll go get working on that. I have tons of features I'd like to see, and I still don't have a talk for the upcoming meeting.
Update: the first thumbnail and linked image were corrected.
The Inevitable Failure of Content-Aware Network Devices
Wanted to point out the new IDC White Paper we posted to the site last week. The report is titled, "IDC Vendor Spotlight: The Inevitable Failure of Content-Aware/DPI Network Devices - and How to Mitigate the Risk." For those of you in the industry, you may recognize the author, Elisabeth Rainge, who also put together IDC's "Worldwide Network Test and Measurement 2008-2012 Forecast."
The report highlights the benefits of testing network equipment using
a broad range of applications and live security strikes at 10 Gigabits
per second speeds with heavy network traffic. Some good data in the report for anyone looking at network equipment testing. Check out the report and let us know what you think.
Real Clickjacking?
After Michael Zalewski's WHATWG post spilled enough beans to show definitely that yesterday's pop-up evaders weren't "clickjacking," I put together another demo this afternoon (link below), which uses a combination of opacity and z-index settings on an iframe. Again, it's just speculation.
http://www.planb-security.net/notclickjacking/iframetrick.html
This seems to fit the bill: No Javascript required, uses iframes, and gives the ability to seemingly overlay one UI on top of another. By the way, the demo is mostly harmless -- it just turns your Myspace profile from private to public. I started down the path of masking my brokerage's trading app, but masking out keystrokes for stock orders seemed to be overkill for a simple speculative demo.
Not Clickjacking (Almost Certainly)
A couple months back, I got to thinking about how to weasel around the built-in pop up blockers for Mozilla Firefox and Microsoft Internet Explorer. I can't remember now why I started down this path, but it consumed my afternoon, and in the end I had about a dozen variations on three or so core strategies.
Since I have no fake watches or mortgages to sell, I thusly shelved it all once I was done. We couldn't think of a reasonable application for these techniques here in BreakingPoint.
Well, maybe something was in the air, because it looks like RSnake and Jeremiah Grossman were working on something similar at around the same time. Only, they were going to present their findings at OWASP this week.
Unfortunately, we can't know what the extent of their trickery is, since some vendors asked them to keep it quiet -- but it's almost certainly more devious than simply skipping around the pop up blockers.
So, in the interest of quelling any speculation on the nature of "ClickJacking," here's two or three mildly interesting examples of what it almost certainly isn't. Note, the below example links are all in an iframe from planb-security.net. So, if you have a tabbed browser, you may notice an illusion the iframe's parent window (this blog post) being replaced by the fake pop up link (in this case, example.com).
If you're in the business of speculating on what other people's findings are, then you can safely check these techniques off your list. Since they all require Javascript to work their anti-anti-pop up magic, NoScript is quite effective in preventing these attacks. Based simply on this observation, and discounting for the moment the inherent lameness of merely subverting mouse clicks, it's virtually certain Jeremiah and RSnake's stuff is much, much more interesting / dangerous.
Network Management Practices
Just finished reading the document Comcast supplied the FCC (PDF) on their network management practices and reading the recaps from CircleID and Wired's Threat Level. This is, as folks point out, more of a historical document since Comcast has already submitted their new practices, but nonetheless it is interesting on a few points. Since we talk constantly about traffic analysis and of course P2P, I've been keeping up with the stories throughout.
What I found the most interesting in light of today's news was not the info around P2P or their new "application-agnostic" practices, it was the fact that Comcast's Jason Livingood who is in their National Engineering & Technical Operations, is commenting on these blog posts and supplying info, links and in some cases taking suggestions. Certainly a thankless job, but he seems to be helping bring a technical voice to the discussion. What have you thought as you've read these stories?
Obviously people will be watching all of this closely and I'm looking forward to hearing your opinions on the new practices they outlined. I'm going to bet some folks have a few things to say about the 250GB cap?
BreakingPoint LiveLook: SSL, Emmy Awards and a Harley
In San Francisco right now attending the Frost & Sullivan Award ceremony tonite and hoping to get some video to you all tomorrow, but I really wanted to post this video that Dennis shot with Mohsin Lari in our QA group. Mohsin and Dennis talk about QA and SSL, but also the fact that Mohsin has an Emmy Award winner in the family and is on a quest for a Harley. Something we all can relate to...well except for winning an Emmy, the award tonite will have to do for now.
Let us know if you have anything that you would like us to video at BreakingPoint or folks you want to see and hear.
Testing Network Devices using Proprietary Applications
As you know, BreakingPoint has a Custom Application Toolkit feature, which allows you to generate stateful application traffic from proprietary apps and then integrate this traffic with BreakingPoint’s testing platform. You can blend the proprietary apps with our standard apps and security strikes or just run them alone, and test their effects on high performance network devices.
Mike Hamilton, our Director of Product Marketing and Tod Beardsley in BreakingPoint Labs got together to demonstrate how to put this feature to good use. The full screencast can be found here (along with our other screencasts), but I thought that this would be of interest to many of you so I've also embedded it below.
Bit Twiddling with Ruby: PacketFu
Since I started here at BreakingPoint, I've become a huge fanboy of Ruby. The Application Simulators I write are about 90% Ruby and 10% XML, and over all, it's a pretty fun language to code in. However, outside of BreakingPoint, Ruby is almost universally perceived as a language specifically for driving web applications. As a result, it can be hard to find good libraries for non-web chores -- in my case, I couldn't come up with a good way to natively read and write raw network packets.
So, I wrote a packet library for Ruby. It's called PacketFu, and over the weekend, I talked about it at Lone Star Ruby Conf here in Austin. In my talk, I spent a fair amount of time discussing the design ideas behind PacketFu, the most excellent BinData Ruby gem, and demoing various tasks that PacketFu endeavors to assist in -- packet sniffing, parsing, and rewriting.
PacketFu, now at version 0.0.3, is triple-secret alpha pre-release level code, but the demos and example applications work, and the external interfaces are pretty much nailed down and documented here. Look for a more sane beta release in the next few weeks, and a major version release before the end of the year. If all goes well, at next year's LSRC, I'll be talking about all the fab apps written using PacketFu.
DPI Coverage Getting it Wrong
Over the past two days news has trickled out that NebuAd's CEO has resigned and the company is now suspending their web-monitoring plans. Blogs and mainstream media are not only calling this a win for online privacy, but a few have talked about how this will kill off deep packet inspection (DPI) as a technology. ZDnet's "Between the Lines" more specifically wrote:
"...it doesn’t sound like there’s a long prosperous future ahead for deep packet inspection..."
As Dennis commented on the ZDnet post, this is a bit of 'throwing the baby out with the bathwater'. DPI capabilities go way beyond marketing uses, including enhancing network security and data leak prevention. To collectively dismiss DPI because of the NebuAd example is short-sighted and a bit irresponsible.
To elaborate on these points, here are the DPI slides we shared a few weeks ago from Dennis' portion of the DPI webinar. These should be helpful for folks who are only looking at one angle of this story.